pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedLarge number of short lived connections - could a connection pool help?
4+ messages / 3 participants
[nested] [flat]
* Large number of short lived connections - could a connection pool help?
@ 2011-11-15 00:42 Cody Caughlan <toolbag@gmail.com>
0 siblings, 2 replies; 4+ messages in thread
From: Cody Caughlan @ 2011-11-15 00:42 UTC (permalink / raw)
To: pgsql-performance
We have anywhere from 60-80 background worker processes connecting to
Postgres, performing a short task and then disconnecting. The lifetime
of these tasks averages 1-3 seconds.
I know that there is some connection overhead to Postgres, but I dont
know what would be the best way to measure this overheard and/or to
determine if its currently an issue at all.
If there is a substantial overheard I would think that employing a
connection pool like pgbouncer to keep a static list of these
connections and then dole them out to the transient workers on demand.
So the overall cumulative number of connections wouldnt change, I
would just attempt to alleviate the setup/teardown of them so quickly.
Is this something that I should look into or is it not much of an
issue? Whats the best way to determine if I could benefit from using a
connection pool?
Thanks.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Large number of short lived connections - could a connection pool help?
@ 2011-11-15 00:59 Ben Chobot <bench@silentmedia.com>
parent: Cody Caughlan <toolbag@gmail.com>
1 sibling, 1 reply; 4+ messages in thread
From: Ben Chobot @ 2011-11-15 00:59 UTC (permalink / raw)
To: Cody Caughlan <toolbag@gmail.com>; +Cc: pgsql-performance
On Nov 14, 2011, at 4:42 PM, Cody Caughlan wrote:
> We have anywhere from 60-80 background worker processes connecting to
> Postgres, performing a short task and then disconnecting. The lifetime
> of these tasks averages 1-3 seconds.
[snip]
> Is this something that I should look into or is it not much of an
> issue? Whats the best way to determine if I could benefit from using a
> connection pool?
Yes, this is precisely a kind of situation a connection pooler will help with. Not only with the the connection set up/tear down overhead, but also by using resources on your server better.... you probably don't actually have 60-80 cores on your server, so reducing that number down to just a few that are actually working will the Postgres finish them faster to work on others. Basically, the queueing happens off the postgres server, letting postgres use the box with less interruptions.
Now, is it a problem to not use a pooler? That depends on if it's causing you grief or not. But if you think you'll get more connection churn or larger numbers of workers, then a connection pooler will only help more.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Large number of short lived connections - could a connection pool help?
@ 2011-11-15 01:04 Cody Caughlan <toolbag@gmail.com>
parent: Ben Chobot <bench@silentmedia.com>
0 siblings, 0 replies; 4+ messages in thread
From: Cody Caughlan @ 2011-11-15 01:04 UTC (permalink / raw)
To: Ben Chobot <bench@silentmedia.com>; +Cc: pgsql-performance
On Mon, Nov 14, 2011 at 4:59 PM, Ben Chobot <bench@silentmedia.com> wrote:
> On Nov 14, 2011, at 4:42 PM, Cody Caughlan wrote:
>
>> We have anywhere from 60-80 background worker processes connecting to
>> Postgres, performing a short task and then disconnecting. The lifetime
>> of these tasks averages 1-3 seconds.
>
> [snip]
>
>> Is this something that I should look into or is it not much of an
>> issue? Whats the best way to determine if I could benefit from using a
>> connection pool?
>
> Yes, this is precisely a kind of situation a connection pooler will help with. Not only with the the connection set up/tear down overhead, but also by using resources on your server better.... you probably don't actually have 60-80 cores on your server, so reducing that number down to just a few that are actually working will the Postgres finish them faster to work on others. Basically, the queueing happens off the postgres server, letting postgres use the box with less interruptions.
>
> Now, is it a problem to not use a pooler? That depends on if it's causing you grief or not. But if you think you'll get more connection churn or larger numbers of workers, then a connection pooler will only help more.
Thanks for your input. Its not causing me grief per se. The load on
the pg machine is small. I guess I am just wondering if I am being
stupid and leaving resources and/or performance on the table.
But it sounds that as a whole it would be a good use case for
pgbouncer and in the long run will prove beneficial. But no, its not
obviously killing me right now.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Large number of short lived connections - could a connection pool help?
@ 2011-11-15 08:04 Mario Weilguni <roadrunner6@gmx.at>
parent: Cody Caughlan <toolbag@gmail.com>
1 sibling, 0 replies; 4+ messages in thread
From: Mario Weilguni @ 2011-11-15 08:04 UTC (permalink / raw)
To: pgsql-performance
Am 15.11.2011 01:42, schrieb Cody Caughlan:
> We have anywhere from 60-80 background worker processes connecting to
> Postgres, performing a short task and then disconnecting. The lifetime
> of these tasks averages 1-3 seconds.
>
> I know that there is some connection overhead to Postgres, but I dont
> know what would be the best way to measure this overheard and/or to
> determine if its currently an issue at all.
>
> If there is a substantial overheard I would think that employing a
> connection pool like pgbouncer to keep a static list of these
> connections and then dole them out to the transient workers on demand.
>
> So the overall cumulative number of connections wouldnt change, I
> would just attempt to alleviate the setup/teardown of them so quickly.
>
> Is this something that I should look into or is it not much of an
> issue? Whats the best way to determine if I could benefit from using a
> connection pool?
>
> Thanks.
>
I had a case where a pooler (in this case pgpool) resulted in a 140%
application improvement - so - yes, it is probably a win to use a
pooling solution.
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2011-11-15 08:04 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-11-15 00:42 Large number of short lived connections - could a connection pool help? Cody Caughlan <toolbag@gmail.com>
2011-11-15 00:59 ` Ben Chobot <bench@silentmedia.com>
2011-11-15 01:04 ` Cody Caughlan <toolbag@gmail.com>
2011-11-15 08:04 ` Mario Weilguni <roadrunner6@gmx.at>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox