pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedrecommendations for web/db connection pooling or DBD::Gofer reviews
5+ messages / 4 participants
[nested] [flat]
* recommendations for web/db connection pooling or DBD::Gofer reviews
@ 2008-04-07 18:36 Mark Stosberg <mark@summersault.com>
2008-04-07 18:44 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews Joshua D. Drake <jd@commandprompt.com>
2008-04-08 22:40 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews PFC <lists@peufeu.com>
0 siblings, 2 replies; 5+ messages in thread
From: Mark Stosberg @ 2008-04-07 18:36 UTC (permalink / raw)
To: pgsql-performance
When traffic to our PostgreSQL-backed website spikes, the first resource
we see being exhausted is the DB slots on the master server (currently
set to about 400).
I expect that as new Apache/mod_perl children are being put to us, they
are creating new database connections.
I'm interested in recommendations to funnel more of that traffic through
fewer DB slots, if that's possible. (We could also consider increasing
the handles available, since the DB server has some CPU and memory to
spare).
I'm particularly interested in review of DBD::Gofer, which seems like it
would help with this in our Perl application:
http://search.cpan.org/dist/DBI/lib/DBD/Gofer.pm
I realize it has limitations, like "no transactions", but I think we
would still able to use it selectively in our application.
Mark
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: recommendations for web/db connection pooling or DBD::Gofer reviews
2008-04-07 18:36 recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
@ 2008-04-07 18:44 ` Joshua D. Drake <jd@commandprompt.com>
1 sibling, 0 replies; 5+ messages in thread
From: Joshua D. Drake @ 2008-04-07 18:44 UTC (permalink / raw)
To: Mark Stosberg <mark@summersault.com>; +Cc: pgsql-performance
On Mon, 07 Apr 2008 14:36:00 -0400
Mark Stosberg <mark@summersault.com> wrote:
> I'm particularly interested in review of DBD::Gofer, which seems like
> it would help with this in our Perl application:
> http://search.cpan.org/dist/DBI/lib/DBD/Gofer.pm
>
> I realize it has limitations, like "no transactions", but I think we
> would still able to use it selectively in our application.
I would stick to proven postgresql technologies such as pgbouncer.
Sincerely,
Joshua D. Drake
>
> Mark
>
>
--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: recommendations for web/db connection pooling or DBD::Gofer reviews
2008-04-07 18:36 recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
@ 2008-04-08 22:40 ` PFC <lists@peufeu.com>
2008-04-10 21:28 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
1 sibling, 1 reply; 5+ messages in thread
From: PFC @ 2008-04-08 22:40 UTC (permalink / raw)
To: Mark Stosberg <mark@summersault.com>; pgsql-performance
> When traffic to our PostgreSQL-backed website spikes, the first resource
> we see being exhausted is the DB slots on the master server (currently
> set to about 400).
>
> I expect that as new Apache/mod_perl children are being put to us, they
> are creating new database connections.
>
> I'm interested in recommendations to funnel more of that traffic through
> fewer DB slots, if that's possible. (We could also consider increasing
> the handles available, since the DB server has some CPU and memory to
> spare).
>
> I'm particularly interested in review of DBD::Gofer, which seems like it
> would help with this in our Perl application:
> http://search.cpan.org/dist/DBI/lib/DBD/Gofer.pm
>
> I realize it has limitations, like "no transactions", but I think we
> would still able to use it selectively in our application.
Under heavy load, Apache has the usual failure mode of spawning so many
threads/processes and database connections that it just exhausts all the
memory on the webserver and also kills the database.
As usual, I would use lighttpd as a frontend (also serving static files)
to handle the large number of concurrent connections to clients, and then
have it funnel this to a reasonable number of perl backends, something
like 10-30. I don't know if fastcgi works with perl, but with PHP it
certainly works very well. If you can't use fastcgi, use lighttpd as a
HTTP proxy and apache with mod_perl behind.
Recipe for good handling of heavy load is using an asynchronous server
(which by design can handle any number of concurrent connections up to the
OS' limit) in front of a small number of dynamic webpage generating
threads/processes.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: recommendations for web/db connection pooling or DBD::Gofer reviews
2008-04-07 18:36 recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
2008-04-08 22:40 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews PFC <lists@peufeu.com>
@ 2008-04-10 21:28 ` Mark Stosberg <mark@summersault.com>
2008-04-11 14:41 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews Vivek Khera <vivek@khera.org>
0 siblings, 1 reply; 5+ messages in thread
From: Mark Stosberg @ 2008-04-10 21:28 UTC (permalink / raw)
To: pgsql-performance
> Under heavy load, Apache has the usual failure mode of spawning so
> many threads/processes and database connections that it just exhausts
> all the memory on the webserver and also kills the database.
> As usual, I would use lighttpd as a frontend (also serving static
> files) to handle the large number of concurrent connections to clients,
> and then have it funnel this to a reasonable number of perl backends,
> something like 10-30. I don't know if fastcgi works with perl, but with
> PHP it certainly works very well. If you can't use fastcgi, use lighttpd
> as a HTTP proxy and apache with mod_perl behind.
> Recipe for good handling of heavy load is using an asynchronous
> server (which by design can handle any number of concurrent connections
> up to the OS' limit) in front of a small number of dynamic webpage
> generating threads/processes.
Thanks for the response.
To be clear, it sounds like you are advocating solving the problem with
scaling the number of connections with a different approach, by limiting
the number of web server processes.
So, the front-end proxy would have a number of max connections, say 200,
and it would connect to another httpd/mod_perl server behind with a
lower number of connections, say 20. If the backend httpd server was
busy, the proxy connection to it would just wait in a queue until it was
available.
Is that the kind of design you had in mind?
That seems like a reasonable option as well. We already have some
lightweight Apache servers in use on the project which currently just
serve static content.
Mark
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: recommendations for web/db connection pooling or DBD::Gofer reviews
2008-04-07 18:36 recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
2008-04-08 22:40 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews PFC <lists@peufeu.com>
2008-04-10 21:28 ` Re: recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
@ 2008-04-11 14:41 ` Vivek Khera <vivek@khera.org>
0 siblings, 0 replies; 5+ messages in thread
From: Vivek Khera @ 2008-04-11 14:41 UTC (permalink / raw)
To: pgsql-performance
On Apr 10, 2008, at 5:28 PM, Mark Stosberg wrote:
> So, the front-end proxy would have a number of max connections, say
> 200, and it would connect to another httpd/mod_perl server behind
> with a lower number of connections, say 20. If the backend httpd
> server was busy, the proxy connection to it would just wait in a
> queue until it was available.
If you read the mod_perl performance tuning guide, it will tell you to
do exactly this. These are solved problems for many, many years now.
The apache mod_proxy really does wonders...
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2008-04-11 14:41 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-04-07 18:36 recommendations for web/db connection pooling or DBD::Gofer reviews Mark Stosberg <mark@summersault.com>
2008-04-07 18:44 ` Joshua D. Drake <jd@commandprompt.com>
2008-04-08 22:40 ` PFC <lists@peufeu.com>
2008-04-10 21:28 ` Mark Stosberg <mark@summersault.com>
2008-04-11 14:41 ` Vivek Khera <vivek@khera.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox