Subject: Re: Inadequate hosting for www.postgresql.org?

From: "Dave Page" <[email-masked]>
Date: Tue, 2 Nov 2004 11:00:48 -0000

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Alexey Borzov
> Sent: 02 November 2004 10:25
> To: Marc G. Fournier
> Cc: [email-masked]
> Subject: Re: [pgsql-www] Inadequate hosting for www.postgresql.org?
> 
> Hi,
> 
> Marc G. Fournier wrote:
> > Am looking into this with Alexey offlist ...
> 
> Looking into this stuff offlist highlighted the following obvious
> problem: postgresql.org (as well as wwwdevel) runs on a 
> shared server that has *lots* of other tasks that can 
> negatively affect performance. 
> It is difficult to even point finger at one particular task 
> that does this.
> 
> So the solution (as was also pointed out in a private mail to 
> me) is to get a dedicated server for running the "master" website.

A /dedicated/ server is *way* over the top as I pointed out in my
previous private message to you. The current site regens all 7000+ pages
in a few minutes for example. I agree though that we could use a little
more horsepower though - I'll see if I can setup something on one of my
boxes here over the next few days.

Regards, Dave

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 02 Nov 2004 03:06:32 -0800

>boxes here over the next few days.
Hello,

A /dedicated/ server may be more than what WWW needs, but
it is not more than it deserves. Although I do not know what else
is going on with the WWW machine, a dedicated machine definitely
would not hurt.

Sincerely,

Joshua D. Drake

>Regards, Dave
>
>---------------------------(end of broadcast)---------------------------
>TIP 2: you can get off all lists at once with the unregister command
>    (send "unregister YourEmailAddressHere" to [email-masked])
>  
>

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - [email-masked] - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL

---

From: Alexey Borzov <[email-masked]>
Date: Tue, 02 Nov 2004 16:05:26 +0300

Hi,

> boxes here over the next few days.

"Dedicated" does not necessarily mean "dedicated to www only", but 
"dedicated to PostgreSQL project". Current infrastructure with shared 
servers was already found inadequate to run search, archives, etc.

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 2 Nov 2004 11:52:57 -0400 (AST)

On Tue, 2 Nov 2004, Alexey Borzov wrote:

>>> problem: postgresql.org (as well as wwwdevel) runs on a shared server that 
>>> has *lots* of other tasks that can negatively affect performance. It is 
>>> difficult to even point finger at one particular task that does this.
>>> 
>>> So the solution (as was also pointed out in a private mail to me) is to 
>>> get a dedicated server for running the "master" website.
>> 
>> 
> "Dedicated" does not necessarily mean "dedicated to www only", but "dedicated 
> to PostgreSQL project". Current infrastructure with shared servers was 
> already found inadequate to run search, archives, etc.

You still haven't answered any of the questions that I asked in a private 
email ... Is this a time of day issue (from your two samples, it looks 
like it is)?  If you set it up to run hourly, what do the #s look like for 
each run throughout the day?  What does the loadavg look like on the 
server when you are running the script?  Or is the database being slow?

Considering that Dave states above that the current script takes minutes 
to generate >7000 pages, what are you doing differently that makes it so 
much slower?  And, why exactly are we changing from the current method if 
the new method is going to require a dedicated server to run it?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [email-masked]           Yahoo!: yscrappy              ICQ: 7615664

---

From: Alexey Borzov <[email-masked]>
Date: Tue, 02 Nov 2004 23:04:52 +0300

Hi,

Marc G. Fournier wrote:
> You still haven't answered any of the questions that I asked in a 
> private email ... Is this a time of day issue (from your two samples, it 
> looks like it is)?  If you set it up to run hourly, what do the #s look 
> like for each run throughout the day?  What does the loadavg look like 
> on the server when you are running the script?  Or is the database being 
> slow?

I tried running mirror later today:

Nov 02 12:42:47 mirror [info] Mirroring finished. 423 page(s) saved, 2354 
second(s) spent

I also am running it now, getting the same ~5 sec per page response time and 
uptime command states:

  7:49PM  up 10 days,  4:36, 4 users, load averages: 4.00, 3.22, 3.04

The database itself does not look slow, nothing like these 5 seconds to connect 
/ send the standard query.

> much slower?  And, why exactly are we changing from the current method 
> if the new method is going to require a dedicated server to run it?

Yes, that's one way to ask these questions. The other way is: why are we having 
response times of ~5 seconds for not-too-complex pages?

The only thing I'm doing differently from the current one is that I'm sending 
HTTP requests to get the pages. I already outlined benefits of this approach 
either here or in a private mail.

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 2 Nov 2004 16:43:54 -0400 (AST)

On Tue, 2 Nov 2004, Alexey Borzov wrote:

>> email ... Is this a time of day issue (from your two samples, it looks like 
>> it is)?  If you set it up to run hourly, what do the #s look like for each 
>> run throughout the day?  What does the loadavg look like on the server when 
>> you are running the script?  Or is the database being slow?
>
> The database itself does not look slow, nothing like these 5 seconds to 
> connect / send the standard query.
>
>> Considering that Dave states above that the current script takes minutes to 
>> generate >7000 pages, what are you doing differently that makes it so much 
>> slower?  And, why exactly are we changing from the current method if the 
>> new method is going to require a dedicated server to run it?
>
> Yes, that's one way to ask these questions. The other way is: why are we 
> having response times of ~5 seconds for not-too-complex pages?

Your guess is as good as mine ... using my browser from here, I'm not seen 
5sec delays, and I'm two countries away from it ... using lynx on the VM 
itself, goign through the web site, I'm seeing instantaneous responses ... 
I'm definitely not seeing 5sec delays ...

Can you extend your script so that it outputs URL spider'd and time taken 
in msec for it?  The pages that I'm testing definitely aren't giving me 
5sec lag times from the server itself ... so if you can give me a 'slow 
URL' that I can look at, that would help ...

----

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 2 Nov 2004 16:49:27 -0400 (AST)

On Tue, 2 Nov 2004, Marc G. Fournier wrote:

>> I also am running it now, getting the same ~5 sec per page response time 
>> and uptime command states:
>> 
> Can you extend your script so that it outputs URL spider'd and time taken in 
> msec for it?  The pages that I'm testing definitely aren't giving me 5sec lag 
> times from the server itself ... so if you can give me a 'slow URL' that I 
> can look at, that would help ...

'k, I just looked at your mirror.php script .. and realized that its not 
the main web site thats the problem, its your new site ... why is your 
site so much slower to load up with lynx then the site that is there now? 
http://www.postgresql.org comes up instantly, 
http://www.alexey.beta.postgresql.org/index.html.en takes forever to load 
... why?

I can actually get coffee waiting for your site to load in my browser 
here, where there is almost zero delay loading the current site ...

----

---

From: Alexey Borzov <[email-masked]>
Date: Wed, 03 Nov 2004 00:32:32 +0300

Hi,

Marc G. Fournier wrote:

> site so much slower to load up with lynx then the site that is there 
> now? http://www.postgresql.org comes up instantly, 
> http://www.alexey.beta.postgresql.org/index.html.en takes forever to 
> load ... why?

That's easy: www.postgresql.org is static HTML, alexey.beta is dynamic.

I added profiling information to the pages (look at the bottom, only the index 
page has a full set of "markers") and the results are the following: ~80% of the 
page generation time (which is 3+ seconds!) is spent loading local files and 
parsing and executing PHP code. Database access is actually *very fast*. DB 
connection is slow, but that takes only ~10% of time and can be fixed by using 
persistant connections.

That also answers the question why current generation scripts work in acceptable 
time.

BTW, may there be some problems with local file access on this setup?

Dave may test the same on wwwdevel if he copies
system/handler.php
system/page.php
system/page/news-and-events.php
from alexey.beta's webroot to wwwdevel, though I doubt the results will have any 
significant difference...

> here, where there is almost zero delay loading the current site ...

OK, here is the answer: the box alexey.beta is upon is only capable of serving 
static content. It chokes on even moderately dynamic one. Buy some real 
hardware, it's not that pricey now.

---

From: Oleg Bartunov <[email-masked]>
Date: Wed, 3 Nov 2004 04:25:55 +0300 (MSK)

Its interesting to see how webperf.org sees different postgresql.org
pages. For example,

http://www.webperf.org/breakdown.html?URL=www.alexey.beta.postgresql.org
Connect 	DNS 	FirstByte 	End 	Size(Bytes) 	URL 	Status
0.033 	0.074 	3.226 	4.225 	17654 	http://www.alexey.beta.postgresql.org 	200
3.258 	3.258 	3.854 	3.933 	43 	http://www.alexey.beta.postgresql.org/layout/images/0.gif 	200
3.379 	3.379 	3.850 	4.356 	8190 	http://www.alexey.beta.postgresql.org/layout/images/logo.png 	200
3.519 	3.520 	3.930 	4.070 	2101 	http://www.alexey.beta.postgresql.org/layout/css/new.css 	200
3.710 	3.729 	4.067 	4.363 	11470 	http://ads.postgresql.org/display_image.php?lvl=sponsor 	200
3.935 	3.936 	8.678 	8.801 	2644 	http://www.alexey.beta.postgresql.org/news.rss 	200
4.072 	4.072 	7.736 	7.739 	864 	http://www.alexey.beta.postgresql.org/events.rss 	200
4.228 	4.228 	4.496 	4.499 	207 	http://www.alexey.beta.postgresql.org/layout/images/mark_menu.png 	200

It shows, that possible problem could be news.rss, events.rss.
www.postgresql.org behaves much better
http://www.webperf.org/breakdown.html?URL=http%3A%2F%2Fwww.postgresql.org
Connect 	DNS 	FirstByte 	End 	Size(Bytes) 	URL 	Status
0.033 	0.074 	0.403 	1.118 	21429 	http://www.postgresql.org 	200
0.417 	0.417 	1.082 	1.099 	929 	http://www.postgresql.org/postgresql.css 	200
0.581 	0.581 	1.220 	1.354 	4447 	http://www.postgresql.org/images/banner-left.jpg 	200
0.700 	0.700 	1.211 	1.336 	2929 	http://www.postgresql.org/images/banner-right.jpg 	200
0.823 	0.841 	1.259 	1.522 	14397 	http://ads.postgresql.org/display_image.php?lvl=sponsor 	200
0.955 	0.955 	1.252 	1.498 	14397 	http://ads.postgresql.org/display_image.php?lvl=free 	200

But, I again want to  stress that current architecture of postgresql.org sites
are very-very bad and you will never get nice performance and scalability
especially for dynamic sites without:

1.  Redirecting different traffic to different servers.

     a)images and other binary things should be served by small and fast
     server like thttpd or lighttpd. Currently, images are served by
     apache+php server which is wasting of resources ! These servers
     could easily serve several thousands req/sec, so one images.postgresql.org
     could serve all such kind of traffic.

     b)requests for dynamic pages should be forwarded to backend servers
       with php compiled and keeping persistent connection to db.
       These backends are fully hidden and accessable only by localhost,
       so they will not stay busy while serving could-be-slow client
       (slow connectivity), That means resources (memory)  will be not wasted
       and there is no demand to have many servers online, because interaction
       with clients will be server only by frontends, see below

     c)requests for static pages are served by plain httpd server like apache -
       frontend. These servers should have cacheing capability to cache
       answers from dynamic servers, so many-many requests for dynamic pages
       will not actualy forwarded to heavy backends ! Our pages are not actually
       very dynamic.

     This is a classical 3 servers setup and it described in many books and
     success stories.

 	Oleg
On Tue, 2 Nov 2004, Marc G. Fournier wrote:

>>>> like it is)?  If you set it up to run hourly, what do the #s look like 
>>>> for each run throughout the day?  What does the loadavg look like on the 
> 'k, I just looked at your mirror.php script .. and realized that its not the 
> main web site thats the problem, its your new site ... why is your site so 
> much slower to load up with lynx then the site that is there now? 
> http://www.alexey.beta.postgresql.org/index.html.en takes forever to load ... 
> why?
>
> I can actually get coffee waiting for your site to load in my browser here, 
> where there is almost zero delay loading the current site ...
>
> ----
> TIP 7: don't forget to increase your free space map settings
>

 	Regards,
 		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email-masked], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

---

From: Alexey Borzov <[email-masked]>
Date: Wed, 03 Nov 2004 13:35:56 +0300

Hi,

> It shows, that possible problem could be news.rss, events.rss.

That's because these pages are dynamic, too. But I think it's an error in 
webperf that it considers them part of index page.

>     This is a classical 3 servers setup and it described in many books and

Your suggestion is nice and all, but right now we are discussing the fact that 
we don't even have *one* dedicated server. Only a shared account on substandard 
  hardware, incapable of running PHP scripts. :(

---

From: "Marc G. Fournier" <[email-masked]>
Date: Wed, 3 Nov 2004 08:04:27 -0400 (AST)

On Wed, 3 Nov 2004, Alexey Borzov wrote:

> Your suggestion is nice and all, but right now we are discussing the fact 
> that we don't even have *one* dedicated server. Only a shared account on 
> substandard  hardware, incapable of running PHP scripts. :(

This has yet to be proven, IMHO ... Dave has already stated that the old 
build scripts aren't nearly as slow as yours, and it was having to deal 
with PHP as well ...

Your first "complaint" was that it wasn't loading the files fast enough 
... so, I enabled mmcache on that server with shared memory use only, so 
that it doesn't use disk except for the first load ... the build is still 
atrociously slow ... the servers are all setup so that shared memory is 
locked into physical RAM, so now what?  The RAM is too slow?

If anyone wants to look at the PHP settings on that server, check out 
http://www.postgresql.org/phpinfo.php ... is there something that should 
be different on compile?  or something that is turned on that might cause 
a slowdown?  I'm using pretty much default FreeBSD ports options, in so 
far as build and php.ini is concerned ... maybe there is something in 
there that should be changed to optimize better?  The server itself is 
doing neglible swapping (<6M) and unless I've been reading top wrong all 
my life (which is definitely a possibility) its reporting 1G of 'Inact' 
RAM, which I believe means that available for processes ... ?

----

---

From: Alexey Borzov <[email-masked]>
Date: Wed, 03 Nov 2004 17:52:28 +0300

Hi,

Marc G. Fournier wrote:
>> That's because these pages are dynamic, too. But I think it's an error 
>> in webperf that it considers them part of index page.
>>
>> Your suggestion is nice and all, but right now we are discussing the 
>> fact that we don't even have *one* dedicated server. Only a shared 
>> account on substandard  hardware, incapable of running PHP scripts. :(
> 
> that it doesn't use disk except for the first load ... the build is 
> still atrociously slow ... the servers are all setup so that shared 
> memory is locked into physical RAM, so now what?  The RAM is too slow?

In case I was unclear before: due to some f**kup in server configuration 
(probably well below the PHP level) file access is EXTREMELY slow. And there 
*is* file access now due to templates and gettext, even if PHP scripts are cached.

If you look at profiling information at http://alexey.beta.postgresql.org/, 
you'll see that just reading files (loadedTemplate, loadedInnerTemplate, 
loadedIndexTemplate) takes 0.2--0.5 seconds. Initializing gettext (which relies 
on files, too) takes 0.1 seconds (setLanguage).

One also has to wonder whether there is another configuration f**kup that makes 
script connect to PostgreSQL for 0.1--0.5 seconds.

One *also* has to wonder what (if not server overload) makes these times 
fluctuate a lot.

A well tuned webserver on adequate hardware will generate the whole page of 
alexey.beta in less than 0.1 second.

Current build scripts DO NOT DO FILE ACCESS, that's why they work faster. But 
that's irrelevant, 'cause there are pages on new website that should be dynamic 
and thus will be accessed by visitors on the master server. Page generation 
times of ~2 seconds will make PostgreSQL project look BAD.

As for the nature of aforementioned configuration f**kups --- no idea. *You* are 
the admin.

---

From: Mitch Pirtle <[email-masked]>
Date: Wed, 3 Nov 2004 11:57:57 -0500

>   hardware, incapable of running PHP scripts. :(

And several people have offered over the last few months, including
myself.  I am thinking about a dual xeon with 1 GB RAM that has a load
average of 0.01...  Nightly backups too, with disk a non-factor.  I
asked about bandwidth usage estimates but never got that far :-(

This I simply do not understand.  If there is a need for dedicated
hardware, and it is being offered, then why is it not being accepted? 
Didn't CommandPrompt also offer dedicated hardware?

Not wanting to start another heated debate, just wanting to understand
what the deal is - as this topic keeps getting regurgitated without a
conclusion.

-- Mitch