From: Matthew Nuzum <mattnuzum@gmail.com>
To: Postgresql Performance list <pgsql-performance@postgresql.org>
Subject: Logarithmic change (decrease) in performance
Date: Wed, 28 Sep 2005 15:02:51 -0500
Message-ID: <f3c0b40805092813026ad4466c@mail.gmail.com> (raw)
Something interesting is going on. I wish I could show you the graphs,
but I'm sure this will not be a surprise to the seasoned veterans.
A particular application server I have has been running for over a
year now. I've been logging cpu load since mid-april.
It took 8 months or more to fall from excellent performance to
"acceptable." Then, over the course of about 5 weeks it fell from
"acceptable" to "so-so." Then, in the last four weeks it's gone from
"so-so" to alarming.
I've been working on this performance drop since Friday but it wasn't
until I replied to Arnau's post earlier today that I remembered I'd
been logging the server load. I grabbed the data and charted it in
Excel and to my surprise, the graph of the server's load average looks
kind of like the graph of y=x^2.
I've got to make a recomendation for a solution to the PHB and my
analysis is showing that as the dataset becomes larger, the amount of
time the disk spends seeking is increasing. This causes processes to
take longer to finish, which causes more processes to pile up, which
cuases processes to take longer to finish, which causes more processes
to pile up etc. It is this growing dataset that seems to be the source
of the sharp decrease in performance.
I knew this day would come, but I'm actually quite surprised that when
it came, there was little time between the warning and the grande
finale. I guess this message is being sent to the list to serve as a
warning to other data warehouse admins that when you reach your
capacity, the downward spiral happens rather quickly.
Crud... Outlook just froze while composing the PHB memo. I've been
working on that for an hour. What a bad day.
--
Matthew Nuzum
www.bearfruit.org
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: mattnuzum@gmail.com
Subject: Re: Logarithmic change (decrease) in performance
In-Reply-To: <f3c0b40805092813026ad4466c@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox