Message-ID: <50013279.4090407@ringerc.id.au>
Date: Sat, 14 Jul 2012 16:48:57 +0800
From: Craig Ringer <ringerc@ringerc.id.au>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:13.0) Gecko/20120605 Thunderbird/13.0
MIME-Version: 1.0
To: B Sreejith <bsreejithin@gmail.com>
CC: Robert Klemme <shortcutter@googlemail.com>,
	pgsql-performance@postgresql.org,
	"sreejith. balakrishnan" <sreejith.balakrishnan@tcs.com>
Subject: Re: Any tool/script available which can be used to measure
	scalability of an application's database.
References: 
 <OF43A6984C.A368D80E-ON65257A37.00211A41-65257A37.002DEE2E@tcs.com>
	<CAM9pMnMcsU0qWb_a2xPHdJ5oobscptE5votqY2T95GnRw7c=pA@mail.gmail.com>
	<CANVt3iY4ipHJG3+j=c6iO7GDRBT4m-hKAMD=mkf9P8k6q8S_Uw@mail.gmail.com>
In-Reply-To: 
 <CANVt3iY4ipHJG3+j=c6iO7GDRBT4m-hKAMD=mkf9P8k6q8S_Uw@mail.gmail.com>
Content-Type: multipart/alternative;
	boundary="------------050700070405040603070904"

This is a multi-part message in MIME format.
--------------050700070405040603070904
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 07/14/2012 09:26 AM, B Sreejith wrote:
>
> Dear Robert,
>
> We need to scale up both size and load.
> Could you please provide steps I need to follow.
>

For load, first you need to build a representative sample of your 
application's querying patterns by logging queries and analysing the 
logs. Produce a load generator based on that data, set up a test copy of 
your database, and start pushing the query rate up to see what happens.

For simpler loads you can write a transaction script for pgbench based 
on your queries.

For size: Copy your data set, then start duplicating it with munged 
copies. Repeat, then use the load generator you wrote for the first part 
to see how scaling the data up affects your queries. See if anything is 
unacceptably slow (the "auto_explain" module is useful here) and examine it.

The truth is that predicting how complex database driven apps will scale 
is insanely hard, because access patterns change as data sizes and user 
counts grow. You're likely to land up tuning for a scenario that's quite 
different to the one that you actually face when you start hitting 
scaling limitations. This doesn't mean you should not investigate, it 
just means your trials don't prove anything and the optimisations you 
make based on what you learn may not gain you much.

--
Craig Ringer

--------------050700070405040603070904
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 07/14/2012 09:26 AM, B Sreejith
      wrote:<br>
    </div>
    <blockquote
cite="mid:CANVt3iY4ipHJG3+j=c6iO7GDRBT4m-hKAMD=mkf9P8k6q8S_Uw@mail.gmail.com"
      type="cite">
      <meta http-equiv="Context-Type" content="text/html;
        charset=ISO-8859-1">
      <p>Dear Robert,</p>
      <p>We need to scale up both size and load.<br>
        Could you please provide steps I need to follow.<br>
      </p>
    </blockquote>
    <br>
    For load, first you need to build a representative sample of your
    application's querying patterns by logging queries and analysing the
    logs. Produce a load generator based on that data, set up a test
    copy of your database, and start pushing the query rate up to see
    what happens.<br>
    <br>
    For simpler loads you can write a transaction script for pgbench
    based on your queries.<br>
    <br>
    For size: Copy your data set, then start duplicating it with munged
    copies. Repeat, then use the load generator you wrote for the first
    part to see how scaling the data up affects your queries. See if
    anything is unacceptably slow (the "auto_explain" module is useful
    here) and examine it.<br>
    <br>
    The truth is that predicting how complex database driven apps will
    scale is insanely hard, because access patterns change as data sizes
    and user counts grow. You're likely to land up tuning for a scenario
    that's quite different to the one that you actually face when you
    start hitting scaling limitations. This doesn't mean you should not
    investigate, it just means your trials don't prove anything and the
    optimisations you make based on what you learn may not gain you
    much.<br>
    <br>
    --<br>
    Craig Ringer<br>
  </body>
</html>

--------------050700070405040603070904--