pg.ddx.io  pgsql-rrreviewers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Craig Ringer <craig@2ndquadrant.com>
To: Josh Berkus <josh@agliodbs.com>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>
Cc: pgsql-rrreviewers@postgresql.org
Subject: Re: [HACKERS] [9.4 CF] Free VMs for Reviewers & Testers
Date: Tue, 09 Jul 2013 10:25:50 +0800
Message-ID: <51DB74AE.9070301@2ndquadrant.com> (raw)
In-Reply-To: <51DB5AE3.8030104@agliodbs.com>
References: <51DB5AE3.8030104@agliodbs.com>
List-Unsubscribe:  <mailto:majordomo@postgresql.org?body=unsub%20pgsql-rrreviewers>

On 07/09/2013 08:35 AM, Josh Berkus wrote:
> Since these are cloud servers, they won't work well for performance
> testing. 

I did some work on that a while ago, and found that I was able to get
_astonishingly_ stable performance results out of EC2 EBS instances
using provisioned IOPS volumes. Running them as "EBS Optimized" didn't
seem to be required for the workloads I was testing on.

What isn't made very clear in the EBS provisioned I/O docs is that for
PIOPS volumes the provisioned I/O count is both a minimum and a limit;
it's a *target*, not just a guaranteed minimum. This is really handy for
testing.

For a 500 iops volume I consistently got 500 +- 5 in pg_test_fsync.

Real world performance was a bit more variable - my pgbench runs were
stable within 5-10% over 5 minute runs with a forced CHECKPOINT first.
They settled considerably over longer runs and bigger
checkpoint_segments, though, and I see the same kind of short-term
jitter in pgbench on real hardware.

So - don't write cloud hosts off for benchmarking. With proper QoS
they're actually really very good. While they're somewhat costly, being
able to spawn them for a day's runs and destroy them at the end is quite
handy, as is being able to easily automate their setup.

For testing of logical changeset streaming replication / BDR performance
I was using scripted sets of EC2 c1.medium instances with 500 piops
'io1' EBS volumes and found it to be an exceedingly useful way of
measuring relative performance.

Results were also pretty consistent across multiple launches of the same
VM, so multiple people should be able to compare results obtained with
different launches of the same VM type and configuration.

These VMs aren't well suited to vertical scaling performance tests and
pushing extremes, but they're really, really good for "what impact does
this patch have on regular real-world workloads".

Just remember to run pgbench from a different vm within the same
availability zone (with appropriate security group setup) if you try to
use EC2 for this sort of thing.

Use of instance store or regular EBS volumes will get you performance
that's absolutely all over the place, of course; only the provisioned
I/O feature seems to be any good for perf testing.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-rrreviewers mailing list (pgsql-rrreviewers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-rrreviewers



view thread (6+ messages)  latest in thread

Message-ID: <51DB74AE.9070301@2ndquadrant.com>
Permalink:  ../51DB74AE.9070301@2ndquadrant.com/
Also on:    postgresql.org/message-id/51DB74AE.9070301@2ndquadrant.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-rrreviewers@postgresql.org
  Cc: craig@2ndquadrant.com, josh@agliodbs.com, pgsql-hackers@postgresql.org
  Subject: Re: [HACKERS] [9.4 CF] Free VMs for Reviewers & Testers
  In-Reply-To: <51DB74AE.9070301@2ndquadrant.com>
  References: <51DB5AE3.8030104@agliodbs.com> <51DB74AE.9070301@2ndquadrant.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox