Message-ID: <4FF63E73.1080104@2ndQuadrant.com>
Date: Thu, 05 Jul 2012 21:25:07 -0400
From: Greg Smith <greg@2ndQuadrant.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.24) Gecko/20111120 Icedove/3.1.16
MIME-Version: 1.0
To: Reza Taheri <rtaheri@vmware.com>
CC: "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>,
	"Andy Bond (abond@redhat.com)" <abond@redhat.com>,
	Greg Kopczynski <gregwk@vmware.com>, Jignesh Shah <jshah@vmware.com>
Subject: Re: Introducing the TPC-V benchmark, and its relationship
	to PostgreSQL
References: 
 <66CE997FB523C04E9749452273184C6C137CB88A82@exch-mbx-113.vmware.com>
In-Reply-To: 
 <66CE997FB523C04E9749452273184C6C137CB88A82@exch-mbx-113.vmware.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

On 07/03/2012 07:08 PM, Reza Taheri wrote:
> TPC-V is a new benchmark under development for virtualized databases. A
> TPC-V configuration has:
>
> - multiple virtual machines running a mix of DSS, OLTP, and business
> logic apps
>
> - VMs running with throughputs ranging from 10% to 40% of the total system
> ..

I think this would be a lot more interesting to the traditional, 
dedicated hardware part of the PostgreSQL community if there was a clear 
way to run this with only a single active machine too.  If it's possible 
for us to use this to compare instances of PostgreSQL on dedicated 
hardware, too, that is enormously more valuable to people with larger 
installations.  It might be helpful to VMWare as well.  Being able to 
say "this VM install gets X% of the performance of a bare-metal install" 
answers a question I get asked all the time--when people want to decide 
between dedicated and virtual setups.

The PostgreSQL community could use a benchmark like this for its own 
performance regression testing too.  A lot of that work is going to 
happen on a dedicated machines.

 > After waving our hands through a number
> of small differences between the platforms, we have calculated a CPU
> cost of around 3.2ms/transaction for the published MS SQL results,
> versus a measurement of 8.6ms/transaction for PostgreSQL. (TPC
> benchmarks are typically pushed to full CPU utilization. One removes all
> bottlenecks in storage, networking, etc., to achieve the 100% CPU usage.
> So CPU cost/tran is the final decider of performance.) So we need to cut
> the CPU cost of transactions in half to make publications with
> PostgreSQL comparable to commercial databases.

I appreciate that getting close to parity here is valuable.  This 
situation is so synthetic though--removing other bottlenecks and looking 
at CPU timing--that it's hard to get too excited about optimizing for 
it.  There's a lot of things in PostgreSQL that we know are slower than 
commercial databases because they're optimized for flexibility (the way 
operators are implemented is the best example) or for long-term code 
maintenance.  Microsoft doesn't care if they have terribly ugly code 
that runs faster, because no one sees that code.  PostgreSQL does care.

The measure that's more fair is a system cost based ones.  What I've 
found is that a fair number of people note PostgreSQL's low-level code 
isn't quite as fast as some of the less flexible alternatives--hard 
coding operators is surely cheaper than looking them up each time--but 
the license cost savings more than pays for bigger hardware to offset 
that.  I wish I had any customer whose database was CPU bound, that 
would be an awesome world to live in.

Anyway, guessing at causes here is premature speculation. When there's 
some code for the test kit published, at that point discussing the 
particulars of why it's not running well will get interesting.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com