Received: from localhost (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id E3A8864FD14 for ; Mon, 22 Dec 2008 04:50:24 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 71460-06 for ; Mon, 22 Dec 2008 04:50:22 -0400 (AST) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.232]) by mail.postgresql.org (Postfix) with ESMTP id 0B94B64FC2F for ; Mon, 22 Dec 2008 04:50:21 -0400 (AST) Received: by rv-out-0506.google.com with SMTP id b25so3291721rvf.43 for ; Mon, 22 Dec 2008 00:50:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=D5qHiqK/7jpic88Qi9zhirOxniAwP+++jx+/gRL5GSI=; b=q+13rjCPKQxvrOp7g3Wge6x/w4og6bea+WQAWO6uUUo4Hg8M6/iTih12ASL3Jw8ktf S0yVZN/Pt3GY7g9upkYTCVGiaYh3GIZVBOR/FkdPur3Kq1OqquEjoPNLbmpKcsmQXFRN f1zn+4nIZ/Yd2S+QOJrgxVZ9xygGN0JCcvXyE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=bnkvyr6sZ8i8qGWH63tn96zB53bhTfpMd5BXoIVg7Xs5f2dJTWn1/QZ4mgqHEhjFkd /P23PnWUBCBDoacPttkv2C2fHc7URYBdrEtcf2AeZjnEPYP5IyzvJhhznt/QomOQXQgI ix/nDxoPpFMMVD3x1c513ZrKNJc9Cr81t0s0E= Received: by 10.140.164.6 with SMTP id m6mr3086777rve.29.1229935821021; Mon, 22 Dec 2008 00:50:21 -0800 (PST) Received: by 10.141.87.11 with HTTP; Mon, 22 Dec 2008 00:50:20 -0800 (PST) Message-ID: <70c01d1d0812220050kb1f755etea3ecada3f3b8631@mail.gmail.com> Date: Mon, 22 Dec 2008 00:50:20 -0800 From: "Mark Wong" To: "Gregory Stark" Subject: Re: dbt-2 tuning results with postgresql-8.3.5 Cc: "pgsql-performance@postgresql.org" , "Selena Deckelmann" , "Gabrielle Roth" In-Reply-To: <87d4fkvgtj.fsf@oxford.xeocode.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <70c01d1d0812201654i5f55150ep619a6c7ff094470@mail.gmail.com> <87hc4yuxbl.fsf@oxford.xeocode.com> <552F108E-9732-4ED6-B45D-6C651C63565E@gmail.com> <87d4fkvgtj.fsf@oxford.xeocode.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=NORMAL_HTTP_TO_IP=0.001 X-Spam-Level: X-Archive-Number: 200812/181 X-Sequence-Number: 32217 On Sun, Dec 21, 2008 at 10:56 PM, Gregory Stark wrote: > Mark Wong writes: > >> On Dec 20, 2008, at 5:33 PM, Gregory Stark wrote: >> >>> "Mark Wong" writes: >>> >>>> To recap, dbt2 is a fair-use derivative of the TPC-C benchmark. We >>>> are using a 1000 warehouse database, which amounts to about 100GB of >>>> raw text data. >>> >>> Really? Do you get conforming results with 1,000 warehouses? What's the 95th >>> percentile response time? >> >> No, the results are not conforming. You and others have pointed that out >> already. The 95th percentile response time are calculated on each page of the >> previous links. > > Where exactly? Maybe I'm blind but I don't see them. Here's an example: http://207.173.203.223/~markwkm/community6/dbt2/baseline.1000.1/report/ The links on the blog entries should be pointing to their respective reports. I spot checked a few and it seems I got some right. I probably didn't make it clear you needed to click on the results to see the reports. >> I find your questions a little odd for the input I'm asking for. Are you >> under the impression we are trying to publish benchmarking results? Perhaps >> this is a simple misunderstanding? > > Hm, perhaps. The "conventional" way to run TPC-C is to run it with larger and > larger scale factors until you find out the largest scale factor you can get a > conformant result at. In other words the scale factor is an output, not an > input variable. > > You're using TPC-C just as an example workload and looking to see how to > maximize the TPM for a given scale factor. I guess there's nothing wrong with > that as long as everyone realizes it's not a TPC-C benchmark. Perhaps, but we're not trying to run a TPC-C benchmark. We're trying to illustrate how performance changes with an understood OLTP workload. The purpose is to show how the system bahaves more so than what the maximum transactions are. We try to advertise the kit the and work for self learning, we never try to pass dbt-2 off as a benchmarking kit. > Except that if the 95th percentile response times are well above a second I > have to wonder whether the situation reflects an actual production OLTP system > well. It implies there are so many concurrent sessions that any given query is > being context switched out for seconds at a time. > > I have to imagine that a real production system would consider the system > overloaded as soon as queries start taking significantly longer than they take > on an unloaded system. People monitor the service wait times and queue depths > for i/o systems closely and having several seconds of wait time is a highly > abnormal situation. We attempt to illustrate the response times on the reports. For example, there is a histogram (drawn as a scatter plot) illustrating the number of transactions vs. the response time for each transaction. This is for the New Order transaction: http://207.173.203.223/~markwkm/community6/dbt2/baseline.1000.1/report/dist_n.png We also plot the response time for a transaction vs the elapsed time (also as a scatter plot). Again, this is for the New Order transaction: http://207.173.203.223/~markwkm/community6/dbt2/baseline.1000.1/report/rt_n.png > I'm not sure how bad that is for the benchmarks. The only effect that comes to > mind is that it might exaggerate the effects of some i/o intensive operations > that under normal conditions might not cause any noticeable impact like wal > log file switches or even checkpoints. I'm not sure I'm following. Is this something than can be shown by any stats collection or profiling? This vaguely reminds me of the the significant spikes in system time (and dips everywhere else) when the operating system is fsyncing during a checkpoint that we've always observed when running this in the past. > If you have a good i/o controller it might confuse your results a bit when > you're comparing random and sequential i/o because the controller might be > able to sort requests by physical position better than in a typical oltp > environment where the wait queues are too short to effectively do that. Thanks for the input. Regards, Mark