X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id 66519D1D064 for ; Sun, 18 Apr 2004 20:32:29 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 58765-09 for ; Sun, 18 Apr 2004 20:32:29 -0300 (ADT) Received: from net2.micro-automation.com (net2.micro-automation.com [64.7.141.29]) by svr1.postgresql.org (Postfix) with SMTP id 916B0D1CACC for ; Sun, 18 Apr 2004 20:32:25 -0300 (ADT) Received: (qmail 10377 invoked from network); 18 Apr 2004 23:32:23 -0000 Received: from dcdsl.ebox.com (HELO ?192.168.1.36?) (davec@64.7.143.116) by net2.micro-automation.com with SMTP; 18 Apr 2004 23:32:24 -0000 Subject: Re: Wierd context-switching issue on Xeon From: Dave Cramer Reply-To: pg@fastcrypt.com To: Tom Lane Cc: lutzeb@aeccom.com, Josh Berkus , pgsql-performance@postgreSQL.org, Neil Conway In-Reply-To: <11437.1082324861@sss.pgh.pa.us> References: <200311251419.36771.josh@agliodbs.com> <200404151039.17763.josh@agliodbs.com> <407EE93F.3090705@aeccom.com> <200404151335.27762.josh@agliodbs.com> <407EFC95.5070805@aeccom.com> <11437.1082324861@sss.pgh.pa.us> Content-Type: text/plain Organization: Cramer Consulting Message-Id: <1082331281.1557.47.camel@localhost.localdomain> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Sun, 18 Apr 2004 19:34:41 -0400 Content-Transfer-Encoding: 7bit X-AntiVirus: scanned for viruses by AMaViS 0.2.1 (http://amavis.org/) X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200404/252 X-Sequence-Number: 6552 So the the kernel/OS is irrelevant here ? this happens on any dual xeon? What about hypterthreading does it still happen if HTT is turned off ? Dave On Sun, 2004-04-18 at 17:47, Tom Lane wrote: > After some further digging I think I'm starting to understand what's up > here, and the really fundamental answer is that a multi-CPU Xeon MP box > sucks for running Postgres. > > I did a bunch of oprofile measurements on a machine belonging to one of > Josh's clients, using a test case that involved heavy concurrent access > to a relatively small amount of data (little enough to fit into Postgres > shared buffers, so that no I/O or kernel calls were really needed once > the test got going). I found that by nearly any measure --- elapsed > time, bus transactions, or machine-clear events --- the spinlock > acquisitions associated with grabbing and releasing the BufMgrLock took > an unreasonable fraction of the time. I saw about 15% of elapsed time, > 40% of bus transactions, and nearly 100% of pipeline-clear cycles going > into what is essentially two instructions out of the entire backend. > (Pipeline clears occur when the cache coherency logic detects a memory > write ordering problem.) > > I am not completely clear on why this machine-level bottleneck manifests > as a lot of context swaps at the OS level. I think what is happening is > that because SpinLockAcquire is so slow, a process is much more likely > than you'd normally expect to arrive at SpinLockAcquire while another > process is also acquiring the spinlock. This puts the two processes > into a "lockstep" condition where the second process is nearly certain > to observe the BufMgrLock as locked, and be forced to suspend itself, > even though the time the first process holds the BufMgrLock is not > really very long at all. > > If you google for Xeon and "cache coherency" you'll find quite a bit of > suggestive information about why this might be more true on the Xeon > setup than others. A couple of interesting hits: > > http://www.theinquirer.net/?article=10797 > says that Xeon MP uses a *slower* FSB than Xeon DP. This would > translate directly to more time needed to transfer a dirty cache line > from one processor to the other, which is the basic operation that we're > talking about here. > > http://www.aceshardware.com/Spades/read.php?article_id=30000187 > says that Opterons use a different cache coherency protocol that is > fundamentally superior to the Xeon's, because dirty cache data can be > transferred directly between two processor caches without waiting for > main memory. > > So in the short term I think we have to tell people that Xeon MP is not > the most desirable SMP platform to run Postgres on. (Josh thinks that > the specific motherboard chipset being used in these machines might > share some of the blame too. I don't have any evidence for or against > that idea, but it's certainly possible.) > > In the long run, however, CPUs continue to get faster than main memory > and the price of cache contention will continue to rise. So it seems > that we need to give up the assumption that SpinLockAcquire is a cheap > operation. In the presence of heavy contention it won't be. > > One thing we probably have got to do soon is break up the BufMgrLock > into multiple finer-grain locks so that there will be less contention. > However I am wary of doing this incautiously, because if we do it in a > way that makes for a significant rise in the number of locks that have > to be acquired to access a buffer, we might end up with a net loss. > > I think Neil Conway was looking into how the bufmgr might be > restructured to reduce lock contention, but if he had come up with > anything he didn't mention exactly what. Neil? > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > > > > !DSPAM:4082feb7326901956819835! > > -- Dave Cramer 519 939 0336 ICQ # 14675561