X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (neptune.hub.org [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id 154F1D1B435 for ; Tue, 25 Nov 2003 23:41:18 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 55137-10 for ; Tue, 25 Nov 2003 19:40:48 -0400 (AST) Received: from sss.pgh.pa.us (unknown [192.204.191.242]) by svr1.postgresql.org (Postfix) with ESMTP id E055DD1B432 for ; Tue, 25 Nov 2003 19:40:46 -0400 (AST) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.12.10/8.12.10) with ESMTP id hAPNel19009471; Tue, 25 Nov 2003 18:40:47 -0500 (EST) To: josh@agliodbs.com Cc: pgsql-performance@postgresql.org Subject: Re: Wierd context-switching issue on Xeon In-reply-to: <200311251419.36771.josh@agliodbs.com> References: <200311251419.36771.josh@agliodbs.com> Comments: In-reply-to Josh Berkus message dated "Tue, 25 Nov 2003 14:19:36 -0800" Date: Tue, 25 Nov 2003 18:40:47 -0500 Message-ID: <9470.1069803647@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200311/259 X-Sequence-Number: 4819 Josh Berkus writes: > We're seeing some odd issues with hyperthreading-capable Xeons, whether or not > hyperthreading is enabled. Basically, when a small number of really-heavy > duty queries hit the system and push all of the CPUs to more than 70% used > (about 1/2 user & 1/2 kernel), the system goes to 100,000+ context switcthes > per second and performance degrades. Strictly a WAG ... but what this sounds like to me is disastrously bad behavior of the spinlock code under heavy contention. We thought we'd fixed the spinlock code for SMP machines awhile ago, but maybe hyperthreading opens some new vistas for misbehavior ... > I know that there's other Xeon users on this list ... has anyone else seen > anything like that? The machines are Dells running Red Hat 7.3. What Postgres version? Is it easy for you to try 7.4? If we were really lucky, the random-backoff algorithm added late in 7.4 development would cure this. If you can't try 7.4, or want to gather more data first, it would be good to try to confirm or disprove the theory that the context switches are coming from spinlock delays. If they are, they'd be coming from the select() calls in s_lock() in s_lock.c. Can you strace or something to see what kernel calls the context switches occur on? Another line of thought is that RH 7.3 is a long ways back, and it wasn't so very long ago that Linux still had lots of SMP bugs. Maybe what you really need is a kernel update? regards, tom lane