X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id ABDC0D1DA2C for ; Thu, 12 Feb 2004 22:58:16 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 49726-01 for ; Thu, 12 Feb 2004 18:58:15 -0400 (AST) Received: from polaris.pinpointresearch.com (66-7-238-130.cust.telepacific.net [66.7.238.130]) by svr1.postgresql.org (Postfix) with ESMTP id 6A93FD1DA74 for ; Thu, 12 Feb 2004 18:58:12 -0400 (AST) Received: from betelgeuse.pinpointresearch.com (betelgeuse.pinpointresearch.com [192.168.1.179]) by polaris.pinpointresearch.com (Postfix) with ESMTP id 37FC2103C7; Thu, 12 Feb 2004 14:57:59 -0800 (PST) From: Steve Crawford Organization: Pinpoint Research To: gjm@caledoncard.com (Greg Mennie), pgsql-admin@postgresql.org Subject: Re: hanging for 30sec when checkpointing Date: Thu, 12 Feb 2004 14:57:59 -0800 User-Agent: KMail/1.5.1 References: <40202216.4010608@shanewright.co.uk> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200402121457.59468.scrawford@pinpointresearch.com> X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200402/189 X-Sequence-Number: 12363 > > I'm running a reasonable sized (~30Gb) 7.3.4 database on Linux > > and I'm getting some weird performance at times. > I am having a similar problem and this is what I've found so far: > > During the checkpoint the volume of data that's written isn't very > high and it goes on for a fairly long time (up to 20 seconds) at a > rate that appears to be well below our disk array's potential. The > volume of data written is usually 1-5 MB/sec on an array that we've > tested to sustain over 50 MB/sec (sequential writes, of course). > > It turns out that what's going on is that the command queue for the > RAID array (3Ware RAID card) is filling up during the checkpoint > and is staying at the max (254 commands) for most of the > checkpoint. The odd lucky insert appears to work, but is extremely > slow. In our case, the WAL files are on the same array as the data > files, so everything grinds to a halt. I spoke with some 3Ware reps at a trade show and they recommended adding the following to /etc/sysctl.conf: vm.max-readahead = 256 vm.min-readahead = 128 These settings take effect at boot. To change on a running system: echo 256 > /proc/sys/vm/max-readahead echo 128 > /proc/sys/vm/min-readahead This advice was specific to the 3Ware card on Linux. Cheers, Steve