X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (neptune.hub.org [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id EB0F3D1D56A for ; Sun, 11 Jan 2004 02:36:03 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 85381-03 for ; Sat, 10 Jan 2004 22:35:35 -0400 (AST) Received: from news.hub.org (news.hub.org [200.46.204.72]) by svr1.postgresql.org (Postfix) with ESMTP id 2DCC1D1D643 for ; Sat, 10 Jan 2004 22:35:32 -0400 (AST) Received: from news.hub.org (news.hub.org [200.46.204.72]) by news.hub.org (8.12.9/8.12.9) with ESMTP id i0B2ZVU6099942 for ; Sun, 11 Jan 2004 02:35:31 GMT (envelope-from news@news.hub.org) Received: (from news@localhost) by news.hub.org (8.12.9/8.12.9/Submit) id i0B2A1gH065715 for pgsql-performance@postgresql.org; Sun, 11 Jan 2004 02:10:01 GMT From: Christopher Browne X-Newsgroups: comp.databases.postgresql.performance Subject: Re: failures on machines using jfs Date: Sat, 10 Jan 2004 21:08:50 -0500 Organization: Hub.Org Networking Services Lines: 56 Message-ID: <60k73zck3x.fsf@dev6.int.libertyrms.info> References: <20040107230608.GN12531@libertyrms.info> <20040109202816.38072cf8.Robert_Creager@LogicalChaos.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Complaints-To: usenet@news.hub.org User-Agent: Gnus/5.1004 (Gnus v5.10.4) XEmacs/21.4 (Reasonable Discussion, linux) Cancel-Lock: sha1:WSkb524vYimex3rYkNGK2HYPAYI= To: pgsql-performance@postgresql.org X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200401/103 X-Sequence-Number: 5278 Robert_Creager@LogicalChaos.org (Robert Creager) writes: > When grilled further on (Wed, 7 Jan 2004 18:06:08 -0500), > Andrew Sullivan confessed: > >> We have lately had a couple of cases where machines either locked >> up, slowed down to the point of complete unusability, or died >> completely while using jfs. We are _not_ sure that jfs is in fact >> the culprit. In one case, a kernel panic appeared to be referring >> to the jfs kernel module, but I can't be sure as I lost the output >> immediately thereafter. Yesterday, we had a problem of data >> corruption on a failed jfs volume. >> >> None of this is to say that jfs is in fact to blame, nor even that, >> if it is, it does not have something to do with the age of our >> installations, &c. (these are all RH 8). In fact, I suspect >> hardware in both cases. But I thought I'd mention it just in case >> other people are seeing strange behaviour, on the principle of >> "better safe than sorry." > > Interestingly enough, I'm using JFS on a new scsi disk with Mandrake > 9.1 and was having similar problems. I was generating heavy disk > usage through database and astronomical data reductions. My machine > (dual AMD) would suddenly hang. No new jobs would run, just > increase the load, until I reboot the machine. > > I solved my problems by creating a 128Mb ram disk (using EXT2) for > the temp data produced my reduction runs. > > I believe JFS was to blame, not hardware, but you never know... Interesting. The set of concurrent factors that came together to appear when this happened "consistently" were thus: 1. Heavy DB updates taking place on JFS filesystems; 2. SMP (we suspected Xeon hyperthreading as a possible factor, but shut it off and still saw the same problem...) 3. The third factor that appeared a catalyst was copying, via scp, a file > 2GB in size onto the system. The third piece was a particularly interesting aspect; the file would get copied over successfully, and the scp process would hang (to the point of "kill -9" being unable to touch it) immediately thereafter. At that point, processes on the system that were accessing files on the hung-up filesystem were locked, also unkillable by "kill 9." That's certainly consistent with JFS being at the root of the problem, whether it was the cause or not... -- let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];; Christopher Browne (416) 646 3304 x124 (land)