Received: from magus.postgresql.org (magus.postgresql.org [87.238.57.229]) by mail.postgresql.org (Postfix) with ESMTP id CF8E72C777F for ; Tue, 17 Jul 2012 21:32:13 -0300 (ADT) Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1SrIBU-0001gx-9k for pgsql-hackers@postgresql.org; Wed, 18 Jul 2012 00:32:12 +0000 Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.5/8.14.5) with ESMTP id q6I0Vqhp011100; Tue, 17 Jul 2012 20:31:52 -0400 (EDT) From: Tom Lane To: Craig Ringer cc: Robert Haas , Jeff Janes , pgsql-hackers , Daniel Farina , =?ISO-8859-1?Q?=22Harold_A=2E_Gim=E9nez=22?= Subject: Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) In-reply-to: <5005FF9F.4040706@ringerc.id.au> References: <4FFCCAC4.4030503@ringerc.id.au> <32078D4B40DD40ECB739D0ECD7D95996@gmail.com> <19949.1342376967@sss.pgh.pa.us> <9603.1342454919@sss.pgh.pa.us> <11952.1342456591@sss.pgh.pa.us> <13011.1342457859@sss.pgh.pa.us> <15955.1342466333@sss.pgh.pa.us> <9247.1342565810@sss.pgh.pa.us> <5005FF9F.4! 040706@ringerc.id.au> Comments: In-reply-to Craig Ringer message dated "Wed, 18 Jul 2012 08:13:19 +0800" Date: Tue, 17 Jul 2012 20:31:52 -0400 Message-ID: <11099.1342571512@sss.pgh.pa.us> X-Pg-Spam-Score: -1.9 (-) X-Archive-Number: 201207/773 X-Sequence-Number: 211456 Craig Ringer writes: > On 07/18/2012 06:56 AM, Tom Lane wrote: >> This implies that nobody has done pull-the-plug testing on either HEAD >> or 9.2 since the checkpointer split went in (2011-11-01) > That makes me wonder if on top of the buildfarm, extending some > buildfarm machines into a "crashfarm" is needed: Not sure if we need a whole "farm", but certainly having at least one machine testing this sort of stuff on a regular basis would make me feel a lot better. > The main challenge would be coming up with suitable tests to run, ones > that could then be checked to make sure nothing was broken. One fairly simple test scenario could go like this: * run the regression tests * pg_dump the regression database * run the regression tests again * hard-kill immediately upon completion * restart database, allow it to perform recovery * pg_dump the regression database * diff previous and new dumps; should be the same The main thing this wouldn't cover is discrepancies in user indexes, since pg_dump doesn't do anything that's likely to result in indexscans on user tables. It ought to be enough to detect the sort of system-wide problem we're talking about here, though. In general I think the hard part is automated reproduction of an OS-crash scenario, but your ideas about how to do that sound promising. Once we have that going, it shouldn't be hard to come up with tests of the form "do X, hard-crash, recover, check X still looks sane". > What else should be checked? The main thing that comes to mind for me is > something I've worried about for a while: that Pg might not always > handle out-of-disk-space anywhere near as gracefully as it's often > claimed to. +1 regards, tom lane