pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Craig Ringer <ringerc@ringerc.id.au>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Jeff Janes <jeff.janes@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Daniel Farina <daniel@heroku.com>
Cc: "Harold A. Giménez" <harold.gimenez@gmail.com>
Subject: Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
Date: Wed, 18 Jul 2012 12:57:53 +0800
Message-ID: <50064251.5010908@ringerc.id.au> (raw)
In-Reply-To: <11099.1342571512@sss.pgh.pa.us>
References: <4FFCCAC4.4030503@ringerc.id.au>
	<CAMkU=1ydaypgAeZxyaMc=KhJJw5riGMBiLpvSqOO42=qArRZmQ@mail.gmail.com>
	<32078D4B40DD40ECB739D0ECD7D95996@gmail.com>
	<CAMkU=1zJK6WiS9-6eO8rNmRpKSSB4=uw2mTsD=k64mNh__=Dcg@mail.gmail.com>
	<CAMkU=1w7vLA63hf-+Uc_j61CvOdKixsOHph9d0H-aLNBYgfX=w@mail.gmail.com>
	<CAMkU=1yLXvODRZZ_=fgrEeJfk2tvZPTTD-8n8BwrAhNz_WBT0A@mail.gmail.com>
	<19949.1342376967@sss.pgh.pa.us>
	<CA+TgmoaTf=zjP37-oG7fr1VbGfuvHSfpajgLZ65EeSJX2mmquA@mail.gmail.com>
	<9603.1342454919@sss.pgh.pa.us>
	<CA+Tgmoa4ta60Y-rZpEPDRCQKJV8-Uqe0P610Jgj2q7e+fondig@mail.gmail.com>
	<11952.1342456591@sss.pgh.pa.us>
	<CA+TgmoYRXMWQ_DGmmqu3Zp8oW3Lt-9Lc1q3EHaV_ZmjWqaPkuA@mail.gmail.com>
	<13011.1342457859@sss.pgh.pa.us>
	<CA+TgmoaEPLo5fmBX1JSkGK3umAamFVghwY7qDb4_QqzteZgUPA@mail.gmail.com>
	<15955.1342466333@sss.pgh.pa.us>
	<CA+TgmoYnqP! f03rYJo3G6dMrqShQF2iMtfpYBPgWdxtEdacaX-w@mail.gmail.com>
	<9247.1342565810@sss.pgh.pa.us>
	<5005FF9F.4! 040706@ringerc.id.au>
	<11099.1342571512@sss.pgh.pa.us>

On 07/18/2012 08:31 AM, Tom Lane wrote:
> Not sure if we need a whole "farm", but certainly having at least one
> machine testing this sort of stuff on a regular basis would make me feel
> a lot better.

OK. That's something I can actually be useful for.

My current qemu/kvm test harness control code is in Python since that's 
what all the other tooling for the project I was using it for is in. Is 
it likely to be useful for me to adapt that code for use for a Pg 
crash-test harness, or will you need a particular tool/language to be 
used? If so, which/what? I'll do pretty much anything except Perl. I'll 
have a result for you more quickly working in Python, though I'm happy 
enough to write it in C (or Java, but I'm guessing that won't get any 
enthusiasm around here).

> One fairly simple test scenario could go like this:
>
> 	* run the regression tests
> 	* pg_dump the regression database
> 	* run the regression tests again
> 	* hard-kill immediately upon completion
> 	* restart database, allow it to perform recovery
> 	* pg_dump the regression database
> 	* diff previous and new dumps; should be the same
>
> The main thing this wouldn't cover is discrepancies in user indexes,
> since pg_dump doesn't do anything that's likely to result in indexscans
> on user tables.  It ought to be enough to detect the sort of system-wide
> problem we're talking about here, though.

It also won't detect issues that only occur during certain points in 
execution, under concurrent load, etc. Still, a start, and I could look 
at extending it into some kind of "crash fuzzing" once the basics were 
working.

> In general I think the hard part is automated reproduction of an
> OS-crash scenario, but your ideas about how to do that sound promising.

It's worked well for other testing I've done. Any writes that're still 
in the guest OS's memory, write queues, etc are lost when kvm is killed, 
just like a hard crash. Anything the kvm guest has flushed to "disk" is 
on the host and preserved - either on the host's disks 
(cache=writethrough) or at least in dirty writeback buffers in ram 
(cache=writeback).

kvm can even do a decent job of simulating a BBU-equipped write-through 
volume by allowing the host OS to do write-back caching of KVM's backing 
device/files. You don't get to set a max write-back cache size directly, 
but Linux I/O writeback settings provide some control.

My favourite thing about kvm is that it's just another command. It can 
be run headless and controlled via virtual serial console and/or its 
monitor socket. It doesn't require special privileges and can operate on 
ordinary files. It's very well suited for hooking into test harnesses.

The only challenge with using kvm/qemu is that there have been some 
breaking changes and a couple of annoying bugs that mean I won't be able 
to support anything except pretty much the latest versions initially. 
kvm is easy to compile and has limited dependencies, so I don't expect 
that to be an issue, but thought it was worth raising.

--
Craig Ringer



view thread (21+ messages)  latest in thread

Message-ID: <50064251.5010908@ringerc.id.au>
Permalink:  ../50064251.5010908@ringerc.id.au/
Also on:    postgresql.org/message-id/50064251.5010908@ringerc.id.au

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: ringerc@ringerc.id.au, tgl@sss.pgh.pa.us, robertmhaas@gmail.com, jeff.janes@gmail.com, pgsql-hackers@postgresql.org, daniel@heroku.com, harold.gimenez@gmail.com
  Subject: Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
  In-Reply-To: <50064251.5010908@ringerc.id.au>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox