pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedCheckpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
21+ messages / 11 participants
[nested] [flat]
* Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
@ 2012-07-17 22:56 Tom Lane <tgl@sss.pgh.pa.us>
2012-07-17 23:48 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Peter Geoghegan <peter@2ndquadrant.com>
2012-07-18 00:13 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
2012-07-18 00:31 ` Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 04:00 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Greg Smith <greg@2ndQuadrant.com>
2012-07-18 05:56 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 12:26 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Robert Haas <robertmhaas@gmail.com>
2012-07-18 21:45 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Simon Riggs <simon@2ndQuadrant.com>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
0 siblings, 8 replies; 21+ messages in thread
From: Tom Lane @ 2012-07-17 22:56 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; Harold A. Giménez <harold.gimenez@gmail.com>
Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> BTW, while we are on the subject: hasn't this split completely broken
>> the statistics about backend-initiated writes?
> Yes, it seems to have done just that.
So I went to fix this in the obvious way (attached), but while testing
it I found that the number of buffers_backend events reported during
a regression test run barely changed; which surprised the heck out of
me, so I dug deeper. The cause turns out to be extremely scary:
ForwardFsyncRequest isn't getting called at all in the bgwriter process,
because the bgwriter process has a pendingOpsTable. So it just queues
its fsync requests locally, and then never acts on them, since it never
runs any checkpoints anymore.
This implies that nobody has done pull-the-plug testing on either HEAD
or 9.2 since the checkpointer split went in (2011-11-01), because even
a modicum of such testing would surely have shown that we're failing to
fsync a significant fraction of our write traffic.
Furthermore, I would say that any performance testing done since then,
if it wasn't looking at purely read-only scenarios, isn't worth the
electrons it's written on. In particular, any performance gain that
anybody might have attributed to the checkpointer splitup is very
probably hogwash.
This is not giving me a warm feeling about our testing practices.
As far as fixing the bug is concerned, the reason for the foulup
is that mdinit() looks to IsBootstrapProcessingMode() to decide
whether to create a pendingOpsTable. That probably was all right
when it was coded, but what it means today is that *any* process
started via AuxiliaryProcessMain will have one; thus not only do
bgwriters have one, but so do walwriter and walreceiver processes;
which might not represent a bug today but it's pretty scary anyway.
I think we need to fix that so it's more directly dependent on the
auxiliary process type. We can't use flags set by the respective
FooMain() functions, such as am_bg_writer, because mdinit is called
from BaseInit() which happens before reaching those functions.
My suggestion is that bootstrap.c ought to make the process's
AuxProcType value available and then mdinit should consult that to
decide what to do. (Having done that, we might consider getting rid
of the "retail" process-type flags am_bg_writer etc.)
regards, tom lane
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-17 23:48 ` Peter Geoghegan <peter@2ndquadrant.com>
2012-07-18 07:30 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
7 siblings, 1 reply; 21+ messages in thread
From: Peter Geoghegan @ 2012-07-17 23:48 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; Harold A. Giménez <harold.gimenez@gmail.com>
On 17 July 2012 23:56, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> This implies that nobody has done pull-the-plug testing on either HEAD
> or 9.2 since the checkpointer split went in (2011-11-01), because even
> a modicum of such testing would surely have shown that we're failing to
> fsync a significant fraction of our write traffic.
>
> Furthermore, I would say that any performance testing done since then,
> if it wasn't looking at purely read-only scenarios, isn't worth the
> electrons it's written on. In particular, any performance gain that
> anybody might have attributed to the checkpointer splitup is very
> probably hogwash.
>
> This is not giving me a warm feeling about our testing practices.
The checkpointer slit-up was not justified as a performance
optimisation so much as a re-factoring effort that might have some
concomitant performance benefits. While I agree that it is regrettable
that this was allowed to go undetected for so long, I do not find it
especially surprising that some performance testing results post-split
didn't strike somebody as fool's gold. Much of the theory surrounding
checkpoint tuning, if followed, results in relatively little work
being done during the sync phase of a checkpoint, especially if an I/O
scheduler like deadline is used.
--
Peter Geoghegan http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-17 23:48 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Peter Geoghegan <peter@2ndquadrant.com>
@ 2012-07-18 07:30 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 0 replies; 21+ messages in thread
From: Heikki Linnakangas @ 2012-07-18 07:30 UTC (permalink / raw)
To: Peter Geoghegan <peter@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 18.07.2012 02:48, Peter Geoghegan wrote:
> On 17 July 2012 23:56, Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> This implies that nobody has done pull-the-plug testing on either HEAD
>> or 9.2 since the checkpointer split went in (2011-11-01), because even
>> a modicum of such testing would surely have shown that we're failing to
>> fsync a significant fraction of our write traffic.
>>
>> Furthermore, I would say that any performance testing done since then,
>> if it wasn't looking at purely read-only scenarios, isn't worth the
>> electrons it's written on. In particular, any performance gain that
>> anybody might have attributed to the checkpointer splitup is very
>> probably hogwash.
>>
>> This is not giving me a warm feeling about our testing practices.
>
> The checkpointer slit-up was not justified as a performance
> optimisation so much as a re-factoring effort that might have some
> concomitant performance benefits.
Agreed, but it means that we need to re-run the tests that were done to
make sure the extra fsync-request traffic is not causing a performance
regression,
http://archives.postgresql.org/pgsql-hackers/2011-10/msg01321.php.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 00:13 ` Craig Ringer <ringerc@ringerc.id.au>
2012-07-24 13:33 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Greg Stark <stark@mit.edu>
7 siblings, 1 reply; 21+ messages in thread
From: Craig Ringer @ 2012-07-18 00:13 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/18/2012 06:56 AM, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> BTW, while we are on the subject: hasn't this split completely broken
>>> the statistics about backend-initiated writes?
>> Yes, it seems to have done just that.
> So I went to fix this in the obvious way (attached), but while testing
> it I found that the number of buffers_backend events reported during
> a regression test run barely changed; which surprised the heck out of
> me, so I dug deeper. The cause turns out to be extremely scary:
> ForwardFsyncRequest isn't getting called at all in the bgwriter process,
> because the bgwriter process has a pendingOpsTable. So it just queues
> its fsync requests locally, and then never acts on them, since it never
> runs any checkpoints anymore.
>
> This implies that nobody has done pull-the-plug testing on either HEAD
> or 9.2 since the checkpointer split went in (2011-11-01)
That makes me wonder if on top of the buildfarm, extending some
buildfarm machines into a "crashfarm" is needed:
- Keep kvm instances with copy-on-write snapshot disks and the build env
on them
- Fire up the VM, do a build, and start the server
- From outside the vm have the test controller connect to the server and
start a test run
- Hard-kill the OS instance at a random point in time.
- Start the OS instance back up
- Start Pg back up and connect to it again
- From the test controller, test the Pg install for possible corruption
by reading the indexes and tables, doing some test UPDATEs, etc.
The main challenge would be coming up with suitable tests to run, ones
that could then be checked to make sure nothing was broken. The test
controller would know how far a test got before the OS got killed and
would know which test it was running, so it'd be able to check for
expected data if provided with appropriate test metadata. Use of enable_
flags should permit scans of indexes and table heaps to be forced.
What else should be checked? The main thing that comes to mind for me is
something I've worried about for a while: that Pg might not always
handle out-of-disk-space anywhere near as gracefully as it's often
claimed to. There's no automated testing for that, so it's hard to
really know. A harnessed VM could be used to test that. Instead of
virtual plug pull tests it could generate a virtual disk of constrained
random size, run its tests until out-of-disk caused failure, stop Pg,
expand the disk, restart Pg, and run its checks.
Variants where WAL was on a separate disk and only WAL or only the main
non-WAL disk run out of space would also make sense and be easy to
produce with such a harness.
I've written some automated kvm test harnesses, so I could have a play
with this idea. I would probably need some help with the test design,
though, and the guest OS would be Linux, Linux, or Linux at least to
start with.
Opinions?
--
Craig Ringer
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 00:13 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
@ 2012-07-24 13:33 ` Greg Stark <stark@mit.edu>
0 siblings, 0 replies; 21+ messages in thread
From: Greg Stark @ 2012-07-24 13:33 UTC (permalink / raw)
To: Craig Ringer <ringerc@ringerc.id.au>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On Wed, Jul 18, 2012 at 1:13 AM, Craig Ringer <ringerc@ringerc.id.au> wrote:
> That makes me wonder if on top of the buildfarm, extending some buildfarm
> machines into a "crashfarm" is needed:
>
> - Keep kvm instances with copy-on-write snapshot disks and the build env
> on them
> - Fire up the VM, do a build, and start the server
> - From outside the vm have the test controller connect to the server and
> start a test run
> - Hard-kill the OS instance at a random point in time.
>
For what it's worth you don't need to do a hard kill of the vm and start
over repeatedly to kill at different times. You could take a snapshot of
the disk storage and keep running. You could take many snapshots from a
single run. Each snapshot would represent the storage that would exist if
the machine had crashed at the point in time that the snapshot was taken.
You do want the snapshots to be taken using something outside the virtual
machine. Either the kvm storage layer or using lvm on the host. But not
using lvm on the guest virtual machine.
And yes, the hard part that always stopped me from looking at this was
having any way to test the correctness of the data.
--
greg
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 00:31 ` Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 04:57 ` Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
2012-07-18 06:00 ` Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
7 siblings, 2 replies; 21+ messages in thread
From: Tom Lane @ 2012-07-18 00:31 UTC (permalink / raw)
To: Craig Ringer <ringerc@ringerc.id.au>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
Craig Ringer <ringerc@ringerc.id.au> writes:
> On 07/18/2012 06:56 AM, Tom Lane wrote:
>> This implies that nobody has done pull-the-plug testing on either HEAD
>> or 9.2 since the checkpointer split went in (2011-11-01)
> That makes me wonder if on top of the buildfarm, extending some
> buildfarm machines into a "crashfarm" is needed:
Not sure if we need a whole "farm", but certainly having at least one
machine testing this sort of stuff on a regular basis would make me feel
a lot better.
> The main challenge would be coming up with suitable tests to run, ones
> that could then be checked to make sure nothing was broken.
One fairly simple test scenario could go like this:
* run the regression tests
* pg_dump the regression database
* run the regression tests again
* hard-kill immediately upon completion
* restart database, allow it to perform recovery
* pg_dump the regression database
* diff previous and new dumps; should be the same
The main thing this wouldn't cover is discrepancies in user indexes,
since pg_dump doesn't do anything that's likely to result in indexscans
on user tables. It ought to be enough to detect the sort of system-wide
problem we're talking about here, though.
In general I think the hard part is automated reproduction of an
OS-crash scenario, but your ideas about how to do that sound promising.
Once we have that going, it shouldn't be hard to come up with tests
of the form "do X, hard-crash, recover, check X still looks sane".
> What else should be checked? The main thing that comes to mind for me is
> something I've worried about for a while: that Pg might not always
> handle out-of-disk-space anywhere near as gracefully as it's often
> claimed to.
+1
regards, tom lane
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 00:31 ` Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 04:57 ` Craig Ringer <ringerc@ringerc.id.au>
1 sibling, 0 replies; 21+ messages in thread
From: Craig Ringer @ 2012-07-18 04:57 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/18/2012 08:31 AM, Tom Lane wrote:
> Not sure if we need a whole "farm", but certainly having at least one
> machine testing this sort of stuff on a regular basis would make me feel
> a lot better.
OK. That's something I can actually be useful for.
My current qemu/kvm test harness control code is in Python since that's
what all the other tooling for the project I was using it for is in. Is
it likely to be useful for me to adapt that code for use for a Pg
crash-test harness, or will you need a particular tool/language to be
used? If so, which/what? I'll do pretty much anything except Perl. I'll
have a result for you more quickly working in Python, though I'm happy
enough to write it in C (or Java, but I'm guessing that won't get any
enthusiasm around here).
> One fairly simple test scenario could go like this:
>
> * run the regression tests
> * pg_dump the regression database
> * run the regression tests again
> * hard-kill immediately upon completion
> * restart database, allow it to perform recovery
> * pg_dump the regression database
> * diff previous and new dumps; should be the same
>
> The main thing this wouldn't cover is discrepancies in user indexes,
> since pg_dump doesn't do anything that's likely to result in indexscans
> on user tables. It ought to be enough to detect the sort of system-wide
> problem we're talking about here, though.
It also won't detect issues that only occur during certain points in
execution, under concurrent load, etc. Still, a start, and I could look
at extending it into some kind of "crash fuzzing" once the basics were
working.
> In general I think the hard part is automated reproduction of an
> OS-crash scenario, but your ideas about how to do that sound promising.
It's worked well for other testing I've done. Any writes that're still
in the guest OS's memory, write queues, etc are lost when kvm is killed,
just like a hard crash. Anything the kvm guest has flushed to "disk" is
on the host and preserved - either on the host's disks
(cache=writethrough) or at least in dirty writeback buffers in ram
(cache=writeback).
kvm can even do a decent job of simulating a BBU-equipped write-through
volume by allowing the host OS to do write-back caching of KVM's backing
device/files. You don't get to set a max write-back cache size directly,
but Linux I/O writeback settings provide some control.
My favourite thing about kvm is that it's just another command. It can
be run headless and controlled via virtual serial console and/or its
monitor socket. It doesn't require special privileges and can operate on
ordinary files. It's very well suited for hooking into test harnesses.
The only challenge with using kvm/qemu is that there have been some
breaking changes and a couple of annoying bugs that mean I won't be able
to support anything except pretty much the latest versions initially.
kvm is easy to compile and has limited dependencies, so I don't expect
that to be an issue, but thought it was worth raising.
--
Craig Ringer
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 00:31 ` Re: Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 06:00 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 21+ messages in thread
From: Tom Lane @ 2012-07-18 06:00 UTC (permalink / raw)
To: Craig Ringer <ringerc@ringerc.id.au>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
Craig Ringer <ringerc@ringerc.id.au> writes:
> On 07/18/2012 08:31 AM, Tom Lane wrote:
>> Not sure if we need a whole "farm", but certainly having at least one
>> machine testing this sort of stuff on a regular basis would make me feel
>> a lot better.
> OK. That's something I can actually be useful for.
> My current qemu/kvm test harness control code is in Python since that's
> what all the other tooling for the project I was using it for is in. Is
> it likely to be useful for me to adapt that code for use for a Pg
> crash-test harness, or will you need a particular tool/language to be
> used? If so, which/what? I'll do pretty much anything except Perl. I'll
> have a result for you more quickly working in Python, though I'm happy
> enough to write it in C (or Java, but I'm guessing that won't get any
> enthusiasm around here).
If we were talking about code that was going to end up in the PG
distribution, I'd kind of want it to be in C or Perl, just to keep down
the number of languages we're depending on. However, it's not obvious
that a tool like this would ever go into our distribution. I'd suggest
working with what you're comfortable with, and we can worry about
translation when and if there's a reason to.
regards, tom lane
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 04:00 ` Greg Smith <greg@2ndQuadrant.com>
2012-07-18 04:20 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
7 siblings, 1 reply; 21+ messages in thread
From: Greg Smith @ 2012-07-18 04:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/17/2012 06:56 PM, Tom Lane wrote:
> So I went to fix this in the obvious way (attached), but while testing
> it I found that the number of buffers_backend events reported during
> a regression test run barely changed; which surprised the heck out of
> me, so I dug deeper. The cause turns out to be extremely scary:
> ForwardFsyncRequest isn't getting called at all in the bgwriter process,
> because the bgwriter process has a pendingOpsTable.
When I did my testing early this year to look at checkpointer
performance (among other 9.2 write changes like group commit), I did see
some cases where buffers_backend was dramatically different on 9.2 vs.
9.1 There were plenty of cases where the totals across a 10 minute
pgbench were almost identical though, so this issue didn't stick out
then. That's a very different workload than the regression tests though.
> This implies that nobody has done pull-the-plug testing on either HEAD
> or 9.2 since the checkpointer split went in (2011-11-01), because even
> a modicum of such testing would surely have shown that we're failing to
> fsync a significant fraction of our write traffic.
Ugh. Most of my pull the plug testing the last six months has been
focused on SSD tests with older versions. I want to duplicate this (and
any potential fix) now that you've highlighted it.
> Furthermore, I would say that any performance testing done since then,
> if it wasn't looking at purely read-only scenarios, isn't worth the
> electrons it's written on. In particular, any performance gain that
> anybody might have attributed to the checkpointer splitup is very
> probably hogwash.
There hasn't been any performance testing that suggested the
checkpointer splitup was justified. The stuff I did showed it being
flat out negative for a subset of pgbench oriented cases, which didn't
seem real-world enough to disprove it as the right thing to do though.
I thought there were two valid justifications for the checkpointer split
(which is not a feature I have any corporate attachment to--I'm as
isolated from how it was developed as you are). The first is that it
seems like the right architecture to allow reworking checkpoints and
background writes for future write path optimization. A good chunk of
the time when I've tried to improve one of those (like my spread sync
stuff from last year), the code was complicated by the background writer
needing to follow the drum of checkpoint timing, and vice-versa. Being
able to hack on those independently got a sign of relief from me. And
while this adds some code duplication in things like the process setup,
I thought the result would be cleaner for people reading the code to
follow too. This problem is terrible, but I think part of how it crept
in is that the single checkpoint+background writer process was doing way
too many things to even follow all of them some days.
The second justification for the split was that it seems easier to get a
low power result from, which I believe was the angle Peter Geoghegan was
working when this popped up originally. The checkpointer has to run
sometimes, but only at a 50% duty cycle as it's tuned out of the box.
It seems nice to be able to approach that in a way that's power
efficient without coupling it to whatever heartbeat the BGW is running
at. I could even see people changing the frequencies for each
independently depending on expected system load. Tune for lower power
when you don't expect many users, that sort of thing.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 04:00 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Greg Smith <greg@2ndQuadrant.com>
@ 2012-07-18 04:20 ` Craig Ringer <ringerc@ringerc.id.au>
0 siblings, 0 replies; 21+ messages in thread
From: Craig Ringer @ 2012-07-18 04:20 UTC (permalink / raw)
To: Greg Smith <greg@2ndQuadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/18/2012 12:00 PM, Greg Smith wrote:
> The second justification for the split was that it seems easier to get
> a low power result from, which I believe was the angle Peter Geoghegan
> was working when this popped up originally. The checkpointer has to
> run sometimes, but only at a 50% duty cycle as it's tuned out of the
> box. It seems nice to be able to approach that in a way that's power
> efficient without coupling it to whatever heartbeat the BGW is running
> at. I could even see people changing the frequencies for each
> independently depending on expected system load. Tune for lower power
> when you don't expect many users, that sort of thing.
>
Yeah - I'm already seeing benefits from that on my laptop, with much
less need to stop Pg when I'm not using it.
--
Craig Ringer
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 05:56 ` Tom Lane <tgl@sss.pgh.pa.us>
7 siblings, 0 replies; 21+ messages in thread
From: Tom Lane @ 2012-07-18 05:56 UTC (permalink / raw)
To: Greg Smith <greg@2ndQuadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
Greg Smith <greg@2ndQuadrant.com> writes:
> On 07/17/2012 06:56 PM, Tom Lane wrote:
>> Furthermore, I would say that any performance testing done since then,
>> if it wasn't looking at purely read-only scenarios, isn't worth the
>> electrons it's written on. In particular, any performance gain that
>> anybody might have attributed to the checkpointer splitup is very
>> probably hogwash.
> There hasn't been any performance testing that suggested the
> checkpointer splitup was justified. The stuff I did showed it being
> flat out negative for a subset of pgbench oriented cases, which didn't
> seem real-world enough to disprove it as the right thing to do though.
Just to clarify, I'm not saying that this means we should revert the
checkpointer split. What I *am* worried about is that we may have been
hacking other things on the basis of faulty performance tests.
regards, tom lane
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 12:26 ` Robert Haas <robertmhaas@gmail.com>
7 siblings, 0 replies; 21+ messages in thread
From: Robert Haas @ 2012-07-18 12:26 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; Harold A. Giménez <harold.gimenez@gmail.com>
On Tue, Jul 17, 2012 at 6:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> So I went to fix this in the obvious way (attached), but while testing
> it I found that the number of buffers_backend events reported during
> a regression test run barely changed; which surprised the heck out of
> me, so I dug deeper. The cause turns out to be extremely scary:
> ForwardFsyncRequest isn't getting called at all in the bgwriter process,
> because the bgwriter process has a pendingOpsTable. So it just queues
> its fsync requests locally, and then never acts on them, since it never
> runs any checkpoints anymore.
:-(
> This implies that nobody has done pull-the-plug testing on either HEAD
> or 9.2 since the checkpointer split went in (2011-11-01), because even
> a modicum of such testing would surely have shown that we're failing to
> fsync a significant fraction of our write traffic.
>
> Furthermore, I would say that any performance testing done since then,
> if it wasn't looking at purely read-only scenarios, isn't worth the
> electrons it's written on. In particular, any performance gain that
> anybody might have attributed to the checkpointer splitup is very
> probably hogwash.
I don't think anybody thought that was going to result in a direct
performance gain, but I agree the performance testing needs to be
redone. I suspect that the impact on my testing is limited, because I
do mostly pgbench testing, and the lost fsync requests were probably
duplicated by non-lost fsync requests from backend writes. But I
agree that it needs to be redone once this is fixed.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-18 21:45 ` Simon Riggs <simon@2ndQuadrant.com>
7 siblings, 0 replies; 21+ messages in thread
From: Simon Riggs @ 2012-07-18 21:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; Harold A. Giménez <harold.gimenez@gmail.com>
On 17 July 2012 23:56, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> BTW, while we are on the subject: hasn't this split completely broken
>>> the statistics about backend-initiated writes?
>
>> Yes, it seems to have done just that.
>
> So I went to fix this in the obvious way (attached), but while testing
> it I found that the number of buffers_backend events reported during
> a regression test run barely changed; which surprised the heck out of
> me, so I dug deeper. The cause turns out to be extremely scary:
> ForwardFsyncRequest isn't getting called at all in the bgwriter process,
> because the bgwriter process has a pendingOpsTable. So it just queues
> its fsync requests locally, and then never acts on them, since it never
> runs any checkpoints anymore.
>
> This implies that nobody has done pull-the-plug testing on either HEAD
> or 9.2 since the checkpointer split went in (2011-11-01), because even
> a modicum of such testing would surely have shown that we're failing to
> fsync a significant fraction of our write traffic.
That problem was reported to me on list some time ago, and I made note
to fix that after last CF.
I added a note to 9.2 open items about it myself, but it appears my
fix was too simple and fixed only the reported problem not the
underlying issue. Reading your patch gave me strong deja vu, so not
sure what happened there.
Not very good from me. Feel free to thwack me to fix such things if I
seem not to respond quickly enough.
I'm now looking at the other open items in my area.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
@ 2012-07-23 04:37 ` David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
7 siblings, 1 reply; 21+ messages in thread
From: David Fetter @ 2012-07-23 04:37 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; Harold A. Giménez <harold.gimenez@gmail.com>
On Tue, Jul 17, 2012 at 06:56:50PM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> BTW, while we are on the subject: hasn't this split completely
> >> broken the statistics about backend-initiated writes?
>
> > Yes, it seems to have done just that.
>
> This implies that nobody has done pull-the-plug testing on either
> HEAD or 9.2 since the checkpointer split went in (2011-11-01),
> because even a modicum of such testing would surely have shown that
> we're failing to fsync a significant fraction of our write traffic.
>
> Furthermore, I would say that any performance testing done since
> then, if it wasn't looking at purely read-only scenarios, isn't
> worth the electrons it's written on. In particular, any performance
> gain that anybody might have attributed to the checkpointer splitup
> is very probably hogwash.
>
> This is not giving me a warm feeling about our testing practices.
Is there any part of this that the buildfarm, or some other automation
framework, might be able to handle?
Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
@ 2012-07-23 12:29 ` Andrew Dunstan <andrew@dunslane.net>
2012-07-23 12:41 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 13:04 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
0 siblings, 2 replies; 21+ messages in thread
From: Andrew Dunstan @ 2012-07-23 12:29 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/23/2012 12:37 AM, David Fetter wrote:
> On Tue, Jul 17, 2012 at 06:56:50PM -0400, Tom Lane wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> BTW, while we are on the subject: hasn't this split completely
>>>> broken the statistics about backend-initiated writes?
>>> Yes, it seems to have done just that.
>> This implies that nobody has done pull-the-plug testing on either
>> HEAD or 9.2 since the checkpointer split went in (2011-11-01),
>> because even a modicum of such testing would surely have shown that
>> we're failing to fsync a significant fraction of our write traffic.
>>
>> Furthermore, I would say that any performance testing done since
>> then, if it wasn't looking at purely read-only scenarios, isn't
>> worth the electrons it's written on. In particular, any performance
>> gain that anybody might have attributed to the checkpointer splitup
>> is very probably hogwash.
>>
>> This is not giving me a warm feeling about our testing practices.
> Is there any part of this that the buildfarm, or some other automation
> framework, might be able to handle?
>
I'm not sure how you automate testing a pull-the-plug scenario.
The buildfarm is not at all designed to test performance. That's why we
want a performance farm.
cheers
andrew
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
@ 2012-07-23 12:41 ` David Fetter <david@fetter.org>
2012-07-23 12:56 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
2012-07-23 15:02 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Jeff Janes <jeff.janes@gmail.com>
1 sibling, 2 replies; 21+ messages in thread
From: David Fetter @ 2012-07-23 12:41 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On Mon, Jul 23, 2012 at 08:29:16AM -0400, Andrew Dunstan wrote:
>
> On 07/23/2012 12:37 AM, David Fetter wrote:
> >On Tue, Jul 17, 2012 at 06:56:50PM -0400, Tom Lane wrote:
> >>Robert Haas <robertmhaas@gmail.com> writes:
> >>>On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>>>BTW, while we are on the subject: hasn't this split completely
> >>>>broken the statistics about backend-initiated writes?
> >>>Yes, it seems to have done just that.
> >>This implies that nobody has done pull-the-plug testing on either
> >>HEAD or 9.2 since the checkpointer split went in (2011-11-01),
> >>because even a modicum of such testing would surely have shown that
> >>we're failing to fsync a significant fraction of our write traffic.
> >>
> >>Furthermore, I would say that any performance testing done since
> >>then, if it wasn't looking at purely read-only scenarios, isn't
> >>worth the electrons it's written on. In particular, any performance
> >>gain that anybody might have attributed to the checkpointer splitup
> >>is very probably hogwash.
> >>
> >>This is not giving me a warm feeling about our testing practices.
> >Is there any part of this that the buildfarm, or some other automation
> >framework, might be able to handle?
> >
>
> I'm not sure how you automate testing a pull-the-plug scenario.
I have a dim memory of how the FreeBSD project was alleged to have
done it, namely by rigging a serial port (yes, it was that long ago)
to the power supply of another machine and randomly cycling the power.
> The buildfarm is not at all designed to test performance. That's why
> we want a performance farm.
Right. Apart from hardware, what are we stalled on?
Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
2012-07-23 12:41 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
@ 2012-07-23 12:56 ` Andrew Dunstan <andrew@dunslane.net>
1 sibling, 0 replies; 21+ messages in thread
From: Andrew Dunstan @ 2012-07-23 12:56 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/23/2012 08:41 AM, David Fetter wrote:
>> The buildfarm is not at all designed to test performance. That's why
>> we want a performance farm.
> Right. Apart from hardware, what are we stalled on?
>
Software :-)
I am trying to find some cycles to get something going.
cheers
andrew
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
2012-07-23 12:41 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
@ 2012-07-23 15:02 ` Jeff Janes <jeff.janes@gmail.com>
1 sibling, 0 replies; 21+ messages in thread
From: Jeff Janes @ 2012-07-23 15:02 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; Craig Ringer <ringerc@ringerc.id.au>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On Mon, Jul 23, 2012 at 5:41 AM, David Fetter <david@fetter.org> wrote:
> On Mon, Jul 23, 2012 at 08:29:16AM -0400, Andrew Dunstan wrote:
>>
>>
>> I'm not sure how you automate testing a pull-the-plug scenario.
>
> I have a dim memory of how the FreeBSD project was alleged to have
> done it, namely by rigging a serial port (yes, it was that long ago)
> to the power supply of another machine and randomly cycling the power.
On Linux,
echo b > /proc/sysrq-trigger
Is supposed to take it down instantly, with no flushing of dirty buffers.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
@ 2012-07-23 13:04 ` Craig Ringer <ringerc@ringerc.id.au>
2012-07-23 13:47 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 21+ messages in thread
From: Craig Ringer @ 2012-07-23 13:04 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: David Fetter <david@fetter.org>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/23/2012 08:29 PM, Andrew Dunstan wrote:
> I'm not sure how you automate testing a pull-the-plug scenario.
fire up kvm or qemu instances, then kill 'em.
--
Craig Ringer
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
2012-07-23 13:04 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
@ 2012-07-23 13:47 ` Andrew Dunstan <andrew@dunslane.net>
2012-07-23 13:58 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
0 siblings, 1 reply; 21+ messages in thread
From: Andrew Dunstan @ 2012-07-23 13:47 UTC (permalink / raw)
To: Craig Ringer <ringerc@ringerc.id.au>; +Cc: David Fetter <david@fetter.org>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/23/2012 09:04 AM, Craig Ringer wrote:
> On 07/23/2012 08:29 PM, Andrew Dunstan wrote:
>
>> I'm not sure how you automate testing a pull-the-plug scenario.
>
> fire up kvm or qemu instances, then kill 'em.
>
>
Yeah, maybe. Knowing just when to kill them might be an interesting
question.
I'm also unsure how much nice cleanup the host supervisor does in such
cases. VMs are wonderful things, but they aren't always the answer. I'm
not saying they aren't here, just wondering.
cheers
andrew
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-23 04:37 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) David Fetter <david@fetter.org>
2012-07-23 12:29 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
2012-07-23 13:04 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Craig Ringer <ringerc@ringerc.id.au>
2012-07-23 13:47 ` Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Andrew Dunstan <andrew@dunslane.net>
@ 2012-07-23 13:58 ` Craig Ringer <ringerc@ringerc.id.au>
0 siblings, 0 replies; 21+ messages in thread
From: Craig Ringer @ 2012-07-23 13:58 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: David Fetter <david@fetter.org>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; Daniel Farina <daniel@heroku.com>; "Harold A. Giménez" <harold.gimenez@gmail.com>
On 07/23/2012 09:47 PM, Andrew Dunstan wrote:
>
> On 07/23/2012 09:04 AM, Craig Ringer wrote:
>> On 07/23/2012 08:29 PM, Andrew Dunstan wrote:
>>
>>> I'm not sure how you automate testing a pull-the-plug scenario.
>>
>> fire up kvm or qemu instances, then kill 'em.
>>
>>
>
> Yeah, maybe. Knowing just when to kill them might be an interesting
> question.
>
> I'm also unsure how much nice cleanup the host supervisor does in such
> cases. VMs are wonderful things, but they aren't always the answer.
> I'm not saying they aren't here, just wondering.
I've done some testing with this, and what it boils down to is that any
data that made it to the virtual disk is persistent after a VM kill.
Anything in dirty buffers on the VM guest is lost. It's a very close
match for real hardware. I haven't tried to examine the details of the
handling of virtualised disk hardware write caches, but disks should be
in write-through mode anyway. A `kill -9` will clear 'em for sure,
anyway, as the guest has no chance to do any cleanup.
One of the great things about kvm and qemu for this sort of testing is
that it's just another program. There's very little magic, and it's
quite easy to test and trace.
I have a qemu/kvm test harness I've been using for another project that
I need to update and clean up as it'd be handy for this. It's just a
matter of making the time, as it's been a busy few days.
--
Craig Ringer
^ permalink raw reply [nested|flat] 21+ messages in thread
end of thread, other threads:[~2012-07-24 13:33 UTC | newest]
Thread overview: 21+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-07-17 22:56 Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation) Tom Lane <tgl@sss.pgh.pa.us>
2012-07-17 23:48 ` Peter Geoghegan <peter@2ndquadrant.com>
2012-07-18 07:30 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2012-07-18 00:13 ` Craig Ringer <ringerc@ringerc.id.au>
2012-07-24 13:33 ` Greg Stark <stark@mit.edu>
2012-07-18 00:31 ` Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 04:57 ` Craig Ringer <ringerc@ringerc.id.au>
2012-07-18 06:00 ` Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 04:00 ` Greg Smith <greg@2ndQuadrant.com>
2012-07-18 04:20 ` Craig Ringer <ringerc@ringerc.id.au>
2012-07-18 05:56 ` Tom Lane <tgl@sss.pgh.pa.us>
2012-07-18 12:26 ` Robert Haas <robertmhaas@gmail.com>
2012-07-18 21:45 ` Simon Riggs <simon@2ndQuadrant.com>
2012-07-23 04:37 ` David Fetter <david@fetter.org>
2012-07-23 12:29 ` Andrew Dunstan <andrew@dunslane.net>
2012-07-23 12:41 ` David Fetter <david@fetter.org>
2012-07-23 12:56 ` Andrew Dunstan <andrew@dunslane.net>
2012-07-23 15:02 ` Jeff Janes <jeff.janes@gmail.com>
2012-07-23 13:04 ` Craig Ringer <ringerc@ringerc.id.au>
2012-07-23 13:47 ` Andrew Dunstan <andrew@dunslane.net>
2012-07-23 13:58 ` Craig Ringer <ringerc@ringerc.id.au>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox