From: Karl Denninger <karl@denninger.net>
To: Brad Nicholson <bnichols@ca.afilias.info>
Cc: Scott Marlowe <scott.marlowe@gmail.com>
Cc: Greg Smith <greg@2ndquadrant.com>
Cc: Scott Carey <scott@richrelevance.com>
Cc: Michael March <mmarch@gmail.com>
Cc: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: Re: Completely un-tuned Postgresql benchmark results: SSD vs desktop HDD
Date: Tue, 10 Aug 2010 14:28:18 -0500
Message-ID: <4C61A852.9040309@denninger.net> (raw)
In-Reply-To: <4C61A054.6080400@ca.afilias.info>
References: <AANLkTikXLYc32Jyr5Fb8VeVNt6R86mCcF+0CkTHALd8W@mail.gmail.com>
<BAE01AAD-6C69-48C7-BD2E-77E463D49903@richrelevance.com>
<AANLkTi=km+H+mZ=yyx0_HTFZ-yU2sh85D+p5cs+ZcUSF@mail.gmail.com>
<A1640F99-A7C5-4F04-9A25-782C46BE7DAB@richrelevance.com>
<4C617C80.9080509@2ndquadrant.com>
<4C619300.3080400@ca.afilias.info>
<4C6196DB.8050307@denninger.net>
<AANLkTi=pehZj1Bb8KVusGykyEzWn__pif-87Tz2beJYj@mail.gmail.com>
<4C619C90.3090005@denninger.net>
<4C61A054.6080400@ca.afilias.info>
Brad Nicholson wrote:
> On 8/10/2010 2:38 PM, Karl Denninger wrote:
>> Scott Marlowe wrote:
>>> On Tue, Aug 10, 2010 at 12:13 PM, Karl Denninger <karl@denninger.net> wrote:
>>>
>>>> ANY disk that says "write is complete" when it really is not is entirely
>>>> unsuitable for ANY real database use. It is simply a matter of time
>>>>
>>>
>>> What about read only slaves where there's a master with 100+spinning
>>> hard drives "getting it right" and you need a half dozen or so read
>>> slaves? I can imagine that being ok, as long as you don't restart a
>>> server after a crash without checking on it.
>>>
>> A read-only slave isn't read-only, is it?
>>
>> I mean, c'mon - how does the data get there?
>>
> A valid case is a Slony replica if used for query offloading (not for
> DR). It's considered a read-only subscriber from the perspective of
> Slony as only Slony can modify the data (although you are technically
> correct, it is not read only - controlled write may be more accurate).
>
> In case of failure, a rebuild + resubscribe gets you back to the same
> consistency. If you have high IO requirements, and don't have the
> budget to rack up extra disk arrays to meet them, it could be an option.
CAREFUL with that model and beliefs.
Specifically, the following will hose you without warning:
1. SLONY gets a change on the master.
2. SLONY commits it to the (read-only) slave.
3. Confirmation comes back to the master that the change was propagated.
4. Slave CRASHES without actually committing the changed data to stable
storage.
When the slave restarts it will not know that the transaction was lost.
Neither will the master, since it was told that it was committed. Slony
will happily go on its way and replicate forward, without any indication
of a problem - except that on the slave, there are one or more
transactions that are **missing**.
Some time later you issue an update that goes to the slave, but the
change previously lost causes the slave commit to violate referential
integrity. SLONY will fail to propagate that change and all behind it
- it effectively locks at that point in time.
You can recover from this by dropping the slave from replication and
re-inserting it, but that forces a full-table copy of everything in the
replication set. The bad news is that the queries to the slave in
question may have been returning erroneous data for some unknown period
of time prior to the lockup in replication (which hopefully you detect
reasonably quickly - you ARE watching SLONY queue depth with some
automated process, right?)
I can both cause this in the lab and have had it happen in the field.
It's a nasty little problem that bit me on a series of disks that
claimed to have write caching off, but in fact did not. I was very
happy that the data on the master was good at that point, as if I had
needed to failover to the slave (thinking it was a "good" copy) I would
have been in SERIOUS trouble.
-- Karl
Attachments:karl.vcf (text/x-vcard, 122B) download
begin:vcard
fn:Karl Denninger
n:Denninger;Karl
email;internet:karl@denninger.net
x-mozilla-html:TRUE
version:2.1
end:vcard
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: karl@denninger.net, bnichols@ca.afilias.info, scott.marlowe@gmail.com, greg@2ndquadrant.com, scott@richrelevance.com, mmarch@gmail.com
Subject: Re: Completely un-tuned Postgresql benchmark results: SSD vs desktop HDD
In-Reply-To: <4C61A852.9040309@denninger.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox