pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: SSD + RAID
36+ messages / 13 participants
[nested] [flat]
* Re: SSD + RAID
@ 2010-02-20 18:28 Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
0 siblings, 1 reply; 36+ messages in thread
From: Bruce Momjian @ 2010-02-20 18:28 UTC (permalink / raw)
To: Matthew Wakeling <matthew@flymine.org>; +Cc: Greg Smith <greg@2ndquadrant.com>; Laszlo Nagy <gandalf@shopzeus.com>; pgsql-performance
Matthew Wakeling wrote:
> On Fri, 13 Nov 2009, Greg Smith wrote:
> > In order for a drive to work reliably for database use such as for
> > PostgreSQL, it cannot have a volatile write cache. You either need a write
> > cache with a battery backup (and a UPS doesn't count), or to turn the cache
> > off. The SSD performance figures you've been looking at are with the drive's
> > write cache turned on, which means they're completely fictitious and
> > exaggerated upwards for your purposes. In the real world, that will result
> > in database corruption after a crash one day.
>
> Seagate are claiming to be on the ball with this one.
>
> http://www.theregister.co.uk/2009/12/08/seagate_pulsar_ssd/
I have updated our documentation to mention that even SSD drives often
have volatile write-back caches. Patch attached and applied.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +
Attachments:
/rtmp/diff (text/x-diff, 1.4K) download
Index: doc/src/sgml/wal.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/wal.sgml,v
retrieving revision 1.61
diff -c -c -r1.61 wal.sgml
*** doc/src/sgml/wal.sgml 3 Feb 2010 17:25:06 -0000 1.61
--- doc/src/sgml/wal.sgml 20 Feb 2010 18:26:40 -0000
***************
*** 59,65 ****
same concerns about data loss exist for write-back drive caches as
exist for disk controller caches. Consumer-grade IDE and SATA drives are
particularly likely to have write-back caches that will not survive a
! power failure. To check write caching on <productname>Linux</> use
<command>hdparm -I</>; it is enabled if there is a <literal>*</> next
to <literal>Write cache</>; <command>hdparm -W</> to turn off
write caching. On <productname>FreeBSD</> use
--- 59,66 ----
same concerns about data loss exist for write-back drive caches as
exist for disk controller caches. Consumer-grade IDE and SATA drives are
particularly likely to have write-back caches that will not survive a
! power failure. Many solid-state drives also have volatile write-back
! caches. To check write caching on <productname>Linux</> use
<command>hdparm -I</>; it is enabled if there is a <literal>*</> next
to <literal>Write cache</>; <command>hdparm -W</> to turn off
write caching. On <productname>FreeBSD</> use
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-02-20 23:07 ` Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 36+ messages in thread
From: Dan Langille @ 2010-02-20 23:07 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Matthew Wakeling <matthew@flymine.org>; Greg Smith <greg@2ndquadrant.com>; Laszlo Nagy <gandalf@shopzeus.com>; pgsql-performance
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Bruce Momjian wrote:
> Matthew Wakeling wrote:
>> On Fri, 13 Nov 2009, Greg Smith wrote:
>>> In order for a drive to work reliably for database use such as for
>>> PostgreSQL, it cannot have a volatile write cache. You either need a write
>>> cache with a battery backup (and a UPS doesn't count), or to turn the cache
>>> off. The SSD performance figures you've been looking at are with the drive's
>>> write cache turned on, which means they're completely fictitious and
>>> exaggerated upwards for your purposes. In the real world, that will result
>>> in database corruption after a crash one day.
>> Seagate are claiming to be on the ball with this one.
>>
>> http://www.theregister.co.uk/2009/12/08/seagate_pulsar_ssd/
>
> I have updated our documentation to mention that even SSD drives often
> have volatile write-back caches. Patch attached and applied.
Hmmm. That got me thinking: consider ZFS and HDD with volatile cache.
Do the characteristics of ZFS avoid this issue entirely?
- --
Dan Langille
BSDCan - The Technical BSD Conference : http://www.bsdcan.org/
PGCon - The PostgreSQL Conference: http://www.pgcon.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.13 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkuAayQACgkQCgsXFM/7nTyMggCgnZUbVzldxjp/nPo8EL1Nq6uG
6+IAoNGIB9x8/mwUQidjM9nnAADRbr9j
=3RJi
-----END PGP SIGNATURE-----
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
@ 2010-02-20 23:19 ` Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
0 siblings, 1 reply; 36+ messages in thread
From: Bruce Momjian @ 2010-02-20 23:19 UTC (permalink / raw)
To: Dan Langille <dan@langille.org>; +Cc: Matthew Wakeling <matthew@flymine.org>; Greg Smith <greg@2ndquadrant.com>; Laszlo Nagy <gandalf@shopzeus.com>; pgsql-performance
Dan Langille wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Bruce Momjian wrote:
> > Matthew Wakeling wrote:
> >> On Fri, 13 Nov 2009, Greg Smith wrote:
> >>> In order for a drive to work reliably for database use such as for
> >>> PostgreSQL, it cannot have a volatile write cache. You either need a write
> >>> cache with a battery backup (and a UPS doesn't count), or to turn the cache
> >>> off. The SSD performance figures you've been looking at are with the drive's
> >>> write cache turned on, which means they're completely fictitious and
> >>> exaggerated upwards for your purposes. In the real world, that will result
> >>> in database corruption after a crash one day.
> >> Seagate are claiming to be on the ball with this one.
> >>
> >> http://www.theregister.co.uk/2009/12/08/seagate_pulsar_ssd/
> >
> > I have updated our documentation to mention that even SSD drives often
> > have volatile write-back caches. Patch attached and applied.
>
> Hmmm. That got me thinking: consider ZFS and HDD with volatile cache.
> Do the characteristics of ZFS avoid this issue entirely?
No, I don't think so. ZFS only avoids partial page writes. ZFS still
assumes something sent to the drive is permanent or it would have no way
to operate.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-02-21 09:34 ` Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 36+ messages in thread
From: Scott Carey @ 2010-02-21 09:34 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Dan Langille <dan@langille.org>; Matthew Wakeling <matthew@flymine.org>; Greg Smith <greg@2ndquadrant.com>; Laszlo Nagy <gandalf@shopzeus.com>; pgsql-performance
On Feb 20, 2010, at 3:19 PM, Bruce Momjian wrote:
> Dan Langille wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Bruce Momjian wrote:
>>> Matthew Wakeling wrote:
>>>> On Fri, 13 Nov 2009, Greg Smith wrote:
>>>>> In order for a drive to work reliably for database use such as for
>>>>> PostgreSQL, it cannot have a volatile write cache. You either need a write
>>>>> cache with a battery backup (and a UPS doesn't count), or to turn the cache
>>>>> off. The SSD performance figures you've been looking at are with the drive's
>>>>> write cache turned on, which means they're completely fictitious and
>>>>> exaggerated upwards for your purposes. In the real world, that will result
>>>>> in database corruption after a crash one day.
>>>> Seagate are claiming to be on the ball with this one.
>>>>
>>>> http://www.theregister.co.uk/2009/12/08/seagate_pulsar_ssd/
>>>
>>> I have updated our documentation to mention that even SSD drives often
>>> have volatile write-back caches. Patch attached and applied.
>>
>> Hmmm. That got me thinking: consider ZFS and HDD with volatile cache.
>> Do the characteristics of ZFS avoid this issue entirely?
>
> No, I don't think so. ZFS only avoids partial page writes. ZFS still
> assumes something sent to the drive is permanent or it would have no way
> to operate.
>
ZFS is write-back cache aware, and safe provided the drive's cache flushing and write barrier related commands work. It will flush data in 'transaction groups' and flush the drive write caches at the end of those transactions. Since its copy on write, it can ensure that all the changes in the transaction group appear on disk, or all are lost. This all works so long as the cache flush commands do.
> --
> Bruce Momjian <bruce@momjian.us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
> PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
> + If your life is a hard drive, Christ can be your backup. +
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
@ 2010-02-21 14:10 ` Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
0 siblings, 1 reply; 36+ messages in thread
From: Bruce Momjian @ 2010-02-21 14:10 UTC (permalink / raw)
To: Scott Carey <scott@richrelevance.com>; +Cc: Dan Langille <dan@langille.org>; Matthew Wakeling <matthew@flymine.org>; Greg Smith <greg@2ndquadrant.com>; Laszlo Nagy <gandalf@shopzeus.com>; pgsql-performance
Scott Carey wrote:
> On Feb 20, 2010, at 3:19 PM, Bruce Momjian wrote:
>
> > Dan Langille wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Bruce Momjian wrote:
> >>> Matthew Wakeling wrote:
> >>>> On Fri, 13 Nov 2009, Greg Smith wrote:
> >>>>> In order for a drive to work reliably for database use such as for
> >>>>> PostgreSQL, it cannot have a volatile write cache. You either need a write
> >>>>> cache with a battery backup (and a UPS doesn't count), or to turn the cache
> >>>>> off. The SSD performance figures you've been looking at are with the drive's
> >>>>> write cache turned on, which means they're completely fictitious and
> >>>>> exaggerated upwards for your purposes. In the real world, that will result
> >>>>> in database corruption after a crash one day.
> >>>> Seagate are claiming to be on the ball with this one.
> >>>>
> >>>> http://www.theregister.co.uk/2009/12/08/seagate_pulsar_ssd/
> >>>
> >>> I have updated our documentation to mention that even SSD drives often
> >>> have volatile write-back caches. Patch attached and applied.
> >>
> >> Hmmm. That got me thinking: consider ZFS and HDD with volatile cache.
> >> Do the characteristics of ZFS avoid this issue entirely?
> >
> > No, I don't think so. ZFS only avoids partial page writes. ZFS still
> > assumes something sent to the drive is permanent or it would have no way
> > to operate.
> >
>
> ZFS is write-back cache aware, and safe provided the drive's
> cache flushing and write barrier related commands work. It will
> flush data in 'transaction groups' and flush the drive write
> caches at the end of those transactions. Since its copy on
> write, it can ensure that all the changes in the transaction
> group appear on disk, or all are lost. This all works so long
> as the cache flush commands do.
Agreed, thought I thought the problem was that SSDs lie about their
cache flush like SATA drives do, or is there something I am missing?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-02-21 14:54 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:39 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 36+ messages in thread
From: Ron Mayer @ 2010-02-21 14:54 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Bruce Momjian wrote:
> Agreed, thought I thought the problem was that SSDs lie about their
> cache flush like SATA drives do, or is there something I am missing?
There's exactly one case I can find[1] where this century's IDE
drives lied more than any other drive with a cache:
Under 120GB Maxtor drives from late 2003 to early 2004.
and it's apparently been worked around for years.
Those drives claimed to support the "FLUSH_CACHE_EXT" feature (IDE
command 0xEA), but did not support sending 48-bit commands which
was needed to send the cache flushing command.
And for that case a workaround for Linux was quickly identified by
checking for *both* the support for 48-bit commands and support for the
flush cache extension[2].
Beyond those 2004 drive + 2003 kernel systems, I think most the rest
of such reports have been various misfeatures in some of Linux's
filesystems (like EXT3 that only wants to send drives cache-flushing
commands when inode change[3]) and linux software raid misfeatures....
...and ISTM those would affect SSDs the same way they'd affect SATA drives.
[1] http://lkml.org/lkml/2004/5/12/132
[2] http://lkml.org/lkml/2004/5/12/200
[3] http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg272253.html
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
@ 2010-02-22 05:39 ` Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
1 sibling, 3 replies; 36+ messages in thread
From: Greg Smith @ 2010-02-22 05:39 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Bruce Momjian <bruce@momjian.us>; pgsql-performance
Ron Mayer wrote:
> Bruce Momjian wrote:
>
>> Agreed, thought I thought the problem was that SSDs lie about their
>> cache flush like SATA drives do, or is there something I am missing?
>>
>
> There's exactly one case I can find[1] where this century's IDE
> drives lied more than any other drive with a cache:
Ron is correct that the problem of mainstream SATA drives accepting the
cache flush command but not actually doing anything with it is long gone
at this point. If you have a regular SATA drive, it almost certainly
supports proper cache flushing. And if your whole software/storage
stacks understands all that, you should not end up with corrupted data
just because there's a volative write cache in there.
But the point of this whole testing exercise coming back into vogue
again is that SSDs have returned this negligent behavior to the
mainstream again. See
http://opensolaris.org/jive/thread.jspa?threadID=121424 for a discussion
of this in a ZFS context just last month. There are many documented
cases of Intel SSDs that will fake a cache flush, such that the only way
to get good reliable writes is to totally disable their writes
caches--at which point performance is so bad you might as well have
gotten a RAID10 setup instead (and longevity is toast too).
This whole area remains a disaster area and extreme distrust of all the
SSD storage vendors is advisable at this point. Basically, if I don't
see the capacitor responsible for flushing outstanding writes, and get a
clear description from the manufacturer how the cached writes are going
to be handled in the event of a power failure, at this point I have to
assume the answer is "badly and your data will be eaten". And the
prices for SSDs that meet that requirement are still quite steep. I
keep hoping somebody will address this market at something lower than
the standard "enterprise" prices. The upcoming SandForce designs seem
to have thought this through correctly:
http://www.anandtech.com/storage/showdoc.aspx?i=3702&p=6 But the
product's not out to the general public yet (just like the Seagate units
that claim to have capacitor backups--I heard a rumor those are also
Sandforce designs actually, so they may be the only ones doing this
right and aiming at a lower price).
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-02-22 07:10 ` Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2 siblings, 1 reply; 36+ messages in thread
From: Arjen van der Meijden @ 2010-02-22 07:10 UTC (permalink / raw)
To: pgsql-performance
On 22-2-2010 6:39 Greg Smith wrote:
> But the point of this whole testing exercise coming back into vogue
> again is that SSDs have returned this negligent behavior to the
> mainstream again. See
> http://opensolaris.org/jive/thread.jspa?threadID=121424 for a discussion
> of this in a ZFS context just last month. There are many documented
> cases of Intel SSDs that will fake a cache flush, such that the only way
> to get good reliable writes is to totally disable their writes
> caches--at which point performance is so bad you might as well have
> gotten a RAID10 setup instead (and longevity is toast too).
That's weird. Intel's SSD's didn't have a write cache afaik:
"I asked Intel about this and it turns out that the DRAM on the Intel
drive isn't used for user data because of the risk of data loss, instead
it is used as memory by the Intel SATA/flash controller for deciding
exactly where to write data (I'm assuming for the wear
leveling/reliability algorithms)."
http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403&p=10
But that is the old version, perhaps the second generation does have a
bit of write caching.
I can understand a SSD might do unexpected things when it loses power
all of a sudden. It will probably try to group writes to fill a single
block (and those blocks vary in size but are normally way larger than
those of a normal spinning disk, they are values like 256 or 512KB) and
it might loose that "waiting until a full block can be written"-data or
perhaps it just couldn't complete a full block-write due to the power
failure.
Although that behavior isn't really what you want, it would be incorrect
to blame write caching for the behavior if the device doesn't even have
a write cache ;)
Best regards,
Arjen
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
@ 2010-02-23 01:04 ` Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
0 siblings, 1 reply; 36+ messages in thread
From: Greg Smith @ 2010-02-23 01:04 UTC (permalink / raw)
To: Arjen van der Meijden <acmmailing@tweakers.net>; +Cc: pgsql-performance
Arjen van der Meijden wrote:
> That's weird. Intel's SSD's didn't have a write cache afaik:
> "I asked Intel about this and it turns out that the DRAM on the Intel
> drive isn't used for user data because of the risk of data loss,
> instead it is used as memory by the Intel SATA/flash controller for
> deciding exactly where to write data (I'm assuming for the wear
> leveling/reliability algorithms)."
> http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403&p=10
Read further down:
"Despite the presence of the external DRAM, both the Intel controller
and the JMicron rely on internal buffers to cache accesses to the
SSD...Intel's controller has a 256KB SRAM on-die."
That's the problematic part: the Intel controllers have a volatile
256KB write cache stored deep inside the SSD controller, and issuing a
standard SATA write cache flush command doesn't seem to clear it. Makes
the drives troublesome for database use.
> I can understand a SSD might do unexpected things when it loses power
> all of a sudden. It will probably try to group writes to fill a single
> block (and those blocks vary in size but are normally way larger than
> those of a normal spinning disk, they are values like 256 or 512KB)
> and it might loose that "waiting until a full block can be
> written"-data or perhaps it just couldn't complete a full block-write
> due to the power failure.
> Although that behavior isn't really what you want, it would be
> incorrect to blame write caching for the behavior if the device
> doesn't even have a write cache ;)
If you write data and that write call returns before the data hits disk,
it's a write cache, period. And if that write cache loses its contents
if power is lost, it's a volatile write cache that can cause database
corruption. The fact that the one on the Intel devices is very small,
basically just dealing with the block chunking behavior you describe,
doesn't change either of those facts.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-02-23 01:11 ` Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 36+ messages in thread
From: Mark Mielke @ 2010-02-23 01:11 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
On 02/22/2010 08:04 PM, Greg Smith wrote:
> Arjen van der Meijden wrote:
>> That's weird. Intel's SSD's didn't have a write cache afaik:
>> "I asked Intel about this and it turns out that the DRAM on the Intel
>> drive isn't used for user data because of the risk of data loss,
>> instead it is used as memory by the Intel SATA/flash controller for
>> deciding exactly where to write data (I'm assuming for the wear
>> leveling/reliability algorithms)."
>> http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403&p=10
>
> Read further down:
>
> "Despite the presence of the external DRAM, both the Intel controller
> and the JMicron rely on internal buffers to cache accesses to the
> SSD...Intel's controller has a 256KB SRAM on-die."
>
> That's the problematic part: the Intel controllers have a volatile
> 256KB write cache stored deep inside the SSD controller, and issuing a
> standard SATA write cache flush command doesn't seem to clear it.
> Makes the drives troublesome for database use.
I had read the above when posted, and then looked up SRAM. SRAM seems to
suggest it will hold the data even after power loss, but only for a
period of time. As long as power can restore within a few minutes, it
seemed like this would be ok?
>> I can understand a SSD might do unexpected things when it loses power
>> all of a sudden. It will probably try to group writes to fill a
>> single block (and those blocks vary in size but are normally way
>> larger than those of a normal spinning disk, they are values like 256
>> or 512KB) and it might loose that "waiting until a full block can be
>> written"-data or perhaps it just couldn't complete a full block-write
>> due to the power failure.
>> Although that behavior isn't really what you want, it would be
>> incorrect to blame write caching for the behavior if the device
>> doesn't even have a write cache ;)
>
> If you write data and that write call returns before the data hits
> disk, it's a write cache, period. And if that write cache loses its
> contents if power is lost, it's a volatile write cache that can cause
> database corruption. The fact that the one on the Intel devices is
> very small, basically just dealing with the block chunking behavior
> you describe, doesn't change either of those facts.
>
The SRAM seems to suggest that it does not necessarily lose its contents
if power is lost - it just doesn't say how long you have to plug it back
in. Isn't this similar to a battery-backed cache or capacitor-backed cache?
I'd love to have a better guarantee - but is SRAM really such a bad model?
Cheers,
mark
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
@ 2010-02-23 01:39 ` Greg Smith <greg@2ndquadrant.com>
2010-02-23 02:21 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 36+ messages in thread
From: Greg Smith @ 2010-02-23 01:39 UTC (permalink / raw)
To: Mark Mielke <mark@mark.mielke.cc>; +Cc: Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
Mark Mielke wrote:
> I had read the above when posted, and then looked up SRAM. SRAM seems
> to suggest it will hold the data even after power loss, but only for a
> period of time. As long as power can restore within a few minutes, it
> seemed like this would be ok?
The normal type of RAM everyone uses is DRAM, which requires constrant
"refresh" cycles to keep it working and is pretty power hungry as a
result. Power gone, data gone an instant later.
There is also Non-volatile SRAM that includes an integrated battery (
http://www.maxim-ic.com/quick_view2.cfm/qv_pk/2648 is a typical
example), and that sort of design can be used to build the sort of
battery-backed caches that RAID controllers provide. If Intel's drives
were built using a NV-SRAM implementation, I'd be a happy owner of one
instead of a constant critic of their drives.
But regular old SRAM is still completely volatile and loses its contents
very quickly after power fails. I doubt you'd even get minutes of time
before it's gone. The ease at which data loss failures with these Intel
drives continue to be duplicated in the field says their design isn't
anywhere near good enough to be considered non-volatile.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-02-23 02:21 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 02:22 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 36+ messages in thread
From: Scott Marlowe @ 2010-02-23 02:21 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Mark Mielke <mark@mark.mielke.cc>; Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
On Mon, Feb 22, 2010 at 6:39 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Mark Mielke wrote:
>>
>> I had read the above when posted, and then looked up SRAM. SRAM seems to
>> suggest it will hold the data even after power loss, but only for a period
>> of time. As long as power can restore within a few minutes, it seemed like
>> this would be ok?
>
> The normal type of RAM everyone uses is DRAM, which requires constrant
> "refresh" cycles to keep it working and is pretty power hungry as a result.
> Â Power gone, data gone an instant later.
Actually, oddly enough, per bit stored dram is much lower power usage
than sram, because it only has something like 2 transistors per bit,
while sram needs something like 4 or 5 (it's been a couple decades
since I took the classes on each). Even with the constant refresh,
dram has a lower power draw than sram.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 02:21 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
@ 2010-02-23 02:22 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 11:49 ` Re: SSD + RAID Pierre C <lists@peufeu.com>
0 siblings, 1 reply; 36+ messages in thread
From: Scott Marlowe @ 2010-02-23 02:22 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Mark Mielke <mark@mark.mielke.cc>; Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
On Mon, Feb 22, 2010 at 7:21 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> On Mon, Feb 22, 2010 at 6:39 PM, Greg Smith <greg@2ndquadrant.com> wrote:
>> Mark Mielke wrote:
>>>
>>> I had read the above when posted, and then looked up SRAM. SRAM seems to
>>> suggest it will hold the data even after power loss, but only for a period
>>> of time. As long as power can restore within a few minutes, it seemed like
>>> this would be ok?
>>
>> The normal type of RAM everyone uses is DRAM, which requires constrant
>> "refresh" cycles to keep it working and is pretty power hungry as a result.
>> Â Power gone, data gone an instant later.
>
> Actually, oddly enough, per bit stored dram is much lower power usage
> than sram, because it only has something like 2 transistors per bit,
> while sram needs something like 4 or 5 (it's been a couple decades
> since I took the classes on each). Â Even with the constant refresh,
> dram has a lower power draw than sram.
Note that's power draw per bit. dram is usually much more densely
packed (it can be with fewer transistors per cell) so the individual
chips for each may have similar power draws while the dram will be 10
times as densely packed as the sram.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 02:21 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 02:22 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
@ 2010-02-23 11:49 ` Pierre C <lists@peufeu.com>
2010-02-23 12:07 ` Re: SSD + RAID Nikolas Everett <nik9000@gmail.com>
2010-02-23 18:36 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
0 siblings, 2 replies; 36+ messages in thread
From: Pierre C @ 2010-02-23 11:49 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; Greg Smith <greg@2ndquadrant.com>; +Cc: Mark Mielke <mark@mark.mielke.cc>; Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
> Note that's power draw per bit. dram is usually much more densely
> packed (it can be with fewer transistors per cell) so the individual
> chips for each may have similar power draws while the dram will be 10
> times as densely packed as the sram.
Differences between SRAM and DRAM :
- price per byte (DRAM much cheaper)
- silicon area per byte (DRAM much smaller)
- random access latency
SRAM = fast, uniform, and predictable, usually 0/1 cycles
DRAM = "a few" up to "a lot" of cycles depending on chip type,
which page/row/column you want to access, wether it's R or W,
wether the page is already open, etc
In fact, DRAM is the new harddisk. SRAM is used mostly when low-latency is
needed (caches, etc).
- ease of use :
SRAM very easy to use : address, data, read, write, clock.
SDRAM needs a smart controller.
SRAM easier to instantiate on a silicon chip
- power draw
When used at high speeds, SRAM ist't power-saving at all, it's used for
speed.
However when not used, the power draw is really negligible.
While it is true that you can recover *some* data out of a SRAM/DRAM chip
that hasn't been powered for a few seconds, you can't really trust that
data. It's only a forensics tool.
Most DRAM now (especially laptop DRAM) includes special power-saving modes
which only keep the data retention logic (refresh, etc) powered, but not
the rest of the chip (internal caches, IO buffers, etc). Laptops, PDAs,
etc all use this feature in suspend-to-RAM mode. In this mode, the power
draw is higher than SRAM, but still pretty minimal, so a laptop can stay
in suspend-to-RAM mode for days.
Anyway, the SRAM vs DRAM isn't really relevant for the debate of SSD data
integrity. You can backup both with a small battery of ultra-cap.
What is important too is that the entire SSD chipset must have been
designed with this in mind : it must detect power loss, and correctly
react to it, and especially not reset itself or do funny stuff to the
memory when the power comes back. Which means at least some parts of the
chipset must stay powered to keep their state.
Now I wonder about something. SSDs use wear-leveling which means the
information about which block was written where must be kept somewhere.
Which means this information must be updated. I wonder how crash-safe and
how atomic these updates are, in the face of a power loss. This is just
like a filesystem. You've been talking only about data, but the block
layout information (metadata) is subject to the same concerns. If the
drive says it's written, not only the data must have been written, but
also the information needed to locate that data...
Therefore I think the yank-the-power-cord test should be done with random
writes happening on an aged and mostly-full SSD... and afterwards, I'd be
interested to know if not only the last txn really committed, but if some
random parts of other stuff weren't "wear-leveled" into oblivion at the
power loss...
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 02:21 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 02:22 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 11:49 ` Re: SSD + RAID Pierre C <lists@peufeu.com>
@ 2010-02-23 12:07 ` Nikolas Everett <nik9000@gmail.com>
1 sibling, 0 replies; 36+ messages in thread
From: Nikolas Everett @ 2010-02-23 12:07 UTC (permalink / raw)
To: Pierre C <lists@peufeu.com>; +Cc: Scott Marlowe <scott.marlowe@gmail.com>; Greg Smith <greg@2ndquadrant.com>; Mark Mielke <mark@mark.mielke.cc>; Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
On Tue, Feb 23, 2010 at 6:49 AM, Pierre C <lists@peufeu.com> wrote:
> Note that's power draw per bit. dram is usually much more densely
>> packed (it can be with fewer transistors per cell) so the individual
>> chips for each may have similar power draws while the dram will be 10
>> times as densely packed as the sram.
>>
>
> Differences between SRAM and DRAM :
>
> [lots of informative stuff]
>
I've been slowly reading the paper at
http://people.redhat.com/drepper/cpumemory.pdf which has a big section on
SRAM vs DRAM with nice pretty pictures. While not strictly relevant its been
illuminating and I wanted to share.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Re: SSD + RAID Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 02:21 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 02:22 ` Re: SSD + RAID Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 11:49 ` Re: SSD + RAID Pierre C <lists@peufeu.com>
@ 2010-02-23 18:36 ` Scott Carey <scott@richrelevance.com>
1 sibling, 0 replies; 36+ messages in thread
From: Scott Carey @ 2010-02-23 18:36 UTC (permalink / raw)
To: Pierre C <lists@peufeu.com>; +Cc: Scott Marlowe <scott.marlowe@gmail.com>; Greg Smith <greg@2ndquadrant.com>; Mark Mielke <mark@mark.mielke.cc>; Arjen van der Meijden <acmmailing@tweakers.net>; pgsql-performance
On Feb 23, 2010, at 3:49 AM, Pierre C wrote:
> Now I wonder about something. SSDs use wear-leveling which means the
> information about which block was written where must be kept somewhere.
> Which means this information must be updated. I wonder how crash-safe and
> how atomic these updates are, in the face of a power loss. This is just
> like a filesystem. You've been talking only about data, but the block
> layout information (metadata) is subject to the same concerns. If the
> drive says it's written, not only the data must have been written, but
> also the information needed to locate that data...
>
> Therefore I think the yank-the-power-cord test should be done with random
> writes happening on an aged and mostly-full SSD... and afterwards, I'd be
> interested to know if not only the last txn really committed, but if some
> random parts of other stuff weren't "wear-leveled" into oblivion at the
> power loss...
>
A couple years ago I postulated that SSD's could do random writes fast if they remapped blocks. Microsoft's SSD whitepaper at the time hinted at this too.
Persisting the remap data is not hard. It goes in the same location as the data, or a separate area that can be written to linearly.
Each block may contain its LBA and a transaction ID or other atomic count. Or another block can have that info. When the SSD
powers up, it can build its table of LBA > block by looking at that data and inverting it and keeping the highest transaction ID for duplicate LBA claims.
Although SSD's have to ERASE data in a large block at a time (256K to 2M typically), they can write linearly to an erased block in much smaller chunks.
Thus, to commit a write, either:
Data, LBA tag, and txID in same block (may require oddly sized blocks).
or
Data written to one block (not committed yet), then LBA tag and txID written elsewhere (which commits the write). Since its all copy on write, partial writes can't happen.
If a block is being moved or compressed when power fails data should never be lost since the old data still exists, the new version just didn't commit. But new data that is being written may not be committed yet in the case of a power failure unless other measures are taken.
>
>
>
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-02-22 14:37 ` Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2 siblings, 1 reply; 36+ messages in thread
From: Bruce Momjian @ 2010-02-22 14:37 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Greg Smith wrote:
> Ron Mayer wrote:
> > Bruce Momjian wrote:
> >
> >> Agreed, thought I thought the problem was that SSDs lie about their
> >> cache flush like SATA drives do, or is there something I am missing?
> >>
> >
> > There's exactly one case I can find[1] where this century's IDE
> > drives lied more than any other drive with a cache:
>
> Ron is correct that the problem of mainstream SATA drives accepting the
> cache flush command but not actually doing anything with it is long gone
> at this point. If you have a regular SATA drive, it almost certainly
> supports proper cache flushing. And if your whole software/storage
> stacks understands all that, you should not end up with corrupted data
> just because there's a volative write cache in there.
OK, but I have a few questions. Is a write to the drive and a cache
flush command the same? Which file systems implement both? I thought a
write to the drive was always assumed to flush it to the platters,
assuming the drive's cache is set to write-through.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-02-22 18:00 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 01:14 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-23 08:23 ` Re: SSD + RAID david@lang.hm
0 siblings, 2 replies; 36+ messages in thread
From: Ron Mayer @ 2010-02-22 18:00 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Greg Smith <greg@2ndquadrant.com>; Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Bruce Momjian wrote:
> Greg Smith wrote:
>> .... If you have a regular SATA drive, it almost certainly
>> supports proper cache flushing....
>
> OK, but I have a few questions. Is a write to the drive and a cache
> flush command the same?
I believe they're different as of ATAPI-6 from 2001.
> Which file systems implement both?
Seems ZFS and recent ext4 have thought these interactions out
thoroughly. Find a slow ext4 that people complain about, and
that's the one doing it right :-).
Ext3 has some particularly odd annoyances where it flushes and waits
for certain writes (ones involving inode changes) but doesn't bother
to flush others (just data changes). As far as I can tell, with
ext3 you need userspace utilities to make sure flushes occur when
you need them. At one point I was tempted to try to put such
userspace hacks into postgres.
I know less about other file systems. Apparently the NTFS guys
are aware of such stuff - but don't know what kinds of fsync equivalent
you'd need to make it happen.
Also worth noting - Linux's software raid stuff (MD and LVM)
need to handle this right as well - and last I checked (sometime
last year) the default setups didn't.
> I thought a
> write to the drive was always assumed to flush it to the platters,
> assuming the drive's cache is set to write-through.
Apparently somewhere around here:
http://www.t10.org/t13/project/d1410r3a-ATA-ATAPI-6.pdf
they were separated in the IDE world.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
@ 2010-02-23 01:14 ` Greg Smith <greg@2ndquadrant.com>
1 sibling, 0 replies; 36+ messages in thread
From: Greg Smith @ 2010-02-23 01:14 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Bruce Momjian <bruce@momjian.us>; pgsql-performance
Ron Mayer wrote:
> I know less about other file systems. Apparently the NTFS guys
> are aware of such stuff - but don't know what kinds of fsync equivalent
> you'd need to make it happen.
>
It's actually pretty straightforward--better than ext3. Windows with
NTFS has been perfectly aware how to do write-through on drives that
support it when you execute _commit for some time:
http://msdn.microsoft.com/en-us/library/17618685(VS.80).aspx
If you switch the postgresql.conf setting to fsync_writethrough on
Windows, it will execute _commit where it would execute fsync on other
platforms, and that pushes through the drive's caches as it should
(unlike fsync in many cases). More about this at
http://archives.postgresql.org/pgsql-hackers/2005-08/msg00227.php and
http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm (which
also covers OS X).
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
@ 2010-02-23 08:23 ` david@lang.hm
2010-02-23 19:35 ` Re: SSD + RAID david@lang.hm
1 sibling, 1 reply; 36+ messages in thread
From: david@lang.hm @ 2010-02-23 08:23 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On Mon, 22 Feb 2010, Ron Mayer wrote:
>
> Also worth noting - Linux's software raid stuff (MD and LVM)
> need to handle this right as well - and last I checked (sometime
> last year) the default setups didn't.
>
I think I saw some stuff in the last few months on this issue on the
kernel mailing list. you may want to doublecheck this when 2.6.33 gets
released (probably this week)
David Lang
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 08:23 ` Re: SSD + RAID david@lang.hm
@ 2010-02-23 19:35 ` david@lang.hm
2010-02-23 20:34 ` Re: SSD + RAID Aidan Van Dyk <aidan@highrise.ca>
0 siblings, 1 reply; 36+ messages in thread
From: david@lang.hm @ 2010-02-23 19:35 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On Tue, 23 Feb 2010, david@lang.hm wrote:
> On Mon, 22 Feb 2010, Ron Mayer wrote:
>
>>
>> Also worth noting - Linux's software raid stuff (MD and LVM)
>> need to handle this right as well - and last I checked (sometime
>> last year) the default setups didn't.
>>
>
> I think I saw some stuff in the last few months on this issue on the kernel
> mailing list. you may want to doublecheck this when 2.6.33 gets released
> (probably this week)
to clarify further (after getting more sleep ;-)
I believe that the linux software raid always did the right thing if you
did a fsync/fdatacync. however barriers that filesystems attempted to use
to avoid the need for a hard fsync used to be silently ignored. I believe
these are now honored (in at least some configurations)
However, one thing that you do not get protection against with software
raid is the potential for the writes to hit some drives but not others. If
this happens the software raid cannot know what the correct contents of
the raid stripe are, and so you could loose everything in that stripe
(including contents of other files that are not being modified that
happened to be in the wrong place on the array)
If you have critical data, you _really_ want to use a raid controller with
battery backup so that if you loose power you have a chance of eventually
completing the write.
David Lang
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 08:23 ` Re: SSD + RAID david@lang.hm
2010-02-23 19:35 ` Re: SSD + RAID david@lang.hm
@ 2010-02-23 20:34 ` Aidan Van Dyk <aidan@highrise.ca>
2010-02-23 21:22 ` Re: SSD + RAID david@lang.hm
0 siblings, 1 reply; 36+ messages in thread
From: Aidan Van Dyk @ 2010-02-23 20:34 UTC (permalink / raw)
To: david@lang.hm; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; Bruce Momjian <bruce@momjian.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance
* david@lang.hm <david@lang.hm> [100223 15:05]:
> However, one thing that you do not get protection against with software
> raid is the potential for the writes to hit some drives but not others.
> If this happens the software raid cannot know what the correct contents
> of the raid stripe are, and so you could loose everything in that stripe
> (including contents of other files that are not being modified that
> happened to be in the wrong place on the array)
That's for stripe-based raid. Mirror sets like raid-1 should give you
either the old data, or the new data, both acceptable responses since
the fsync/barreir hasn't "completed".
Or have I missed another subtle interaction?
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 08:23 ` Re: SSD + RAID david@lang.hm
2010-02-23 19:35 ` Re: SSD + RAID david@lang.hm
2010-02-23 20:34 ` Re: SSD + RAID Aidan Van Dyk <aidan@highrise.ca>
@ 2010-02-23 21:22 ` david@lang.hm
2010-02-23 21:32 ` Re: SSD + RAID Mark Mielke <mark@mark.mielke.cc>
2010-02-24 08:32 ` Re: SSD + RAID Dave Crooke <dcrooke@gmail.com>
0 siblings, 2 replies; 36+ messages in thread
From: david@lang.hm @ 2010-02-23 21:22 UTC (permalink / raw)
To: Aidan Van Dyk <aidan@highrise.ca>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; Bruce Momjian <bruce@momjian.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On Tue, 23 Feb 2010, Aidan Van Dyk wrote:
> * david@lang.hm <david@lang.hm> [100223 15:05]:
>
>> However, one thing that you do not get protection against with software
>> raid is the potential for the writes to hit some drives but not others.
>> If this happens the software raid cannot know what the correct contents
>> of the raid stripe are, and so you could loose everything in that stripe
>> (including contents of other files that are not being modified that
>> happened to be in the wrong place on the array)
>
> That's for stripe-based raid. Mirror sets like raid-1 should give you
> either the old data, or the new data, both acceptable responses since
> the fsync/barreir hasn't "completed".
>
> Or have I missed another subtle interaction?
one problem is that when the system comes back up and attempts to check
the raid array, it is not going to know which drive has valid data. I
don't know exactly what it does in that situation, but this type of error
in other conditions causes the system to take the array offline.
David Lang
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 08:23 ` Re: SSD + RAID david@lang.hm
2010-02-23 19:35 ` Re: SSD + RAID david@lang.hm
2010-02-23 20:34 ` Re: SSD + RAID Aidan Van Dyk <aidan@highrise.ca>
2010-02-23 21:22 ` Re: SSD + RAID david@lang.hm
@ 2010-02-23 21:32 ` Mark Mielke <mark@mark.mielke.cc>
1 sibling, 0 replies; 36+ messages in thread
From: Mark Mielke @ 2010-02-23 21:32 UTC (permalink / raw)
To: david@lang.hm; +Cc: Aidan Van Dyk <aidan@highrise.ca>; Ron Mayer <rm_pg@cheapcomplexdevices.com>; Bruce Momjian <bruce@momjian.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On 02/23/2010 04:22 PM, david@lang.hm wrote:
> On Tue, 23 Feb 2010, Aidan Van Dyk wrote:
>
>> * david@lang.hm <david@lang.hm> [100223 15:05]:
>>
>>> However, one thing that you do not get protection against with software
>>> raid is the potential for the writes to hit some drives but not others.
>>> If this happens the software raid cannot know what the correct contents
>>> of the raid stripe are, and so you could loose everything in that
>>> stripe
>>> (including contents of other files that are not being modified that
>>> happened to be in the wrong place on the array)
>>
>> That's for stripe-based raid. Mirror sets like raid-1 should give you
>> either the old data, or the new data, both acceptable responses since
>> the fsync/barreir hasn't "completed".
>>
>> Or have I missed another subtle interaction?
>
> one problem is that when the system comes back up and attempts to
> check the raid array, it is not going to know which drive has valid
> data. I don't know exactly what it does in that situation, but this
> type of error in other conditions causes the system to take the array
> offline.
I think the real concern here is that depending on how the data is read
later - and depending on which disks it reads from - it could read
*either* old or new, at any time in the future. I.e. it reads "new" from
disk 1 the first time, and then an hour later it reads "old" from disk 2.
I think this concern might be invalid for a properly running system,
though. When a RAID array is not cleanly shut down, the RAID array
should run in "degraded" mode until it can be sure that the data is
consistent. In this case, it should pick one drive, and call it the
"live" one, and then rebuild the other from the "live" one. Until it is
re-built, it should only satisfy reads from the "live" one, or parts of
the "rebuilding" one that are known to be clean.
I use mdadm software RAID, and all of me reading (including some of its
source code) and experience (shutting down the box uncleanly) tells me,
it is working properly. In fact, the "rebuild" process can get quite
ANNOYING as the whole system becomes much slower during rebuild, and
rebuild of large partitions can take hours to complete.
For mdadm, there is a not-so-well-known "write-intent bitmap"
capability. Once enabled, mdadm will embed a small bitmap (128 bits?)
into the partition, and each bit will indicate a section of the
partition. Before writing to a section, it will mark that section as
dirty using this bitmap. It will leave this bit set for some time after
the partition is "clean" (lazy clear). The effect of this, is that at
any point in time, only certain sections of the drive are dirty, and on
recovery, it is a lot cheaper to only rebuild the dirty sections. It
works really well.
So, I don't think this has to be a problem. There are solutions, and any
solution that claims to be complete should offer these sorts of
capabilities.
Cheers,
mark
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-22 14:37 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 08:23 ` Re: SSD + RAID david@lang.hm
2010-02-23 19:35 ` Re: SSD + RAID david@lang.hm
2010-02-23 20:34 ` Re: SSD + RAID Aidan Van Dyk <aidan@highrise.ca>
2010-02-23 21:22 ` Re: SSD + RAID david@lang.hm
@ 2010-02-24 08:32 ` Dave Crooke <dcrooke@gmail.com>
1 sibling, 0 replies; 36+ messages in thread
From: Dave Crooke @ 2010-02-24 08:32 UTC (permalink / raw)
To: david@lang.hm; +Cc: Aidan Van Dyk <aidan@highrise.ca>; Ron Mayer <rm_pg@cheapcomplexdevices.com>; Bruce Momjian <bruce@momjian.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance
It's always possible to rebuild into a consistent configuration by assigning
a precedence order; for parity RAID, the data drives take precedence over
parity drives, and for RAID-1 sets it assigns an arbitrary master.
You *should* never lose a whole stripe ... for example, RAID-5 updates do
"read old data / parity, write new data, write new parity" ... there is no
need to touch any other data disks, so they will be preserved through the
rebuild. Similarly, if only one block is being updated there is no need to
update the entire stripe.
David - what caused /dev/md to decide to take an array offline?
Cheers
Dave
On Tue, Feb 23, 2010 at 3:22 PM, <david@lang.hm> wrote:
> On Tue, 23 Feb 2010, Aidan Van Dyk wrote:
>
> * david@lang.hm <david@lang.hm> [100223 15:05]:
>>
>> However, one thing that you do not get protection against with software
>>> raid is the potential for the writes to hit some drives but not others.
>>> If this happens the software raid cannot know what the correct contents
>>> of the raid stripe are, and so you could loose everything in that stripe
>>> (including contents of other files that are not being modified that
>>> happened to be in the wrong place on the array)
>>>
>>
>> That's for stripe-based raid. Mirror sets like raid-1 should give you
>> either the old data, or the new data, both acceptable responses since
>> the fsync/barreir hasn't "completed".
>>
>> Or have I missed another subtle interaction?
>>
>
> one problem is that when the system comes back up and attempts to check the
> raid array, it is not going to know which drive has valid data. I don't know
> exactly what it does in that situation, but this type of error in other
> conditions causes the system to take the array offline.
>
>
> David Lang
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-02-27 01:40 ` Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2 siblings, 1 reply; 36+ messages in thread
From: Bruce Momjian @ 2010-02-27 01:40 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
I have added documentation about the ATAPI drive flush command, and the
typical SSD behavior.
---------------------------------------------------------------------------
Greg Smith wrote:
> Ron Mayer wrote:
> > Bruce Momjian wrote:
> >
> >> Agreed, thought I thought the problem was that SSDs lie about their
> >> cache flush like SATA drives do, or is there something I am missing?
> >>
> >
> > There's exactly one case I can find[1] where this century's IDE
> > drives lied more than any other drive with a cache:
>
> Ron is correct that the problem of mainstream SATA drives accepting the
> cache flush command but not actually doing anything with it is long gone
> at this point. If you have a regular SATA drive, it almost certainly
> supports proper cache flushing. And if your whole software/storage
> stacks understands all that, you should not end up with corrupted data
> just because there's a volative write cache in there.
>
> But the point of this whole testing exercise coming back into vogue
> again is that SSDs have returned this negligent behavior to the
> mainstream again. See
> http://opensolaris.org/jive/thread.jspa?threadID=121424 for a discussion
> of this in a ZFS context just last month. There are many documented
> cases of Intel SSDs that will fake a cache flush, such that the only way
> to get good reliable writes is to totally disable their writes
> caches--at which point performance is so bad you might as well have
> gotten a RAID10 setup instead (and longevity is toast too).
>
> This whole area remains a disaster area and extreme distrust of all the
> SSD storage vendors is advisable at this point. Basically, if I don't
> see the capacitor responsible for flushing outstanding writes, and get a
> clear description from the manufacturer how the cached writes are going
> to be handled in the event of a power failure, at this point I have to
> assume the answer is "badly and your data will be eaten". And the
> prices for SSDs that meet that requirement are still quite steep. I
> keep hoping somebody will address this market at something lower than
> the standard "enterprise" prices. The upcoming SandForce designs seem
> to have thought this through correctly:
> http://www.anandtech.com/storage/showdoc.aspx?i=3702&p=6 But the
> product's not out to the general public yet (just like the Seagate units
> that claim to have capacitor backups--I heard a rumor those are also
> Sandforce designs actually, so they may be the only ones doing this
> right and aiming at a lower price).
>
> --
> Greg Smith 2ndQuadrant US Baltimore, MD
> PostgreSQL Training, Services and Support
> greg@2ndQuadrant.com www.2ndQuadrant.us
>
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +
Attachments:
/rtmp/diff (text/x-diff, 1.6K) download
Index: doc/src/sgml/wal.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/wal.sgml,v
retrieving revision 1.62
diff -c -c -r1.62 wal.sgml
*** doc/src/sgml/wal.sgml 20 Feb 2010 18:28:37 -0000 1.62
--- doc/src/sgml/wal.sgml 27 Feb 2010 01:37:03 -0000
***************
*** 59,66 ****
same concerns about data loss exist for write-back drive caches as
exist for disk controller caches. Consumer-grade IDE and SATA drives are
particularly likely to have write-back caches that will not survive a
! power failure. Many solid-state drives also have volatile write-back
! caches. To check write caching on <productname>Linux</> use
<command>hdparm -I</>; it is enabled if there is a <literal>*</> next
to <literal>Write cache</>; <command>hdparm -W</> to turn off
write caching. On <productname>FreeBSD</> use
--- 59,69 ----
same concerns about data loss exist for write-back drive caches as
exist for disk controller caches. Consumer-grade IDE and SATA drives are
particularly likely to have write-back caches that will not survive a
! power failure, though <acronym>ATAPI-6</> introduced a drive cache
! flush command that some file systems use, e.g. <acronym>ZFS</>.
! Many solid-state drives also have volatile write-back
! caches, and many do not honor cache flush commands by default.
! To check write caching on <productname>Linux</> use
<command>hdparm -I</>; it is enabled if there is a <literal>*</> next
to <literal>Write cache</>; <command>hdparm -W</> to turn off
write caching. On <productname>FreeBSD</> use
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-02-27 19:38 ` Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 36+ messages in thread
From: Greg Smith @ 2010-02-27 19:38 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Bruce Momjian wrote:
> I have added documentation about the ATAPI drive flush command, and the
> typical SSD behavior.
>
If one of us goes back into that section one day to edit again it might
be worth mentioning that FLUSH CACHE EXT is the actual ATAPI-6 command
that a drive needs to support properly. I wouldn't bother with another
doc edit commit just for that specific part though, pretty obscure.
I find it kind of funny how many discussions run in parallel about even
really detailed technical implementation details around the world. For
example, doesn't
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg30585.html
look exactly like the exchange between myself and Arjen the other day,
referencing the same AnandTech page?
Could be worse; one of us could be the poor sap at
http://opensolaris.org/jive/thread.jspa;jsessionid=41B679C30D136C059E1BB7C06CA7DCE0?messageID=397730
who installed Windows XP, VirtualBox for Windows, an OpenSolaris VM
inside of it, and then was shocked that cache flushes didn't make their
way all the way through that chain and had his 10TB ZFS pool corrupted
as a result. Hurray for virtualization!
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-02-27 20:16 ` Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
0 siblings, 1 reply; 36+ messages in thread
From: Bruce Momjian @ 2010-02-27 20:16 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Greg Smith wrote:
> Bruce Momjian wrote:
> > I have added documentation about the ATAPI drive flush command, and the
> > typical SSD behavior.
> >
>
> If one of us goes back into that section one day to edit again it might
> be worth mentioning that FLUSH CACHE EXT is the actual ATAPI-6 command
> that a drive needs to support properly. I wouldn't bother with another
> doc edit commit just for that specific part though, pretty obscure.
That setting name was not easy to find so I added it to the
documentation.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-02-28 04:05 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-28 05:06 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-03-02 03:33 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 36+ messages in thread
From: Ron Mayer @ 2010-02-28 04:05 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Greg Smith <greg@2ndquadrant.com>; Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Bruce Momjian wrote:
> Greg Smith wrote:
>> Bruce Momjian wrote:
>>> I have added documentation about the ATAPI drive flush command, and the
>>
>> If one of us goes back into that section one day to edit again it might
>> be worth mentioning that FLUSH CACHE EXT is the actual ATAPI-6 command
>> that a drive needs to support properly. I wouldn't bother with another
>> doc edit commit just for that specific part though, pretty obscure.
>
> That setting name was not easy to find so I added it to the
> documentation.
If we're spelling out specific IDE commands, it might be worth
noting that the corresponding SCSI command is "SYNCHRONIZE CACHE"[1].
Linux apparently sends FLUSH_CACHE commands to IDE drives in the
exact sample places it sends SYNCHRONIZE CACHE commands to SCSI
drives[2].
It seems that the same file systems, SW raid layers,
virtualization platforms, and kernels that have a problem
sending FLUSH CACHE commands to SATA drives have he same exact
same problems sending SYNCHRONIZE CACHE commands to SCSI drives.
With the exact same effect of not getting writes all the way
through disk caches.
No?
[1] http://linux.die.net/man/8/sg_sync
[2] http://hardware.slashdot.org/comments.pl?sid=149349&cid=12519114
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
@ 2010-02-28 05:06 ` Greg Smith <greg@2ndquadrant.com>
2010-03-02 03:34 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 36+ messages in thread
From: Greg Smith @ 2010-02-28 05:06 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Bruce Momjian <bruce@momjian.us>; pgsql-performance
Ron Mayer wrote:
> Linux apparently sends FLUSH_CACHE commands to IDE drives in the
> exact sample places it sends SYNCHRONIZE CACHE commands to SCSI
> drives[2].
> [2] http://hardware.slashdot.org/comments.pl?sid=149349&cid=12519114
>
Well, that's old enough to not even be completely right anymore about
SATA disks and kernels. It's FLUSH_CACHE_EXT that's been added to ATA-6
to do the right thing on modern drives and that gets used nowadays, and
that doesn't necessarily do so on most of the SSDs out there; all of
which Bruce's recent doc additions now talk about correctly.
There's this one specific area we know about that the most popular
systems tend to get really wrong all the time; that's got the
appropriate warning now with the right magic keywords that people can
look into it more if motivated. While it would be nice to get super
thorough and document everything, I think there's already more docs in
there than this project would prefer to have to maintain in this area.
Are we going to get into IDE, SATA, SCSI, SAS, FC, and iSCSI? If the
idea is to be complete that's where this would go. I don't know that
the documentation needs to address every possible way every possible
filesystem can be flushed.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-28 05:06 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-03-02 03:34 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 36+ messages in thread
From: Bruce Momjian @ 2010-03-02 03:34 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Greg Smith wrote:
> Ron Mayer wrote:
> > Linux apparently sends FLUSH_CACHE commands to IDE drives in the
> > exact sample places it sends SYNCHRONIZE CACHE commands to SCSI
> > drives[2].
> > [2] http://hardware.slashdot.org/comments.pl?sid=149349&cid=12519114
> >
>
> Well, that's old enough to not even be completely right anymore about
> SATA disks and kernels. It's FLUSH_CACHE_EXT that's been added to ATA-6
> to do the right thing on modern drives and that gets used nowadays, and
> that doesn't necessarily do so on most of the SSDs out there; all of
> which Bruce's recent doc additions now talk about correctly.
>
> There's this one specific area we know about that the most popular
> systems tend to get really wrong all the time; that's got the
> appropriate warning now with the right magic keywords that people can
> look into it more if motivated. While it would be nice to get super
> thorough and document everything, I think there's already more docs in
> there than this project would prefer to have to maintain in this area.
>
> Are we going to get into IDE, SATA, SCSI, SAS, FC, and iSCSI? If the
> idea is to be complete that's where this would go. I don't know that
> the documentation needs to address every possible way every possible
> filesystem can be flushed.
The bottom line is that the reason we have so much detailed
documentation about this is that mostly only database folks care about
such issues, so we end up having to research and document this
ourselves --- I don't see any alternatives.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
@ 2010-03-02 03:33 ` Bruce Momjian <bruce@momjian.us>
2010-03-02 06:13 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-03-02 08:36 ` Re: SSD + RAID Pierre C <lists@peufeu.com>
1 sibling, 2 replies; 36+ messages in thread
From: Bruce Momjian @ 2010-03-02 03:33 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Greg Smith <greg@2ndquadrant.com>; pgsql-performance
Ron Mayer wrote:
> Bruce Momjian wrote:
> > Greg Smith wrote:
> >> Bruce Momjian wrote:
> >>> I have added documentation about the ATAPI drive flush command, and the
> >>
> >> If one of us goes back into that section one day to edit again it might
> >> be worth mentioning that FLUSH CACHE EXT is the actual ATAPI-6 command
> >> that a drive needs to support properly. I wouldn't bother with another
> >> doc edit commit just for that specific part though, pretty obscure.
> >
> > That setting name was not easy to find so I added it to the
> > documentation.
>
> If we're spelling out specific IDE commands, it might be worth
> noting that the corresponding SCSI command is "SYNCHRONIZE CACHE"[1].
>
>
> Linux apparently sends FLUSH_CACHE commands to IDE drives in the
> exact sample places it sends SYNCHRONIZE CACHE commands to SCSI
> drives[2].
>
> It seems that the same file systems, SW raid layers,
> virtualization platforms, and kernels that have a problem
> sending FLUSH CACHE commands to SATA drives have he same exact
> same problems sending SYNCHRONIZE CACHE commands to SCSI drives.
> With the exact same effect of not getting writes all the way
> through disk caches.
I always assumed SCSI disks had a write-through cache and therefore
didn't need a drive cache flush comment.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-03-02 03:33 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-03-02 06:13 ` Greg Smith <greg@2ndquadrant.com>
2010-03-03 15:16 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
1 sibling, 1 reply; 36+ messages in thread
From: Greg Smith @ 2010-03-02 06:13 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Bruce Momjian wrote:
> I always assumed SCSI disks had a write-through cache and therefore
> didn't need a drive cache flush comment.
>
There's more detail on all this mess at
http://wiki.postgresql.org/wiki/SCSI_vs._IDE/SATA_Disks and it includes
this perception, which I've recently come to believe isn't actually
correct anymore. Like the IDE crowd, it looks like one day somebody
said "hey, we lose every write heavy benchmark badly because we only
have a write-through cache", and that principle got lost along the
wayside. What has been true, and I'm staring to think this is what
we've all been observing rather than a write-through cache, is that the
proper cache flushing commands have been there in working form for so
much longer that it's more likely your SCSI driver and drive do the
right thing if the filesystem asks them to. SCSI SYNCHRONIZE CACHE has
a much longer and prouder history than IDE's FLUSH_CACHE and SATA's
FLUSH_CACHE_EXT.
It's also worth noting that many current SAS drives, the current SCSI
incarnation, are basically SATA drives with a bridge chipset stuck onto
them, or with just the interface board swapped out. This one reason why
top-end SAS capacities lag behind consumer SATA drives. They use the
consumers as beta testers to get the really fundamental firmware issues
sorted out, and once things are stable they start stamping out the
version with the SAS interface instead. (Note that there's a parallel
manufacturing approach that makes much smaller SAS drives, the 2.5"
server models or those at higher RPMs, that doesn't go through this
path. Those are also the really expensive models, due to economy of
scale issues). The idea that these would have fundamentally different
write cache behavior doesn't really follow from that development model.
At this point, there are only two common differences between "consumer"
and "enterprise" hard drives of the same size and RPM when there are
directly matching ones:
1) You might get SAS instead of SATA as the interface, which provides
the more mature command set I was talking about above--and therefore may
give you a sane write-back cache with proper flushing, which is all the
database really expects.
2) The timeouts when there's a read/write problem are tuned down in the
enterprise version, to be more compatible with RAID setups where you
want to push the drive off-line when this happens rather than presuming
you can fix it. Consumers would prefer that the drive spent a lot of
time doing heroics to try and save their sole copy of the apparently
missing data.
You might get a slightly higher grade of parts if you're lucky too; I
wouldn't count on it though. That seems to be saved for the high RPM or
smaller size drives only.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-03-02 03:33 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-03-02 06:13 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
@ 2010-03-03 15:16 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
0 siblings, 0 replies; 36+ messages in thread
From: Ron Mayer @ 2010-03-03 15:16 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Ron Mayer <rm_pg@cheapcomplexdevices.com>; pgsql-performance
Greg Smith wrote:
> Bruce Momjian wrote:
>> I always assumed SCSI disks had a write-through cache and therefore
>> didn't need a drive cache flush comment.
Some do. SCSI disks have write-back caches.
Some have both(!) - a write-back cache but the user can explicitly
send write-through requests.
Microsoft explains it well (IMHO) here:
http://msdn.microsoft.com/en-us/library/aa508863.aspx
"For example, suppose that the target is a SCSI device with
a write-back cache. If the device supports write-through
requests, the initiator can bypass the write cache by
setting the force unit access (FUA) bit in the command
descriptor block (CDB) of the write command."
> this perception, which I've recently come to believe isn't actually
> correct anymore. ... I'm staring to think this is what
> we've all been observing rather than a write-through cache
I think what we've been observing is that guys with SCSI drives
are more likely to either
(a) have battery-backed RAID controllers that insure writes succeed,
or
(b) have other decent RAID controllers that understand details
like that FUA bit to send write-through requests even if
a SCSI devices has a write-back cache.
In contrast, most guys with PATA drives are probably running
software RAID (if any) with a RAID stack (older LVM and MD)
known to lose the cache flushing commands.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 01:40 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Re: SSD + RAID Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-03-02 03:33 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
@ 2010-03-02 08:36 ` Pierre C <lists@peufeu.com>
1 sibling, 0 replies; 36+ messages in thread
From: Pierre C @ 2010-03-02 08:36 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: Greg Smith <greg@2ndquadrant.com>; pgsql-performance
> I always assumed SCSI disks had a write-through cache and therefore
> didn't need a drive cache flush comment.
Maximum performance can only be reached with a writeback cache so the
drive can reorder and cluster writes, according to the realtime position
of the heads and platter rotation.
The problem is not the write cache itself, it is that, for your data to be
safe, the "flush cache" or "barrier" command must get all the way through
the application / filesystem to the hardware, going through a nondescript
number of software/firmware/hardware layers, all of which may :
- not specify if they honor or ignore flush/barrier commands, and which
ones
- not specify if they will reordre writes ignoring barriers/flushes or not
- have been written by people who are not aware of such issues
- have been written by companies who are perfectly aware of such issues
but chose to ignore them to look good in benchmarks
- have some incompatibilities that result in broken behaviour
- have bugs
As far as I'm concerned, a configuration that doesn't properly respect the
commands needed for data integrity is broken.
The sad truth is that given a software/hardware IO stack, there's no way
to be sure, and testing isn't easy, if at all possible to do. Some cache
flushes might be ignored under some circumstances.
For this to change, you don't need a hardware change, but a mentality
change.
Flash filesystem developers use flash simulators which measure wear
leveling, etc.
We'd need a virtual box with a simulated virtual harddrive which is able
to check this.
What a mess.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: SSD + RAID
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Re: SSD + RAID Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Re: SSD + RAID Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Re: SSD + RAID Ron Mayer <rm_pg@cheapcomplexdevices.com>
@ 2010-02-22 14:39 ` Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 36+ messages in thread
From: Bruce Momjian @ 2010-02-22 14:39 UTC (permalink / raw)
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>; +Cc: pgsql-performance
Ron Mayer wrote:
> Bruce Momjian wrote:
> > Agreed, thought I thought the problem was that SSDs lie about their
> > cache flush like SATA drives do, or is there something I am missing?
>
> There's exactly one case I can find[1] where this century's IDE
> drives lied more than any other drive with a cache:
>
> Under 120GB Maxtor drives from late 2003 to early 2004.
>
> and it's apparently been worked around for years.
>
> Those drives claimed to support the "FLUSH_CACHE_EXT" feature (IDE
> command 0xEA), but did not support sending 48-bit commands which
> was needed to send the cache flushing command.
>
> And for that case a workaround for Linux was quickly identified by
> checking for *both* the support for 48-bit commands and support for the
> flush cache extension[2].
>
>
> Beyond those 2004 drive + 2003 kernel systems, I think most the rest
> of such reports have been various misfeatures in some of Linux's
> filesystems (like EXT3 that only wants to send drives cache-flushing
> commands when inode change[3]) and linux software raid misfeatures....
>
> ...and ISTM those would affect SSDs the same way they'd affect SATA drives.
I think the point is not that drives lie about their write-back and
write-through behavior, but rather that many SATA/IDE drives default to
write-back, and not write-through, and many administrators an file
systems are not aware of this behavior.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +
^ permalink raw reply [nested|flat] 36+ messages in thread
end of thread, other threads:[~2010-03-03 15:16 UTC | newest]
Thread overview: 36+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-02-20 18:28 Re: SSD + RAID Bruce Momjian <bruce@momjian.us>
2010-02-20 23:07 ` Dan Langille <dan@langille.org>
2010-02-20 23:19 ` Bruce Momjian <bruce@momjian.us>
2010-02-21 09:34 ` Scott Carey <scott@richrelevance.com>
2010-02-21 14:10 ` Bruce Momjian <bruce@momjian.us>
2010-02-21 14:54 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-22 05:39 ` Greg Smith <greg@2ndquadrant.com>
2010-02-22 07:10 ` Arjen van der Meijden <acmmailing@tweakers.net>
2010-02-23 01:04 ` Greg Smith <greg@2ndquadrant.com>
2010-02-23 01:11 ` Mark Mielke <mark@mark.mielke.cc>
2010-02-23 01:39 ` Greg Smith <greg@2ndquadrant.com>
2010-02-23 02:21 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 02:22 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-02-23 11:49 ` Pierre C <lists@peufeu.com>
2010-02-23 12:07 ` Nikolas Everett <nik9000@gmail.com>
2010-02-23 18:36 ` Scott Carey <scott@richrelevance.com>
2010-02-22 14:37 ` Bruce Momjian <bruce@momjian.us>
2010-02-22 18:00 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-23 01:14 ` Greg Smith <greg@2ndquadrant.com>
2010-02-23 08:23 ` david@lang.hm
2010-02-23 19:35 ` david@lang.hm
2010-02-23 20:34 ` Aidan Van Dyk <aidan@highrise.ca>
2010-02-23 21:22 ` david@lang.hm
2010-02-23 21:32 ` Mark Mielke <mark@mark.mielke.cc>
2010-02-24 08:32 ` Dave Crooke <dcrooke@gmail.com>
2010-02-27 01:40 ` Bruce Momjian <bruce@momjian.us>
2010-02-27 19:38 ` Greg Smith <greg@2ndquadrant.com>
2010-02-27 20:16 ` Bruce Momjian <bruce@momjian.us>
2010-02-28 04:05 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-02-28 05:06 ` Greg Smith <greg@2ndquadrant.com>
2010-03-02 03:34 ` Bruce Momjian <bruce@momjian.us>
2010-03-02 03:33 ` Bruce Momjian <bruce@momjian.us>
2010-03-02 06:13 ` Greg Smith <greg@2ndquadrant.com>
2010-03-03 15:16 ` Ron Mayer <rm_pg@cheapcomplexdevices.com>
2010-03-02 08:36 ` Pierre C <lists@peufeu.com>
2010-02-22 14:39 ` Bruce Momjian <bruce@momjian.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox