From: Karl Denninger <karl@denninger.net>
To: pgsql-performance@postgresql.org
Subject: Re: New server: SSD/RAID recommendations?
Date: Tue, 07 Jul 2015 07:39:01 -0500
Message-ID: <559BC865.9080804@denninger.net> (raw)
In-Reply-To: <B9C4FEE4-1527-4CB0-B941-CF6F2CF38D4E@skogoglandskap.no>
References: <CAFwQ8rfwyBMD7AiL659qtG+UhsOUZthLSBYF7ODTn6TKxJoTPg@mail.gmail.com>
<VisenaEmail.44.70ed435cef1739b1.14e4c07dd3d@tc7-visena>
<31701C89A44B714FB8D242F806E7286C5AED32FA@NTXBOIMBX01.micron.com>
<559AB332.50304@pinpointresearch.com>
<52A08B87-2A8A-43F2-92CE-2514F5AD141B@skogoglandskap.no>
<1253229687.3798367.1436264898828.JavaMail.zimbra@desy.de>
<985C5CEF-2E55-46AD-A29F-90D48D057A3F@skogoglandskap.no>
<1052663055.3807152.1436266613895.JavaMail.zimbra@desy.de>
<559BB7D8.6090701@denninger.net>
<B9C4FEE4-1527-4CB0-B941-CF6F2CF38D4E@skogoglandskap.no>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
On 7/7/2015 06:52, Graeme B. Bell wrote:
> Hi Karl,
>
> Great post, thanks.
>
> Though I don't think it's against conventional wisdom to aggregate writes into larger blocks rather than rely on 4k performance on ssds :-)
>
> 128kb blocks + compression certainly makes sense. But it might make less sense I suppose if you had some incredibly high rate of churn in your rows.
> But for the work we do here, we could use 16MB blocks for all the difference it would make. (Tip to others: don't do that. 128kb block performance is already enough out the IO bus to most ssds)
>
> Do you have your WAL log on a compressed zfs fs?
>
> Graeme Bell
Yes.
Data goes on one mirrored set of vdevs, pg_xlog goes on a second,
separate pool. WAL goes on a third pool on RaidZ2. WAL typically goes
on rotating storage since I use it (and a basebackup) as disaster
recovery (and in hot spare apps the source for the syncing hot standbys)
and that's nearly a big-block-write-only data stream. Rotating media is
fine for that in most applications. I take a new basebackup on
reasonable intervals and rotate the WAL logs to keep that from growing
without boundary.
I use LSI host adapters for the drives themselves (no hardware RAID);
I'm currently running on FreeBSD 10.1. Be aware that ZFS on FreeBSD has
some fairly nasty issues that I developed (and publish) a patch for;
without it some workloads can result in very undesirable behavior where
working set gets paged out in favor of ZFS ARC; if that happens your
performance will go straight into the toilet.
Back before FreeBSD 9 when ZFS was simply not stable enough for me I
used ARECA hardware RAID adapters and rotating media with BBUs and large
cache memory installed on them with UFS filesystems. Hardware adapters
are, however, a net lose in a ZFS environment even when they nominally
work well (and they frequently interact very badly with ZFS during
certain operations making them just flat-out unsuitable.) All-in I far
prefer ZFS on a host adapter to UFS on a RAID adapter both from a data
integrity and performance standpoint.
My SSD drives of choice are all Intel; for lower-end requirements the
730s work very well; the S3500 is next and if your write volume is high
enough the S3700 has much greater endurance (but at a correspondingly
higher price.) All three are properly power-fail protected. All three
are much, much faster than rotating storage. If you can saturate the
SATA channels and need still more I/O throughput NVMe drives are the
next quantum up in performance; I'm not there with our application at
the present time.
Incidentally while there are people who have questioned the 730 series
power loss protection I've tested it with plug-pulls and in addition it
watchdogs its internal power loss capacitors -- from the smartctl -a
display of one of them on an in-service machine here:
175 Power_Loss_Cap_Test 0x0033 100 100 010 Pre-fail
Always - 643 (4 6868)
--
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
Attachments:
smime.p7s (application/pkcs7-signature, 2.9K) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: karl@denninger.net
Subject: Re: New server: SSD/RAID recommendations?
In-Reply-To: <559BC865.9080804@denninger.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox