From: Mel Llaguno <mllaguno@coverity.com>
To: Josh Berkus <josh@agliodbs.com>
To: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: Re: HFS+ pg_test_fsync performance
Date: Wed, 23 Apr 2014 18:19:19 +0000
Message-ID: <CF7D5F4F.17BAE%mllaguno@coverity.com> (raw)
In-Reply-To: <535800CB.7060609@agliodbs.com>
References: <CF71BE2D.15126%mllaguno@coverity.com>
<WM!a9b4ed7dac906302c038db25f5e0fe365f6533dbcbf6355ee220f9ad29ac8ea5e508e511293b65b1f860ae24da369fdb!@asav-2.01.com>
<535800CB.7060609@agliodbs.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
Josh,
Thanks for the feedback. Given the prevalence of SSDs/VMs, it would be
useful to start collecting stats/tuning for different operating systems
for things like fsync (and possibly bonnie++/dd). If the community is
interested, I¹ve got a performance lab that I¹d be willing to help run
tests on. Having this information would only improve our ability to
support our customers.
M.
On 4/23/14, 12:04 PM, "Josh Berkus" <josh@agliodbs.com> wrote:
>Mel,
>
>> I was given anecdotal information regarding HFS+ performance under OSX
>>as
>> being unsuitable for production PG deployments and that pg_test_fsync
>> could be used to measure the relative speed versus other operating
>>systems
>
>You're welcome to identify your source of anecdotal evidence: me. And
>it's based on experience and testing, although I'm not allowed to share
>the raw results. Let's just say that I was part of two different
>projects where we moved from using OSX on Apple hardware do using Linux
>on the *same* hardware ... because of intolerable performance on OSX.
>Switching to Linux more than doubled our real write throughput.
>
>> Compare file sync methods using one 8kB write:
>> (in wal_sync_method preference order, except fdatasync
>> is Linux's default)
>> open_datasync 8752.240 ops/sec 114
>>usecs/op
>> fdatasync 8556.469 ops/sec 117
>>usecs/op
>> fsync 8831.080 ops/sec 113
>>usecs/op
>==========================================================================
>==
>> fsync_writethrough 735.362 ops/sec 1360
>>usecs/op
>==========================================================================
>==
>> open_sync 8967.000 ops/sec 112
>>usecs/op
>
>fsync_writethrough is the *only* relevant stat above. For all of the
>other fsync operations, OSX is "faking it"; returning to the calling
>code without ever flushing to disk. This would result in data
>corruption in the event of an unexpected system shutdown.
>
>Both OSX and Windows do this, which is why we *have* fsync_writethrough.
> Mind you, I'm a little shocked that performance is still so bad on
>SSDs; I'd attributed HFS's slow fsync mostly to waiting for a full disk
>rotation, but apparently the primary cause is something else.
>
>You'll notice that the speed of fsync_writethrough is 1/4 that of
>comparable speed on Linux. You can get similar performance on Linux by
>putting your WAL on a ramdisk, but I don't recommend that for production.
>
>But: things get worse. In addition to the very slow speed on real
>fsyncs, HFS+ has very primitive IO scheduling for multiprocessor
>workloads; the filesystem was designed for single-core machines (in
>1995!) and has no ability to interleave writes from multiple concurrent
>processes. This results in "stuttering" as the IO system tries to
>service first one write request, then another, and ends up stalling
>both. If you do, for example, a serious ETL workload with parallelism
>on OSX, you'll see that IO throughput describes a sawtooth from full
>speed to zero, being near-zero about half the time.
>
>So not only are fsyncs slower, real throughput for sustained writes on
>HFS+ are 50% or less of the hardware maximum in any real multi-user
>workload.
>
>In order to test this, you'd need a workload which required loading and
>sorting several tables larger than RAM, at least two in parallel.
>
>In the words of the lead HFS+ developer, Don Brady: "Since we believed
>it was only a stop gap solution, we just went from 16 to 32 bits. Had we
>known that it would still be in use 15 years later with multi-terabyte
>drives, we probably would have done more design changes!"
>
>HFS+ was written in about 6 months, and is largely unimproved since its
>release in 1995. Ext2 doesn't perform too well, either; the difference
>is that Linux users have alternative filesystems available.
>
>--
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com
>
>
>--
>Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: mllaguno@coverity.com, josh@agliodbs.com
Subject: Re: HFS+ pg_test_fsync performance
In-Reply-To: <CF7D5F4F.17BAE%mllaguno@coverity.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox