From: Mel Llaguno <mllaguno@coverity.com>
To: Josh Berkus <josh@agliodbs.com>, "pgsql-performance@postgresql.org"
 <pgsql-performance@postgresql.org>
Subject: Re: HFS+ pg_test_fsync performance
Thread-Topic: [PERFORM] HFS+ pg_test_fsync performance
Thread-Index: AQHPWDFwlR15YRY8f0KSgkk6+/NESpsfjViX//+ewgA=
Date: Wed, 23 Apr 2014 18:19:19 +0000
Message-ID: <CF7D5F4F.17BAE%mllaguno@coverity.com>
References: <CF71BE2D.15126%mllaguno@coverity.com>
 <WM!a9b4ed7dac906302c038db25f5e0fe365f6533dbcbf6355ee220f9ad29ac8ea5e508e511293b65b1f860ae24da369fdb!@asav-2.01.com>
 <535800CB.7060609@agliodbs.com>
In-Reply-To: <535800CB.7060609@agliodbs.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.9.131030
received-spf: None (: coverity.com does not designate permitted sender hosts)
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <5B6538DF74FA44409C5A5E51E906F1CB@namprd05.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

Josh,

Thanks for the feedback. Given the prevalence of SSDs/VMs, it would be
useful to start collecting stats/tuning for different operating systems
for things like fsync (and possibly bonnie++/dd). If the community is
interested, I=B9ve got a performance lab that I=B9d be willing to help run
tests on. Having this information would only improve our ability to
support our customers.

M.

On 4/23/14, 12:04 PM, "Josh Berkus" <josh@agliodbs.com> wrote:

>Mel,
>
>> I was given anecdotal information regarding HFS+ performance under OSX
>>as
>> being unsuitable for production PG deployments and that pg_test_fsync
>> could be used to measure the relative speed versus other operating
>>systems
>
>You're welcome to identify your source of anecdotal evidence: me.  And
>it's based on experience and testing, although I'm not allowed to share
>the raw results.  Let's just say that I was part of two different
>projects where we moved from using OSX on Apple hardware do using Linux
>on the *same* hardware ... because of intolerable performance on OSX.
>Switching to Linux more than doubled our real write throughput.
>
>> Compare file sync methods using one 8kB write:
>> (in wal_sync_method preference order, except fdatasync
>> is Linux's default)
>>         open_datasync                    8752.240 ops/sec     114
>>usecs/op
>>         fdatasync                        8556.469 ops/sec     117
>>usecs/op
>>         fsync                            8831.080 ops/sec     113
>>usecs/op
>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=3D=3D
>>         fsync_writethrough                735.362 ops/sec    1360
>>usecs/op
>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=3D=3D
>>         open_sync                        8967.000 ops/sec     112
>>usecs/op
>
>fsync_writethrough is the *only* relevant stat above.  For all of the
>other fsync operations, OSX is "faking it"; returning to the calling
>code without ever flushing to disk.  This would result in data
>corruption in the event of an unexpected system shutdown.
>
>Both OSX and Windows do this, which is why we *have* fsync_writethrough.
> Mind you, I'm a little shocked that performance is still so bad on
>SSDs; I'd attributed HFS's slow fsync mostly to waiting for a full disk
>rotation, but apparently the primary cause is something else.
>
>You'll notice that the speed of fsync_writethrough is 1/4 that of
>comparable speed on Linux.   You can get similar performance on Linux by
>putting your WAL on a ramdisk, but I don't recommend that for production.
>
>But: things get worse.  In addition to the very slow speed on real
>fsyncs, HFS+ has very primitive IO scheduling for multiprocessor
>workloads; the filesystem was designed for single-core machines (in
>1995!) and has no ability to interleave writes from multiple concurrent
>processes.  This results in "stuttering" as the IO system tries to
>service first one write request, then another, and ends up stalling
>both.  If you do, for example, a serious ETL workload with parallelism
>on OSX, you'll see that IO throughput describes a sawtooth from full
>speed to zero, being near-zero about half the time.
>
>So not only are fsyncs slower, real throughput for sustained writes on
>HFS+ are 50% or less of the hardware maximum in any real multi-user
>workload.
>
>In order to test this, you'd need a workload which required loading and
>sorting several tables larger than RAM, at least two in parallel.
>
>In the words of the lead HFS+ developer, Don Brady:  "Since we believed
>it was only a stop gap solution, we just went from 16 to 32 bits. Had we
>known that it would still be in use 15 years later with multi-terabyte
>drives, we probably would have done more design changes!"
>
>HFS+ was written in about 6 months, and is largely unimproved since its
>release in 1995.  Ext2 doesn't perform too well, either; the difference
>is that Linux users have alternative filesystems available.
>
>--=20
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com
>
>
>--=20
>Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance


--=20
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance