pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedNFS, file system cache and shared_buffers
13+ messages / 8 participants
[nested] [flat]
* NFS, file system cache and shared_buffers
@ 2014-05-27 11:06 Albe Laurenz <laurenz.albe@wien.gv.at>
0 siblings, 3 replies; 13+ messages in thread
From: Albe Laurenz @ 2014-05-27 11:06 UTC (permalink / raw)
To: pgsql-performance
I just learned that NFS does not use a file system cache on the client side.
On the other hand, PostgreSQL relies on the file system cache for performance,
because beyond a certain amount of shared_buffers performance will suffer.
Together these things seem to indicate that you cannot get good performance
with a large database over NFS since you can leverage memory speed.
Now I wonder if there are any remedies (CacheFS?) and what experiences
people have made with the performance of large databases over NFS.
Yours,
Laurenz Albe
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 12:26 Heikki Linnakangas <hlinnakangas@vmware.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
2 siblings, 1 reply; 13+ messages in thread
From: Heikki Linnakangas @ 2014-05-27 12:26 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; pgsql-performance
On 05/27/2014 02:06 PM, Albe Laurenz wrote:
> I just learned that NFS does not use a file system cache on the client side.
>
> On the other hand, PostgreSQL relies on the file system cache for performance,
> because beyond a certain amount of shared_buffers performance will suffer.
>
> Together these things seem to indicate that you cannot get good performance
> with a large database over NFS since you can leverage memory speed.
>
> Now I wonder if there are any remedies (CacheFS?) and what experiences
> people have made with the performance of large databases over NFS.
I have no personal experience with NFS, but sounds like a
higher-than-usual shared_buffers value would be good.
- Heikki
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 14:40 Stephen Frost <sfrost@snowman.net>
parent: Heikki Linnakangas <hlinnakangas@vmware.com>
0 siblings, 1 reply; 13+ messages in thread
From: Stephen Frost @ 2014-05-27 14:40 UTC (permalink / raw)
To: Heikki Linnakangas <hlinnakangas@vmware.com>; +Cc: Albe Laurenz <laurenz.albe@wien.gv.at>; pgsql-performance
* Heikki Linnakangas (hlinnakangas@vmware.com) wrote:
> On 05/27/2014 02:06 PM, Albe Laurenz wrote:
> >I just learned that NFS does not use a file system cache on the client side.
> >
> >On the other hand, PostgreSQL relies on the file system cache for performance,
> >because beyond a certain amount of shared_buffers performance will suffer.
> >
> >Together these things seem to indicate that you cannot get good performance
> >with a large database over NFS since you can leverage memory speed.
> >
> >Now I wonder if there are any remedies (CacheFS?) and what experiences
> >people have made with the performance of large databases over NFS.
>
> I have no personal experience with NFS, but sounds like a
> higher-than-usual shared_buffers value would be good.
It would certainly be worthwhile to test it. In the end you would,
hopefully, end up with a situation where you're maximizing RAM usage-
the NFS server is certainly caching in *its* filesystem cache, while on
the PG server you're getting the benefit of shared_buffers without the
drawback of double-buffering (since you couldn't ever use the NFS
server's memory for shared_buffers anyway).
All that said, there has always been a recommendation of caution around
using NFS as a backing store for PG, or any RDBMS..
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 835B) download
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 15:00 Albe Laurenz <laurenz.albe@wien.gv.at>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 13+ messages in thread
From: Albe Laurenz @ 2014-05-27 15:00 UTC (permalink / raw)
To: Stephen Frost *EXTERN* <sfrost@snowman.net>; Heikki Linnakangas <hlinnakangas@vmware.com>; +Cc: pgsql-performance
Stephen Frost wrote:
> All that said, there has always been a recommendation of caution around
> using NFS as a backing store for PG, or any RDBMS..
I know that Oracle recommends it - they even built an NFS client
into their database server to make the most of it.
Yours,
Laurenz Albe
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 15:09 Shaun Thomas <sthomas@optionshouse.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
1 sibling, 1 reply; 13+ messages in thread
From: Shaun Thomas @ 2014-05-27 15:09 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; Stephen Frost *EXTERN* <sfrost@snowman.net>; Heikki Linnakangas <hlinnakangas@vmware.com>; +Cc: pgsql-performance
On 05/27/2014 10:00 AM, Albe Laurenz wrote:
> I know that Oracle recommends it - they even built an NFS client
> into their database server to make the most of it.
That's odd. Every time the subject of NFS comes up, it's almost
immediately shot down with explicit advice to Never Do That(tm). It can
be kinda safe-ish if mounted in sync mode with caching disabled, but I'd
never use it on any of our systems.
We also have this in the Wiki:
http://wiki.postgresql.org/wiki/Shared_Storage
--
Shaun Thomas
OptionsHouse, LLC | 141 W. Jackson Blvd. | Suite 800 | Chicago IL, 60604
312-676-8870
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 15:18 David Boreham <david_list@boreham.org>
parent: Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 0 replies; 13+ messages in thread
From: David Boreham @ 2014-05-27 15:18 UTC (permalink / raw)
To: pgsql-performance
On 5/27/2014 9:09 AM, Shaun Thomas wrote:
> On 05/27/2014 10:00 AM, Albe Laurenz wrote:
>
>> I know that Oracle recommends it - they even built an NFS client
>> into their database server to make the most of it.
>
> That's odd. Every time the subject of NFS comes up, it's almost
> immediately shot down with explicit advice to Never Do That(tm). It
> can be kinda safe-ish if mounted in sync mode with caching disabled,
> but I'd never use it on any of our systems.
It has been a long time since I was in the weeds of this issue, but the
crux is that it was (still is?) hard to be sure that the filesystem's
behavior was exactly as expected. My recollection of the Oracle story
was that they had to verify the end-to-end behavior, and essentially
certify its correctness to guarantee database acid. So you needed to be
running a very specific version of the NFS code, configured in a very
specific way. This isn't entirely inconsistent with the reference above
that they "built an NFS client". That's something you might need to do
in order to be sure it behaves in the way you expect. Possibly the NFS
implementations deployed today are more consistent and correct than was
the case in the past. I wouldn't use a network filesystem for any kind
of database storage myself though.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 15:32 Jeff Janes <jeff.janes@gmail.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
1 sibling, 1 reply; 13+ messages in thread
From: Jeff Janes @ 2014-05-27 15:32 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; +Cc: Stephen Frost *EXTERN* <sfrost@snowman.net>; Heikki Linnakangas <hlinnakangas@vmware.com>; pgsql-performance
On Tue, May 27, 2014 at 8:00 AM, Albe Laurenz <laurenz.albe@wien.gv.at>wrote:
> Stephen Frost wrote:
> > All that said, there has always been a recommendation of caution around
> > using NFS as a backing store for PG, or any RDBMS..
>
> I know that Oracle recommends it - they even built an NFS client
> into their database server to make the most of it.
>
Last I heard (which has been a while), Oracle supported specific brand
named implementations of NFS, and warned against any others on a data
integrity basis.
Why would they implement their own client? Did they have to do something
special in their client to make it safe?
Cheers,
Jeff
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 16:54 Jeff Janes <jeff.janes@gmail.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
2 siblings, 0 replies; 13+ messages in thread
From: Jeff Janes @ 2014-05-27 16:54 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; +Cc: pgsql-performance
On Tue, May 27, 2014 at 4:06 AM, Albe Laurenz <laurenz.albe@wien.gv.at>wrote:
> I just learned that NFS does not use a file system cache on the client
> side.
>
My experience suggested that it did something a little weirder than that.
It would cache read data as long as it was clean, but once the data was
dirtied and written back, it would drop it from the cache. But it probably
depends on lot of variables and details I don't recall anymore.
>
> On the other hand, PostgreSQL relies on the file system cache for
> performance,
> because beyond a certain amount of shared_buffers performance will suffer.
>
Some people have some problems sometimes, and they are not readily
reproducible (at least not in a publicly disclosable way, that I know of).
Other people use large shared_buffers and have no problems at all, or none
that are fixed by lowering shared_buffers.
We should not elevate a rumor to a law.
>
> Together these things seem to indicate that you cannot get good performance
> with a large database over NFS since you can leverage memory speed.
>
> Now I wonder if there are any remedies (CacheFS?) and what experiences
> people have made with the performance of large databases over NFS.
>
I've only used it in cases where I didn't consider durability important,
and even then didn't find it worth pursuing due to the performance. But I
was piggybacking on existing resource, I didn't have an impressive NFS
server tuned specifically for this usage, so my experience probably doesn't
mean much performance wise.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-27 18:20 John Melesky <john.melesky@rentrakmail.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
2 siblings, 1 reply; 13+ messages in thread
From: John Melesky @ 2014-05-27 18:20 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; +Cc: pgsql-performance
On Tue, May 27, 2014 at 4:06 AM, Albe Laurenz <laurenz.albe@wien.gv.at>wrote:
> I just learned that NFS does not use a file system cache on the client
> side.
>
That's ... incorrect. NFS is cache-capable. NFSv3 (I think? It may have
been v2) started sending metadata on file operations that was intended to
allow for client-side caches. NFSv4 added all sorts of stateful behavior
which allows for much more aggressive caching.
Where did you read that you could not use caching with NFS?
--
John Melesky | Sr Database Administrator
503.284.7581 x204 | john.melesky@rentrak.com <john.melesky@rentrakmail.com>
RENTRAK | www.rentrak.com | NASDAQ: RENT
Notice: This message is confidential and is intended only for the
recipient(s) named above. If you have received this message in error,
or are not the named recipient(s), please immediately notify the
sender and delete this message.
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-28 07:50 Albe Laurenz <laurenz.albe@wien.gv.at>
parent: John Melesky <john.melesky@rentrakmail.com>
0 siblings, 0 replies; 13+ messages in thread
From: Albe Laurenz @ 2014-05-28 07:50 UTC (permalink / raw)
To: John Melesky *EXTERN* <john.melesky@rentrakmail.com>; +Cc: pgsql-performance
John Melesky wrote:
>> I just learned that NFS does not use a file system cache on the client side.
>
> That's ... incorrect. NFS is cache-capable. NFSv3 (I think? It may have been v2) started sending
> metadata on file operations that was intended to allow for client-side caches. NFSv4 added all sorts
> of stateful behavior which allows for much more aggressive caching.
What do you mean by "allows"? Does it cache files in memory or not?
Do you need additional software? Special configuration?
> Where did you read that you could not use caching with NFS?
I have it by hearsay from somebody who seemed knowledgable, and the
existence of CacheFS seemed to indicate it was true.
Yours,
Laurenz Albe
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-28 08:41 Albe Laurenz <laurenz.albe@wien.gv.at>
parent: Jeff Janes <jeff.janes@gmail.com>
0 siblings, 2 replies; 13+ messages in thread
From: Albe Laurenz @ 2014-05-28 08:41 UTC (permalink / raw)
To: Jeff Janes *EXTERN* <jeff.janes@gmail.com>; +Cc: Stephen Frost *EXTERN* <sfrost@snowman.net>; Heikki Linnakangas <hlinnakangas@vmware.com>; pgsql-performance
Jeff Janes wrote:
>>> All that said, there has always been a recommendation of caution around
>>> using NFS as a backing store for PG, or any RDBMS..
>>
>> I know that Oracle recommends it - they even built an NFS client
>> into their database server to make the most of it.
>
> Last I heard (which has been a while), Oracle supported specific brand named implementations of NFS,
> and warned against any others on a data integrity basis.
I couldn't find any detailed information, but it seems that only certain
NFS devices are supported.
> Why would they implement their own client? Did they have to do something special in their client to
> make it safe?
I think it is mostly a performance issue. Each backend mounts its own copy
of the data files it needs.
Yours,
Laurenz Albe
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-29 00:36 Josh Berkus <josh@agliodbs.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
1 sibling, 0 replies; 13+ messages in thread
From: Josh Berkus @ 2014-05-29 00:36 UTC (permalink / raw)
To: pgsql-performance
>> Why would they implement their own client? Did they have to do something special in their client to
>> make it safe?
>
> I think it is mostly a performance issue. Each backend mounts its own copy
> of the data files it needs.
I personally would never put PostgreSQL on an NFS share on Linux.
Unless things have changed radically in the last couple years, Linux's
NFS code is flaky and unreliable, including flat-out lying about whether
stuff has been sent and received or not. This is why NetApp's NFS
servers came with their own, proprietary, Linux kernel module.
NFS on Solaris/Illumos is a different story. Not sure about FreeBSD.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: NFS, file system cache and shared_buffers
@ 2014-05-30 07:12 Albe Laurenz <laurenz.albe@wien.gv.at>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
1 sibling, 0 replies; 13+ messages in thread
From: Albe Laurenz @ 2014-05-30 07:12 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; Jeff Janes *EXTERN* <jeff.janes@gmail.com>; +Cc: Stephen Frost *EXTERN* <sfrost@snowman.net>; Heikki Linnakangas <hlinnakangas@vmware.com>; pgsql-performance
I wrote:
>Jeff Janes wrote:
>>>> All that said, there has always been a recommendation of caution around
>>>> using NFS as a backing store for PG, or any RDBMS..
>>>
>>> I know that Oracle recommends it - they even built an NFS client
>>> into their database server to make the most of it.
>>
>> Last I heard (which has been a while), Oracle supported specific brand named implementations of NFS,
>> and warned against any others on a data integrity basis.
>
> I couldn't find any detailed information, but it seems that only certain
> NFS devices are supported.
For the record: Oracle support told me that all NFS is supported on Linux,
regardless of the device. "Supported" does not mean "recommended", of course.
Yours,
Laurenz Albe
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 13+ messages in thread
end of thread, other threads:[~2014-05-30 07:12 UTC | newest]
Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2014-05-27 11:06 NFS, file system cache and shared_buffers Albe Laurenz <laurenz.albe@wien.gv.at>
2014-05-27 12:26 ` Heikki Linnakangas <hlinnakangas@vmware.com>
2014-05-27 14:40 ` Stephen Frost <sfrost@snowman.net>
2014-05-27 15:00 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2014-05-27 15:09 ` Shaun Thomas <sthomas@optionshouse.com>
2014-05-27 15:18 ` David Boreham <david_list@boreham.org>
2014-05-27 15:32 ` Jeff Janes <jeff.janes@gmail.com>
2014-05-28 08:41 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2014-05-29 00:36 ` Josh Berkus <josh@agliodbs.com>
2014-05-30 07:12 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2014-05-27 16:54 ` Jeff Janes <jeff.janes@gmail.com>
2014-05-27 18:20 ` John Melesky <john.melesky@rentrakmail.com>
2014-05-28 07:50 ` Albe Laurenz <laurenz.albe@wien.gv.at>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox