From: Bruce Momjian <bruce@momjian.us>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: PostgreSQL-documentation <pgsql-docs@postgresql.org>
Subject: Re: [HACKERS] PG on NFS may be just a bad idea
Date: Sun, 4 Nov 2007 16:51:38 -0500 (EST)
Message-ID: <200711042151.lA4LpcP29113@momjian.us> (raw)
In-Reply-To: <25517.1191038302@sss.pgh.pa.us>
Based on this analysis, I have added an NFS section to the tablespaces
portion of the documentation, and linked to it from 'Creating a database
cluster'. Patch attached.
---------------------------------------------------------------------------
Tom Lane wrote:
> I spent a bit of time tonight poking at the issue reported here:
> http://archives.postgresql.org/pgsql-novice/2007-08/msg00123.php
>
> It turns out to be quite easy to reproduce, at least for me: start CVS
> HEAD on an NFS-mounted $PGDATA directory, and run the contrib regression
> tests ("make installcheck" in contrib/). I see more than half of the
> DROP DATABASE commands complaining in exactly the way Miya describes.
> This failure rate might be an artifact of the particular environment
> (I tested NFS client = Fedora Core 6, server = HPUX 10.20 on a much
> slower machine) but the problem is clearly real.
>
> In the earlier thread I cited suggestions that this behavior comes from
> client programs holding files open longer than they should. However,
> strace'ing this behavior shows no evidence at all that that is happening
> in Postgres. I have an strace that shows conclusively that the bgwriter
> never opened any file in the target database at all, and all earlier
> backends exited before the one doing the DROP DATABASE began its dirty
> work, and yet:
>
> [pid 19211] 22:50:30.517077 rmdir("base/18193") = -1 ENOTEMPTY (Directory not empty)
> [pid 19211] 22:50:30.517863 write(2, "WARNING: could not remove file "..., 79WARNING: could not remove file or directory "base/18193": Directory not empty
> ) = 79
> [pid 19211] 22:50:30.517974 sendto(7, "N\0\0\0rSWARNING\0C01000\0Mcould not "..., 115, 0, NULL, 0) = 115
>
> After some googling I think that the damage may actually be getting done
> at the kernel level. According to
> http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
> it is fairly common for NFS clients to cache writes, meaning that the
> kernel itself may be holding an old write and not sending it to the NFS
> server until after the file deletion command has been sent.
>
> (I don't have the network-fu needed to prove that this is happening by
> sniffing the network traffic; anyone want to try?)
>
> If this is what's happening I'd claim it is a kernel bug, but seeing
> that I see it on FC6 and Miya sees it on Solaris 10, it would be a bug
> widespread enough that we'd not be likely to get it killed off soon.
>
> Maybe we need to actively discourage people from running Postgres
> against NFS-mounted data directories. Shane Kerr's paper cited above
> mentions some other rather scary properties, including O_EXCL file
> creation not really working properly.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Attachments:/rtmp/diff (text/x-diff, 2.4K) download
Index: doc/src/sgml/manage-ag.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/manage-ag.sgml,v
retrieving revision 2.55
retrieving revision 2.56
diff -c -r2.55 -r2.56
*** doc/src/sgml/manage-ag.sgml 4 Nov 2007 19:43:33 -0000 2.55
--- doc/src/sgml/manage-ag.sgml 4 Nov 2007 21:40:02 -0000 2.56
***************
*** 495,499 ****
--- 495,525 ----
the old tablespace locations.)
</para>
+ <sect2 id="manage-ag-tablespaces-nfs">+ <title>Network File Systems</title>+ + <indexterm zone="manage-ag-tablespaces-nfs">+ <primary>Network File Systems</primary>+ </indexterm>+ <indexterm><primary><acronym>NFS</></><see>Network File Systems</></>+ <indexterm><primary>Network Attached Storage (<acronym>NAS</>)</><see>Network File Systems</></>+ + <para>+ Many installations create tablespace on network file systems.+ Sometimes this is done directly via <acronym>NFS</>, or by using a+ Network Attached Storage (<acronym>NAS</>) device that uses+ <acronym>NFS</> internally. <productname>PostgreSQL</> does nothing+ special for <acronym>NFS</> file systems, meaning it assumes+ <acronym>NFS</> behaves exactly like locally-connected drives. If+ client and server <acronym>NFS</> implementations have non-standard+ semantics, this can cause reliability problems (see <ulink+ url="http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html"></ulink>).+ Specifically, delayed (asynchonous) writes to the <acronym>NFS</>+ server can cause reliability problems; if possible, mount+ <acronym>NFS</> file systems synchonously to avoid this.+ </para>+ + </sect2>+
</sect1>
</chapter>
Index: doc/src/sgml/runtime.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/runtime.sgml,v
retrieving revision 1.382
retrieving revision 1.383
diff -c -r1.382 -r1.383
*** doc/src/sgml/runtime.sgml 1 Nov 2007 19:06:01 -0000 1.382
--- doc/src/sgml/runtime.sgml 4 Nov 2007 21:48:03 -0000 1.383
***************
*** 159,164 ****
--- 159,170 ----
for the database cluster. Normally this should be chosen to match the
locale setting. For details see <xref linkend="multibyte">.
</para>
+ + <para>+ If using non-local (network) file systems, see <xref+ linkend="manage-ag-tablespaces-nfs">.+ </para>+
</sect1>
<sect1 id="server-start">
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-docs@postgresql.org
Cc: bruce@momjian.us, tgl@sss.pgh.pa.us
Subject: Re: [HACKERS] PG on NFS may be just a bad idea
In-Reply-To: <200711042151.lA4LpcP29113@momjian.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox