Subject: Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)

From: Laurenz Albe <[email-masked]>
Date: Mon, 14 Jul 2025 14:20:14 +0200

On Mon, 2025-07-14 at 11:19 +0530, Amol Inamdar wrote:
> I'm currently running PostgreSQL version 16.6 inside a Docker container
> (base image: UBI 9), using Docker Compose. The PostgreSQL data directory
> is mounted from an NFS volume hosted on a z/OS NFS server.
> 
> The environment has a few constraints:
> 
> - It’s a highly secure and access-controlled setup.
> - Due to platform restrictions on z/OS, the mounted NFS directory cannot
>   be owned by the PostgreSQL user (e.g., `postgres`) inside the container.
> - As a result, PostgreSQL fails to start because of the directory
>   ownership validation check.

It is not a good idea to have a mount point be the data directory.
The proper solution is to create the data directory inside the
mount point.  That way, the permissions of the data directory don't
have to be the same as the permissions of the mount point.

Yours,
Laurenz Albe

---

From: Amol Inamdar <[email-masked]>
Date: Mon, 14 Jul 2025 17:59:12 +0530

Hi Laurenz

Thanks for the reply,

If I am not mistaken, below is my understanding of your suggestion.

Suppose that My mount point on the NFS server is say /nfs-mount/postgres/
and you are suggesting to have a data directory as say
/nfs-mount/postgres/db or something like that ?
and assign this value to the PGDATA ?

If that is the case, then when and who should be creating the directory DB
?

Please correct me if I am wrong about the understanding.

Thanks,
Amol

-- 
-regards
Amol

---

From: Laurenz Albe <[email-masked]>
Date: Mon, 14 Jul 2025 21:44:19 +0900

On Mon, 2025-07-14 at 17:59 +0530, Amol Inamdar wrote:
> and you are suggesting to have a data directory as say /nfs-mount/postgres/db or something like that ? 
> If that is the case, then when and who should be creating the directory DB ? 
> 

You understood me perfectly well.

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.

Alternatively, the root user could create the data directory with the
correct ownership and permissions prior to running "initdb".

Yours,
Laurenz Albe

---

From: Tom Lane <[email-masked]>
Date: Mon, 14 Jul 2025 10:07:20 -0400

> It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory.  There is a sobering tale
in this old thread:

https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected.  But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk.  To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Do not bypass that ownership/permissions check.  It is there
for very good reasons.

			regards, tom lane

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 14 Jul 2025 20:20:16 +0200

> different ownership/permissions) just below the mount.

Be careful: There are two different directorys involved in a mount
point. The one in the parent filesystem and the one in the mounted file
system.

E.g.:

# mkdir /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 254,0	Inode: 317         Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

So here we have inode 317 on device 254,0 and unsurprisingly it belongs
to root and is only writable by root.

Now let's mount a filesystem:

# mount /dev/vgroot/demo /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 254,13	Inode: 2           Links: 3
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

This is now inode 2 on device 254,13 (i.e. the root directory on
/dev/vgroot/demo). Since I just created that it also belongs to root.
Let's change that:

# chown postgres: /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 254,13	Inode: 2           Links: 3
Access: (0755/drwxr-xr-x)  Uid: (  108/postgres)   Gid: (  114/postgres)

Ok, now it belongs to postgres and postgres could create files,
subdirectories, etc.

But when I unmount it:

# umount /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 254,0	Inode: 317         Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

Lo and behold: The original Inode is back and unchanged.

The user postgres couldn't write to that directory if it tried to.

That said, there are of course some remaining failure modes:

1) An admin might notice the wrong permissions and "fix" them without
   investigating the cause
2) Some automated procedure (running as root) might "fix" the
   permissions at startup (like the initdb in the example) or during an
   upgrade.
3) use your imagination ;-).

        hjp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Tom Lane <[email-masked]>
Date: Mon, 14 Jul 2025 14:30:45 -0400

"Peter J. Holzer" <[email-masked]> writes:
>> underlying mount-point directory.

> point. The one in the parent filesystem and the one in the mounted file

True, and the safety requirement really is only that the parent
filesystem's mount-point directory not be writable by us.
But normal practice is that both directories are root-owned,
or at least owned by highly privileged users.

(I have a vague idea that there are system-level security hazards,
not specific to Postgres, if mount-point directories are publicly
writable.  Don't feel like researching that though.)

			regards, tom lane

---

From: Laurenz Albe <[email-masked]>
Date: Tue, 15 Jul 2025 08:02:06 +0200

> writable.  Don't feel like researching that though.)

Well, if you are using an ext? file system, there is a lost+found
directory where fsck places links to orphaned inodes.
If the PostgreSQL user owns the mount point and wants to use
"initdb" to create a data directory in it, the program will fail
and complain that the directory is not empty.  The danger is great
that the user removes the lost+found directory to solve the problem.

True, one could re-create it with "mklost+found", but if a DBA
is uneducated enough to remove the directory in the first place,
the risk is high that he wouldn't think of creating it again,
which is a problem if the file system ever becomes corrupted.

All this doesn't apply to NFS, but it is yet another reason
(apart from the safety of a subdirectory that doesn't exist
on the file system underlying the mount point) why we should
continue to recommend that the data directory be not a mount point.

Yours,
Laurenz Albe

---

From: Amol Inamdar <[email-masked]>
Date: Tue, 15 Jul 2025 17:06:29 +0530

Thanks Tom and Laurenz for the explanation.
Let me try out a few things and get back to you if needed.

Thanks,
Amol

-- 
-regards
Amol

---

From: Amol Inamdar <[email-masked]>
Date: Wed, 16 Jul 2025 18:54:50 +0530

Hi All,

I would like to rephrase the question a little bit, below is how our setup
going to be

   1. NFS mount point is for /nfs-mount/postgres (and permissions locked
   down so that Postgres cannot create directories in here)
   2. Postgres data directory is /nfs-mount/postgres/db
   3.

   With secured NFS + AT-TLS setup Postgres will be able to write to data
   directory but not parent dir, however the file ownership information
   Postgres sees from the stat() call will not match the Postgres user in the
   container (even though the AT-TLS strict access control will ensure only
   the Posgres user can read/write to this directory)

Considering the above scenario/setup, what is the danger of removing the
ownership check in miscinit.c checkDataDir() function ?

-- 
-regards
Amol

---

From: Ron Johnson <[email-masked]>
Date: Wed, 16 Jul 2025 10:17:43 -0400

Quoting Tom's earlier email:
"(But I too *would not use Postgres-over-NFS for any critical data*.
Too many moving parts.  It's tough enough to ensure crash safety
with local storage.)"

You're going through a lot of security effort to implement a Worst Practice.

-- 
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

---

From: Greg Sabino Mullane <[email-masked]>
Date: Wed, 16 Jul 2025 20:41:44 -0400

On Wed, Jul 16, 2025 at 9:25 AM Amol Inamdar <[email-masked]> wrote:

> This thread is fascinating. It's like combining two of the most annoying
technologies in the world, NFS and SELinux, into something worse than
either of them.

Many people use Docker, and NFS, and Postgres all the time. Stop trying to
push on a string.  Conform your process to Postgres' fairly minimal and
sane requirements, rather than the other way around.

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support