public inbox for [email protected]
help / color / mirror / Atom feedRe: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
11+ messages / 6 participants
[nested] [flat]
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-14 12:20 Laurenz Albe <[email protected]>
0 siblings, 2 replies; 11+ messages in thread
From: Laurenz Albe @ 2025-07-14 12:20 UTC (permalink / raw)
To: Amol Inamdar <[email protected]>; [email protected]
On Mon, 2025-07-14 at 11:19 +0530, Amol Inamdar wrote:
> I'm currently running PostgreSQL version 16.6 inside a Docker container
> (base image: UBI 9), using Docker Compose. The PostgreSQL data directory
> is mounted from an NFS volume hosted on a z/OS NFS server.
>
> The environment has a few constraints:
>
> - It’s a highly secure and access-controlled setup.
> - Due to platform restrictions on z/OS, the mounted NFS directory cannot
> be owned by the PostgreSQL user (e.g., `postgres`) inside the container.
> - As a result, PostgreSQL fails to start because of the directory
> ownership validation check.
It is not a good idea to have a mount point be the data directory.
The proper solution is to create the data directory inside the
mount point. That way, the permissions of the data directory don't
have to be the same as the permissions of the mount point.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-14 12:29 Amol Inamdar <[email protected]>
parent: Laurenz Albe <[email protected]>
1 sibling, 1 reply; 11+ messages in thread
From: Amol Inamdar @ 2025-07-14 12:29 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: [email protected]
Hi Laurenz
Thanks for the reply,
If I am not mistaken, below is my understanding of your suggestion.
Suppose that My mount point on the NFS server is say /nfs-mount/postgres/
and you are suggesting to have a data directory as say
/nfs-mount/postgres/db or something like that ?
and assign this value to the PGDATA ?
If that is the case, then when and who should be creating the directory DB
?
Please correct me if I am wrong about the understanding.
Thanks,
Amol
On Mon, Jul 14, 2025 at 5:50 PM Laurenz Albe <[email protected]>
wrote:
> On Mon, 2025-07-14 at 11:19 +0530, Amol Inamdar wrote:
> > I'm currently running PostgreSQL version 16.6 inside a Docker container
> > (base image: UBI 9), using Docker Compose. The PostgreSQL data directory
> > is mounted from an NFS volume hosted on a z/OS NFS server.
> >
> > The environment has a few constraints:
> >
> > - It’s a highly secure and access-controlled setup.
> > - Due to platform restrictions on z/OS, the mounted NFS directory cannot
> > be owned by the PostgreSQL user (e.g., `postgres`) inside the
> container.
> > - As a result, PostgreSQL fails to start because of the directory
> > ownership validation check.
>
> It is not a good idea to have a mount point be the data directory.
> The proper solution is to create the data directory inside the
> mount point. That way, the permissions of the data directory don't
> have to be the same as the permissions of the mount point.
>
> Yours,
> Laurenz Albe
>
--
-regards
Amol
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-14 12:44 Laurenz Albe <[email protected]>
parent: Amol Inamdar <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: Laurenz Albe @ 2025-07-14 12:44 UTC (permalink / raw)
To: Amol Inamdar <[email protected]>; +Cc: [email protected]
On Mon, 2025-07-14 at 17:59 +0530, Amol Inamdar wrote:
> If I am not mistaken, below is my understanding of your suggestion.
>
> Suppose that My mount point on the NFS server is say /nfs-mount/postgres/
> and you are suggesting to have a data directory as say /nfs-mount/postgres/db or something like that ?
> and assign this value to the PGDATA ?
>
> If that is the case, then when and who should be creating the directory DB ?
>
> Please correct me if I am wrong about the understanding.
You understood me perfectly well.
The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.
Alternatively, the root user could create the data directory with the
correct ownership and permissions prior to running "initdb".
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-14 14:07 Tom Lane <[email protected]>
parent: Laurenz Albe <[email protected]>
1 sibling, 2 replies; 11+ messages in thread
From: Tom Lane @ 2025-07-14 14:07 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: Amol Inamdar <[email protected]>; [email protected]
Laurenz Albe <[email protected]> writes:
> It is not a good idea to have a mount point be the data directory.
^^^ This. ^^^
That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory. There is a sobering tale
in this old thread:
https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com
Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected. But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk. To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.
Do not bypass that ownership/permissions check. It is there
for very good reasons.
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-14 18:20 Peter J. Holzer <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 11+ messages in thread
From: Peter J. Holzer @ 2025-07-14 18:20 UTC (permalink / raw)
To: [email protected]
On 2025-07-14 10:07:20 -0400, Tom Lane wrote:
> Laurenz Albe <[email protected]> writes:
> > It is not a good idea to have a mount point be the data directory.
>
> ^^^ This. ^^^
>
> That is primarily for safety reasons: if for some reason the
> filesystem gets dismounted, or hasn't come on-line yet during
> a reboot, you do not want Postgres to be able to write on the
> underlying mount-point directory. There is a sobering tale
> in this old thread:
>
> https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com
>
> Now it didn't help any that they were using a start script that
> would automatically run initdb if it didn't see a data directory
> where expected. But even without that, you are in for a world of
> hurt if the mount drops while the server is running and the server
> has any ability to write on the underlying storage; it will think
> whatever it was able to write is safely down on disk. To prevent
> that, the server must not have write permissions on the mount
> point, which dictates making a separate data directory (with
> different ownership/permissions) just below the mount.
Be careful: There are two different directorys involved in a mount
point. The one in the parent filesystem and the one in the mounted file
system.
E.g.:
# mkdir /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,0 Inode: 317 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
So here we have inode 317 on device 254,0 and unsurprisingly it belongs
to root and is only writable by root.
Now let's mount a filesystem:
# mount /dev/vgroot/demo /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,13 Inode: 2 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
This is now inode 2 on device 254,13 (i.e. the root directory on
/dev/vgroot/demo). Since I just created that it also belongs to root.
Let's change that:
# chown postgres: /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,13 Inode: 2 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 108/postgres) Gid: ( 114/postgres)
Ok, now it belongs to postgres and postgres could create files,
subdirectories, etc.
But when I unmount it:
# umount /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,0 Inode: 317 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Lo and behold: The original Inode is back and unchanged.
The user postgres couldn't write to that directory if it tried to.
That said, there are of course some remaining failure modes:
1) An admin might notice the wrong permissions and "fix" them without
investigating the cause
2) Some automated procedure (running as root) might "fix" the
permissions at startup (like the initdb in the example) or during an
upgrade.
3) use your imagination ;-).
hjp
--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | [email protected] | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
Attachments:
[application/pgp-signature] signature.asc (833B, 2-signature.asc)
download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-14 18:30 Tom Lane <[email protected]>
parent: Peter J. Holzer <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Tom Lane @ 2025-07-14 18:30 UTC (permalink / raw)
To: Peter J. Holzer <[email protected]>; +Cc: [email protected]
"Peter J. Holzer" <[email protected]> writes:
> On 2025-07-14 10:07:20 -0400, Tom Lane wrote:
>> That is primarily for safety reasons: if for some reason the
>> filesystem gets dismounted, or hasn't come on-line yet during
>> a reboot, you do not want Postgres to be able to write on the
>> underlying mount-point directory.
> Be careful: There are two different directorys involved in a mount
> point. The one in the parent filesystem and the one in the mounted file
> system.
True, and the safety requirement really is only that the parent
filesystem's mount-point directory not be writable by us.
But normal practice is that both directories are root-owned,
or at least owned by highly privileged users.
(I have a vague idea that there are system-level security hazards,
not specific to Postgres, if mount-point directories are publicly
writable. Don't feel like researching that though.)
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-15 06:02 Laurenz Albe <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: Laurenz Albe @ 2025-07-15 06:02 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter J. Holzer <[email protected]>; +Cc: [email protected]
On Mon, 2025-07-14 at 14:30 -0400, Tom Lane wrote:
> (I have a vague idea that there are system-level security hazards,
> not specific to Postgres, if mount-point directories are publicly
> writable. Don't feel like researching that though.)
Well, if you are using an ext? file system, there is a lost+found
directory where fsck places links to orphaned inodes.
If the PostgreSQL user owns the mount point and wants to use
"initdb" to create a data directory in it, the program will fail
and complain that the directory is not empty. The danger is great
that the user removes the lost+found directory to solve the problem.
True, one could re-create it with "mklost+found", but if a DBA
is uneducated enough to remove the directory in the first place,
the risk is high that he wouldn't think of creating it again,
which is a problem if the file system ever becomes corrupted.
All this doesn't apply to NFS, but it is yet another reason
(apart from the safety of a subdirectory that doesn't exist
on the file system underlying the mount point) why we should
continue to recommend that the data directory be not a mount point.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-15 11:36 Amol Inamdar <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 11+ messages in thread
From: Amol Inamdar @ 2025-07-15 11:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Laurenz Albe <[email protected]>; [email protected]
Thanks Tom and Laurenz for the explanation.
Let me try out a few things and get back to you if needed.
Thanks,
Amol
On Mon, Jul 14, 2025 at 7:37 PM Tom Lane <[email protected]> wrote:
> Laurenz Albe <[email protected]> writes:
> > It is not a good idea to have a mount point be the data directory.
>
> ^^^ This. ^^^
>
> That is primarily for safety reasons: if for some reason the
> filesystem gets dismounted, or hasn't come on-line yet during
> a reboot, you do not want Postgres to be able to write on the
> underlying mount-point directory. There is a sobering tale
> in this old thread:
>
> https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com
>
> Now it didn't help any that they were using a start script that
> would automatically run initdb if it didn't see a data directory
> where expected. But even without that, you are in for a world of
> hurt if the mount drops while the server is running and the server
> has any ability to write on the underlying storage; it will think
> whatever it was able to write is safely down on disk. To prevent
> that, the server must not have write permissions on the mount
> point, which dictates making a separate data directory (with
> different ownership/permissions) just below the mount.
>
> Do not bypass that ownership/permissions check. It is there
> for very good reasons.
>
> regards, tom lane
>
--
-regards
Amol
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-16 13:24 Amol Inamdar <[email protected]>
parent: Amol Inamdar <[email protected]>
0 siblings, 2 replies; 11+ messages in thread
From: Amol Inamdar @ 2025-07-16 13:24 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Laurenz Albe <[email protected]>; [email protected]
Hi All,
I would like to rephrase the question a little bit, below is how our setup
going to be
1. NFS mount point is for /nfs-mount/postgres (and permissions locked
down so that Postgres cannot create directories in here)
2. Postgres data directory is /nfs-mount/postgres/db
3.
With secured NFS + AT-TLS setup Postgres will be able to write to data
directory but not parent dir, however the file ownership information
Postgres sees from the stat() call will not match the Postgres user in the
container (even though the AT-TLS strict access control will ensure only
the Posgres user can read/write to this directory)
Considering the above scenario/setup, what is the danger of removing the
ownership check in miscinit.c checkDataDir() function ?
On Tue, Jul 15, 2025 at 5:06 PM Amol Inamdar <[email protected]> wrote:
> Thanks Tom and Laurenz for the explanation.
> Let me try out a few things and get back to you if needed.
>
> Thanks,
> Amol
>
> On Mon, Jul 14, 2025 at 7:37 PM Tom Lane <[email protected]> wrote:
>
>> Laurenz Albe <[email protected]> writes:
>> > It is not a good idea to have a mount point be the data directory.
>>
>> ^^^ This. ^^^
>>
>> That is primarily for safety reasons: if for some reason the
>> filesystem gets dismounted, or hasn't come on-line yet during
>> a reboot, you do not want Postgres to be able to write on the
>> underlying mount-point directory. There is a sobering tale
>> in this old thread:
>>
>>
>> https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com
>>
>> Now it didn't help any that they were using a start script that
>> would automatically run initdb if it didn't see a data directory
>> where expected. But even without that, you are in for a world of
>> hurt if the mount drops while the server is running and the server
>> has any ability to write on the underlying storage; it will think
>> whatever it was able to write is safely down on disk. To prevent
>> that, the server must not have write permissions on the mount
>> point, which dictates making a separate data directory (with
>> different ownership/permissions) just below the mount.
>>
>> Do not bypass that ownership/permissions check. It is there
>> for very good reasons.
>>
>> regards, tom lane
>>
>
>
> --
> -regards
> Amol
>
--
-regards
Amol
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-16 14:17 Ron Johnson <[email protected]>
parent: Amol Inamdar <[email protected]>
1 sibling, 0 replies; 11+ messages in thread
From: Ron Johnson @ 2025-07-16 14:17 UTC (permalink / raw)
To: pgsql-generallists.postgresql.org <[email protected]>
Quoting Tom's earlier email:
"(But I too *would not use Postgres-over-NFS for any critical data*.
Too many moving parts. It's tough enough to ensure crash safety
with local storage.)"
You're going through a lot of security effort to implement a Worst Practice.
On Wed, Jul 16, 2025 at 9:25 AM Amol Inamdar <[email protected]> wrote:
> Hi All,
>
> I would like to rephrase the question a little bit, below is how our setup
> going to be
>
> 1. NFS mount point is for /nfs-mount/postgres (and permissions locked
> down so that Postgres cannot create directories in here)
> 2. Postgres data directory is /nfs-mount/postgres/db
> 3.
>
> With secured NFS + AT-TLS setup Postgres will be able to write to data
> directory but not parent dir, however the file ownership information
> Postgres sees from the stat() call will not match the Postgres user in the
> container (even though the AT-TLS strict access control will ensure only
> the Posgres user can read/write to this directory)
>
> Considering the above scenario/setup, what is the danger of removing the
> ownership check in miscinit.c checkDataDir() function ?
>
>
> On Tue, Jul 15, 2025 at 5:06 PM Amol Inamdar <[email protected]> wrote:
>
>> Thanks Tom and Laurenz for the explanation.
>> Let me try out a few things and get back to you if needed.
>>
>> Thanks,
>> Amol
>>
>> On Mon, Jul 14, 2025 at 7:37 PM Tom Lane <[email protected]> wrote:
>>
>>> Laurenz Albe <[email protected]> writes:
>>> > It is not a good idea to have a mount point be the data directory.
>>>
>>> ^^^ This. ^^^
>>>
>>> That is primarily for safety reasons: if for some reason the
>>> filesystem gets dismounted, or hasn't come on-line yet during
>>> a reboot, you do not want Postgres to be able to write on the
>>> underlying mount-point directory. There is a sobering tale
>>> in this old thread:
>>>
>>>
>>> https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com
>>>
>>> Now it didn't help any that they were using a start script that
>>> would automatically run initdb if it didn't see a data directory
>>> where expected. But even without that, you are in for a world of
>>> hurt if the mount drops while the server is running and the server
>>> has any ability to write on the underlying storage; it will think
>>> whatever it was able to write is safely down on disk. To prevent
>>> that, the server must not have write permissions on the mount
>>> point, which dictates making a separate data directory (with
>>> different ownership/permissions) just below the mount.
>>>
>>> Do not bypass that ownership/permissions check. It is there
>>> for very good reasons.
>>>
>>> regards, tom lane
>>>
>>
>>
>> --
>> -regards
>> Amol
>>
>
>
> --
> -regards
> Amol
>
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)
@ 2025-07-17 00:41 Greg Sabino Mullane <[email protected]>
parent: Amol Inamdar <[email protected]>
1 sibling, 0 replies; 11+ messages in thread
From: Greg Sabino Mullane @ 2025-07-17 00:41 UTC (permalink / raw)
To: Amol Inamdar <[email protected]>; +Cc: Tom Lane <[email protected]>; Laurenz Albe <[email protected]>; [email protected]
On Wed, Jul 16, 2025 at 9:25 AM Amol Inamdar <[email protected]> wrote:
>
> 1. NFS mount point is for /nfs-mount/postgres (and permissions locked
> down so that Postgres cannot create directories in here)
> 2. Postgres data directory is /nfs-mount/postgres/db
> 3.
>
> With secured NFS + AT-TLS setup Postgres will be able to write to data
> directory but not parent dir, however the file ownership information
> Postgres sees from the stat() call will not match the Postgres user in the
> container (even though the AT-TLS strict access control will ensure only
> the Posgres user can read/write to this directory)
>
> This thread is fascinating. It's like combining two of the most annoying
technologies in the world, NFS and SELinux, into something worse than
either of them.
Many people use Docker, and NFS, and Postgres all the time. Stop trying to
push on a string. Conform your process to Postgres' fairly minimal and
sane requirements, rather than the other way around.
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2025-07-17 00:41 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-07-14 12:20 Re: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS) Laurenz Albe <[email protected]>
2025-07-14 12:29 ` Amol Inamdar <[email protected]>
2025-07-14 12:44 ` Laurenz Albe <[email protected]>
2025-07-14 14:07 ` Tom Lane <[email protected]>
2025-07-14 18:20 ` Peter J. Holzer <[email protected]>
2025-07-14 18:30 ` Tom Lane <[email protected]>
2025-07-15 06:02 ` Laurenz Albe <[email protected]>
2025-07-15 11:36 ` Amol Inamdar <[email protected]>
2025-07-16 13:24 ` Amol Inamdar <[email protected]>
2025-07-16 14:17 ` Ron Johnson <[email protected]>
2025-07-17 00:41 ` Greg Sabino Mullane <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox