public inbox for [email protected]
help / color / mirror / Atom feedFile locks for data directory lockfile in the context of Linux namespaces
7+ messages / 3 participants
[nested] [flat]
* File locks for data directory lockfile in the context of Linux namespaces
@ 2025-12-19 14:27 Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Dolgov @ 2025-12-19 14:27 UTC (permalink / raw)
To: pgsql-hackers
Hi,
TL;DR This is a proposal to use file locking with a data directory lockfile at
startup, which helps to avoid potential Linux PID namespace visibility issues.
Recently I've stumbled upon a quite annoying problem, which will require a bit
of explanation. Currently at startup if the data directory lock exists,
postgres inspects it and assumes that if it contains the same PID as the
current process, it must be a state file after a system reboot and assigning
of the same PID again. But it seems there is another possible scenario: two
postgres instances are running concurrently inside different PID namespaces,
they don't see each other and have the same PID assigned withing the respective
namespace.
It's relatively easy to use pid/ipc/net namespaces to construct a situation,
when two postgres instances run in parallel on the same data directory and do
not notice a thing, something like this:
sudo unshare --ipc --net --pid --fork --mount-proc \
bash -c 'sudo -u postgres postgres -D data'
This of course can lead to all sorts of nasty issues, but looks very artificial
at first -- obviously whoever is responsible for namespace management must also
take care about data access isolation.
But it turns out situations like this indeed could happen in practice, when it
comes to container orchestration, mostly due to lack of knowledge or
misunderstanding of documentation. Kubernetes has one particular access mode
for volumes, ReadWriteOnce [1], which often assumed to be good enough -- but it
guarantees only a single mount per node, not per pod. Kubernetes also allows a
forced pod termination [2], which removes the pod from the API, but still gives
some grace period for the pod to finish. All of this can lead to an unfortunate
sequence of events:
* A postgres pod got forcefully terminated and removed from the API right away.
* A new postgres pod is started instead (there is nothing in the API, so
why not), while the old one is still terminating.
* If they were utilizing RWO mode, the new pod will immediately get the data
volume and can access it while the old pod is still terminating.
In the end we get a situation similar to what I've described above, and
strangely enough it looks like this indeed happens in the field.
It's fair to say that it's a Kubernetes issue (there are warnings about
that in the documentation), and PostgreSQL doesn't have anything to do
with that. But taking into account general possibility of confusing
PostgreSQL with Linux namespaces it looks to me like one of those "shoot
yourself in the foot" situation, and I became curious if there are any
easy way to improve things.
The root of the problem is lack of any time related information that PostgreSQL
could use to distinguish between two scenarios: when a single container was
killed and started again later; and when two containers run at the same time.
After some experimenting it looks like the only plausible answer could be file
locking for data directory lockfile.
This approach was discussed many times in hackers, and from what I see there
are few arguments against using file locking as the main mechanism for
protecting the data directory content:
* Portability. There are two types of file locks, advisory record locks (POSIX)
and open file description locks (was non-POSIX). The former has set of flaws,
but most importantly for this discussion is that advisory record locks are
associated with a process and thus affected by PID namespace isolation. The
later are associated with open file descriptors and are suitable solution to
fix the problem. Originally open file description locks were non-POSIX, but
looks like they have become a part of POSIX.1 2024, (see F_OFD_SETLK) [3].
* Issues with NFS. It turns out NFSv3 does not support open file description
locks and convert them into advisory locks. For our purposes it means that
the aproach will not change anything for NFSv3. Regarding NFSv4, it uses some
sort of lease system for locking, and I haven't found anything claiming that
locks will be converted to advisory.
With this in mind, it seems to me that adding file locking to data directory
lockfile as a "best efforts" approach (i.e. if it doesn't work, we continue as
before) on top of already existing mechanism will improve most of things, while
keeping status quo for some others. I've attached a quick sketch of how the
patch might look like.
Any thoughts / commentaries on that?
[1]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
[2]: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-forced
[3]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: File locks for data directory lockfile in the context of Linux namespaces
@ 2026-06-05 12:37 Ilmar Yunusov <[email protected]>
parent: Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Ilmar Yunusov @ 2026-06-05 12:37 UTC (permalink / raw)
To: [email protected]; +Cc: Dmitry Dolgov <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested
Hi,
I looked at v2, focusing on apply/build status and the PID namespace scenario
described in the cover letter.
I used the v2 patch from Dmitry's 2026-01-17 message, on origin/master at
4cb2a9863d89b320f37eb1bd76822f6f65e59311.
The patch applies cleanly with git am, and git diff --check reports no issues.
I built locally with:
./configure --prefix="$PWD/pg-install" --without-readline --without-zlib --without-icu
make -s -j8
make -s install
The build completed successfully. Configure found F_OFD_SETLK on this macOS
host too.
I also built and tested on Linux with the same configure options:
make -s -j3
make -s install
There, configure also found F_OFD_SETLK, and:
make -C src/test/regress check
passed; all regression tests passed.
For the namespace behavior, I used two PostgreSQL builds on the same Linux host:
unpatched master and v2. The test starts the first postmaster in one
pid/ipc/net namespace, then tries to start a second postmaster in another
pid/ipc/net namespace on the same data directory.
On unpatched master, both postmasters started. Both saw themselves as PID 2 in
their own PID namespace, and the second postmaster rewrote postmaster.pid. After
the second startup, the lock file contained the second socket directory:
2
.../pidns-base/data
1780661958
65437
.../pidns-base/sock2
With v2, the first postmaster started, and the second startup failed with:
FATAL: cannot lock the lock file "postmaster.pid"
HINT: Another server is starting.
So the patch fixes the concrete Linux PID namespace failure mode I tested.
One behavior I wanted to ask about: v2 also OFD-locks Unix socket lock files,
not only the data directory lockfile. That follows from calling
FlockDataDirLockFile() from the generic CreateLockFile() path, and I also saw
it at runtime. With one v2 postmaster using a Unix socket directory, /proc/locks
showed OFD write locks for both files:
.../data/postmaster.pid
.../sock/.s.PGSQL.65436.lock
The corresponding postmaster fds were:
/proc/835058/fd/5 -> .../data/postmaster.pid
/proc/835058/fd/8 -> .../sock/.s.PGSQL.65436.lock
I had expected the new lock to apply only to postmaster.pid, because the patch
title, commit message, helper name, and comments all describe the data directory
lockfile. The socket lockfile behavior therefore looked like either an
intentional scope expansion that should be named as such, or an accidental side
effect of using the generic CreateLockFile() path.
Is locking the socket lock file intentional here? If so, maybe the helper name,
comment, and fd tracking should reflect that broader scope. If not, perhaps the
new lock should be applied only for the isDDLock case; otherwise v2 changes
socket lockfile semantics too, not only postmaster.pid.
While reading that code, I also noticed a small error-path issue: DataDirLockFD
starts as 0, but if fcntl(F_OFD_SETLK) fails with a non-EAGAIN error the code
only emits a warning and does not assign a duplicate fd. UnlinkLockFiles()
then closes DataDirLockFD unconditionally. Initializing it to -1, closing it
conditionally, and checking dup(fd) would make that path more explicit.
Aside from those questions, this looks like a useful best-effort improvement to
me, and the namespace failure mode appears to be addressed by the OFD lock in
the Linux test above.
I did not test NFS behavior, older stable branches, Windows behavior, or
installcheck-world. Unprivileged namespace creation was not permitted on the
Linux host I used, so the namespace repro was run with sudo.
Regards,
Ilmar Yunusov
The new status of this patch is: Waiting on Author
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: File locks for data directory lockfile in the context of Linux namespaces
@ 2026-06-19 15:11 Dmitry Dolgov <[email protected]>
parent: Ilmar Yunusov <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Dolgov @ 2026-06-19 15:11 UTC (permalink / raw)
To: Ilmar Yunusov <[email protected]>; +Cc: [email protected]
> On Fri, Jun 05, 2026 at 12:37:26PM +0000, Ilmar Yunusov wrote:
> The following review has been posted through the commitfest application:
> make installcheck-world: not tested
> Implements feature: tested, passed
> Spec compliant: not tested
> Documentation: not tested
>
> Hi,
>
> I looked at v2, focusing on apply/build status and the PID namespace scenario
> described in the cover letter.
Thanks for the review and testing!
> Is locking the socket lock file intentional here? If so, maybe the helper name,
> comment, and fd tracking should reflect that broader scope. If not, perhaps the
> new lock should be applied only for the isDDLock case; otherwise v2 changes
> socket lockfile semantics too, not only postmaster.pid.
It's been a while, quite frankly I don't remember. On the face of it, I
think both the directory and socket lock files should be locked in the
same way, since both are equally susceptible for the failure scenario
described in this thread, even if the consequences of a failure are
different. Let me do some renaming to clarify that.
> While reading that code, I also noticed a small error-path issue: DataDirLockFD
> starts as 0, but if fcntl(F_OFD_SETLK) fails with a non-EAGAIN error the code
> only emits a warning and does not assign a duplicate fd. UnlinkLockFiles()
> then closes DataDirLockFD unconditionally. Initializing it to -1, closing it
> conditionally, and checking dup(fd) would make that path more explicit.
Good point, I'll address this in the next version.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: File locks for data directory lockfile in the context of Linux namespaces
@ 2026-06-23 14:29 Dmitry Dolgov <[email protected]>
parent: Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Dolgov @ 2026-06-23 14:29 UTC (permalink / raw)
To: Ilmar Yunusov <[email protected]>; +Cc: [email protected]
> On Fri, Jun 19, 2026 at 05:11:11PM +0200, Dmitry Dolgov wrote:
> > Is locking the socket lock file intentional here? If so, maybe the helper name,
> > comment, and fd tracking should reflect that broader scope. If not, perhaps the
> > new lock should be applied only for the isDDLock case; otherwise v2 changes
> > socket lockfile semantics too, not only postmaster.pid.
>
> It's been a while, quite frankly I don't remember. On the face of it, I
> think both the directory and socket lock files should be locked in the
> same way, since both are equally susceptible for the failure scenario
> described in this thread, even if the consequences of a failure are
> different. Let me do some renaming to clarify that.
>
> > While reading that code, I also noticed a small error-path issue: DataDirLockFD
> > starts as 0, but if fcntl(F_OFD_SETLK) fails with a non-EAGAIN error the code
> > only emits a warning and does not assign a duplicate fd. UnlinkLockFiles()
> > then closes DataDirLockFD unconditionally. Initializing it to -1, closing it
> > conditionally, and checking dup(fd) would make that path more explicit.
>
> Good point, I'll address this in the next version.
Something like this should be sufficient I think.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: File locks for data directory lockfile in the context of Linux namespaces
@ 2026-07-06 07:30 Ilmar Yunusov <[email protected]>
parent: Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Ilmar Yunusov @ 2026-07-06 07:30 UTC (permalink / raw)
To: [email protected]; +Cc: Dmitry Dolgov <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested
Hi,
I looked at v2, focusing on apply/build status and the PID namespace scenario
described in the cover letter.
I used the v2 patch from Dmitry's 2026-01-17 message, on origin/master at
4cb2a9863d89b320f37eb1bd76822f6f65e59311.
The patch applies cleanly with git am, and git diff --check reports no issues.
I built locally with:
./configure --prefix="$PWD/pg-install" --without-readline --without-zlib --without-icu
make -s -j8
make -s install
The build completed successfully. Configure found F_OFD_SETLK on this macOS
host too.
I also built and tested on Linux with the same configure options:
make -s -j3
make -s install
There, configure also found F_OFD_SETLK, and:
make -C src/test/regress check
passed; all regression tests passed.
For the namespace behavior, I used two PostgreSQL builds on the same Linux host:
unpatched master and v2. The test starts the first postmaster in one
pid/ipc/net namespace, then tries to start a second postmaster in another
pid/ipc/net namespace on the same data directory.
On unpatched master, both postmasters started. Both saw themselves as PID 2 in
their own PID namespace, and the second postmaster rewrote postmaster.pid. After
the second startup, the lock file contained the second socket directory:
2
.../pidns-base/data
1780661958
65437
.../pidns-base/sock2
With v2, the first postmaster started, and the second startup failed with:
FATAL: cannot lock the lock file "postmaster.pid"
HINT: Another server is starting.
So the patch fixes the concrete Linux PID namespace failure mode I tested.
One behavior I wanted to ask about: v2 also OFD-locks Unix socket lock files,
not only the data directory lockfile. That follows from calling
FlockDataDirLockFile() from the generic CreateLockFile() path, and I also saw
it at runtime. With one v2 postmaster using a Unix socket directory, /proc/locks
showed OFD write locks for both files:
.../data/postmaster.pid
.../sock/.s.PGSQL.65436.lock
The corresponding postmaster fds were:
/proc/835058/fd/5 -> .../data/postmaster.pid
/proc/835058/fd/8 -> .../sock/.s.PGSQL.65436.lock
I had expected the new lock to apply only to postmaster.pid, because the patch
title, commit message, helper name, and comments all describe the data directory
lockfile. The socket lockfile behavior therefore looked like either an
intentional scope expansion that should be named as such, or an accidental side
effect of using the generic CreateLockFile() path.
Is locking the socket lock file intentional here? If so, maybe the helper name,
comment, and fd tracking should reflect that broader scope. If not, perhaps the
new lock should be applied only for the isDDLock case; otherwise v2 changes
socket lockfile semantics too, not only postmaster.pid.
While reading that code, I also noticed a small error-path issue: DataDirLockFD
starts as 0, but if fcntl(F_OFD_SETLK) fails with a non-EAGAIN error the code
only emits a warning and does not assign a duplicate fd. UnlinkLockFiles()
then closes DataDirLockFD unconditionally. Initializing it to -1, closing it
conditionally, and checking dup(fd) would make that path more explicit.
Aside from those questions, this looks like a useful best-effort improvement to
me, and the namespace failure mode appears to be addressed by the OFD lock in
the Linux test above.
I did not test NFS behavior, older stable branches, Windows behavior, or
installcheck-world. Unprivileged namespace creation was not permitted on the
Linux host I used, so the namespace repro was run with sudo.
Regards,
Ilmar Yunusov
## v3 follow-up draft
Suggested CommitFest form:
- In response to: Dmitry's 2026-06-23 v3 message
- make installcheck-world: not tested
- Implements feature: tested, passed
- Spec compliant: not tested
- Documentation: not tested
- New status: Waiting on Author
- Attachments: none
Subject:
Re: File locks for data directory lockfile in the context of Linux namespaces
Body:
Hi,
I looked at v3, focusing on whether it addresses the two points from my
previous review, the Linux runtime behavior, and the current CFBot failures.
I used the v3 patch from Dmitry's 2026-06-23 message, on origin/master at
9d1188f29865e66c4196578501e74e8c815fba8d.
The patch applies cleanly with git am, and git diff --check reports no issues.
On Ubuntu 24.04.4, configure found F_OFD_SETLK, and this passed:
./configure --prefix=/home/master/pgcf-6335-v3-20260706/pg-install --without-readline --without-zlib --without-icu
make -s -j3
make -s install
make -C src/test/regress check
All 245 regression tests passed.
I re-ran the PID namespace reproducer from the earlier review against v3. The
first postmaster started in one pid/ipc/net namespace on the test data
directory. A second postmaster in another pid/ipc/net namespace on the same
data directory failed with:
FATAL: cannot lock the lock file "postmaster.pid"
HINT: Another server is starting.
So the concrete Linux PID namespace failure mode I tested still looks addressed
in v3.
I also checked the lockfile scope at runtime. With one v3 postmaster using a
Unix socket directory, /proc/<pid>/fd showed fds for both files:
.../data/postmaster.pid
.../sock/.s.PGSQL.65442.lock
/proc/locks also showed OFDLCK ADVISORY WRITE entries for both file inodes.
That matches the v3 scope where socket lockfiles are intentionally locked too.
The other question from my previous review also looks addressed: the single
static DataDirLockFD issue is replaced by per-lockfile fd tracking in the
lock_files list, with conditional close in UnlinkLockFiles().
There is still a build blocker, matching the current CFBot failures. v3 adds a
typedef named LockFile in src/backend/utils/init/miscinit.c, but that name
conflicts with the Windows LockFile() function. The CompilerWarnings log
reports, for example:
miscinit.c:79:3: error: 'LockFile' redeclared as different kind of symbol
and the Windows logs show the same issue:
miscinit.c(79): error C2365: 'LockFile': redefinition; previous definition was 'function'
There is also a missing semicolon in the non-F_OFD_SETLK branch:
miscinit.c:1169:18: error: expected ';' before '}' token
As a quick local compile check, in the temporary worktree only, renaming the
new typedef to LockFileData and adding the missing semicolon made the normal
macOS build pass. With HAVE_DECL_F_OFD_SETLK temporarily forced to 0 in the
generated pg_config.h, the affected miscinit.o target also compiled after those
two mechanical changes. I did not run a local Windows build.
I did not run installcheck-world, and did not test NFS behavior or
stable-branch backpatch applicability.
Regards,
Ilmar Yunusov
The new status of this patch is: Waiting on Author
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: File locks for data directory lockfile in the context of Linux namespaces
@ 2026-07-06 12:10 Dmitry Dolgov <[email protected]>
parent: Ilmar Yunusov <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Dolgov @ 2026-07-06 12:10 UTC (permalink / raw)
To: Ilmar Yunusov <[email protected]>; +Cc: [email protected]
> On Mon, Jul 06, 2026 at 07:30:04AM +0000, Ilmar Yunusov wrote:
>
> There is still a build blocker, matching the current CFBot failures. v3 adds a
> typedef named LockFile in src/backend/utils/init/miscinit.c, but that name
> conflicts with the Windows LockFile() function. The CompilerWarnings log
> reports, for example:
Thanks for looking into the build failure. I wanted to check it out what
was happening on Windows, but after the migration from Cirrus to Github
one have to be logged in to see the build logs, and at that moment I
found myself logged off.
Regarding the name, I'm afraid LockFileData has a chance of causing some
confusion due to a common pattern, where structures are named with
"Data" suffix and a pointer type definition without. Since it's about a
name clash with an external library, I suggest PGLockFile instead, but
open for better suggestions.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: File locks for data directory lockfile in the context of Linux namespaces
@ 2026-07-06 22:07 Zsolt Parragi <[email protected]>
parent: Dmitry Dolgov <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Zsolt Parragi @ 2026-07-06 22:07 UTC (permalink / raw)
To: [email protected]
Hello!
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
According to fcntl.2, this should handle both EACCESS and EAGAIN:
ERRORS
EACCES or EAGAIN
Operation is prohibited by locks held by other processes.
+static int
+OFDLockFile(int fd, const char *filename)
+...
+ else
+ return dup(fd);
Isn't this missing an FD_CLOEXEC, so that launched processes doesn't
inherit it and keep the lock open possibly longer than needed?
Also, shouldn't the code verify the result of dup? (!= -1 / errno)
+ flock_fd = OFDLockFile(fd, filename);
Can't we leak flock_fd in the stale path?
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
Shouldn't this check for >= 0?
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
This probably should be:
ereport(WARNING, (errcode_for_file_access(), errmsg("could not lock
file \"%s\": %m", filename)))
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-07-06 22:07 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-12-19 14:27 File locks for data directory lockfile in the context of Linux namespaces Dmitry Dolgov <[email protected]>
2026-06-05 12:37 ` Ilmar Yunusov <[email protected]>
2026-06-19 15:11 ` Dmitry Dolgov <[email protected]>
2026-06-23 14:29 ` Dmitry Dolgov <[email protected]>
2026-07-06 07:30 ` Ilmar Yunusov <[email protected]>
2026-07-06 12:10 ` Dmitry Dolgov <[email protected]>
2026-07-06 22:07 ` Zsolt Parragi <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox