public inbox for [email protected]
help / color / mirror / Atom feedRe: Logical replication timeout problem
25+ messages / 4 participants
[nested] [flat]
* Re: Logical replication timeout problem
@ 2023-01-16 16:36 Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Ashutosh Bapat @ 2023-01-16 16:36 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Wed, Jan 11, 2023 at 4:11 PM [email protected]
<[email protected]> wrote:
>
> On Mon, Jan 9, 2023 at 13:04 PM Amit Kapila <[email protected]> wrote:
> >
>
> Thanks for your comments.
>
> > One more thing, I think it would be better to expose a new callback
> > API via reorder buffer as suggested previously [2] similar to other
> > reorder buffer APIs instead of directly using reorderbuffer API to
> > invoke plugin API.
>
> Yes, I agree. I think it would be better to add a new callback API on the HEAD.
> So, I improved the fix approach:
> Introduce a new optional callback to update the process. This callback function
> is invoked at the end inside the main loop of the function
> ReorderBufferProcessTXN() for each change. In this way, I think it seems that
> similar timeout problems could be avoided.
I am a bit worried about the indirections that the wrappers and hooks
create. Output plugins call OutputPluginUpdateProgress() in callbacks
but I don't see why ReorderBufferProcessTXN() needs a callback to
call OutputPluginUpdateProgress. I don't think output plugins are
going to do anything special with that callback than just call
OutputPluginUpdateProgress. Every output plugin will need to implement
it and if they do not they will face the timeout problem. That would
be unnecessary. Instead ReorderBufferUpdateProgress() in your first
patch was more direct and readable. That way the fix works for any
output plugin. In fact, I am wondering whether we could have a call in
ReorderBufferProcessTxn() at the end of transaction
(commit/prepare/commit prepared/abort prepared) instead of the
corresponding output plugin callbacks calling
OutputPluginUpdateProgress().
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-17 10:04 Amit Kapila <[email protected]>
parent: Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Amit Kapila @ 2023-01-17 10:04 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: [email protected] <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Mon, Jan 16, 2023 at 10:06 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Wed, Jan 11, 2023 at 4:11 PM [email protected]
> <[email protected]> wrote:
> >
> > On Mon, Jan 9, 2023 at 13:04 PM Amit Kapila <[email protected]> wrote:
> > >
> >
> > Thanks for your comments.
> >
> > > One more thing, I think it would be better to expose a new callback
> > > API via reorder buffer as suggested previously [2] similar to other
> > > reorder buffer APIs instead of directly using reorderbuffer API to
> > > invoke plugin API.
> >
> > Yes, I agree. I think it would be better to add a new callback API on the HEAD.
> > So, I improved the fix approach:
> > Introduce a new optional callback to update the process. This callback function
> > is invoked at the end inside the main loop of the function
> > ReorderBufferProcessTXN() for each change. In this way, I think it seems that
> > similar timeout problems could be avoided.
>
> I am a bit worried about the indirections that the wrappers and hooks
> create. Output plugins call OutputPluginUpdateProgress() in callbacks
> but I don't see why ReorderBufferProcessTXN() needs a callback to
> call OutputPluginUpdateProgress.
>
Yeah, I think we can do it as we are doing the previous approach but
we need an additional wrapper (update_progress_cb_wrapper()) as the
current patch has so that we can add error context information. This
is similar to why we have a wrapper for all other callbacks like
change_cb_wrapper.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-17 13:11 Ashutosh Bapat <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Ashutosh Bapat @ 2023-01-17 13:11 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila <[email protected]> wrote:
> >
> > I am a bit worried about the indirections that the wrappers and hooks
> > create. Output plugins call OutputPluginUpdateProgress() in callbacks
> > but I don't see why ReorderBufferProcessTXN() needs a callback to
> > call OutputPluginUpdateProgress.
> >
>
> Yeah, I think we can do it as we are doing the previous approach but
> we need an additional wrapper (update_progress_cb_wrapper()) as the
> current patch has so that we can add error context information. This
> is similar to why we have a wrapper for all other callbacks like
> change_cb_wrapper.
>
Ultimately OutputPluginUpdateProgress() will be called - which in turn
will call ctx->update_progress. I don't see wrappers around
OutputPluginWrite or OutputPluginPrepareWrite. But I see that those
two are called always from output plugin, so indirectly those are
called through a wrapper. I also see that update_progress_cb_wrapper()
is similar, as far as wrapper is concerned, to
ReorderBufferUpdateProgress() in the earlier patch.
ReorderBufferUpdateProgress() looks more readable than the wrapper.
If we want to keep the wrapper at least we should use a different
variable name. update_progress is also there LogicalDecodingContext
and will be indirectly called from ReorderBuffer::update_progress.
Somebody might think that there's some recursion involved there.
That's a mighty confusion.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-18 05:28 Amit Kapila <[email protected]>
parent: Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Amit Kapila @ 2023-01-18 05:28 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: [email protected] <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila <[email protected]> wrote:
>
> > >
> > > I am a bit worried about the indirections that the wrappers and hooks
> > > create. Output plugins call OutputPluginUpdateProgress() in callbacks
> > > but I don't see why ReorderBufferProcessTXN() needs a callback to
> > > call OutputPluginUpdateProgress.
> > >
> >
> > Yeah, I think we can do it as we are doing the previous approach but
> > we need an additional wrapper (update_progress_cb_wrapper()) as the
> > current patch has so that we can add error context information. This
> > is similar to why we have a wrapper for all other callbacks like
> > change_cb_wrapper.
> >
>
> Ultimately OutputPluginUpdateProgress() will be called - which in turn
> will call ctx->update_progress.
>
No, update_progress_cb_wrapper() should directly call
ctx->update_progress(). The key reason to have a
update_progress_cb_wrapper() is that it allows us to add error context
information (see the usage of output_plugin_error_callback).
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 25+ messages in thread
* RE: Logical replication timeout problem
@ 2023-01-18 08:19 [email protected] <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: [email protected] @ 2023-01-18 08:19 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>; +Cc: Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Wed, Jan 18, 2023 at 13:29 PM Amit Kapila <[email protected]> wrote:
> On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat
> <[email protected]> wrote:
> >
> > On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila <[email protected]> wrote:
> >
> > > >
> > > > I am a bit worried about the indirections that the wrappers and hooks
> > > > create. Output plugins call OutputPluginUpdateProgress() in callbacks
> > > > but I don't see why ReorderBufferProcessTXN() needs a callback to
> > > > call OutputPluginUpdateProgress.
> > > >
> > >
> > > Yeah, I think we can do it as we are doing the previous approach but
> > > we need an additional wrapper (update_progress_cb_wrapper()) as the
> > > current patch has so that we can add error context information. This
> > > is similar to why we have a wrapper for all other callbacks like
> > > change_cb_wrapper.
> > >
> >
> > Ultimately OutputPluginUpdateProgress() will be called - which in turn
> > will call ctx->update_progress.
> >
>
> No, update_progress_cb_wrapper() should directly call
> ctx->update_progress(). The key reason to have a
> update_progress_cb_wrapper() is that it allows us to add error context
> information (see the usage of output_plugin_error_callback).
I think it makes sense. This also avoids the need for every output plugin to
implement the callback. So I tried to improve the patch based on this approach.
And I tried to add some comments for this new callback to distinguish it from
ctx->update_progress.
Attach the new patch.
Regards,
Wang Wei
Attachments:
[application/octet-stream] v3-0001-Fix-the-logical-replication-timeout-during-proces.patch (11.5K, ../../OS3PR01MB6275E5C6DC1E9E7CD2D11E039EC79@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v3-0001-Fix-the-logical-replication-timeout-during-proces.patch)
download | inline diff:
From fe7015a8a000ecee60c2498e871a5a70c9eacddf Mon Sep 17 00:00:00 2001
From: wangw <[email protected]>
Date: Wed, 18 Jan 2023 13:45:32 +0800
Subject: [PATCH v3] Fix the logical replication timeout during processing of
DDL.
The problem is when there is a DDL in a transaction that generates lots of
temporary data due to rewrite rules, these temporary data will not be processed
by the pgoutput - plugin. Therefore, the previous fix (f95d53e) for DML had no
impact on this case.
To fix this, we introduced a new ReorderBuffer callback -
'ReorderBufferUpdateProgressCB'. This callback is called to try to update the
process after each change has been processed during sending data of a
transaction (and its subtransactions) to the output plugin.
---
src/backend/replication/logical/logical.c | 83 +++++++++++++++++++
.../replication/logical/reorderbuffer.c | 2 +
src/backend/replication/pgoutput/pgoutput.c | 54 ++----------
src/include/replication/reorderbuffer.h | 12 +++
src/tools/pgindent/typedefs.list | 1 +
5 files changed, 104 insertions(+), 48 deletions(-)
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 1a58dd7649..a16f40e8ae 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -93,6 +93,11 @@ static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *tx
static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
int nrelations, Relation relations[], ReorderBufferChange *change);
+/* update progress callback */
+static void update_progress_cb_wrapper(ReorderBuffer *cache,
+ ReorderBufferTXN *txn,
+ ReorderBufferChange *change);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, const char *plugin);
/*
@@ -278,6 +283,12 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit_prepared = commit_prepared_cb_wrapper;
ctx->reorder->rollback_prepared = rollback_prepared_cb_wrapper;
+ /*
+ * Callback to support updating progress during sending data of a
+ * transaction (and its subtransactions) to the output plugin.
+ */
+ ctx->reorder->update_progress = update_progress_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -1584,6 +1595,78 @@ stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+/*
+ * Update progress callback
+ *
+ * Try to update progress and send a keepalive message if too many changes were
+ * processed when processing txn.
+ *
+ * For a large transaction, if we don't send any change to the downstream for a
+ * long time (exceeds the wal_receiver_timeout of standby) then it can timeout.
+ * This can happen when all or most of the changes are either not published or
+ * got filtered out.
+ */
+static void
+update_progress_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ static int changes_count = 0; /* Static variable used to accumulate
+ * the number of changes while
+ * processing txn. */
+
+ Assert(!ctx->fast_forward);
+
+ if (!ctx->update_progress)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "update_progress";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * Report this change's lsn so replies from clients can give an up-to-date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->end_xact = false;
+
+ /*
+ * We don't want to try sending a keepalive message after processing each
+ * change as that can have overhead. Tests revealed that there is no
+ * noticeable overhead in doing it after continuously processing 100 or so
+ * changes.
+ */
+#define CHANGES_THRESHOLD 100
+
+ /*
+ * After continuously processing CHANGES_THRESHOLD changes, we
+ * try to send a keepalive message if required.
+ */
+ if (++changes_count >= CHANGES_THRESHOLD)
+ {
+ ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, false);
+ changes_count = 0;
+ }
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 54ee824e6c..2e134bd011 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2446,6 +2446,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
elog(ERROR, "tuplecid value in changequeue");
break;
}
+
+ rb->update_progress(rb, txn, change);
}
/* speculative insertion record must be freed by now */
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 1a80d67bb9..0ba6b6b39c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -92,8 +92,6 @@ static void send_relation_and_attrs(Relation relation, TransactionId xid,
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
-static void update_replication_progress(LogicalDecodingContext *ctx,
- bool skipped_xact);
/*
* Only 3 publication actions are used for row filtering ("insert", "update",
@@ -586,7 +584,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
* from this transaction has been sent to the downstream.
*/
sent_begin_txn = txndata->sent_begin_txn;
- update_replication_progress(ctx, !sent_begin_txn);
+ OutputPluginUpdateProgress(ctx, !sent_begin_txn);
pfree(txndata);
txn->output_plugin_private = NULL;
@@ -625,7 +623,7 @@ static void
pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr prepare_lsn)
{
- update_replication_progress(ctx, false);
+ OutputPluginUpdateProgress(ctx, false);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -639,7 +637,7 @@ static void
pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn)
{
- update_replication_progress(ctx, false);
+ OutputPluginUpdateProgress(ctx, false);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -655,7 +653,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
XLogRecPtr prepare_end_lsn,
TimestampTz prepare_time)
{
- update_replication_progress(ctx, false);
+ OutputPluginUpdateProgress(ctx, false);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1401,8 +1399,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
TupleTableSlot *old_slot = NULL;
TupleTableSlot *new_slot = NULL;
- update_replication_progress(ctx, false);
-
if (!is_publishable_relation(relation))
return;
@@ -1637,8 +1633,6 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Oid *relids;
TransactionId xid = InvalidTransactionId;
- update_replication_progress(ctx, false);
-
/* Remember the xid for the change in streaming mode. See pgoutput_change. */
if (in_streaming)
xid = change->txn->xid;
@@ -1702,8 +1696,6 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
TransactionId xid = InvalidTransactionId;
- update_replication_progress(ctx, false);
-
if (!data->messages)
return;
@@ -1903,7 +1895,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
Assert(!in_streaming);
Assert(rbtxn_is_streamed(txn));
- update_replication_progress(ctx, false);
+ OutputPluginUpdateProgress(ctx, false);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1924,7 +1916,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
{
Assert(rbtxn_is_streamed(txn));
- update_replication_progress(ctx, false);
+ OutputPluginUpdateProgress(ctx, false);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
OutputPluginWrite(ctx, true);
@@ -2424,37 +2416,3 @@ send_repl_origin(LogicalDecodingContext *ctx, RepOriginId origin_id,
}
}
}
-
-/*
- * Try to update progress and send a keepalive message if too many changes were
- * processed.
- *
- * For a large transaction, if we don't send any change to the downstream for a
- * long time (exceeds the wal_receiver_timeout of standby) then it can timeout.
- * This can happen when all or most of the changes are either not published or
- * got filtered out.
- */
-static void
-update_replication_progress(LogicalDecodingContext *ctx, bool skipped_xact)
-{
- static int changes_count = 0;
-
- /*
- * We don't want to try sending a keepalive message after processing each
- * change as that can have overhead. Tests revealed that there is no
- * noticeable overhead in doing it after continuously processing 100 or so
- * changes.
- */
-#define CHANGES_THRESHOLD 100
-
- /*
- * If we are at the end of transaction LSN, update progress tracking.
- * Otherwise, after continuously processing CHANGES_THRESHOLD changes, we
- * try to send a keepalive message if required.
- */
- if (ctx->end_xact || ++changes_count >= CHANGES_THRESHOLD)
- {
- OutputPluginUpdateProgress(ctx, skipped_xact);
- changes_count = 0;
- }
-}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index f6c4dd75db..6dfb2c971e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -525,6 +525,12 @@ typedef void (*ReorderBufferStreamTruncateCB) (
Relation relations[],
ReorderBufferChange *change);
+/* update progress callback signature */
+typedef void (*ReorderBufferUpdateProgressCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferChange *change);
+
struct ReorderBuffer
{
/*
@@ -588,6 +594,12 @@ struct ReorderBuffer
ReorderBufferStreamMessageCB stream_message;
ReorderBufferStreamTruncateCB stream_truncate;
+ /*
+ * Callback to be called when updating progress during sending data of a
+ * transaction (and its subtransactions) to the output plugin.
+ */
+ ReorderBufferUpdateProgressCB update_progress;
+
/*
* Pointer that will be passed untouched to the callbacks.
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 23bafec5f7..a1721d13bb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2310,6 +2310,7 @@ ReorderBufferToastEnt
ReorderBufferTupleBuf
ReorderBufferTupleCidEnt
ReorderBufferTupleCidKey
+ReorderBufferUpdateProgressCB
ReorderTuple
RepOriginId
ReparameterizeForeignPathByChild_function
--
2.23.0.windows.1
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-18 12:07 Ashutosh Bapat <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Ashutosh Bapat @ 2023-01-18 12:07 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Wed, Jan 18, 2023 at 1:49 PM [email protected]
<[email protected]> wrote:
>
> On Wed, Jan 18, 2023 at 13:29 PM Amit Kapila <[email protected]> wrote:
> > On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat
> > <[email protected]> wrote:
> > >
> > > On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila <[email protected]> wrote:
> > >
> > > > >
> > > > > I am a bit worried about the indirections that the wrappers and hooks
> > > > > create. Output plugins call OutputPluginUpdateProgress() in callbacks
> > > > > but I don't see why ReorderBufferProcessTXN() needs a callback to
> > > > > call OutputPluginUpdateProgress.
> > > > >
> > > >
> > > > Yeah, I think we can do it as we are doing the previous approach but
> > > > we need an additional wrapper (update_progress_cb_wrapper()) as the
> > > > current patch has so that we can add error context information. This
> > > > is similar to why we have a wrapper for all other callbacks like
> > > > change_cb_wrapper.
> > > >
> > >
> > > Ultimately OutputPluginUpdateProgress() will be called - which in turn
> > > will call ctx->update_progress.
> > >
> >
> > No, update_progress_cb_wrapper() should directly call
> > ctx->update_progress(). The key reason to have a
> > update_progress_cb_wrapper() is that it allows us to add error context
> > information (see the usage of output_plugin_error_callback).
>
> I think it makes sense. This also avoids the need for every output plugin to
> implement the callback. So I tried to improve the patch based on this approach.
>
> And I tried to add some comments for this new callback to distinguish it from
> ctx->update_progress.
Comments don't help when using cscope or some such code browsing tool.
Better to use a different variable name.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-18 12:30 Amit Kapila <[email protected]>
parent: Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Amit Kapila @ 2023-01-18 12:30 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: [email protected] <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Wed, Jan 18, 2023 at 5:37 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Wed, Jan 18, 2023 at 1:49 PM [email protected]
> <[email protected]> wrote:
> >
> > On Wed, Jan 18, 2023 at 13:29 PM Amit Kapila <[email protected]> wrote:
> > > On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat
> > > <[email protected]> wrote:
> > > >
> > > > On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila <[email protected]> wrote:
> > > >
> > > > > >
> > > > > > I am a bit worried about the indirections that the wrappers and hooks
> > > > > > create. Output plugins call OutputPluginUpdateProgress() in callbacks
> > > > > > but I don't see why ReorderBufferProcessTXN() needs a callback to
> > > > > > call OutputPluginUpdateProgress.
> > > > > >
> > > > >
> > > > > Yeah, I think we can do it as we are doing the previous approach but
> > > > > we need an additional wrapper (update_progress_cb_wrapper()) as the
> > > > > current patch has so that we can add error context information. This
> > > > > is similar to why we have a wrapper for all other callbacks like
> > > > > change_cb_wrapper.
> > > > >
> > > >
> > > > Ultimately OutputPluginUpdateProgress() will be called - which in turn
> > > > will call ctx->update_progress.
> > > >
> > >
> > > No, update_progress_cb_wrapper() should directly call
> > > ctx->update_progress(). The key reason to have a
> > > update_progress_cb_wrapper() is that it allows us to add error context
> > > information (see the usage of output_plugin_error_callback).
> >
> > I think it makes sense. This also avoids the need for every output plugin to
> > implement the callback. So I tried to improve the patch based on this approach.
> >
> > And I tried to add some comments for this new callback to distinguish it from
> > ctx->update_progress.
>
> Comments don't help when using cscope or some such code browsing tool.
> Better to use a different variable name.
>
+ /*
+ * Callback to be called when updating progress during sending data of a
+ * transaction (and its subtransactions) to the output plugin.
+ */
+ ReorderBufferUpdateProgressCB update_progress;
Are you suggesting changing the name of the above variable? If so, how
about apply_progress, progress, or updateprogress? If you don't like
any of these then feel free to suggest something else. If we change
the variable name then accordingly, we need to update
ReorderBufferUpdateProgressCB as well.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-19 10:43 Ashutosh Bapat <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 25+ messages in thread
From: Ashutosh Bapat @ 2023-01-19 10:43 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Wed, Jan 18, 2023 at 6:00 PM Amit Kapila <[email protected]> wrote:
> + */
> + ReorderBufferUpdateProgressCB update_progress;
>
> Are you suggesting changing the name of the above variable? If so, how
> about apply_progress, progress, or updateprogress? If you don't like
> any of these then feel free to suggest something else. If we change
> the variable name then accordingly, we need to update
> ReorderBufferUpdateProgressCB as well.
>
I would liked to have all the callback names renamed with prefix
"rbcb_xxx" so that they have very less chances of conflicting with
similar names in the code base. But it's probably late to do that :).
How are update_txn_progress since the CB is supposed to be used only
within a transaction? or update_progress_txn?
update_progress_cb_wrapper needs a change of name as well.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: Logical replication timeout problem
@ 2023-01-19 11:37 Amit Kapila <[email protected]>
parent: Ashutosh Bapat <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Amit Kapila @ 2023-01-19 11:37 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: [email protected] <[email protected]>; Fabrice Chapuis <[email protected]>; Euler Taveira <[email protected]>; Masahiko Sawada <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Simon Riggs <[email protected]>; Petr Jelinek <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Ajin Cherian <[email protected]>
On Thu, Jan 19, 2023 at 4:13 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Wed, Jan 18, 2023 at 6:00 PM Amit Kapila <[email protected]> wrote:
>
> > + */
> > + ReorderBufferUpdateProgressCB update_progress;
> >
> > Are you suggesting changing the name of the above variable? If so, how
> > about apply_progress, progress, or updateprogress? If you don't like
> > any of these then feel free to suggest something else. If we change
> > the variable name then accordingly, we need to update
> > ReorderBufferUpdateProgressCB as well.
> >
>
> I would liked to have all the callback names renamed with prefix
> "rbcb_xxx" so that they have very less chances of conflicting with
> similar names in the code base. But it's probably late to do that :).
>
> How are update_txn_progress since the CB is supposed to be used only
> within a transaction? or update_progress_txn?
>
Personally, I would prefer 'apply_progress' as it would be similar to
a few other callbacks like apply_change, apply_truncate, or as is
proposed by patch update_progress again because it is similar to
existing callbacks like commit_prepared. If you and others don't like
any of those then we can go for 'update_progress_txn' as well. Anybody
else has an opinion on this?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v3] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 5f77f3cac29..15bda6c6413 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 61cee42daa7..e24cb7a6f01 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index 568e0e150bf..3d641dc0403 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..26c3324542c 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} LockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ LockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(LockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ LockFile *lock_file = (LockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1969d467c1d..185d69b5520 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1653,6 +1653,7 @@ LocationLen
LockAcquireResult
LockClauseStrength
LockData
+LockFile
LockInfoData
LockInstanceData
LockMethod
base-commit: 031904048aa22e7c70dc8e9c170e2743f9b0f090
--
2.52.0
--zdel3ow7bygx53fm--
^ permalink raw reply [nested|flat] 25+ messages in thread
* [PATCH v4] Use open file description locks for lockfiles
@ 2025-12-18 17:21 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 25+ messages in thread
From: Dmitrii Dolgov @ 2025-12-18 17:21 UTC (permalink / raw)
When starting up, postmaster checks for an existing data directory lockfile. If
this file contains current process PID, it's assumed to be stale. Turns out
there is another possibility: we might be running in a PID namespace, and there
is another postgres running inside another PID namespace using the same data
directory. The result is that we don't see another process due to namespace
isolation and start concurrently with the other.
To prevent such situations, at startup use fcntl to get an exclusive open file
description lock for data directory lockfile. Since such locks are associated
with open file descriptors, meaning they're not affected by PID namespace
isolation. It's a "best effort" locking, intended to work with already existing
mechanism, not replace it.
This approach was discussed multiple times in the past, and usually was
rejected as the main work horse for the data directory lockfile due to:
* Portability issues. Open file description lock was a non-POSIX extension in
Linux and similar flock is from BSD standard. But looks like everybody agrees
that such locks make more sense than a typical advisory locks, and
F_OFD_SETLK made its way into POSIX.1 2024 [1].
* Issues with NFS. The current state of things here looks like this:
- NFSv3 doesn't implement open file description locks, they're converted to
advisory locks instead. Advisory locks are subject to namespace isolation,
meaning that processes in different PID namespaces will not see each other
advisory lock, and it's still possible to run multiple postgres
instances on the same data directory.
- NFSv4 uses a lease system for locking, I haven't found any mention of
conversion to advisory locks neither in the man page nor in RFC [2].
To summarize, the approach is now considered POSIX and should fix the described
problem everywhere, except NFSv3.
Use open file description lock for both data directory and socker
lockfiles, since both are affected in the same way.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fcntl.html
[2]: https://www.rfc-editor.org/rfc/rfc7530
Reviewed-by: Ilmar Yunusov <[email protected]>
---
configure | 14 +++
configure.ac | 3 +
meson.build | 1 +
src/backend/utils/init/miscinit.c | 136 ++++++++++++++++++++++++------
src/include/pg_config.h.in | 4 +
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 134 insertions(+), 25 deletions(-)
diff --git a/configure b/configure
index 35b0b72f0a7..25ebcd3cc47 100755
--- a/configure
+++ b/configure
@@ -16444,6 +16444,20 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# Linux open file descriptor locks
+ac_fn_c_check_decl "$LINENO" "F_OFD_SETLK" "ac_cv_have_decl_F_OFD_SETLK" "#include <fcntl.h>
+"
+if test "x$ac_cv_have_decl_F_OFD_SETLK" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_F_OFD_SETLK $ac_have_decl
+_ACEOF
+
+
ac_fn_c_check_func "$LINENO" "explicit_bzero" "ac_cv_func_explicit_bzero"
if test "x$ac_cv_func_explicit_bzero" = xyes; then :
$as_echo "#define HAVE_EXPLICIT_BZERO 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 0e624fe36b9..677137207e7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1913,6 +1913,9 @@ AC_CHECK_DECLS([memset_s], [], [], [#define __STDC_WANT_LIB_EXT1__ 1
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
+# Linux open file descriptor locks
+AC_CHECK_DECLS([F_OFD_SETLK], [], [], [#include <fcntl.h>])
+
AC_REPLACE_FUNCS(m4_normalize([
explicit_bzero
getopt
diff --git a/meson.build b/meson.build
index d88a7a70308..153fbb477bb 100644
--- a/meson.build
+++ b/meson.build
@@ -2901,6 +2901,7 @@ decl_checks = [
['strlcpy', 'string.h'],
['strsep', 'string.h'],
['timingsafe_bcmp', 'string.h'],
+ ['F_OFD_SETLK', 'fcntl.h'],
]
# Need to check for function declarations for these functions, because
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7ffc808073a..d4c2f80eb46 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -69,6 +69,15 @@ static List *lock_files = NIL;
static Latch LocalLatchData;
+typedef struct
+{
+ /* LockFile name. */
+ const char *filename;
+
+ /* File descriptor for open file description lock. */
+ int fd;
+} PGLockFile;
+
/* ----------------------------------------------------------------
* ignoring system indexes support stuff
*
@@ -1119,6 +1128,48 @@ RestoreClientConnectionInfo(char *conninfo)
*-------------------------------------------------------------------------
*/
+/*
+ * OFD lock the specified lockfile.
+ *
+ * Lock the lockfile with an open file description lock. If the lock is already
+ * taken, it's a hard stop. It's only a best effort test, and any other errors
+ * are ignored. On succes the file descriptor is duplicated, to make sure there
+ * will be at least one open copy of it to keep the lock.
+ *
+ * filename argument is used only for reporting purposes.
+ */
+static int
+OFDLockFile(int fd, const char *filename)
+{
+#if HAVE_DECL_F_OFD_SETLK
+ struct flock lock;
+
+ lock.l_type = F_WRLCK;
+ lock.l_whence = SEEK_SET;
+ lock.l_start = 0;
+ lock.l_len = 0;
+ lock.l_pid = 0;
+
+ if (fcntl(fd, F_OFD_SETLK, &lock) == -1)
+ {
+ if (errno == EAGAIN)
+ ereport(FATAL,
+ (errcode(ERRCODE_LOCK_FILE_EXISTS),
+ errmsg("cannot lock the lock file \"%s\"", filename),
+ errhint("Another server is starting.")));
+ else
+ {
+ elog(WARNING, "Failed locking file \"%s\", %m", filename);
+ return -1;
+ }
+ }
+ else
+ return dup(fd);
+#else
+ return -1;
+#endif
+}
+
/*
* proc_exit callback to remove lockfiles.
*/
@@ -1129,9 +1180,16 @@ UnlinkLockFiles(int status, Datum arg)
foreach(l, lock_files)
{
- char *curfile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
- unlink(curfile);
+ /*
+ * Close the file descriptor, which keeps the open file description
+ * lock.
+ */
+ if (lock_file->fd > 0)
+ close(lock_file->fd);
+
+ unlink(lock_file->filename);
/* Should we complain if the unlink fails? */
}
/* Since we're about to exit, no need to reclaim storage */
@@ -1161,7 +1219,9 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *socketDir,
bool isDDLock, const char *refName)
{
- int fd;
+ int fd,
+ flock_fd = -1;
+ PGLockFile *lock_file;
char buffer[MAXPGPATH * 2 + 256];
int ntries;
int len;
@@ -1173,22 +1233,32 @@ CreateLockFile(const char *filename, bool amPostmaster,
const char *envvar;
/*
- * If the PID in the lockfile is our own PID or our parent's or
- * grandparent's PID, then the file must be stale (probably left over from
- * a previous system boot cycle). We need to check this because of the
- * likelihood that a reboot will assign exactly the same PID as we had in
- * the previous reboot, or one that's only one or two counts larger and
- * hence the lockfile's PID now refers to an ancestor shell process. We
- * allow pg_ctl to pass down its parent shell PID (our grandparent PID)
- * via the environment variable PG_GRANDPARENT_PID; this is so that
- * launching the postmaster via pg_ctl can be just as reliable as
- * launching it directly. There is no provision for detecting
- * further-removed ancestor processes, but if the init script is written
- * carefully then all but the immediate parent shell will be root-owned
- * processes and so the kill test will fail with EPERM. Note that we
- * cannot get a false negative this way, because an existing postmaster
- * would surely never launch a competing postmaster or pg_ctl process
- * directly.
+ * If we find an already existing lockfile containing our own PID, there
+ * are few options:
+ *
+ * - There is another process, that we don't see due to PID namespace
+ * isolation, which is already running in this data directory.
+ *
+ * To prevent two concurrent processes working with the same data
+ * directory, we first try to lock the lockfile exclusively.
+ *
+ * - The file must be stale, probably left over from a previous system
+ * boot cycle. The same if the lockfile contains our parent's or
+ * grandparent's PID.
+ *
+ * We need to check this because of the likelihood that a reboot will
+ * assign exactly the same PID as we had in the previous reboot, or one
+ * that's only one or two counts larger and hence the lockfile's PID now
+ * refers to an ancestor shell process. We allow pg_ctl to pass down its
+ * parent shell PID (our grandparent PID) via the environment variable
+ * PG_GRANDPARENT_PID; this is so that launching the postmaster via pg_ctl
+ * can be just as reliable as launching it directly. There is no
+ * provision for detecting further-removed ancestor processes, but if the
+ * init script is written carefully then all but the immediate parent
+ * shell will be root-owned processes and so the kill test will fail with
+ * EPERM. Note that we cannot get a false negative this way, because an
+ * existing postmaster would surely never launch a competing postmaster or
+ * pg_ctl process directly.
*/
my_pid = getpid();
@@ -1224,7 +1294,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
*/
fd = open(filename, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
if (fd >= 0)
- break; /* Success; exit the retry loop */
+ {
+ /* Success; lock and exit the retry loop */
+ flock_fd = OFDLockFile(fd, filename);
+ break;
+ }
/*
* Couldn't create the pid file. Probably it already exists.
@@ -1238,8 +1312,12 @@ CreateLockFile(const char *filename, bool amPostmaster,
/*
* Read the file to get the old owner's PID. Note race condition
* here: file might have been deleted since we tried to create it.
+ *
+ * We're going to use the same fd for flock, and want to create a
+ * write lock for the latter one. Since both fd and the lock have to
+ * be of the same type, open the file for read and write.
*/
- fd = open(filename, O_RDONLY, pg_file_create_mode);
+ fd = open(filename, O_RDWR, pg_file_create_mode);
if (fd < 0)
{
if (errno == ENOENT)
@@ -1249,6 +1327,10 @@ CreateLockFile(const char *filename, bool amPostmaster,
errmsg("could not open lock file \"%s\": %m",
filename)));
}
+
+ /* Try to lock the file. We stop here, if it's already locked. */
+ flock_fd = OFDLockFile(fd, filename);
+
pgstat_report_wait_start(WAIT_EVENT_LOCK_FILE_CREATE_READ);
if ((len = read(fd, buffer, sizeof(buffer) - 1)) < 0)
ereport(FATAL,
@@ -1448,7 +1530,11 @@ CreateLockFile(const char *filename, bool amPostmaster,
* Use lcons so that the lock files are unlinked in reverse order of
* creation; this is critical!
*/
- lock_files = lcons(pstrdup(filename), lock_files);
+ lock_file = palloc0_object(PGLockFile);
+ lock_file->filename = pstrdup(filename);
+ lock_file->fd = flock_fd;
+
+ lock_files = lcons(lock_file, lock_files);
}
/*
@@ -1495,14 +1581,14 @@ TouchSocketLockFiles(void)
foreach(l, lock_files)
{
- char *socketLockFile = (char *) lfirst(l);
+ PGLockFile *lock_file = (PGLockFile *) lfirst(l);
/* No need to touch the data directory lock file, we trust */
- if (strcmp(socketLockFile, DIRECTORY_LOCK_FILE) == 0)
+ if (strcmp(lock_file->filename, DIRECTORY_LOCK_FILE) == 0)
continue;
/* we just ignore any error here */
- (void) utime(socketLockFile, NULL);
+ (void) utime(lock_file->filename, NULL);
}
}
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 4f8113c144b..cc38c06dc13 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -85,6 +85,10 @@
don't. */
#undef HAVE_DECL_F_FULLFSYNC
+/* Define to 1 if you have the declaration of `F_OFD_SETLK', and to 0 if you
+ don't. */
+#undef HAVE_DECL_F_OFD_SETLK
+
/* Define to 1 if you have the declaration of `memset_s', and to 0 if you
don't. */
#undef HAVE_DECL_MEMSET_S
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..ad34142e7d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1956,6 +1956,7 @@ PGIOAlignedBlock
PGLZ_HistEntry
PGLZ_Strategy
PGLoadBalanceType
+PGLockFile
PGMessageField
PGModuleMagicFunction
PGNoticeHooks
base-commit: 73dfe79fd6034b1e7e41e83d9c82c166dba8eb67
--
2.52.0
--tks2nr37zts5e6h7--
^ permalink raw reply [nested|flat] 25+ messages in thread
end of thread, other threads:[~2025-12-18 17:21 UTC | newest]
Thread overview: 25+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-01-16 16:36 Re: Logical replication timeout problem Ashutosh Bapat <[email protected]>
2023-01-17 10:04 ` Amit Kapila <[email protected]>
2023-01-17 13:11 ` Ashutosh Bapat <[email protected]>
2023-01-18 05:28 ` Amit Kapila <[email protected]>
2023-01-18 08:19 ` [email protected] <[email protected]>
2023-01-18 12:07 ` Ashutosh Bapat <[email protected]>
2023-01-18 12:30 ` Amit Kapila <[email protected]>
2023-01-19 10:43 ` Ashutosh Bapat <[email protected]>
2023-01-19 11:37 ` Amit Kapila <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v4] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
2025-12-18 17:21 [PATCH v3] Use open file description locks for lockfiles Dmitrii Dolgov <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox