public inbox for [email protected]  

help / color / mirror / Atom feed

From: Andres Freund <[email protected]>
To: Tom Lane <[email protected]>
Cc: Tomas Vondra <[email protected]>
Cc: Jelte Fennema-Nio <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Subject: Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup
Date: Tue, 11 Feb 2025 18:13:26 -0500
Message-ID: <v43qmocsld2vbbpylns5yfqnk6w2vqdigly2hwnctr7pior53d@3xdpzthtrzqq> (raw)
In-Reply-To: <[email protected]>
References: <CAGECzQQh6VSy3KG4pN1d=h9J=D1rStFCMR+t7yh_Kwj-g87aLQ@mail.gmail.com>
	<[email protected]>
	<[email protected]>
	<7u7dbn6s2i6bf3hjzkbqaexj2bpoblqxwbkffbetl4rjv6dcom@s2uickjc5z53>
	<[email protected]>
	<xvd2cyrtd4wk42ugweydxfcy3bwtaymu4gqmky5fpfcu6xia4m@qbgeq23yncch>
	<[email protected]>

Hi,

On 2025-02-11 17:55:39 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > So the issue would actually be that we're currently doing set_max_safe_fds()
> > too late, not too early :/
> 
> Well, we'd rather set_max_safe_fds happen after semaphore creation,
> so that it doesn't have to be explicitly aware of whether semaphores
> consume FDs.  Could we have it be aware of how many FDs *will be*
> needed for io_uring, but postpone creation of those until after we
> jack up RLIMIT_NOFILE?

Yes, that should be fairly trivial to do. It'd require a call from postmaster
into the aio subsystem, after set_max_safe_fds() is done, but I think that'd
be ok.  I'd be inclined to not make that a general mechanism for now.


> I guess the other way would be to have two rounds of RLIMIT_NOFILE
> adjustment, before and after shmem creation.  That seems ugly but
> shouldn't be very time-consuming.

Yea, something like that could also work.

At some point I was playing with calling AcquireExternalFD() the appropriate
number of times during shmem reservation. That turned out to work badly right
now, because it relies on max_safe_fds() being set *and* insists on
numExternalFDs < max_safe_fds / 3.

One way to deal with this would be for AcquireExternalFD() to first increase
the rlimit, if necessary and possible, then start to close LRU files and only
then fail.

Arguably that'd have the advantage of e.g. postgres_fdw not stomping quite so
hard on VFDs, because we'd first increase the limit.

Of course it'd also mean we'd increase the limit more often. But I don't think
it's particularly expensive.

Greetings,

Andres Freund

view thread (16+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup
  In-Reply-To: <v43qmocsld2vbbpylns5yfqnk6w2vqdigly2hwnctr7pior53d@3xdpzthtrzqq>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox