MIME-Version: 1.0
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Tue, 11 Feb 2025 19:52:34 +0100
Message-ID: <CAGECzQQh6VSy3KG4pN1d=h9J=D1rStFCMR+t7yh_Kwj-g87aLQ@mail.gmail.com>
Subject: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: Andres Freund <andres@anarazel.de>
Content-Type: multipart/mixed; boundary="000000000000a18be3062de25444"
Archived-At: <https://www.postgresql.org/message-id/CAGECzQQh6VSy3KG4pN1d%3Dh9J%3DD1rStFCMR%2Bt7yh_Kwj-g87aLQ%40mail.gmail.com>
Precedence: bulk

--000000000000a18be3062de25444
Content-Type: text/plain; charset="UTF-8"

The default open file limit of 1024 is extremely low, given modern
resources and kernel architectures. The reason that this hasn't changed
change is because doing so would break legacy programs that use the
select(2) system call in hard to debug ways. So instead programs that
want to opt-in to a higher open file limit are expected to bump their
soft limit to their hard limit on startup. Details on this are very well
explained in a blogpost by the systemd author[1]. There's also a similar
change done by the Go language[2].

So this starts bumping postmaster and pgbench its soft open file limit
to the hard open file limit. Doing so is especially useful for the AIO
work that Andres is doing, because io_uring consumes a lot of file
descriptors. But even without looking at AIO there is a large number of
reports from people that require changing their soft file limit before
starting postgres, sometimes falling back to lowering
max_files_per_process when they fail to do so[3-8]. It's also not all
that strange to fail at setting the soft open file limit because there
are multiple places where one can configure such limits and usually only
one of them is effective (which one depends on how Postgres is started).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9

--000000000000a18be3062de25444
Content-Type: text/x-patch; charset="US-ASCII"; 
	name="v1-0001-Bump-soft-open-file-limit-RLIMIT_NOFILE-to-hard-l.patch"
Content-Disposition: attachment; 
	filename="v1-0001-Bump-soft-open-file-limit-RLIMIT_NOFILE-to-hard-l.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_m70ua7o40>
X-Attachment-Id: f_m70ua7o40

RnJvbSBlY2Y3ZmM1ZjM0YjRlMDI1ZWFhNGRkYTgwOTg3NGYxOGRmZGIyNGI0IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBKZWx0ZSBGZW5uZW1hLU5pbyA8Z2l0aHViLXRlY2hAamVsdGVm
Lm5sPgpEYXRlOiBUdWUsIDExIEZlYiAyMDI1IDE5OjE1OjM2ICswMTAwClN1YmplY3Q6IFtQQVRD
SCB2MV0gQnVtcCBzb2Z0IG9wZW4gZmlsZSBsaW1pdCAoUkxJTUlUX05PRklMRSkgdG8gaGFyZCBs
aW1pdCBvbgogc3RhcnR1cAoKVGhlIGRlZmF1bHQgb3BlbiBmaWxlIGxpbWl0IG9mIDEwMjQgaXMg
ZXh0cmVtZWx5IGxvdywgZ2l2ZW4gbW9kZXJuCnJlc291cmNlcyBhbmQga2VybmVsIGFyY2hpdGVj
dHVyZXMuIFRoZSByZWFzb24gdGhhdCB0aGlzIGhhc24ndCBjaGFuZ2VkCmNoYW5nZSBpcyBiZWNh
dXNlIGRvaW5nIHNvIHdvdWxkIGJyZWFrIGxlZ2FjeSBwcm9ncmFtcyB0aGF0IHVzZSB0aGUKc2Vs
ZWN0KDIpIHN5c3RlbSBjYWxsIGluIGhhcmQgdG8gZGVidWcgd2F5cy4gU28gaW5zdGVhZCBwcm9n
cmFtcyB0aGF0CndhbnQgdG8gb3B0LWluIHRvIGEgaGlnaGVyIG9wZW4gZmlsZSBsaW1pdCBhcmUg
ZXhwZWN0ZWQgdG8gYnVtcCB0aGVpcgpzb2Z0IGxpbWl0IHRvIHRoZWlyIGhhcmQgbGltaXQgb24g
c3RhcnR1cC4gRGV0YWlscyBvbiB0aGlzIGFyZSB2ZXJ5IHdlbGwKZXhwbGFpbmVkIGluIGEgYmxv
Z3Bvc3QgYnkgdGhlIHN5c3RlbWQgYXV0aG9yWzFdLiBUaGVyZSdzIGFsc28gYSBzaW1pbGFyCmNo
YW5nZSBkb25lIGJ5IHRoZSBHbyBsYW5ndWFnZVsyXS4KClNvIHRoaXMgc3RhcnRzIGJ1bXBpbmcg
cG9zdG1hc3RlciBhbmQgcGdiZW5jaCBpdHMgc29mdCBvcGVuIGZpbGUgbGltaXQKdG8gdGhlIGhh
cmQgb3BlbiBmaWxlIGxpbWl0LiBEb2luZyBzbyBpcyBlc3BlY2lhbGx5IHVzZWZ1bCBmb3IgdGhl
IEFJTwp3b3JrIHRoYXQgQW5kcmVzIGlzIGRvaW5nLCBiZWNhdXNlIGlvX3VyaW5nIGNvbnN1bWVz
IGEgbG90IG9mIGZpbGUKZGVzY3JpcHRvcnMuIEJ1dCBldmVuIHdpdGhvdXQgbG9va2luZyBhdCBB
SU8gdGhlcmUgaXMgYSBsYXJnZSBudW1iZXIgb2YKcmVwb3J0cyBmcm9tIHBlb3BsZSB0aGF0IHJl
cXVpcmUgY2hhbmdpbmcgdGhlaXIgc29mdCBmaWxlIGxpbWl0IGJlZm9yZQpzdGFydGluZyBwb3N0
Z3Jlcywgc29tZXRpbWVzIGZhbGxpbmcgYmFjayB0byBsb3dlcmluZwptYXhfZmlsZXNfcGVyX3By
b2Nlc3Mgd2hlbiB0aGV5IGZhaWwgdG8gZG8gc29bMy04XS4gSXQncyBhbHNvIG5vdCBhbGwKdGhh
dCBzdHJhbmdlIHRvIGZhaWwgYXQgc2V0dGluZyB0aGUgc29mdCBvcGVuIGZpbGUgbGltaXQgYmVj
YXVzZSB0aGVyZQphcmUgbXVsdGlwbGUgcGxhY2VzIHdoZXJlIG9uZSBjYW4gY29uZmlndXJlIHN1
Y2ggbGltaXRzIGFuZCB1c3VhbGx5IG9ubHkKb25lIG9mIHRoZW0gaXMgZWZmZWN0aXZlICh3aGlj
aCBvbmUgZGVwZW5kcyBvbiBob3cgUG9zdGdyZXMgaXMgc3RhcnRlZCkuCgpbMV06IGh0dHBzOi8v
MHBvaW50ZXIubmV0L2Jsb2cvZmlsZS1kZXNjcmlwdG9yLWxpbWl0cy5odG1sClsyXTogaHR0cHM6
Ly9naXRodWIuY29tL2dvbGFuZy9nby9pc3N1ZXMvNDYyNzkKWzNdOiBodHRwczovL3NlcnZlcmZh
dWx0LmNvbS9xdWVzdGlvbnMvNzg1MzMwL2dldHRpbmctdG9vLW1hbnktb3Blbi1maWxlcy1lcnJv
ci1mb3ItcG9zdGdyZXMKWzRdOiBodHRwczovL3NlcnZlcmZhdWx0LmNvbS9xdWVzdGlvbnMvNzE2
OTgyL2hvdy10by1yYWlzZS1tYXgtbm8tb2YtZmlsZS1kZXNjcmlwdG9ycy1mb3ItZGFlbW9ucy1y
dW5uaW5nLW9uLWRlYmlhbi1qZXNzaWUKWzVdOiBodHRwczovL3d3dy5wb3N0Z3Jlc3FsLm9yZy9t
ZXNzYWdlLWlkL2ZsYXQvQ0FLdGM4dlhoN052UF9xV2o4RXFxb3JQWTk3YnZ4U2FYM2g1dTdhOVBw
dFJGSFc1eDdnJTQwbWFpbC5nbWFpbC5jb20KWzZdOiBodHRwczovL3d3dy5wb3N0Z3Jlc3FsLm9y
Zy9tZXNzYWdlLWlkL2ZsYXQvMTEzY2UzMWIwOTA4MTIwOTU1dzc3MDI5MDk5aTdlY2VjYzA1MzA4
NDA5NWElNDBtYWlsLmdtYWlsLmNvbQpbN106IGh0dHBzOi8vZ2l0aHViLmNvbS9hYmlvc29mdC9j
b2xpbWEvZGlzY3Vzc2lvbnMvODM2Cls4XTogaHR0cHM6Ly93d3cucG9zdGdyZXNxbC5vcmcvbWVz
c2FnZS1pZC9mbGF0LzI5NjYzLjEwMDc3Mzg5NTclNDBzc3MucGdoLnBhLnVzIzIwNzllYzllMmQ4
YjI1MTU5MzgxMmEzNzExYmZlOWU5Ci0tLQogc3JjL2JhY2tlbmQvcG9zdG1hc3Rlci9wb3N0bWFz
dGVyLmMgfCAyMiArKysrKysrKysrKysrKysrKysrKysrCiBzcmMvYmluL3BnYmVuY2gvcGdiZW5j
aC5jICAgICAgICAgICB8ICA1ICsrKysrCiAyIGZpbGVzIGNoYW5nZWQsIDI3IGluc2VydGlvbnMo
KykKCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9wb3N0bWFzdGVyL3Bvc3RtYXN0ZXIuYyBiL3Ny
Yy9iYWNrZW5kL3Bvc3RtYXN0ZXIvcG9zdG1hc3Rlci5jCmluZGV4IGJiMjJiMTNhZGVmLi4zYzJm
YmViZmE2MiAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvcG9zdG1hc3Rlci9wb3N0bWFzdGVyLmMK
KysrIGIvc3JjL2JhY2tlbmQvcG9zdG1hc3Rlci9wb3N0bWFzdGVyLmMKQEAgLTcyLDYgKzcyLDcg
QEAKICNpbmNsdWRlIDxjdHlwZS5oPgogI2luY2x1ZGUgPHN5cy9zdGF0Lmg+CiAjaW5jbHVkZSA8
c3lzL3NvY2tldC5oPgorI2luY2x1ZGUgPHN5cy9yZXNvdXJjZS5oPgogI2luY2x1ZGUgPGZjbnRs
Lmg+CiAjaW5jbHVkZSA8c3lzL3BhcmFtLmg+CiAjaW5jbHVkZSA8bmV0ZGIuaD4KQEAgLTQwNiw2
ICs0MDcsOCBAQCBzdGF0aWMgRE5TU2VydmljZVJlZiBib25qb3VyX3NkcmVmID0gTlVMTDsKIC8q
CiAgKiBwb3N0bWFzdGVyLmMgLSBmdW5jdGlvbiBwcm90b3R5cGVzCiAgKi8KKworc3RhdGljIHZv
aWQgQ29uZmlndXJlRmlsZUxpbWl0KHZvaWQpOwogc3RhdGljIHZvaWQgQ2xvc2VTZXJ2ZXJQb3J0
cyhpbnQgc3RhdHVzLCBEYXR1bSBhcmcpOwogc3RhdGljIHZvaWQgdW5saW5rX2V4dGVybmFsX3Bp
ZF9maWxlKGludCBzdGF0dXMsIERhdHVtIGFyZyk7CiBzdGF0aWMgdm9pZCBnZXRJbnN0YWxsYXRp
b25QYXRocyhjb25zdCBjaGFyICphcmd2MCk7CkBAIC01NzMsNiArNTc2LDggQEAgUG9zdG1hc3Rl
ck1haW4oaW50IGFyZ2MsIGNoYXIgKmFyZ3ZbXSkKIAkvKiBCZWdpbiBhY2NlcHRpbmcgc2lnbmFs
cy4gKi8KIAlzaWdwcm9jbWFzayhTSUdfU0VUTUFTSywgJlVuQmxvY2tTaWcsIE5VTEwpOwogCisJ
Q29uZmlndXJlRmlsZUxpbWl0KCk7CisKIAkvKgogCSAqIE9wdGlvbnMgc2V0dXAKIAkgKi8KQEAg
LTEzOTMsNiArMTM5OCwyMyBAQCBQb3N0bWFzdGVyTWFpbihpbnQgYXJnYywgY2hhciAqYXJndltd
KQogCWFib3J0KCk7CQkJCQkvKiBub3QgcmVhY2hlZCAqLwogfQogCitzdGF0aWMgdm9pZAorQ29u
ZmlndXJlRmlsZUxpbWl0KHZvaWQpCit7CisjaWZkZWYgSEFWRV9HRVRSTElNSVQKKwlzdHJ1Y3Qg
cmxpbWl0IHJsaW07CisKKwlpZiAoZ2V0cmxpbWl0KFJMSU1JVF9OT0ZJTEUsICZybGltKSAhPSAw
KQorCXsKKwkJZWxvZyhXQVJOSU5HLCAiY291bGQgbm90IGdldCBvcGVuIGZpbGUgbGltaXQ6ICVt
Iik7CisJCXJldHVybjsKKwl9CisKKwlybGltLnJsaW1fY3VyID0gcmxpbS5ybGltX21heDsKKwlp
ZiAoc2V0cmxpbWl0KFJMSU1JVF9OT0ZJTEUsICZybGltKSAhPSAwKQorCQllbG9nKFdBUk5JTkcs
ICJjb3VsZCBub3Qgc2V0IG9wZW4gZmlsZSBsaW1pdDogJW0iKTsKKyNlbmRpZgorfQogCiAvKgog
ICogb25fcHJvY19leGl0IGNhbGxiYWNrIHRvIGNsb3NlIHNlcnZlcidzIGxpc3RlbiBzb2NrZXRz
CmRpZmYgLS1naXQgYS9zcmMvYmluL3BnYmVuY2gvcGdiZW5jaC5jIGIvc3JjL2Jpbi9wZ2JlbmNo
L3BnYmVuY2guYwppbmRleCBmMzAzYmRlZWM4ZC4uMTYxYjIzMWQ1OTEgMTAwNjQ0Ci0tLSBhL3Ny
Yy9iaW4vcGdiZW5jaC9wZ2JlbmNoLmMKKysrIGIvc3JjL2Jpbi9wZ2JlbmNoL3BnYmVuY2guYwpA
QCAtNjc5NCw2ICs2Nzk0LDExIEBAIG1haW4oaW50IGFyZ2MsIGNoYXIgKiphcmd2KQogI2lmZGVm
IEhBVkVfR0VUUkxJTUlUCiAJCQkJaWYgKGdldHJsaW1pdChSTElNSVRfTk9GSUxFLCAmcmxpbSkg
PT0gLTEpCiAJCQkJCXBnX2ZhdGFsKCJnZXRybGltaXQgZmFpbGVkOiAlbSIpOworCisJCQkJcmxp
bS5ybGltX2N1ciA9IHJsaW0ucmxpbV9tYXg7CisJCQkJaWYgKHNldHJsaW1pdChSTElNSVRfTk9G
SUxFLCAmcmxpbSkgPT0gLTEpCisJCQkJCXBnX2ZhdGFsKCJnZXRybGltaXQgZmFpbGVkOiAlbSIp
OworCiAJCQkJaWYgKHJsaW0ucmxpbV9jdXIgPCBuY2xpZW50cyArIDMpCiAJCQkJewogCQkJCQlw
Z19sb2dfZXJyb3IoIm5lZWQgYXQgbGVhc3QgJWQgb3BlbiBmaWxlcywgYnV0IHN5c3RlbSBsaW1p
dCBpcyAlbGQiLAoKYmFzZS1jb21taXQ6IGMzNjZkMmJkYmE3YzNiOWIyY2NhMTQyOWQ0NTM1ODY2
ZTIzMWNhNTUKLS0gCjIuNDMuMAoK
--000000000000a18be3062de25444--