From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Andrew Dunstan <andrew@dunslane.net>
Cc: Kevin McKibbin <kevinmckibbin123@gmail.com>
Cc: pgsql-performance@lists.postgresql.org
Subject: Re: pgbench: could not connect to server: Resource temporarily unavailable
Date: Mon, 22 Aug 2022 12:57:43 -0400
Message-ID: <361175.1661187463@sss.pgh.pa.us> (raw)
In-Reply-To: <114075.1661136913@sss.pgh.pa.us>
References: <CADc_NKg2d+oZY9mg4DdQdoUcGzN2kOYXBu-3--RW_hEe0tUV=g@mail.gmail.com>
<3909995.1661052002@sss.pgh.pa.us>
<35d221aa-3da4-f744-3965-ceca7f71f4ea@dunslane.net>
<8864.1661116501@sss.pgh.pa.us>
<83134bb1-3a21-059f-1cd6-4997de2652b8@dunslane.net>
<33898.1661118503@sss.pgh.pa.us>
<CA+hUKGLTNSJrKYByirsSyqodmp28McpNUCL5-gvCLXtOj0RwBQ@mail.gmail.com>
<68893.1661122531@sss.pgh.pa.us>
<CA+hUKG+MxvFtEYFeSfJZoUUmNSR=fZ=fffWUshU8eLg+CFki+A@mail.gmail.com>
<84184.1661127637@sss.pgh.pa.us>
<CA+hUKGKPyXKf2jrnSUMKc8XvRTYs+kkiZY9GA6nMdMUgLG6EaQ@mail.gmail.com>
<105986.1661134693@sss.pgh.pa.us>
<CA+hUKGLP80EUDitSKLfgr1rS2pBtr91VY-3GuTbduYPkQ+DYwQ@mail.gmail.com>
<114075.1661136913@sss.pgh.pa.us>
OK, here's some proposed patches.
0001 adds a para about how to raise the listen queue length.
0002 isn't quite related, but while writing 0001 I noticed a nearby
use of /proc/sys/... which I thought should be converted to sysctl.
IMO /proc/sys pretty much sucks, at least for documentation purposes,
for multiple reasons:
* It's unlike the way you do things on other platforms.
* "man sysctl" will lead you to useful documentation about how to
use that command. There's no obvious way to find documentation
about /proc/sys.
* It's not at all sudo-friendly. Compare
sudo sh -c 'echo 0 >/proc/sys/kernel/randomize_va_space'
sudo sysctl -w kernel.randomize_va_space=0
The former is a lot longer and it's far from obvious why you have
to do it that way.
* You have to think in sysctl terms anyway if you want to make the
setting persist across reboots, which you almost always do.
* Everywhere else in runtime.sgml, we use sysctl not /proc/sys.
0003 removes PG_SOMAXCONN. While doing that I noticed that this
computation hadn't been touched throughout all the various
changes fooling with exactly what gets counted in MaxBackends.
I think the most appropriate definition for the listen queue
length is now MaxConnections * 2, not MaxBackends * 2, because
the other processes counted in MaxBackends don't correspond to
incoming connections.
I propose 0003 for HEAD only, but the docs changes could be
back-patched.
regards, tom lane
Attachments:
[text/x-diff] 0001-document-somaxconn.patch (1.2K, ../361175.1661187463@sss.pgh.pa.us/2-0001-document-somaxconn.patch)
download | inline diff:diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgmlindex 963b18ed85..1192faa6ae 100644--- a/doc/src/sgml/runtime.sgml+++ b/doc/src/sgml/runtime.sgml@@ -1298,6 +1298,22 @@ default:\
linkend="guc-max-files-per-process"/> configuration parameter to
limit the consumption of open files.
</para>
++ <para>+ Another kernel limit that may be of concern when supporting large+ numbers of client connections is the maximum socket connection queue+ length. If more than that many connection requests arrive within a+ very short period, some may get rejected before the postmaster can+ service the requests, with those clients receiving unhelpful+ connection failure errors such as <quote>Resource temporarily+ unavailable</quote>. The default queue length limit is 128 on many+ platforms. To raise it, adjust the appropriate kernel parameter+ via <application>sysctl</application>, then restart the postmaster.+ The parameter is variously named <varname>net.core.somaxconn</varname>+ on Linux, <varname>kern.ipc.soacceptqueue</varname> on newer FreeBSD,+ and <varname>kern.ipc.somaxconn</varname> on macOS and other BSD+ variants.+ </para>
</sect2>
<sect2 id="linux-memory-overcommit">
[text/x-diff] 0002-avoid-proc-sys-in-docs.patch (1.7K, ../361175.1661187463@sss.pgh.pa.us/3-0002-avoid-proc-sys-in-docs.patch)
download | inline diff:diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgmlindex 963b18ed85..1192faa6ae 100644--- a/doc/src/sgml/runtime.sgml+++ b/doc/src/sgml/runtime.sgml@@ -1258,11 +1258,12 @@ default:\
<itemizedlist>
<listitem>
<para>
- On <productname>Linux</productname>- <filename>/proc/sys/fs/file-max</filename> determines the- maximum number of open files that the kernel will support. It can- be changed by writing a different number into the file or by- adding an assignment in <filename>/etc/sysctl.conf</filename>.+ On <productname>Linux</productname> the kernel parameter+ <varname>fs.file-max</varname> determines the maximum number of open+ files that the kernel will support. It can be changed with+ <literal>sysctl -w fs.file-max=<replaceable>N</replaceable></literal>.+ To make the setting persist across reboots, add an assignment+ in <filename>/etc/sysctl.conf</filename>.
The maximum limit of files per process is fixed at the time the
kernel is compiled; see
<filename>/usr/src/linux/Documentation/proc.txt</filename> for
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.cindex 8a038d1b2a..1664fcee2a 100644--- a/src/backend/postmaster/postmaster.c+++ b/src/backend/postmaster/postmaster.c@@ -4891,7 +4891,7 @@ SubPostmasterMain(int argc, char *argv[])
* If testing EXEC_BACKEND on Linux, you should run this as root before
* starting the postmaster:
*
- * echo 0 >/proc/sys/kernel/randomize_va_space+ * sysctl -w kernel.randomize_va_space=0
*
* This prevents using randomized stack and code addresses that cause the
* child process's memory map to be different from the parent's, making it
[text/x-diff] 0003-remove-PG_SOMAXCONN.patch (1.6K, ../361175.1661187463@sss.pgh.pa.us/4-0003-remove-PG_SOMAXCONN.patch)
download | inline diff:diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.cindex 8ff3be611d..7112e9751b 100644--- a/src/backend/libpq/pqcomm.c+++ b/src/backend/libpq/pqcomm.c@@ -537,13 +537,11 @@ StreamServerPort(int family, const char *hostName, unsigned short portNumber,
}
/*
- * Select appropriate accept-queue length limit. PG_SOMAXCONN is only- * intended to provide a clamp on the request on platforms where an- * overly large request provokes a kernel error (are there any?).+ * Select appropriate accept-queue length limit. It seems reasonable+ * to use a value similar to the maximum number of child processes+ * that the postmaster will permit.
*/
- maxconn = MaxBackends * 2;- if (maxconn > PG_SOMAXCONN)- maxconn = PG_SOMAXCONN;+ maxconn = MaxConnections * 2;
err = listen(fd, maxconn);
if (err < 0)
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.hindex 844c3e0f09..f2a106f983 100644--- a/src/include/pg_config_manual.h+++ b/src/include/pg_config_manual.h@@ -114,17 +114,6 @@
*/
#define MAXPGPATH 1024
-/*- * PG_SOMAXCONN: maximum accept-queue length limit passed to- * listen(2). You'd think we should use SOMAXCONN from- * <sys/socket.h>, but on many systems that symbol is much smaller- * than the kernel's actual limit. In any case, this symbol need be- * twiddled only if you have a kernel that refuses large limit values,- * rather than silently reducing the value to what it can handle- * (which is what most if not all Unixen do).- */-#define PG_SOMAXCONN 10000-
/*
* You can try changing this if you have a machine with bytes of
* another size, but no guarantee...
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tgl@sss.pgh.pa.us, thomas.munro@gmail.com, andrew@dunslane.net, kevinmckibbin123@gmail.com, pgsql-performance@lists.postgresql.org
Subject: Re: pgbench: could not connect to server: Resource temporarily unavailable
In-Reply-To: <361175.1661187463@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox