Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oQAkM-0001lR-Kv for pgsql-performance@arkaria.postgresql.org; Mon, 22 Aug 2022 16:57:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oQAkL-0000g8-8t for pgsql-performance@arkaria.postgresql.org; Mon, 22 Aug 2022 16:57:53 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oQAkK-0000fy-P0 for pgsql-performance@lists.postgresql.org; Mon, 22 Aug 2022 16:57:52 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oQAkD-0004UU-Oe for pgsql-performance@lists.postgresql.org; Mon, 22 Aug 2022 16:57:51 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 27MGvhLS361176; Mon, 22 Aug 2022 12:57:43 -0400 From: Tom Lane To: Thomas Munro cc: Andrew Dunstan , Kevin McKibbin , pgsql-performance@lists.postgresql.org Subject: Re: pgbench: could not connect to server: Resource temporarily unavailable In-reply-to: <114075.1661136913@sss.pgh.pa.us> References: <3909995.1661052002@sss.pgh.pa.us> <35d221aa-3da4-f744-3965-ceca7f71f4ea@dunslane.net> <8864.1661116501@sss.pgh.pa.us> <83134bb1-3a21-059f-1cd6-4997de2652b8@dunslane.net> <33898.1661118503@sss.pgh.pa.us> <68893.1661122531@sss.pgh.pa.us> <84184.1661127637@sss.pgh.pa.us> <105986.1661134693@sss.pgh.pa.us> <114075.1661136913@sss.pgh.pa.us> Comments: In-reply-to Tom Lane message dated "Sun, 21 Aug 2022 22:55:13 -0400" MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" Content-ID: <361027.1661187385.0@sss.pgh.pa.us> Date: Mon, 22 Aug 2022 12:57:43 -0400 Message-ID: <361175.1661187463@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <361027.1661187385.1@sss.pgh.pa.us> OK, here's some proposed patches. 0001 adds a para about how to raise the listen queue length. 0002 isn't quite related, but while writing 0001 I noticed a nearby use of /proc/sys/... which I thought should be converted to sysctl. IMO /proc/sys pretty much sucks, at least for documentation purposes, for multiple reasons: * It's unlike the way you do things on other platforms. * "man sysctl" will lead you to useful documentation about how to use that command. There's no obvious way to find documentation about /proc/sys. * It's not at all sudo-friendly. Compare sudo sh -c 'echo 0 >/proc/sys/kernel/randomize_va_space' sudo sysctl -w kernel.randomize_va_space=0 The former is a lot longer and it's far from obvious why you have to do it that way. * You have to think in sysctl terms anyway if you want to make the setting persist across reboots, which you almost always do. * Everywhere else in runtime.sgml, we use sysctl not /proc/sys. 0003 removes PG_SOMAXCONN. While doing that I noticed that this computation hadn't been touched throughout all the various changes fooling with exactly what gets counted in MaxBackends. I think the most appropriate definition for the listen queue length is now MaxConnections * 2, not MaxBackends * 2, because the other processes counted in MaxBackends don't correspond to incoming connections. I propose 0003 for HEAD only, but the docs changes could be back-patched. regards, tom lane ------- =_aaaaaaaaaa0 Content-Type: text/x-diff; name="0001-document-somaxconn.patch"; charset="us-ascii" Content-ID: <361027.1661187385.2@sss.pgh.pa.us> Content-Description: 0001-document-somaxconn.patch diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 963b18ed85..1192faa6ae 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -1298,6 +1298,22 @@ default:\ linkend="guc-max-files-per-process"/> configuration parameter to limit the consumption of open files. + + + Another kernel limit that may be of concern when supporting large + numbers of client connections is the maximum socket connection queue + length. If more than that many connection requests arrive within a + very short period, some may get rejected before the postmaster can + service the requests, with those clients receiving unhelpful + connection failure errors such as Resource temporarily + unavailable. The default queue length limit is 128 on many + platforms. To raise it, adjust the appropriate kernel parameter + via sysctl, then restart the postmaster. + The parameter is variously named net.core.somaxconn + on Linux, kern.ipc.soacceptqueue on newer FreeBSD, + and kern.ipc.somaxconn on macOS and other BSD + variants. + ------- =_aaaaaaaaaa0 Content-Type: text/x-diff; name="0002-avoid-proc-sys-in-docs.patch"; charset="us-ascii" Content-ID: <361027.1661187385.3@sss.pgh.pa.us> Content-Description: 0002-avoid-proc-sys-in-docs.patch Content-Transfer-Encoding: quoted-printable diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 963b18ed85..1192faa6ae 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -1258,11 +1258,12 @@ default:\ - On Linux - /proc/sys/fs/file-max determines the - maximum number of open files that the kernel will support. It can - be changed by writing a different number into the file or by - adding an assignment in /etc/sysctl.conf. + On Linux the kernel parameter + fs.file-max determines the maximum number of ope= n + files that the kernel will support. It can be changed with + sysctl -w fs.file-max=3DN. + To make the setting persist across reboots, add an assignment + in /etc/sysctl.conf. The maximum limit of files per process is fixed at the time the kernel is compiled; see /usr/src/linux/Documentation/proc.txt for diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/= postmaster.c index 8a038d1b2a..1664fcee2a 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -4891,7 +4891,7 @@ SubPostmasterMain(int argc, char *argv[]) * If testing EXEC_BACKEND on Linux, you should run this as root before * starting the postmaster: * - * echo 0 >/proc/sys/kernel/randomize_va_space + * sysctl -w kernel.randomize_va_space=3D0 * * This prevents using randomized stack and code addresses that cause th= e * child process's memory map to be different from the parent's, making = it ------- =_aaaaaaaaaa0 Content-Type: text/x-diff; name="0003-remove-PG_SOMAXCONN.patch"; charset="us-ascii" Content-ID: <361027.1661187385.4@sss.pgh.pa.us> Content-Description: 0003-remove-PG_SOMAXCONN.patch Content-Transfer-Encoding: quoted-printable diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c index 8ff3be611d..7112e9751b 100644 --- a/src/backend/libpq/pqcomm.c +++ b/src/backend/libpq/pqcomm.c @@ -537,13 +537,11 @@ StreamServerPort(int family, const char *hostName, u= nsigned short portNumber, } = /* - * Select appropriate accept-queue length limit. PG_SOMAXCONN is only - * intended to provide a clamp on the request on platforms where an - * overly large request provokes a kernel error (are there any?). + * Select appropriate accept-queue length limit. It seems reasonable + * to use a value similar to the maximum number of child processes + * that the postmaster will permit. */ - maxconn =3D MaxBackends * 2; - if (maxconn > PG_SOMAXCONN) - maxconn =3D PG_SOMAXCONN; + maxconn =3D MaxConnections * 2; = err =3D listen(fd, maxconn); if (err < 0) diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual= .h index 844c3e0f09..f2a106f983 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -114,17 +114,6 @@ */ #define MAXPGPATH 1024 = -/* - * PG_SOMAXCONN: maximum accept-queue length limit passed to - * listen(2). You'd think we should use SOMAXCONN from - * , but on many systems that symbol is much smaller - * than the kernel's actual limit. In any case, this symbol need be - * twiddled only if you have a kernel that refuses large limit values, - * rather than silently reducing the value to what it can handle - * (which is what most if not all Unixen do). - */ -#define PG_SOMAXCONN 10000 - /* * You can try changing this if you have a machine with bytes of * another size, but no guarantee... ------- =_aaaaaaaaaa0--