Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iolNn-00076G-2u for pgsql-bugs@arkaria.postgresql.org; Tue, 07 Jan 2020 09:42:39 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iolNl-0005Du-Qa for pgsql-bugs@arkaria.postgresql.org; Tue, 07 Jan 2020 09:42:37 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iolNl-0005Dk-Dd for pgsql-bugs@lists.postgresql.org; Tue, 07 Jan 2020 09:42:37 +0000 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1iolNi-0008Td-5o for pgsql-bugs@lists.postgresql.org; Tue, 07 Jan 2020 09:42:36 +0000 Received: by mail-wr1-x441.google.com with SMTP id y11so53115300wrt.6 for ; Tue, 07 Jan 2020 01:42:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=GGYIJvqfo7hcPSN6akISWR4TyH5NoRHNeEylTWCQFmw=; b=FPIMARIo0Ac8Aw0Pd40/eOfGtguyGEeHGm5wBqxG2Zb6ZbDEKR0TbA/KfXgk/Idct5 fIVwY+8U/g2aWq3+AL4lZ3Qp9xmaQ2Cp9oNrIovUc1zls+vfIrUQu0r2iD+zOanwf4KM jhBu1fDbGT7dTXVBeNARC3cju6gc8W9mAz6pBXtHxICWJ9Rq+X4MDsLf7TK4UYwIpHbY X7OZhd6xxQB6Z4vYt4VHx2rFY7NbSAZuNixGrFhuw54Q1bhfajaxpI+PThm+uSe48Mw2 /pzkPNMawhkE+sKin1g8j8DVjjob3k+piQ2BsxVyG6gCoiXRk0itF1wSyZrDgZR+Qr3A xejw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=GGYIJvqfo7hcPSN6akISWR4TyH5NoRHNeEylTWCQFmw=; b=aPtWQTzmVtbSeSHlqyGFbZIZn8lm3jIRc7inOY39s7FdOfxkJJYnAvqYPlYkM2NsC1 wAN0SHDI9d7EITn97u61OO5k/6pXia0LVpHNoYtnqYE42Ziww7Xd0zu7r82UcrS9dNUo khNYbNM0QJ8PHoIMpsHAmyJdmIRB9PwBmfXq5iLSj0RmOqqX0myuegSApvUqtmCbaSZc elSYp9c1P3pvnKeJUgTx39hXxknbmdREtKmRzePnNqkqsk1VtAzpfiHfPU3/Gbu3fEtF KFdTsvzQxJJ0dF4isMj0Z1qQYyjoEeaRuQ+AkanHrR6K2AlsBCNpBYLgQSdhe1tEyhd4 epiw== X-Gm-Message-State: APjAAAUMeyFqzbeZijsgmXe7Bg8QLOuKLO2FN4uyLHmuR+0VBGaRFL3e T1WoBpcmTTnkNwz0GdKIpOtxGTeDj7w= X-Google-Smtp-Source: APXvYqxx2tkHCr4+XOBvgTQSAsSyzbZfSQesuIUhV0QhnsZpVXbnedq9Ak0i1V957EOd81Jr7BLFZw== X-Received: by 2002:adf:edd0:: with SMTP id v16mr66387875wro.310.1578390150882; Tue, 07 Jan 2020 01:42:30 -0800 (PST) Received: from localhost (ip-86-49-253-92.net.upcbroadband.cz. [86.49.253.92]) by smtp.gmail.com with ESMTPSA id s3sm26294800wmh.25.2020.01.07.01.42.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2020 01:42:30 -0800 (PST) Date: Tue, 7 Jan 2020 10:42:28 +0100 From: Tomas Vondra To: Chatur Gopalkrishna Cc: pgsql-bugs@lists.postgresql.org Subject: Re: Unable to fork: Resource Unavailable Message-ID: <20200107094228.eupftsq24tn4asr4@development> References: <007001d5c53b$0db0ca90$29125fb0$@ridenext.co.in> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <007001d5c53b$0db0ca90$29125fb0$@ridenext.co.in> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Tue, Jan 07, 2020 at 02:45:44PM +0530, Chatur Gopalkrishna wrote: >Hi! > >Our system works in hot/standby mode. There are multiple tables with >multiple entries in the postgres sql database. > >We observed that at times, observing the following prints: > >2019-12-28T14:11:12.522653+09:00 localhost postgres[1355]: [3-1] 2019-12-28 >05:11:12.521 GMT LOG: could not fork autovacuum worker process: Resource >temporarily unavailable > >2019-12-28T14:11:13.050796+09:00 localhost postgres[1355]: [4-1] 2019-12-28 >05:11:13.050 GMT LOG: archiver process (PID 27698) exited with exit code 1 > >2019-12-28T14:11:13.051514+09:00 localhost postgres[1355]: [5-1] 2019-12-28 >05:11:13.051 GMT LOG: archiver process (PID 17226) exited with exit code 1 > >2019-12-28T14:12:23.259291+09:00 localhost postgres[1355]: [6-1] 2019-12-28 >05:12:23.259 GMT LOG: could not fork new process for connection: Resource >temporarily unavailable > >2019-12-28T14:12:23.267624+09:00 localhost postgres[1355]: [7-1] 2019-12-28 >05:12:23.267 GMT LOG: could not fork new process for connection: Resource >temporarily unavailable > >2019-12-28T14:14:08.384115+09:00 localhost postgres[1355]: [8-1] 2019-12-28 >05:14:08.383 GMT LOG: could not fork new process for connection: Resource >temporarily unavailable > >2019-12-28T14:14:08.384348+09:00 localhost postgres[1355]: [9-1] 2019-12-28 >05:14:08.384 GMT LOG: could not fork new process for connection: Resource >temporarily unavailable > >2019-12-28T14:14:12.581409+09:00 localhost postgres[1355]: [10-1] 2019-12-28 >05:14:12.581 GMT LOG: could not fork autovacuum worker process: Resource >temporarily unavailable > >2019-12-28T14:14:13.391192+09:00 localhost postgres[1355]: [11-1] 2019-12-28 >05:14:13.391 GMT LOG: could not fork new process for connection: Resource >temporarily unavailable > >2019-12-28T14:14:13.391356+09:00 localhost postgres[1355]: [12-1] 2019-12-28 >05:14:13.391 GMT LOG: could not fork new process for connection: Resource >temporarily unavailable > >2019-12-28T14:14:13.581992+09:00 localhost postgres[1355]: [13-1] 2019-12-28 >05:14:13.581 GMT LOG: could not fork autovacuum worker process: Resource >temporarily unavailable > >Due to the above, our applications are unable to connect to Postgres and our >systems are rebooting. > >Can someone please help me understand the reason of "Resource temporarily >unavailable" and any configuration / debug pointers? > This seems like some sort of resorce exhaustion, likely at the kernel level (file descriptors, limits, ...). You haven't provided any info about which OS you're using, but this is unlikely a postgres bug, so posting it on pgsql-general would be more appropriate. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services