Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nvKQf-0002EW-VQ for pgsql-performance@arkaria.postgresql.org; Sun, 29 May 2022 15:02:06 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nvKQe-0003al-SM for pgsql-performance@arkaria.postgresql.org; Sun, 29 May 2022 15:02:04 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nvKQe-0003ac-E4 for pgsql-performance@lists.postgresql.org; Sun, 29 May 2022 15:02:04 +0000 Received: from mail-il1-x12c.google.com ([2607:f8b0:4864:20::12c]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nvKQb-0002t2-61 for pgsql-performance@lists.postgresql.org; Sun, 29 May 2022 15:02:03 +0000 Received: by mail-il1-x12c.google.com with SMTP id e9so6119923ilq.6 for ; Sun, 29 May 2022 08:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=+RiL1nVWvlO8TV2FZV5/R439wDH6P0ZlU1r+qdsJn+U=; b=4x8ABXF0L3DnYL67rlvFe4CQg12dPkfU5b4G4kZwGSc21l+jT4Bm/Y3ZhlirOhTvs8 ONfZashWyLMWmtQLhBNOomJUX005QJPiw+09SHkjVHekIC09/MWjNl4jyzsg/hswqhP1 eP+vZ+yzNz2kII7slSz2OypHoZWI1IeCYhAkfVcEjKduANQu7N8/bChTDsCDSZzef53t 5B1uWiPgzR0SlQbDCwA9SCE3keq+8VU9wv90lW/M9bdNo0Gj/Rk4LqSmQDZmdMwXJFEJ 9WY5i3VIg6JRHDHqJAyQLZg8oZVMX62x0/P+qtIR2d5HnNF/5AZzu8onz8dTj0Xc8s48 g87g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=+RiL1nVWvlO8TV2FZV5/R439wDH6P0ZlU1r+qdsJn+U=; b=NzX1e+p2XHzeDqJToZMJ4h5eWp/slmRZo/omtoMyBMtBtBl4PgKfUbzdKEMd4fgiL/ j3KD7sjb6TDtb7zAEW8cKIEZzZnme4actxdamABsgSYZjZVpMXSfWJxl7uD0E6YHzX2R ATvOrJ00G51YH7Ziy4J7H5errJKk1AtXt/Yv4BeQvcY0V3zgJmgDNZQMiyu07cKNLXtD n9fAKyG2A61gLDl/kBnDSKj6AqwtR+sPoNQvI6zZk6eIijVC4dtRe55JloKtJ5VwdOHM oXJnkUIljvbdltF+T99y6foz552iPxfIGkW2dXGpDXJzPf08U+gTwAYty/GPNhZvJdEv 0sbg== X-Gm-Message-State: AOAM533cOCtdO9qKNFoUIcQK+u0wtHnI5mmn4yLlIaexwTTeSmyehs5u 8MS+O1hI+ZOKo8pb1+l4PtJVNQ== X-Google-Smtp-Source: ABdhPJzpNKDk+25IBJ/MIPjY4Re3CHT/kw3X9ZITZzZjxuGpFWZJ/IBXw+PTDoDomuk0yA7CWJMYnw== X-Received: by 2002:a92:de0d:0:b0:2cd:6d7b:e12f with SMTP id x13-20020a92de0d000000b002cd6d7be12fmr25924138ilm.179.1653836519339; Sun, 29 May 2022 08:01:59 -0700 (PDT) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id cv11-20020a056638498b00b0032ea3ba170dsm2066973jab.86.2022.05.29.08.01.58 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 29 May 2022 08:01:58 -0700 (PDT) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id CE9CE8008C4; Sun, 29 May 2022 10:01:57 -0500 (CDT) Date: Sun, 29 May 2022 10:01:57 -0500 From: Justin Pryzby To: "James Pang (chaolpan)" Cc: pgsql-performance@lists.postgresql.org Subject: Re: postgres backend process hang on " D " state Message-ID: <20220529150157.GM19626@telsasoft.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Sun, May 29, 2022 at 01:20:12PM +0000, James Pang (chaolpan) wrote: > We have a performance test on Postgresql 13.4 on RHEL8.4 , just after connection storm in ( 952 new connections coming in 1 minute), a lot of backends start on " D " state, and when more sessions got disconnected, they do not exit successfully, instead became "defunct". No errors from postgresql.log , just after the connection storm, some pg_cron workers can not started either. The server is a Virtual machine and no IO hang (actually) IO load is very low. Could be a postgresql bug or an OS abnormal behavior? What settings have you set ? https://wiki.postgresql.org/wiki/Server_Configuration What extensions do you have loaded? \dx Send the output of SELECT * FROM pg_stat_activity either as an attachment or in \x mode? What is your data dir ? Is it on the VM's root filesystem or something else ? Show the output of "mount". Are there any kernel messages in /var/log/messages or `dmesg` ? How many relations are in your schema ? Are you using temp tables ? Long-running transactions ? -- Justin