Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pMYJT-0006wq-Ns for pgsql-performance@arkaria.postgresql.org; Mon, 30 Jan 2023 17:51:27 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pMYJS-0001e5-Kf for pgsql-performance@arkaria.postgresql.org; Mon, 30 Jan 2023 17:51:26 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pMYJS-0001dv-5g for pgsql-performance@lists.postgresql.org; Mon, 30 Jan 2023 17:51:26 +0000 Received: from mail-il1-x12c.google.com ([2607:f8b0:4864:20::12c]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1pMYJO-0003Hf-VQ for pgsql-performance@lists.postgresql.org; Mon, 30 Jan 2023 17:51:25 +0000 Received: by mail-il1-x12c.google.com with SMTP id m8so5438300ili.7 for ; Mon, 30 Jan 2023 09:51:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=DzK+ZV6hqyqA5JVM/ywPhQfKjdu+84KnGBStkHhTH9U=; b=k+7/kKf+Gm1o+Lo0gND8dBFseYHIlpeBBoHQwPKq6IoxVJazLdRLKs6ncIbvS5GwyU vl5dLHJjFtRL6slzllB59KM2N5n92sT4uqhuoniuw6VkXB3ecysc+4x7tEBMtSkF18Nb 3ncNQ37r0eIqWrxxEAThCpigpJNvO9cRHf9hYolWikBvpzk+60B635yHYIIQYGfTwAXv 1OTMl5+hzDQoEResQ6WiFiX0F3+9fIViihzRqgMjW4YOOSgBGpS/uxYrxKBErw4aluuH xRNQrUeOfYrn+XGkB+cNHqoCipRR/Wk2Qp5WmWx4t/dZkC5DDiPbGDPbDjEFOnm1Q4oT YObw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DzK+ZV6hqyqA5JVM/ywPhQfKjdu+84KnGBStkHhTH9U=; b=pcIeiUiPILUKIUNPmqpmmPHYieZNyRybJL4fnz47C/i86r/FQ8BH4QnSRyMfvlvQy8 26Q4S3IynCQb4OETN8s9Z8IUpQXzLPCiE1C37yrSia3tor8W+HSj/Ce1FJqGUuLNt6W4 qDm2xZwRgOpnotnslcIdESG97U6KA8WCZjGheyHxiQW2tNlNg1qmPUkAgujuwiimixih jqxQBy50UoSAazBeuH/ykSKZEOT2itaSjIGhYA6AJrA2NdHlAF7KhmDajTeamUfh8O74 n9HnilclNbj4y2PMrXLjzpdFJD4Xkm52WC79FbmrIqJKqKcIU8R4Zug2R3OJ6bo5R4+W ynBg== X-Gm-Message-State: AO0yUKUZKCy6u5G8EDvi7mog0StiWacmmHXSm3L//DxLt0FHavE0sknj s3hvl/n0lex6/pfEL6C4gcQGkw== X-Google-Smtp-Source: AK7set+AWFDQgyrIMxYoOSnMDpgSP72ODWbRzmpc8THIZH+9FbTkaUMVoVg/hAHuMFrt3GNil8RxNA== X-Received: by 2002:a05:6e02:1645:b0:30c:1b0:fcf6 with SMTP id v5-20020a056e02164500b0030c01b0fcf6mr8406523ilu.7.1675101080387; Mon, 30 Jan 2023 09:51:20 -0800 (PST) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id t190-20020a0254c7000000b0038a3b8aaf11sm4911769jaa.37.2023.01.30.09.51.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 09:51:19 -0800 (PST) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id 02467800A8D; Mon, 30 Jan 2023 11:51:18 -0600 (CST) Date: Mon, 30 Jan 2023 11:51:18 -0600 From: Justin Pryzby To: Mok Cc: pgsql-performance@lists.postgresql.org Subject: Re: Database Stalls Message-ID: <20230130175118.GE22427@telsasoft.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Mon, Jan 30, 2023 at 05:47:49PM +0000, Mok wrote: > Hi, > > We've started to observe instances of one of our databases stalling for a > few seconds. > > We see a spike in wal write locks then nothing for a few seconds. After > which we have spike latency as processes waiting to get to the db can do > so. > > There is nothing in the postgres logs that give us any clues to what could > be happening, no locks, unusually high/long running transactions, just a > pause and resume. > > Could anyone give me any advice as to what to look for when it comes to > checking the underlying disk that the db is on? What version postgres? What settings have non-default values ? What OS/version? What environment/hardware? VM/image/provider/... Have you enabled logging for vacuum/checkpoints/locks ? https://wiki.postgresql.org/wiki/Slow_Query_Questions