Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kdXaK-0000cP-7c for pgsql-hackers@arkaria.postgresql.org; Fri, 13 Nov 2020 11:49:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kdXaJ-0000Rj-25 for pgsql-hackers@arkaria.postgresql.org; Fri, 13 Nov 2020 11:49:43 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kdXaI-0000Rc-Nj for pgsql-hackers@lists.postgresql.org; Fri, 13 Nov 2020 11:49:42 +0000 Received: from forwardcorp1p.mail.yandex.net ([77.88.29.217]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kdXaG-0003j3-J3 for pgsql-hackers@postgresql.org; Fri, 13 Nov 2020 11:49:42 +0000 Received: from sas1-5717c3cea310.qloud-c.yandex.net (sas1-5717c3cea310.qloud-c.yandex.net [IPv6:2a02:6b8:c14:3616:0:640:5717:c3ce]) by forwardcorp1p.mail.yandex.net (Yandex) with ESMTP id E68DF2E14FF; Fri, 13 Nov 2020 14:49:38 +0300 (MSK) Received: from sas1-58a37b48fb94.qloud-c.yandex.net (sas1-58a37b48fb94.qloud-c.yandex.net [2a02:6b8:c08:1d1b:0:640:58a3:7b48]) by sas1-5717c3cea310.qloud-c.yandex.net (mxbackcorp/Yandex) with ESMTP id peOBvcWI6O-ncx8hAJt; Fri, 13 Nov 2020 14:49:38 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1605268178; bh=TNFFCQic2/4JWGz8HXhfthGuxGAa7uuPG6nUr6il34U=; h=To:Message-Id:References:Date:Subject:Cc:From:In-Reply-To; b=GVOyw5X50cqngqxPNEcWD2dkNsRztpQD9p4ybL5XpQ/53R5ODjR5uzASsC1CToyVm ftKagC643mdTOkNV3Dk+740LjfO/XGg5bnIkrlgDR2f/QIsrHpD/xfvbqsfcBEHzJI W/af+TBxaqy3avxQhfyPcusZMj8rktHlbFHI2BS0= Authentication-Results: sas1-5717c3cea310.qloud-c.yandex.net; dkim=pass header.i=@yandex-team.ru Received: from dynamic-ekb.dhcp.yndx.net (dynamic-ekb.dhcp.yndx.net [2a02:6b8:0:2807:1b2:b9d:ed58:7c13]) by sas1-58a37b48fb94.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id xi23RJRV6a-nammwlCX; Fri, 13 Nov 2020 14:49:38 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: MultiXact\SLRU buffers configuration From: Andrey Borodin In-Reply-To: <35862787-8b4d-a290-789e-6e12dc6527e8@enterprisedb.com> Date: Fri, 13 Nov 2020 16:49:36 +0500 Cc: Tomas Vondra , Alexander Korotkov , Anastasia Lubennikova , Daniel Gustafsson , Kyotaro Horiguchi , pgsql-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <13F86913-C01B-4983-AE2E-493F5A028280@yandex-team.ru> References: <2087E87D-44CA-4443-8E6A-5087F07443F4@yandex-team.ru> <8A94938B-054C-4439-9866-2C220B4D0DD7@yandex-team.ru> <20201028013651.de5cj2xadgmba5nf@development> <13D8FD63-559A-4737-B7FD-05288D1CEF8B@yandex-team.ru> <20201028233243.ygm6yqlynkqpzekr@development> <43F3DE92-F236-4EA5-B4D6-39BEF6BD849D@yandex-team.ru> <20201029134933.xd4mh2cofuf6tdfz@development> <65C1B4BA-D16F-4939-978B-AC8F370F5A5E@yandex-team.ru> <9b4d17df-b811-8323-16be-3cab913216d1@enterprisedb.com> <35862787-8b4d-a290-789e-6e12dc6527e8@enterprisedb.com> To: Tomas Vondra X-Mailer: Apple Mail (2.3608.120.23.2.4) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk > 10 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3., =D0=B2 23:07, Tomas Vondra = =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB= (=D0=B0): >=20 > On 11/10/20 7:16 AM, Andrey Borodin wrote: >>=20 >>=20 >> but this picture was not stable. >>=20 >=20 > Seems we haven't made much progress in reproducing the issue :-( I = guess > we'll need to know more about the machine where this happens. Is there > anything special about the hardware/config? Are you monitoring size of > the pg_multixact directory? It's Ubuntu 18.04.4 LTS, Intel Xeon E5-2660 v4, 56 CPU cores with 256Gb = of RAM. PostgreSQL 10.14, compiled by gcc 7.5.0, 64-bit No, unfortunately we do not have signals for SLRU sizes. 3.5Tb mdadm raid10 over 28 SSD drives, 82% full. First incident triggering investigation was on 2020-04-19, at that time = cluster was running on PG 10.11. But I think it was happening before. I'd say nothing special... >=20 >> How do you collect wait events for aggregation? just insert into some = table with cron? >>=20 >=20 > No, I have a simple shell script (attached) sampling data from > pg_stat_activity regularly. Then I load it into a table and aggregate = to > get a summary. Thanks! Best regards, Andrey Borodin.