Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lwid7-0006xw-19 for pgsql-bugs@arkaria.postgresql.org; Fri, 25 Jun 2021 10:00:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lwid6-0005Bb-0Z for pgsql-bugs@arkaria.postgresql.org; Fri, 25 Jun 2021 10:00:08 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lwid5-0005BT-P2 for pgsql-bugs@lists.postgresql.org; Fri, 25 Jun 2021 10:00:07 +0000 Received: from mail-lf1-x131.google.com ([2a00:1450:4864:20::131]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lwid1-00006G-WB for pgsql-bugs@lists.postgresql.org; Fri, 25 Jun 2021 10:00:07 +0000 Received: by mail-lf1-x131.google.com with SMTP id f30so15433780lfj.1 for ; Fri, 25 Jun 2021 03:00:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=/RvI8cVvVWLEqzEkBTNkAvYz1PPpjRQ1L46PR+Q8kDA=; b=XAMv/3ch1DxF3FiBEbo4RzObBtDBG1tEQjhjWHSyoXpnEuU707tR9ohUWQGrF6gAJI 769jidU5UhuiQL9L3zLm+59zeeRVT7MQFhgc0aYnsRUbbSUbfMD00l3rKua4Hvgen7np RVjcSyU0PO5noGZ+CBbW59hKG3OLGXfZy9x2v7ylDEQM/xsdh8b7JfgVkXiTuF1JXEE7 F7guMk+XuG3evXYSGB8/KNmWIoFqEnfuFC7qrE4DyIdzOsrl9usIxF+9iPIP+tsVwdwD cQZth4gAIqvBcZtduLKtz/N/GWg3W4uf3l3nZ5/v4t54NiEyZR0IKt17QVsJowRGG/ZG B73w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=/RvI8cVvVWLEqzEkBTNkAvYz1PPpjRQ1L46PR+Q8kDA=; b=DUwNl8wkUEVDT7/mApNJKb5g2H11yER1jTIUhj0zLleMjOtcSCkN0qMXTBFNlHx/LB IlY3AJol8JaE7dp+el/puY1xsYxgPN6P1ttCjWJH1WgxMbZfsizM0gvU5W3rRw2X4ItC AuywnXKBNyh06tmBCePrbMgyrkQtCyWjkqV8PFZxhvW4nNonuGiSOgtwKF+fmshcqKWH CMhkDSVuvAHTupiOpUL69G0oJbYVLV7puxnrbi63LbS2Pt1swnzCk1lqPtAWo5wdToUA wQ41YJZVWcyiNGPO+DYFsYZGpvdq4TsNJdM9+af43+mcYqyOoSed/Zn9We+LHc81X9fO g3Mw== X-Gm-Message-State: AOAM530rZi32ypt7+1cVuoyas6o4uFvNCR1ElFSxk5tktB0xOBOp+SZ2 hZZzy7c5UkgwBwkWCORJxCXRrmrSD/c= X-Google-Smtp-Source: ABdhPJy0PPAcR3Ii75fJOF0MP07U2fmxBlw65sM3B2fsBCYCujzTN9VNJ68koSZ6D1kXGLEGenqsBQ== X-Received: by 2002:a19:490f:: with SMTP id w15mr7557977lfa.228.1624615202228; Fri, 25 Jun 2021 03:00:02 -0700 (PDT) Received: from [1.0.0.7] ([178.155.6.177]) by smtp.gmail.com with ESMTPSA id h5sm470628lfp.193.2021.06.25.03.00.01 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 25 Jun 2021 03:00:01 -0700 (PDT) Subject: Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus() To: pgsql-bugs@lists.postgresql.org References: <17072-2f8764857ef2c92a@postgresql.org> From: Alexander Lakhin Message-ID: Date: Fri, 25 Jun 2021 13:00:00 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <17072-2f8764857ef2c92a@postgresql.org> Content-Type: multipart/mixed; boundary="------------A773E013695984415A83CBF0" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------A773E013695984415A83CBF0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit 24.06.2021 19:00, PG Bug reporting form wrote: > When running multiple installchecks (100-200) in a loop I get the following > fail: If it could be of any use, please look at the draft script that executes multiple installchecks on one instance. (Sorry for a dirty play with the makefle and neighbors — it's not intended for the production use.) I started to develop this script to discover a hardware or software issue on a powerful server (192 cores, 4 TB RAM, ...). Single installcheck is a needle in a haystack for such hardware, so I tried to run multiple instances and multiple installchecks for each. Some tests are not ready to run in a such configuration, but most of the tests work and bring some findings. Best regards, Alexander --------------A773E013695984415A83CBF0 Content-Type: application/x-shellscript; name="test-icx.sh" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="test-icx.sh" IyEvYmluL2Jhc2gKc2V0IC1lCgojIEJ1aWxkIGV4YW1wbGUKIyAuL2NvbmZpZ3VyZSAtLXBy ZWZpeD0vdG1wL3BnIC0tZW5hYmxlLWRlYnVnIC0tZW5hYmxlLWNhc3NlcnQgPi9kZXYvbnVs bCAmJiBtYWtlIC1qOCA+L2Rldi9udWxsICYmIG1ha2UgLWo4IC1DIGNvbnRyaWIgPi9kZXYv bnVsbCAmJiBtYWtlIGluc3RhbGwgJiYgbWFrZSBpbnN0YWxsIC1DIGNvbnRyaWIKCiMgQ2hv b3NlIHNlcnZlciBpZAppZiBbIC16ICQxIF07IHRoZW4KICBTSUQ9MDEKZWxzZQogIFNJRD0k MQpmaQoKIyBTZXQgZ2xvYmFsIHZhcmlhYmxlcwpOVU1fSU5TVEFMTENIRUNLUz0yMDAKTlVN X0lURVJBVElPTlM9NTAKClBHUFJFRklYPS90bXAvcGcvClNSQ0NPUFlESVI9L3RtcC90LwoK UEdEQj1gcHdkYC90bXBkYgpQR0JJTj0kUEdQUkVGSVgvYmluCmV4cG9ydCBQQVRIPSIkUEdC SU46JFBBVEgiCmV4cG9ydCBMRF9MSUJSQVJZX1BBVEg9IiRQR1BSRUZJWC9saWIiCgpleHBv cnQgUEdIT1NUPTEyNy4wLjAuMQpleHBvcnQgUEdQT1JUPTU0MzIKZXhwb3J0IFBHVVNFUj1w b3N0Z3JlcwpleHBvcnQgUEdEQVRBQkFTRT10ZXN0CgojIFN0b3AgcHJldmlvdXMgaW5zdGFu Y2UgaW4gY2FzZSBpdCBpcyBsZWZ0IGZyb20gYSBwcmV2aW91cyBydW4KJFBHQklOL3BnX2N0 bCAtdyAtdCA1IC1EICIkUEdEQiIgc3RvcCB8fCB0cnVlCgojIEFuZCBjbGVhciBkYXRhIGFu ZCBsb2dzCnJtIC1yZiAiJFBHREIiIHNlcnZlcioubG9nIHJtIC1yZiBpbnN0YWxsY2hlY2sq LmxvZyB8fCB0cnVlCgojIFByZXBhcmUgaW5zdGFsbGNoZWNrIGRpcmVjdG9yaWVzCmZvciBj IGluIGBzZXEgJE5VTV9JTlNUQUxMQ0hFQ0tTYDsgZG8KICBybSAtcmYgJFNSQ0NPUFlESVIv aWMkU0lELSRjIHx8IHRydWUKICBta2RpciAkU1JDQ09QWURJUi9pYyRTSUQtJGMvCiAgY3Ag KiAkU1JDQ09QWURJUi9pYyRTSUQtJGMvID4vZGV2L251bGwgMj4mMSB8fCB0cnVlCiAgZWNo byAiZXhpdCAwIiA+ICRTUkNDT1BZRElSL2ljJFNJRC0kYy9jb25maWcuc3RhdHVzCiAgbWtk aXIgJFNSQ0NPUFlESVIvaWMkU0lELSRjL3NyYwogIGNwIHNyYy8qICRTUkNDT1BZRElSL2lj JFNJRC0kYy9zcmMvID4vZGV2L251bGwgMj4mMSB8fCB0cnVlCiAgY3AgLXIgc3JjL3Rlc3Qg JFNSQ0NPUFlESVIvaWMkU0lELSRjL3NyYy8KICBzZWQgLWUgJ3MvXmFsbDovI2FsbDovJyAt aSAkU1JDQ09QWURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvcmVncmVzcy9HTlVtYWtlZmlsZQoK ICAjIEZpeGluZyB0ZXN0cwoKICBjcDM9YHByaW50ZiAiJS0zcyIgJGNgCiAgYzAzPWBwcmlu dGYgIiUwM2QiICRjYAogIHNlZCAtZSAicy9yZWdyZXNzaW9uL3JlZ3Jlc3MkY3AzLyIgLWkg JFNSQ0NPUFlESVIvaWMkU0lELSRjL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvdXBkYXRh YmxlX3ZpZXdzLm91dCAKICBzZWQgLWUgInMvcmVncmVzc2lvbi9yZWdyZXNzJGNwMy8iIC1p ICRTUkNDT1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3NlcXVl bmNlLm91dAoKICBzZWQgLWUgInMvIHJlZ3Jlc3Npb24vIHJlZ3Jlc3MkYy8iIC1pICRTUkND T1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9yZWdyZXNzL3NxbC9wdWJsaWNhdGlvbi5zcWwg XAogICAgJFNSQ0NPUFlESVIvaWMkU0lELSRjL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQv cHVibGljYXRpb24ub3V0CgogIHNlZCAtZSAicy8gcmVncmVzc2lvbi8gcmVncmVzcyRjLyIg LWkgJFNSQ0NPUFlESVIvaWMkU0lELSRjL3NyYy90ZXN0L3JlZ3Jlc3Mvc3FsL2RlcGVuZGVu Y3kuc3FsIFwKICAgICRTUkNDT1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9yZWdyZXNzL2V4 cGVjdGVkL2RlcGVuZGVuY3kub3V0CgogIHNlZCAtZSAicy9TRUxFQ1Qgb2JqX2Rlc2NyaXB0 aW9uKHMub2lkLCAncGdfc3Vic2NyaXB0aW9uJykgRlJPTSBwZ19zdWJzY3JpcHRpb24gczsv U0VMRUNUIG9ial9kZXNjcmlwdGlvbiBGUk9NIChTRUxFQ1Qgb2JqX2Rlc2NyaXB0aW9uKHMu b2lkLCAncGdfc3Vic2NyaXB0aW9uJykgRlJPTSBwZ19zdWJzY3JpcHRpb24gcykgZCBXSEVS RSBvYmpfZGVzY3JpcHRpb24gSVMgTk9UIE5VTEwgTElNSVQgMTsvIiAtaSBcCiAgICAkU1JD Q09QWURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvcmVncmVzcy9zcWwvc3Vic2NyaXB0aW9uLnNx bCBcCiAgICAkU1JDQ09QWURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3Rl ZC9zdWJzY3JpcHRpb24ub3V0CgogIHNlZCAtZSAicy9TRUxFQ1QgcGdfbm90aWZpY2F0aW9u X3F1ZXVlX3VzYWdlKCk7L1NFTEVDVCByb3VuZChwZ19ub3RpZmljYXRpb25fcXVldWVfdXNh Z2UoKSAqIDEwMDApIFxcLyAxMDAwIEFTIHBnX25vdGlmaWNhdGlvbl9xdWV1ZV91c2FnZS8i IC1pIFwKICAgICRTUkNDT1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9yZWdyZXNzL3NxbC9h c3luYy5zcWwgXAogICAgJFNSQ0NPUFlESVIvaWMkU0lELSRjL3NyYy90ZXN0L3JlZ3Jlc3Mv ZXhwZWN0ZWQvYXN5bmMub3V0CgogICMgRGlzYWJsZSB0ZXN0IGZvcmVpZ25fZGF0YSBhcyBu b3QgZml4YWJsZSBieSBhIHNpbXBsZSBzZWQKICBzZWQgLWUgInMvZm9yZWlnbl9kYXRhIC8v IiAtaSAkU1JDQ09QWURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvcmVncmVzcy9wYXJhbGxlbF9z Y2hlZHVsZQogIHNlZCAtZSAicy90ZXN0OiBmb3JlaWduX2RhdGFcJC8vIiAtaSAkU1JDQ09Q WURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvcmVncmVzcy9zZXJpYWxfc2NoZWR1bGUKCiAgIyBE aXNhYmxlIHRlc3QgcGFzc3dvcmQgYXMgbm90IGZpeGFibGUgYnkgYSBzaW1wbGUgc2VkICht ZDUgcGFzc3dvcmQgY29udGFpbnMgdGhlIHVzZXJuYW1lIChyZWdyZXNzX3Bhc3N3ZCkpCiAg c2VkIC1lICJzL3Bhc3N3b3JkIC8vIiAtaSAkU1JDQ09QWURJUi9pYyRTSUQtJGMvc3JjL3Rl c3QvcmVncmVzcy9wYXJhbGxlbF9zY2hlZHVsZQogIHNlZCAtZSAicy90ZXN0OiBwYXNzd29y ZFwkLy8iIC1pICRTUkNDT1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9yZWdyZXNzL3Nlcmlh bF9zY2hlZHVsZQogCiAgIyBEaXNhYmxlIHRlc3RzIHNlbGVjdF9wYXJhbGxlbCBhbmQgYWR2 aXNvcnlfbG9ja3MgYXMgdW5zdGFibGUgaW4gcGFyYWxsZWwgcnVucwogIHNlZCAtZSAicy9z ZWxlY3RfcGFyYWxsZWwgLy8iIC1pICRTUkNDT1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9y ZWdyZXNzL3BhcmFsbGVsX3NjaGVkdWxlCiAgc2VkIC1lICJzL3Rlc3Q6IHNlbGVjdF9wYXJh bGxlbFwkLy8iIC1pICRTUkNDT1BZRElSL2ljJFNJRC0kYy9zcmMvdGVzdC9yZWdyZXNzL3Nl cmlhbF9zY2hlZHVsZQogIHNlZCAtZSAicy9hZHZpc29yeV9sb2NrIC8vIiAtaSAkU1JDQ09Q WURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvcmVncmVzcy9wYXJhbGxlbF9zY2hlZHVsZQogIHNl ZCAtZSAicy90ZXN0OiBhZHZpc29yeV9sb2NrXCQvLyIgLWkgJFNSQ0NPUFlESVIvaWMkU0lE LSRjL3NyYy90ZXN0L3JlZ3Jlc3Mvc2VyaWFsX3NjaGVkdWxlCgogICMgRml4IHJlZ3Jlc3Mg ZW50aXRpZXMKICBmaW5kICAkU1JDQ09QWURJUi9pYyRTSUQtJGMvc3JjL3Rlc3QvIFwoIC1u YW1lICouc3FsIC1vIC1uYW1lICouc291cmNlIC1vIC1uYW1lICoub3V0IFwpIC1leGVjIHNl ZCAtZSBcCiJzL3JlZ3Jlc3NfXCh0YmxzcGFjZVwpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMv cmVncmVzc19cKHZhY3V1bVwpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKG90 aGVyX3BhcnRpdGlvbmVkX2ZrX293bmVyXCkvcmVnciR7YzAzfV9cMS9nOyAiXAoicy9yZWdy ZXNzX1wodGVzdHN1YlwpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKHJlZ3Jv bGVfdGVzdFwpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc2lvbl9cKGF0eF90ZXN0 X2RhdGFiYXNlXCkvcmVncmVzcyR7YzAzfV9cMS9nOyAiXAoicy9yZWdyZXNzX1woY29uc3Ry YWludF9jb21tZW50c1wpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKG5vX2No aWxkX2FjY2Vzc1wpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKHJvbF9sb2Nr MVwpL3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKGFsaWNlXHxib2JcfGNhcm9s XHxkYXZlXCkvcmVnciR7YzAzfV9cMS9nOyAiXAoicy9yZWdyZXNzX1woW2Etel9dKnVzZXJb MC05XSpcKS9yZWdyJHtjMDN9X1wxL2c7ICJcCiJzL3JlZ3Jlc3NfXChbYS16X10qcm9sZVwp L3JlZ3Ike2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKFthLXpfXSpncm91cFwpL3JlZ3Ik e2MwM31fXDEvZzsgIlwKInMvcmVncmVzc19cKHRlc3RfW2EtejAtOV9dKlwpL3JlZ3Ike2Mw M31fXDEvZzsgIlwKInMvcmVncmVzc19cKHByaXZfW2EtejAtOV1cK1wpL3JlZ3Ike2MwM31f XDEvZzsgIlwKInMvcmVncmVzc19cKHJsc19bYS16MC05XVwrXCkvcmVnciR7YzAzfV9cMS9n OyAiXAoicy9yZWdyZXNzX1wocGFzc3dkW2EtejAtOV9dKlwpL3JlZ3Ike2MwM31fXDEvZzsg IiAtaSB7fSBcOwoKZG9uZQoKIyBQcmVwYXJlIG1hc3RlciBub2RlCiRQR0JJTi9pbml0ZGIg LWsgLUQgIiRQR0RCIiAtLXVzZXJuYW1lPXBvc3RncmVzCmVjaG8gIgpzaGFyZWRfYnVmZmVy cyA9IDEwMjRNQgptYXhfY29ubmVjdGlvbnMgPSAxMDAwCm1heF9sb2Nrc19wZXJfdHJhbnNh Y3Rpb24gPSA1MDAKCmxvZ19taW5fbWVzc2FnZXMgPSBub3RpY2UKbG9nX21pbl9lcnJvcl9z dGF0ZW1lbnQgPSBsb2cKCmZzeW5jID0gb2ZmCiIgPj4gIiRQR0RCL3Bvc3RncmVzcWwuYXV0 by5jb25mIgoKIyBTdGFydCBzZXJ2ZXIKJFBHQklOL3BnX2N0bCAtdyAtdCAxNSAtRCAiJFBH REIiIC1sIHNlcnZlciRTSUQubG9nIHN0YXJ0CgojIFJ1biBgbWFrZSBpbnN0YWxsY2hlY2tg IGluIGEgbG9vcApyZXN1bHQ9MAp0aW1lIGZvciBpIGluIGBzZXEgJE5VTV9JVEVSQVRJT05T YDsgZG8KICBlY2hvICJpdGVyYXRpb24gJGk6IGBkYXRlYCIKICBmb3IgYyBpbiBgc2VxICRO VU1fSU5TVEFMTENIRUNLU2A7IGRvCiAgICAoY2QgJFNSQ0NPUFlESVIvaWMkU0lELSRjOyBF WFRSQV9SRUdSRVNTX09QVFM9Ii0tZGJuYW1lPXJlZ3Jlc3MkYyIgdGltZSBtYWtlIGluc3Rh bGxjaGVjayBOT19HRU5FUkFURURfSEVBREVSUz0xKSA+PiBpbnN0YWxsY2hlY2skU0lELSRj LmxvZyAyPiYxICYKICBkb25lCiAgd2FpdAoKICBlY2hvICJpbnN0YWxsY2hlY2sgZmluaXNo ZWQ6IGBkYXRlYCIKICAjIFRoZXJlIGFyZSBzdGlsbCBzb21lIHVuc3RhYmxlIHRlc3RzOiBz dGF0cywgbWlzY19zYW5pdHksIGNyZWF0ZV9pbmRleAogIGlmIGdyZXAgJ3Rlc3QgcHJvY2Vz cyBleGl0ZWQgd2l0aCcgaW5zdGFsbGNoZWNrKi5sb2cgPmZhaWwubG9nIHx8IGdyZXAgJ1RP RE86IEZBSUwnIGluc3RhbGxjaGVjayoubG9nOyB0aGVuCiAgICAgZWNobyAiaW5zdGFsbGNo ZWNrIEZBSUxFRCIKICAgICByZXN1bHQ9MQogICAgIGJyZWFrCiAgZmkKICBzbGVlcCAzCiAg Y29yZWR1bXBjdGwgJiYgcmVzdWx0PTEKICBbICRyZXN1bHQgLWVxIDAgXSB8fCBicmVhawpk b25lCgppZiBbICRyZXN1bHQgLWVxIDAgXTsgdGhlbgogICMgU3RvcCBzZXJ2ZXIgb24gc3Vj Y2VzcwogICRQR0JJTi9wZ19jdGwgLXcgLXQgOTAgLUQgIiRQR0RCIiBzdG9wCiAgZWNobyAi b2siCmZpCmV4aXQgJHJlc3VsdAo= --------------A773E013695984415A83CBF0--