Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rhMAN-00B2Pd-Tn for pgsql-bugs@arkaria.postgresql.org; Tue, 05 Mar 2024 04:12:36 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rhMAM-00DJ02-32 for pgsql-bugs@arkaria.postgresql.org; Tue, 05 Mar 2024 04:12:34 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rhMAL-00DIzt-NN for pgsql-bugs@lists.postgresql.org; Tue, 05 Mar 2024 04:12:34 +0000 Received: from mail-pg1-x52e.google.com ([2607:f8b0:4864:20::52e]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1rhMAI-002uCj-3s for pgsql-bugs@postgresql.org; Tue, 05 Mar 2024 04:12:32 +0000 Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-5e4b775e1d6so1734270a12.1 for ; Mon, 04 Mar 2024 20:12:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=j-davis-com.20230601.gappssmtp.com; s=20230601; t=1709611950; x=1710216750; darn=postgresql.org; h=mime-version:user-agent:date:to:from:subject:message-id:from:to:cc :subject:date:message-id:reply-to; bh=cI+yNj7PJDyW7HFA4YsEG73EjELSQ6HKr0k3nRbF0UQ=; b=Xh59UyCuhYMbyaOFYaLuO397m4TbwW9O8wU3j+jMlKFqHfcl2k0GlExgOtA00+0Uht 4CABPLot8FPOgHyPPAG9lYd9EPZJ2Xh/YJhjBIzUN82WBH7fPR1aKCYogeJ1nBzVIeQD zgB8Dd5UPngw90J3WyNVBfPG65ffEKIW26g5IWp8vcYCDkVdVOQ4KStLpMr3FTGgN9hB SGP3R9j/4qassdM+AdHtKR1ZUM71HkIgTuvf2bepwBTGGOlr+SCP49m7HbH2kFDX1YZq jsa/VtExbpsCJA+d7TYk7l/9yERex4i8xLAtlyt9e5ukI5xMrTf83DJpIYl8+c8biRK9 Y0dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709611950; x=1710216750; h=mime-version:user-agent:date:to:from:subject:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cI+yNj7PJDyW7HFA4YsEG73EjELSQ6HKr0k3nRbF0UQ=; b=bWRLFNXcCNDyuF59YZlN/OpXk5KCxMHb1AzN39OVLCN5knFDL1scbZmu7i0eJNn4LH XQxxulk9yYvZrM/Mq+vtD1RAkdaJFF4gs+SY8h5t0pZkZlljxFHE5rKfG1KI7XqfeNnD oqWgNZOBDz3mbIwnOYE5ph2VXTiUhrku2j/BS/4iMImXBsJON7sF7EPfSUzyIQpYShQv mDPjy2EwKI6CdLkVY3ngyNFCW35jAcKsHLjEDMPU1XHATkTH3tu0NZM3KvnUejB0dFRj mQBlQpowR1CG8RVKWcHASYoabhotnwXPEJIr5k0aYH/1LVhGcAjPSkuuFkiCd62N+P9X W0ag== X-Gm-Message-State: AOJu0YyKuBMIcsXhaZo19cacUMc4g8Hzx5E8RG09hfvPZYX/PcG71eS2 pPQ2BmPwhDetrIJb2CwTIHOH/+pnNLTDpkkbWz24spO5PH+uN/v3ZRIj0St0JuvFMI18qlZANgY = X-Google-Smtp-Source: AGHT+IFMor4rsvKI0herFtyFTD6/8WHO+2CzDveyQVahzcLCAJpxyUTN4pfaXQmbovhHr7U0EhJSyQ== X-Received: by 2002:a17:902:dac9:b0:1dc:c161:bce6 with SMTP id q9-20020a170902dac900b001dcc161bce6mr1360948plx.15.1709611949959; Mon, 04 Mar 2024 20:12:29 -0800 (PST) Received: from [172.18.10.46] ([12.126.244.130]) by smtp.gmail.com with ESMTPSA id h12-20020a170902f7cc00b001dcac08a360sm9393821plw.175.2024.03.04.20.12.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 20:12:29 -0800 (PST) Message-ID: <0f64b4c7fc200890f2055ce4d6650e9c2191fac2.camel@j-davis.com> Subject: Query stuck with wait event IPC / ParallelFinish From: Jeff Davis To: pgsql-bugs@postgresql.org Date: Mon, 04 Mar 2024 20:12:28 -0800 Content-Type: multipart/mixed; boundary="=-TbOpeewhGhDHlYJVJm+D" User-Agent: Evolution 3.44.4-0ubuntu2 MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --=-TbOpeewhGhDHlYJVJm+D Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable If the attached test patch is applied (no code changes) to master, and the "amcheck" target is run with debug_parallel_query =3D regress, I see the following stuck query: # select application_name, now() - state_change AS duration, wait_event_type, wait_event, state, query from pg_stat_activity where pid<>pg_backend_pid() and backend_type=3D'client backend'; application_name | 004_verify_nbtree_unique.pl duration | 00:12:29.456255 wait_event_type | IPC wait_event | ParallelFinish state | active query | DELETE FROM bttest_unique3 WHERE 380 <=3D i AND i <=3D 420; I found this while fixing a parallel query problem with 2af07e2f74, but the problem can be observed on an earlier version of master as well (2c29e7fc95) so it's unrelated to my change. I haven't investigated much, but I don't see an explanation other than a bug. It reproduces on my local machine consistently. Regards, Jeff Davis --=-TbOpeewhGhDHlYJVJm+D Content-Disposition: attachment; filename="v1-0001-IPC-ParallelFinish-not-finishing.patch" Content-Type: text/x-patch; name="v1-0001-IPC-ParallelFinish-not-finishing.patch"; charset="UTF-8" Content-Transfer-Encoding: base64 RnJvbSBlZWE5YmRjNDU0ZDgwM2MyMWMyYzQ3NWVhZWIyYWE3NTIxMzMyMzM3IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBKZWZmIERhdmlzIDxqZWZmQGotZGF2aXMuY29tPgpEYXRlOiBN b24sIDQgTWFyIDIwMjQgMTk6NDQ6NDcgLTA4MDAKU3ViamVjdDogW1BBVENIIHYxXSBJUEMgUGFy YWxsZWxGaW5pc2ggbm90IGZpbmlzaGluZwoKLS0tCiBjb250cmliL2FtY2hlY2svdC8wMDRfdmVy aWZ5X25idHJlZV91bmlxdWUucGwgfCAzMyArKysrKysrKysrKy0tLS0tLS0tCiAxIGZpbGUgY2hh bmdlZCwgMjAgaW5zZXJ0aW9ucygrKSwgMTMgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvY29u dHJpYi9hbWNoZWNrL3QvMDA0X3ZlcmlmeV9uYnRyZWVfdW5pcXVlLnBsIGIvY29udHJpYi9hbWNo ZWNrL3QvMDA0X3ZlcmlmeV9uYnRyZWVfdW5pcXVlLnBsCmluZGV4IDNmNDc0YTE1OGEuLjRiNzA0 ZTY4MTUgMTAwNjQ0Ci0tLSBhL2NvbnRyaWIvYW1jaGVjay90LzAwNF92ZXJpZnlfbmJ0cmVlX3Vu aXF1ZS5wbAorKysgYi9jb250cmliL2FtY2hlY2svdC8wMDRfdmVyaWZ5X25idHJlZV91bmlxdWUu cGwKQEAgLTIwLDggKzIwLDExIEBAICRub2RlLT5zYWZlX3BzcWwoCiAJJ3Bvc3RncmVzJywgcSgK IAlDUkVBVEUgRVhURU5TSU9OIGFtY2hlY2s7CiAKKwlDUkVBVEUgU0NIRU1BIHRlc3RfYW1jaGVj azsKKwlTRVQgc2VhcmNoX3BhdGggPSB0ZXN0X2FtY2hlY2s7CisKIAlDUkVBVEUgRlVOQ1RJT04g b2tfY21wIChpbnQ0LCBpbnQ0KQotCVJFVFVSTlMgaW50IExBTkdVQUdFIHNxbCBBUworCVJFVFVS TlMgaW50IExBTkdVQUdFIHNxbCBTRVQgc2VhcmNoX3BhdGggPSB0ZXN0X2FtY2hlY2sgQVMKIAkk JAogCQlTRUxFQ1QKIAkJCUNBU0UgV0hFTiAkMSA8ICQyIFRIRU4gLTEKQEAgLTM0LDcgKzM3LDcg QEAgJG5vZGUtPnNhZmVfcHNxbCgKIAktLS0gQ2hlY2sgMTogdW5pcXVlbmVzcyB2aW9sYXRpb24u CiAJLS0tCiAJQ1JFQVRFIEZVTkNUSU9OIG9rX2NtcDEgKGludDQsIGludDQpCi0JUkVUVVJOUyBp bnQgTEFOR1VBR0Ugc3FsIEFTCisJUkVUVVJOUyBpbnQgTEFOR1VBR0Ugc3FsIFNFVCBzZWFyY2hf cGF0aCA9IHRlc3RfYW1jaGVjayBBUwogCSQkCiAJCVNFTEVDVCBva19jbXAoJDEsICQyKTsKIAkk JDsKQEAgLTQzLDcgKzQ2LDcgQEAgJG5vZGUtPnNhZmVfcHNxbCgKIAktLS0gTWFrZSB2YWx1ZXMg NzY4IGFuZCA3NjkgbG9vayBlcXVhbC4KIAktLS0KIAlDUkVBVEUgRlVOQ1RJT04gYmFkX2NtcDEg KGludDQsIGludDQpCi0JUkVUVVJOUyBpbnQgTEFOR1VBR0Ugc3FsIEFTCisJUkVUVVJOUyBpbnQg TEFOR1VBR0Ugc3FsIFNFVCBzZWFyY2hfcGF0aCA9IHRlc3RfYW1jaGVjayBBUwogCSQkCiAJCVNF TEVDVAogCQkJQ0FTRSBXSEVOICgkMSA9IDc2OCBBTkQgJDIgPSA3NjkpIE9SCkBAIC01NiwxMyAr NTksMTMgQEAgJG5vZGUtPnNhZmVfcHNxbCgKIAktLS0gQ2hlY2sgMjogdW5pcXVlbmVzcyB2aW9s YXRpb24gd2l0aG91dCBkZWR1cGxpY2F0aW9uLgogCS0tLQogCUNSRUFURSBGVU5DVElPTiBva19j bXAyIChpbnQ0LCBpbnQ0KQotCVJFVFVSTlMgaW50IExBTkdVQUdFIHNxbCBBUworCVJFVFVSTlMg aW50IExBTkdVQUdFIHNxbCBTRVQgc2VhcmNoX3BhdGggPSB0ZXN0X2FtY2hlY2sgQVMKIAkkJAog CQlTRUxFQ1Qgb2tfY21wKCQxLCAkMik7CiAJJCQ7CiAKIAlDUkVBVEUgRlVOQ1RJT04gYmFkX2Nt cDIgKGludDQsIGludDQpCi0JUkVUVVJOUyBpbnQgTEFOR1VBR0Ugc3FsIEFTCisJUkVUVVJOUyBp bnQgTEFOR1VBR0Ugc3FsIFNFVCBzZWFyY2hfcGF0aCA9IHRlc3RfYW1jaGVjayBBUwogCSQkCiAJ CVNFTEVDVAogCQkJQ0FTRSBXSEVOICQxID0gJDIgQU5EICQxID0gNDAwIFRIRU4gLTEKQEAgLTc0 LDEzICs3NywxMyBAQCAkbm9kZS0+c2FmZV9wc3FsKAogCS0tLSBDaGVjayAzOiB1bmlxdWVuZXNz IHZpb2xhdGlvbiB3aXRoIGRlZHVwbGljYXRpb24uCiAJLS0tCiAJQ1JFQVRFIEZVTkNUSU9OIG9r X2NtcDMgKGludDQsIGludDQpCi0JUkVUVVJOUyBpbnQgTEFOR1VBR0Ugc3FsIEFTCisJUkVUVVJO UyBpbnQgTEFOR1VBR0Ugc3FsIFNFVCBzZWFyY2hfcGF0aCA9IHRlc3RfYW1jaGVjayBBUwogCSQk CiAJCVNFTEVDVCBva19jbXAoJDEsICQyKTsKIAkkJDsKIAogCUNSRUFURSBGVU5DVElPTiBiYWRf Y21wMyAoaW50NCwgaW50NCkKLQlSRVRVUk5TIGludCBMQU5HVUFHRSBzcWwgQVMKKwlSRVRVUk5T IGludCBMQU5HVUFHRSBzcWwgU0VUIHNlYXJjaF9wYXRoID0gdGVzdF9hbWNoZWNrIEFTCiAJJCQK IAkJU0VMRUNUIGJhZF9jbXAyKCQxLCAkMik7CiAJJCQ7CkBAIC0xNDIsNyArMTQ1LDcgQEAgbXkg KCRyZXN1bHQsICRzdGRvdXQsICRzdGRlcnIpOwogIyBXZSBoYXZlIG5vdCB5ZXQgYnJva2VuIHRo ZSBpbmRleCwgc28gd2Ugc2hvdWxkIGdldCBubyBjb3JydXB0aW9uCiAkcmVzdWx0ID0gJG5vZGUt PnNhZmVfcHNxbCgKIAkncG9zdGdyZXMnLCBxKAotCVNFTEVDVCBidF9pbmRleF9jaGVjaygnYnR0 ZXN0X3VuaXF1ZV9pZHgxJywgdHJ1ZSwgdHJ1ZSk7CisJU0VMRUNUIGJ0X2luZGV4X2NoZWNrKCd0 ZXN0X2FtY2hlY2suYnR0ZXN0X3VuaXF1ZV9pZHgxJywgdHJ1ZSwgdHJ1ZSk7CiApKTsKIGlzKCRy ZXN1bHQsICcnLCAncnVuIGFtY2hlY2sgb24gbm9uLWJyb2tlbiBidHRlc3RfdW5pcXVlX2lkeDEn KTsKIApAQCAtMTUwLDYgKzE1Myw3IEBAIGlzKCRyZXN1bHQsICcnLCAncnVuIGFtY2hlY2sgb24g bm9uLWJyb2tlbiBidHRlc3RfdW5pcXVlX2lkeDEnKTsKICMgdmFsdWVzIHRvIGJlIGVxdWFsLgog JG5vZGUtPnNhZmVfcHNxbCgKIAkncG9zdGdyZXMnLCBxKAorCVNFVCBzZWFyY2hfcGF0aCA9IHRl c3RfYW1jaGVjazsKIAlVUERBVEUgcGdfY2F0YWxvZy5wZ19hbXByb2MgU0VUCiAJCSAgIGFtcHJv YyA9ICdiYWRfY21wMSc6OnJlZ3Byb2MKIAlXSEVSRSBhbXByb2MgPSAnb2tfY21wMSc6OnJlZ3By b2M7CkBAIC0xNTcsNyArMTYxLDcgQEAgJG5vZGUtPnNhZmVfcHNxbCgKIAogKCRyZXN1bHQsICRz dGRvdXQsICRzdGRlcnIpID0gJG5vZGUtPnBzcWwoCiAJJ3Bvc3RncmVzJywgcSgKLQlTRUxFQ1Qg YnRfaW5kZXhfY2hlY2soJ2J0dGVzdF91bmlxdWVfaWR4MScsIHRydWUsIHRydWUpOworCVNFTEVD VCBidF9pbmRleF9jaGVjaygndGVzdF9hbWNoZWNrLmJ0dGVzdF91bmlxdWVfaWR4MScsIHRydWUs IHRydWUpOwogKSk7CiBvayggJHN0ZGVyciA9fiAvaW5kZXggdW5pcXVlbmVzcyBpcyB2aW9sYXRl ZCBmb3IgaW5kZXggImJ0dGVzdF91bmlxdWVfaWR4MSIvLAogCSdkZXRlY3RlZCB1bmlxdWVuZXNz IHZpb2xhdGlvbiBmb3IgaW5kZXggImJ0dGVzdF91bmlxdWVfaWR4MSInKTsKQEAgLTE3NSwxMyAr MTc5LDE0IEBAIG9rKCAkc3RkZXJyID1+IC9pbmRleCB1bmlxdWVuZXNzIGlzIHZpb2xhdGVkIGZv ciBpbmRleCAiYnR0ZXN0X3VuaXF1ZV9pZHgxIi8sCiAjIGJ1dCBubyB1bmlxdWVuZXNzIHZpb2xh dGlvbi4KICgkcmVzdWx0LCAkc3Rkb3V0LCAkc3RkZXJyKSA9ICRub2RlLT5wc3FsKAogCSdwb3N0 Z3JlcycsIHEoCi0JU0VMRUNUIGJ0X2luZGV4X2NoZWNrKCdidHRlc3RfdW5pcXVlX2lkeDInLCB0 cnVlLCB0cnVlKTsKKwlTRUxFQ1QgYnRfaW5kZXhfY2hlY2soJ3Rlc3RfYW1jaGVjay5idHRlc3Rf dW5pcXVlX2lkeDInLCB0cnVlLCB0cnVlKTsKICkpOwogb2soICRzdGRlcnIgPX4gL2l0ZW0gb3Jk ZXIgaW52YXJpYW50IHZpb2xhdGVkIGZvciBpbmRleCAiYnR0ZXN0X3VuaXF1ZV9pZHgyIi8sCiAJ J2RldGVjdGVkIGl0ZW0gb3JkZXIgaW52YXJpYW50IHZpb2xhdGlvbiBmb3IgaW5kZXggImJ0dGVz dF91bmlxdWVfaWR4MiInKTsKIAogJG5vZGUtPnNhZmVfcHNxbCgKIAkncG9zdGdyZXMnLCBxKAor CVNFVCBzZWFyY2hfcGF0aCA9IHRlc3RfYW1jaGVjazsKIAlVUERBVEUgcGdfY2F0YWxvZy5wZ19h bXByb2MgU0VUCiAJCSAgIGFtcHJvYyA9ICdva19jbXAyJzo6cmVncHJvYwogCVdIRVJFIGFtcHJv YyA9ICdiYWRfY21wMic6OnJlZ3Byb2M7CkBAIC0xODksNyArMTk0LDcgQEAgJG5vZGUtPnNhZmVf cHNxbCgKIAogKCRyZXN1bHQsICRzdGRvdXQsICRzdGRlcnIpID0gJG5vZGUtPnBzcWwoCiAJJ3Bv c3RncmVzJywgcSgKLQlTRUxFQ1QgYnRfaW5kZXhfY2hlY2soJ2J0dGVzdF91bmlxdWVfaWR4Mics IHRydWUsIHRydWUpOworCVNFTEVDVCBidF9pbmRleF9jaGVjaygndGVzdF9hbWNoZWNrLmJ0dGVz dF91bmlxdWVfaWR4MicsIHRydWUsIHRydWUpOwogKSk7CiBvayggJHN0ZGVyciA9fiAvaW5kZXgg dW5pcXVlbmVzcyBpcyB2aW9sYXRlZCBmb3IgaW5kZXggImJ0dGVzdF91bmlxdWVfaWR4MiIvLAog CSdkZXRlY3RlZCB1bmlxdWVuZXNzIHZpb2xhdGlvbiBmb3IgaW5kZXggImJ0dGVzdF91bmlxdWVf aWR4MiInKTsKQEAgLTIwNiw3ICsyMTEsNyBAQCBvayggJHN0ZGVyciA9fiAvaW5kZXggdW5pcXVl bmVzcyBpcyB2aW9sYXRlZCBmb3IgaW5kZXggImJ0dGVzdF91bmlxdWVfaWR4MiIvLAogIyBidXQg bm8gdW5pcXVlbmVzcyB2aW9sYXRpb24uCiAoJHJlc3VsdCwgJHN0ZG91dCwgJHN0ZGVycikgPSAk bm9kZS0+cHNxbCgKIAkncG9zdGdyZXMnLCBxKAotCVNFTEVDVCBidF9pbmRleF9jaGVjaygnYnR0 ZXN0X3VuaXF1ZV9pZHgzJywgdHJ1ZSwgdHJ1ZSk7CisJU0VMRUNUIGJ0X2luZGV4X2NoZWNrKCd0 ZXN0X2FtY2hlY2suYnR0ZXN0X3VuaXF1ZV9pZHgzJywgdHJ1ZSwgdHJ1ZSk7CiApKTsKIG9rKCAk c3RkZXJyID1+IC9pdGVtIG9yZGVyIGludmFyaWFudCB2aW9sYXRlZCBmb3IgaW5kZXggImJ0dGVz dF91bmlxdWVfaWR4MyIvLAogCSdkZXRlY3RlZCBpdGVtIG9yZGVyIGludmFyaWFudCB2aW9sYXRp b24gZm9yIGluZGV4ICJidHRlc3RfdW5pcXVlX2lkeDMiJyk7CkBAIC0yMTUsNiArMjIwLDcgQEAg b2soICRzdGRlcnIgPX4gL2l0ZW0gb3JkZXIgaW52YXJpYW50IHZpb2xhdGVkIGZvciBpbmRleCAi YnR0ZXN0X3VuaXF1ZV9pZHgzIi8sCiAjIHdpdGggZGlmZmVyZW50IHZpc2liaWxpdHkuCiAkbm9k ZS0+c2FmZV9wc3FsKAogCSdwb3N0Z3JlcycsIHEoCisJU0VUIHNlYXJjaF9wYXRoID0gdGVzdF9h bWNoZWNrOwogCURFTEVURSBGUk9NIGJ0dGVzdF91bmlxdWUzIFdIRVJFIDM4MCA8PSBpIEFORCBp IDw9IDQyMDsKIAlJTlNFUlQgSU5UTyBidHRlc3RfdW5pcXVlMyAoU0VMRUNUICogRlJPTSBnZW5l cmF0ZV9zZXJpZXMoMzgwLCA0MjApKTsKIAlJTlNFUlQgSU5UTyBidHRlc3RfdW5pcXVlMyBWQUxV RVMgKDQwMCk7CkBAIC0yMjgsNiArMjM0LDcgQEAgJG5vZGUtPnNhZmVfcHNxbCgKIAogJG5vZGUt PnNhZmVfcHNxbCgKIAkncG9zdGdyZXMnLCBxKAorCVNFVCBzZWFyY2hfcGF0aCA9IHRlc3RfYW1j aGVjazsKIAlVUERBVEUgcGdfY2F0YWxvZy5wZ19hbXByb2MgU0VUCiAJCSAgIGFtcHJvYyA9ICdv a19jbXAzJzo6cmVncHJvYwogCVdIRVJFIGFtcHJvYyA9ICdiYWRfY21wMyc6OnJlZ3Byb2M7CkBA IC0yMzUsNyArMjQyLDcgQEAgJG5vZGUtPnNhZmVfcHNxbCgKIAogKCRyZXN1bHQsICRzdGRvdXQs ICRzdGRlcnIpID0gJG5vZGUtPnBzcWwoCiAJJ3Bvc3RncmVzJywgcSgKLQlTRUxFQ1QgYnRfaW5k ZXhfY2hlY2soJ2J0dGVzdF91bmlxdWVfaWR4MycsIHRydWUsIHRydWUpOworCVNFTEVDVCBidF9p bmRleF9jaGVjaygndGVzdF9hbWNoZWNrLmJ0dGVzdF91bmlxdWVfaWR4MycsIHRydWUsIHRydWUp OwogKSk7CiBvayggJHN0ZGVyciA9fiAvaW5kZXggdW5pcXVlbmVzcyBpcyB2aW9sYXRlZCBmb3Ig aW5kZXggImJ0dGVzdF91bmlxdWVfaWR4MyIvLAogCSdkZXRlY3RlZCB1bmlxdWVuZXNzIHZpb2xh dGlvbiBmb3IgaW5kZXggImJ0dGVzdF91bmlxdWVfaWR4MyInKTsKLS0gCjIuMzQuMQoK --=-TbOpeewhGhDHlYJVJm+D--