Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sqCkV-007ol4-7W for pgsql-hackers@arkaria.postgresql.org; Mon, 16 Sep 2024 14:30:45 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sqCkT-00Fql5-OM for pgsql-hackers@arkaria.postgresql.org; Mon, 16 Sep 2024 14:30:41 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sqCkT-00Fqkw-6c for pgsql-hackers@lists.postgresql.org; Mon, 16 Sep 2024 14:30:41 +0000 Received: from mail.postgrespro.ru ([93.174.131.139]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sqCkO-001UhJ-HW for pgsql-hackers@lists.postgresql.org; Mon, 16 Sep 2024 14:30:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=postgrespro.ru; s=mx2023; t=1726497036; bh=eNkI22aNT3F+FriQ1YvjYyS7Y1acEN3AcQoAGcyRUi4=; h=Message-ID:Date:User-Agent:Subject:To:Cc:References:From: In-Reply-To:From; b=ThI88rVKs4qXDW0fu17qYrBHIMjT/00iJC/lLvyGED63R3G0ofi0a+CSovD5HCy3y zrCnqBZwl7TOZgm0+LzHQAKLpH6eHPKLPbxpPDv0Pokfs+hsUM3R26IYZ9svEFEg+W TV19bMthKOdezdpjNsLKlYXoBmJq7EeFjpf2gbjfffelR2IXHg5ONJxjReZYOJUNxs +yBLSjuskzbZQoGTgBNO2eZGhS4zoMsBMVN/gtkpwklkPnp3kGmTfpDCd3vuw6bkv9 +eZ1jla1QPjlxTx0HSH8S5MryvlJBPHuf0JmLRJBeSaiJehhYJqXYvBaSbMQLKIcKU 76xhb+NLdQJ3Q== Received: from [172.30.32.46] (unknown [172.30.32.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: a.melnikov@postgrespro.ru) by mail.postgrespro.ru (Postfix/587) with ESMTPSA id 27395601EE; Mon, 16 Sep 2024 17:30:36 +0300 (MSK) Content-Type: multipart/mixed; boundary="------------GBBr0owxEivUgVlvfueZfPYT" Message-ID: <77032579-4dc3-4552-9a09-30aaa114c144@postgrespro.ru> Date: Mon, 16 Sep 2024 17:30:35 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: May be BUG. Periodic burst growth of the checkpoint_req counter on replica. To: Fujii Masao , Alexander Korotkov Cc: Magnus Hagander , "Anton A. Melnikov" , Andres Freund , Kyotaro Horiguchi , pgsql-hackers@lists.postgresql.org References: <99b2ccd1-a77a-962a-0837-191cdf56c2b9@inbox.ru> <20220907.163946.2214338139097492905.horikyota.ntt@gmail.com> <84b732cb-2eef-1d4c-e9f3-48514d492f55@inbox.ru> <20221206184453.ebr4eqwbnwfxl476@awork3.anarazel.de> <42223bc5-23ad-c129-7779-92df92b21bae@inbox.ru> <7bd1c12b-4219-45bf-88a6-8cacd008dfdc@postgrespro.ru> <3ed0590b-8132-482e-a9e6-bd694e4a48d7@postgrespro.ru> <9ea77f40-818d-4841-9dee-158ac8f6e690@oss.nttdata.com> Content-Language: en-US From: "Anton A. Melnikov" In-Reply-To: <9ea77f40-818d-4841-9dee-158ac8f6e690@oss.nttdata.com> X-KSMG-AntiPhishing: NotDetected, bases: 2024/09/16 12:54:00 X-KSMG-AntiSpam-Interceptor-Info: not scanned X-KSMG-AntiSpam-Status: not scanned, disabled by settings X-KSMG-AntiVirus: Kaspersky Secure Mail Gateway, version 2.1.0.7854, bases: 2024/09/16 11:48:00 #26596414 X-KSMG-AntiVirus-Status: NotDetected, skipped X-KSMG-LinksScanning: not scanned, disabled by settings X-KSMG-Message-Action: skipped X-KSMG-Rule-ID: 1 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------GBBr0owxEivUgVlvfueZfPYT Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi! On 13.09.2024 18:20, Fujii Masao wrote: > > If I understand correctly, restartpoints_timed and restartpoints_done were > separated because a restartpoint can be skipped. restartpoints_timed counts > when a restartpoint is triggered by a timeout, whether it runs or not, > while restartpoints_done only tracks completed restartpoints. > > Similarly, I believe checkpoints should be handled the same way. > Checkpoints can also be skipped when the system is idle, but currently, > num_timed counts even the skipped ones, despite its documentation stating > it's the "Number of scheduled checkpoints that have been performed." > > Why not separate num_timed into something like checkpoints_timed and > checkpoints_done to reflect these different counters? +1 This idea seems quite tenable to me. There is a small clarification. Now if there were no skipped restartpoints then restartpoints_done will be equal to restartpoints_timed + restartpoints_req. Similar for checkpoints. So i tried to introduce num_done counter for checkpoints in the patch attached. I'm not sure should we include testing for the case when num_done is less than num_timed + num_requested to the regress tests. I haven't been able to get it in a short time yet. E.g. such a case may be obtained when an a error "checkpoints are occurring too frequently" as follows: -set checkpoint_timeout = 30 and checkpoint_warning = 40 in the postgresql.conf -start server -do periodically bulk insertions in the 1st client (e.g. insert into test values (generate_series(1,1E7));) -watch for pg_stat_checkpointer in the 2nd one: # SELECT CURRENT_TIME; select * from pg_stat_checkpointer; # \watch After some time, in the log will appear: 2024-09-16 16:38:47.888 MSK [193733] LOG: checkpoints are occurring too frequently (13 seconds apart) 2024-09-16 16:38:47.888 MSK [193733] HINT: Consider increasing the configuration parameter "max_wal_size". And num_timed + num_requested will become greater than num_done. Would be nice to find some simpler and faster way. With the best regards, -- Anton A. Melnikov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company --------------GBBr0owxEivUgVlvfueZfPYT Content-Type: text/x-patch; charset=UTF-8; name="v1-0001-Introduce-num_done-counter-in-the-pg_stat_checkpointer.patch" Content-Disposition: attachment; filename*0="v1-0001-Introduce-num_done-counter-in-the-pg_stat_checkpoint"; filename*1="er.patch" Content-Transfer-Encoding: base64 RnJvbSBmY2QwYjYxZDFmMTcxOGRiZjY2NGNiMzUwOWFhZDE2NTQzZDY1Mzc1IE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiAiQW50b24gQS4gTWVsbmlrb3YiIDxhLm1lbG5pa292 QHBvc3RncmVzcHJvLnJ1PgpEYXRlOiBNb24sIDE2IFNlcCAyMDI0IDE2OjEyOjA3ICswMzAw ClN1YmplY3Q6IFtQQVRDSF0gSW50cm9kdWNlIG51bV9kb25lIGNvdW50ZXIgaW4gdGhlIHBn X3N0YXRfY2hlY2twb2ludGVyIHZpZXcKIHRoYXQgcmVmbGVjdHMgbnVtYmVyIG9mIHJlYWxs eSBwZXJmb3JtZWQgY2hlY2twb2ludHMuCgotLS0KIGRvYy9zcmMvc2dtbC9tb25pdG9yaW5n LnNnbWwgICAgICAgICAgICAgICAgICB8IDEzICsrKysrLQogc3JjL2JhY2tlbmQvY2F0YWxv Zy9zeXN0ZW1fdmlld3Muc3FsICAgICAgICAgIHwgIDEgKwogc3JjL2JhY2tlbmQvcG9zdG1h c3Rlci9jaGVja3BvaW50ZXIuYyAgICAgICAgIHwgIDIgKwogLi4uL3V0aWxzL2FjdGl2aXR5 L3Bnc3RhdF9jaGVja3BvaW50ZXIuYyAgICAgIHwgIDIgKwogc3JjL2JhY2tlbmQvdXRpbHMv YWR0L3Bnc3RhdGZ1bmNzLmMgICAgICAgICAgIHwgIDYgKysrCiBzcmMvaW5jbHVkZS9jYXRh bG9nL3BnX3Byb2MuZGF0ICAgICAgICAgICAgICAgfCA0MCArKysrKysrKysrKy0tLS0tLS0t CiBzcmMvaW5jbHVkZS9wZ3N0YXQuaCAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgMSAr CiBzcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3J1bGVzLm91dCAgICAgICAgICAgfCAgMSAr CiA4IGZpbGVzIGNoYW5nZWQsIDQ3IGluc2VydGlvbnMoKyksIDE5IGRlbGV0aW9ucygtKQoK ZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9tb25pdG9yaW5nLnNnbWwgYi9kb2Mvc3JjL3Nn bWwvbW9uaXRvcmluZy5zZ21sCmluZGV4IDkzM2RlNmZlMDdmLi5kYWQ3ZTIzNmE0MyAxMDA2 NDQKLS0tIGEvZG9jL3NyYy9zZ21sL21vbml0b3Jpbmcuc2dtbAorKysgYi9kb2Mvc3JjL3Nn bWwvbW9uaXRvcmluZy5zZ21sCkBAIC0zMDUxLDcgKzMwNTEsNyBAQCBkZXNjcmlwdGlvbiB8 IFdhaXRpbmcgZm9yIGEgbmV3bHkgaW5pdGlhbGl6ZWQgV0FMIGZpbGUgdG8gcmVhY2ggZHVy YWJsZSBzdG9yYWdlCiAgICAgICAgPHN0cnVjdGZpZWxkPm51bV90aW1lZDwvc3RydWN0Zmll bGQ+IDx0eXBlPmJpZ2ludDwvdHlwZT4KICAgICAgIDwvcGFyYT4KICAgICAgIDxwYXJhPgot ICAgICAgIE51bWJlciBvZiBzY2hlZHVsZWQgY2hlY2twb2ludHMgdGhhdCBoYXZlIGJlZW4g cGVyZm9ybWVkCisgICAgICAgTnVtYmVyIG9mIHNjaGVkdWxlZCBjaGVja3BvaW50cwogICAg ICAgPC9wYXJhPjwvZW50cnk+CiAgICAgIDwvcm93PgogCkBAIC0zMDYwLDcgKzMwNjAsMTYg QEAgZGVzY3JpcHRpb24gfCBXYWl0aW5nIGZvciBhIG5ld2x5IGluaXRpYWxpemVkIFdBTCBm aWxlIHRvIHJlYWNoIGR1cmFibGUgc3RvcmFnZQogICAgICAgIDxzdHJ1Y3RmaWVsZD5udW1f cmVxdWVzdGVkPC9zdHJ1Y3RmaWVsZD4gPHR5cGU+YmlnaW50PC90eXBlPgogICAgICAgPC9w YXJhPgogICAgICAgPHBhcmE+Ci0gICAgICAgTnVtYmVyIG9mIHJlcXVlc3RlZCBjaGVja3Bv aW50cyB0aGF0IGhhdmUgYmVlbiBwZXJmb3JtZWQKKyAgICAgICBOdW1iZXIgb2YgYmFja2Vu ZCByZXF1ZXN0ZWQgY2hlY2twb2ludHMKKyAgICAgIDwvcGFyYT48L2VudHJ5PgorICAgICA8 L3Jvdz4KKworICAgICAgPHJvdz4KKyAgICAgIDxlbnRyeSByb2xlPSJjYXRhbG9nX3RhYmxl X2VudHJ5Ij48cGFyYSByb2xlPSJjb2x1bW5fZGVmaW5pdGlvbiI+CisgICAgICAgPHN0cnVj dGZpZWxkPm51bV9kb25lPC9zdHJ1Y3RmaWVsZD4gPHR5cGU+YmlnaW50PC90eXBlPgorICAg ICAgPC9wYXJhPgorICAgICAgPHBhcmE+CisgICAgICAgTnVtYmVyIG9mIGNoZWNrcG9pbnRz IHRoYXQgaGF2ZSBiZWVuIHBlcmZvcm1lZAogICAgICAgPC9wYXJhPjwvZW50cnk+CiAgICAg IDwvcm93PgogCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9jYXRhbG9nL3N5c3RlbV92aWV3 cy5zcWwgYi9zcmMvYmFja2VuZC9jYXRhbG9nL3N5c3RlbV92aWV3cy5zcWwKaW5kZXggN2Zk NWQyNTZhMTguLjQ5MTA5ZGJkYzg2IDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9jYXRhbG9n L3N5c3RlbV92aWV3cy5zcWwKKysrIGIvc3JjL2JhY2tlbmQvY2F0YWxvZy9zeXN0ZW1fdmll d3Muc3FsCkBAIC0xMTM4LDYgKzExMzgsNyBAQCBDUkVBVEUgVklFVyBwZ19zdGF0X2NoZWNr cG9pbnRlciBBUwogICAgIFNFTEVDVAogICAgICAgICBwZ19zdGF0X2dldF9jaGVja3BvaW50 ZXJfbnVtX3RpbWVkKCkgQVMgbnVtX3RpbWVkLAogICAgICAgICBwZ19zdGF0X2dldF9jaGVj a3BvaW50ZXJfbnVtX3JlcXVlc3RlZCgpIEFTIG51bV9yZXF1ZXN0ZWQsCisgICAgICAgIHBn X3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9udW1fcGVyZm9ybWVkKCkgQVMgbnVtX2RvbmUsCiAg ICAgICAgIHBnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9yZXN0YXJ0cG9pbnRzX3RpbWVkKCkg QVMgcmVzdGFydHBvaW50c190aW1lZCwKICAgICAgICAgcGdfc3RhdF9nZXRfY2hlY2twb2lu dGVyX3Jlc3RhcnRwb2ludHNfcmVxdWVzdGVkKCkgQVMgcmVzdGFydHBvaW50c19yZXEsCiAg ICAgICAgIHBnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9yZXN0YXJ0cG9pbnRzX3BlcmZvcm1l ZCgpIEFTIHJlc3RhcnRwb2ludHNfZG9uZSwKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3Bv c3RtYXN0ZXIvY2hlY2twb2ludGVyLmMgYi9zcmMvYmFja2VuZC9wb3N0bWFzdGVyL2NoZWNr cG9pbnRlci5jCmluZGV4IGVlYjczYzg1NzI2Li4wNmFkMmY1MmYyNyAxMDA2NDQKLS0tIGEv c3JjL2JhY2tlbmQvcG9zdG1hc3Rlci9jaGVja3BvaW50ZXIuYworKysgYi9zcmMvYmFja2Vu ZC9wb3N0bWFzdGVyL2NoZWNrcG9pbnRlci5jCkBAIC00OTUsNiArNDk1LDggQEAgQ2hlY2tw b2ludGVyTWFpbihjaGFyICpzdGFydHVwX2RhdGEsIHNpemVfdCBzdGFydHVwX2RhdGFfbGVu KQogCiAJCQkJaWYgKGRvX3Jlc3RhcnRwb2ludCkKIAkJCQkJUGVuZGluZ0NoZWNrcG9pbnRl clN0YXRzLnJlc3RhcnRwb2ludHNfcGVyZm9ybWVkKys7CisJCQkJZWxzZQorCQkJCQlQZW5k aW5nQ2hlY2twb2ludGVyU3RhdHMubnVtX3BlcmZvcm1lZCsrOwogCQkJfQogCQkJZWxzZQog CQkJewpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvYWN0aXZpdHkvcGdzdGF0X2No ZWNrcG9pbnRlci5jIGIvc3JjL2JhY2tlbmQvdXRpbHMvYWN0aXZpdHkvcGdzdGF0X2NoZWNr cG9pbnRlci5jCmluZGV4IGJiZmM5YzdlMTgzLi40YTBhMmQxNDkzYSAxMDA2NDQKLS0tIGEv c3JjL2JhY2tlbmQvdXRpbHMvYWN0aXZpdHkvcGdzdGF0X2NoZWNrcG9pbnRlci5jCisrKyBi L3NyYy9iYWNrZW5kL3V0aWxzL2FjdGl2aXR5L3Bnc3RhdF9jaGVja3BvaW50ZXIuYwpAQCAt NDksNiArNDksNyBAQCBwZ3N0YXRfcmVwb3J0X2NoZWNrcG9pbnRlcih2b2lkKQogI2RlZmlu ZSBDSEVDS1BPSU5URVJfQUNDKGZsZCkgc3RhdHNfc2htZW0tPnN0YXRzLmZsZCArPSBQZW5k aW5nQ2hlY2twb2ludGVyU3RhdHMuZmxkCiAJQ0hFQ0tQT0lOVEVSX0FDQyhudW1fdGltZWQp OwogCUNIRUNLUE9JTlRFUl9BQ0MobnVtX3JlcXVlc3RlZCk7CisJQ0hFQ0tQT0lOVEVSX0FD QyhudW1fcGVyZm9ybWVkKTsKIAlDSEVDS1BPSU5URVJfQUNDKHJlc3RhcnRwb2ludHNfdGlt ZWQpOwogCUNIRUNLUE9JTlRFUl9BQ0MocmVzdGFydHBvaW50c19yZXF1ZXN0ZWQpOwogCUNI RUNLUE9JTlRFUl9BQ0MocmVzdGFydHBvaW50c19wZXJmb3JtZWQpOwpAQCAtMTI3LDYgKzEy OCw3IEBAIHBnc3RhdF9jaGVja3BvaW50ZXJfc25hcHNob3RfY2Iodm9pZCkKICNkZWZpbmUg Q0hFQ0tQT0lOVEVSX0NPTVAoZmxkKSBwZ1N0YXRMb2NhbC5zbmFwc2hvdC5jaGVja3BvaW50 ZXIuZmxkIC09IHJlc2V0LmZsZDsKIAlDSEVDS1BPSU5URVJfQ09NUChudW1fdGltZWQpOwog CUNIRUNLUE9JTlRFUl9DT01QKG51bV9yZXF1ZXN0ZWQpOworCUNIRUNLUE9JTlRFUl9DT01Q KG51bV9wZXJmb3JtZWQpOwogCUNIRUNLUE9JTlRFUl9DT01QKHJlc3RhcnRwb2ludHNfdGlt ZWQpOwogCUNIRUNLUE9JTlRFUl9DT01QKHJlc3RhcnRwb2ludHNfcmVxdWVzdGVkKTsKIAlD SEVDS1BPSU5URVJfQ09NUChyZXN0YXJ0cG9pbnRzX3BlcmZvcm1lZCk7CmRpZmYgLS1naXQg YS9zcmMvYmFja2VuZC91dGlscy9hZHQvcGdzdGF0ZnVuY3MuYyBiL3NyYy9iYWNrZW5kL3V0 aWxzL2FkdC9wZ3N0YXRmdW5jcy5jCmluZGV4IDMzYzdiMjU1NjBiLi41NjI0YmVmMThlNyAx MDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3Bnc3RhdGZ1bmNzLmMKKysrIGIv c3JjL2JhY2tlbmQvdXRpbHMvYWR0L3Bnc3RhdGZ1bmNzLmMKQEAgLTExOTEsNiArMTE5MSwx MiBAQCBwZ19zdGF0X2dldF9jaGVja3BvaW50ZXJfbnVtX3JlcXVlc3RlZChQR19GVU5DVElP Tl9BUkdTKQogCVBHX1JFVFVSTl9JTlQ2NChwZ3N0YXRfZmV0Y2hfc3RhdF9jaGVja3BvaW50 ZXIoKS0+bnVtX3JlcXVlc3RlZCk7CiB9CiAKK0RhdHVtCitwZ19zdGF0X2dldF9jaGVja3Bv aW50ZXJfbnVtX3BlcmZvcm1lZChQR19GVU5DVElPTl9BUkdTKQoreworCVBHX1JFVFVSTl9J TlQ2NChwZ3N0YXRfZmV0Y2hfc3RhdF9jaGVja3BvaW50ZXIoKS0+bnVtX3BlcmZvcm1lZCk7 Cit9CisKIERhdHVtCiBwZ19zdGF0X2dldF9jaGVja3BvaW50ZXJfcmVzdGFydHBvaW50c190 aW1lZChQR19GVU5DVElPTl9BUkdTKQogewpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvY2F0 YWxvZy9wZ19wcm9jLmRhdCBiL3NyYy9pbmNsdWRlL2NhdGFsb2cvcGdfcHJvYy5kYXQKaW5k ZXggNTNhMDgxZWQ4ODYuLmM4YzFiNzlmZjYzIDEwMDY0NAotLS0gYS9zcmMvaW5jbHVkZS9j YXRhbG9nL3BnX3Byb2MuZGF0CisrKyBiL3NyYy9pbmNsdWRlL2NhdGFsb2cvcGdfcHJvYy5k YXQKQEAgLTU4MDcsNDIgKzU4MDcsNTggQEAKICAgcHJvYXJnbW9kZXMgPT4gJ3tvLG8sbyxv LG8sbyxvfScsCiAgIHByb2FyZ25hbWVzID0+ICd7YXJjaGl2ZWRfY291bnQsbGFzdF9hcmNo aXZlZF93YWwsbGFzdF9hcmNoaXZlZF90aW1lLGZhaWxlZF9jb3VudCxsYXN0X2ZhaWxlZF93 YWwsbGFzdF9mYWlsZWRfdGltZSxzdGF0c19yZXNldH0nLAogICBwcm9zcmMgPT4gJ3BnX3N0 YXRfZ2V0X2FyY2hpdmVyJyB9LAoteyBvaWQgPT4gJzI3NjknLAoreyBvaWQgPT4gJzYzNDcn LAogICBkZXNjciA9PiAnc3RhdGlzdGljczogbnVtYmVyIG9mIHRpbWVkIGNoZWNrcG9pbnRz IHN0YXJ0ZWQgYnkgdGhlIGNoZWNrcG9pbnRlcicsCiAgIHByb25hbWUgPT4gJ3BnX3N0YXRf Z2V0X2NoZWNrcG9pbnRlcl9udW1fdGltZWQnLCBwcm92b2xhdGlsZSA9PiAncycsCiAgIHBy b3BhcmFsbGVsID0+ICdyJywgcHJvcmV0dHlwZSA9PiAnaW50OCcsIHByb2FyZ3R5cGVzID0+ ICcnLAogICBwcm9zcmMgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9udW1fdGltZWQn IH0sCi17IG9pZCA9PiAnMjc3MCcsCit7IG9pZCA9PiAnNjM0OCcsCiAgIGRlc2NyID0+ICdz dGF0aXN0aWNzOiBudW1iZXIgb2YgYmFja2VuZCByZXF1ZXN0ZWQgY2hlY2twb2ludHMgc3Rh cnRlZCBieSB0aGUgY2hlY2twb2ludGVyJywKICAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRf Y2hlY2twb2ludGVyX251bV9yZXF1ZXN0ZWQnLCBwcm92b2xhdGlsZSA9PiAncycsCiAgIHBy b3BhcmFsbGVsID0+ICdyJywgcHJvcmV0dHlwZSA9PiAnaW50OCcsIHByb2FyZ3R5cGVzID0+ ICcnLAogICBwcm9zcmMgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9udW1fcmVxdWVz dGVkJyB9LAoteyBvaWQgPT4gJzYzMjcnLAoreyBvaWQgPT4gJzYzNDknLAorICBkZXNjciA9 PiAnc3RhdGlzdGljczogbnVtYmVyIG9mIGNoZWNrcG9pbnRzIHBlcmZvcm1lZCBieSB0aGUg Y2hlY2twb2ludGVyJywKKyAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRfY2hlY2twb2ludGVy X251bV9wZXJmb3JtZWQnLAorICBwcm92b2xhdGlsZSA9PiAncycsIHByb3BhcmFsbGVsID0+ ICdyJywgcHJvcmV0dHlwZSA9PiAnaW50OCcsCisgIHByb2FyZ3R5cGVzID0+ICcnLAorICBw cm9zcmMgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9udW1fcGVyZm9ybWVkJyB9LAor eyBvaWQgPT4gJzYzNTAnLAogICBkZXNjciA9PiAnc3RhdGlzdGljczogbnVtYmVyIG9mIHRp bWVkIHJlc3RhcnRwb2ludHMgc3RhcnRlZCBieSB0aGUgY2hlY2twb2ludGVyJywKICAgcHJv bmFtZSA9PiAncGdfc3RhdF9nZXRfY2hlY2twb2ludGVyX3Jlc3RhcnRwb2ludHNfdGltZWQn LCBwcm92b2xhdGlsZSA9PiAncycsCiAgIHByb3BhcmFsbGVsID0+ICdyJywgcHJvcmV0dHlw ZSA9PiAnaW50OCcsIHByb2FyZ3R5cGVzID0+ICcnLAogICBwcm9zcmMgPT4gJ3BnX3N0YXRf Z2V0X2NoZWNrcG9pbnRlcl9yZXN0YXJ0cG9pbnRzX3RpbWVkJyB9LAoteyBvaWQgPT4gJzYz MjgnLAoreyBvaWQgPT4gJzYzNTEnLAogICBkZXNjciA9PiAnc3RhdGlzdGljczogbnVtYmVy IG9mIGJhY2tlbmQgcmVxdWVzdGVkIHJlc3RhcnRwb2ludHMgc3RhcnRlZCBieSB0aGUgY2hl Y2twb2ludGVyJywKICAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRfY2hlY2twb2ludGVyX3Jl c3RhcnRwb2ludHNfcmVxdWVzdGVkJywKICAgcHJvdm9sYXRpbGUgPT4gJ3MnLCBwcm9wYXJh bGxlbCA9PiAncicsIHByb3JldHR5cGUgPT4gJ2ludDgnLAogICBwcm9hcmd0eXBlcyA9PiAn JywKICAgcHJvc3JjID0+ICdwZ19zdGF0X2dldF9jaGVja3BvaW50ZXJfcmVzdGFydHBvaW50 c19yZXF1ZXN0ZWQnIH0sCi17IG9pZCA9PiAnNjMyOScsCit7IG9pZCA9PiAnNjM1MicsCiAg IGRlc2NyID0+ICdzdGF0aXN0aWNzOiBudW1iZXIgb2YgYmFja2VuZCBwZXJmb3JtZWQgcmVz dGFydHBvaW50cycsCiAgIHByb25hbWUgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9y ZXN0YXJ0cG9pbnRzX3BlcmZvcm1lZCcsCiAgIHByb3ZvbGF0aWxlID0+ICdzJywgcHJvcGFy YWxsZWwgPT4gJ3InLCBwcm9yZXR0eXBlID0+ICdpbnQ4JywKICAgcHJvYXJndHlwZXMgPT4g JycsCiAgIHByb3NyYyA9PiAncGdfc3RhdF9nZXRfY2hlY2twb2ludGVyX3Jlc3RhcnRwb2lu dHNfcGVyZm9ybWVkJyB9LAoteyBvaWQgPT4gJzI3NzEnLAoreyBvaWQgPT4gJzYzNTMnLAog ICBkZXNjciA9PiAnc3RhdGlzdGljczogbnVtYmVyIG9mIGJ1ZmZlcnMgd3JpdHRlbiBkdXJp bmcgY2hlY2twb2ludHMgYW5kIHJlc3RhcnRwb2ludHMnLAogICBwcm9uYW1lID0+ICdwZ19z dGF0X2dldF9jaGVja3BvaW50ZXJfYnVmZmVyc193cml0dGVuJywgcHJvdm9sYXRpbGUgPT4g J3MnLAogICBwcm9wYXJhbGxlbCA9PiAncicsIHByb3JldHR5cGUgPT4gJ2ludDgnLCBwcm9h cmd0eXBlcyA9PiAnJywKICAgcHJvc3JjID0+ICdwZ19zdGF0X2dldF9jaGVja3BvaW50ZXJf YnVmZmVyc193cml0dGVuJyB9LAoteyBvaWQgPT4gJzYzMTQnLCBkZXNjciA9PiAnc3RhdGlz dGljczogbGFzdCByZXNldCBmb3IgdGhlIGNoZWNrcG9pbnRlcicsCit7IG9pZCA9PiAnNjM1 NCcsIGRlc2NyID0+ICdzdGF0aXN0aWNzOiBsYXN0IHJlc2V0IGZvciB0aGUgY2hlY2twb2lu dGVyJywKICAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRfY2hlY2twb2ludGVyX3N0YXRfcmVz ZXRfdGltZScsIHByb3ZvbGF0aWxlID0+ICdzJywKICAgcHJvcGFyYWxsZWwgPT4gJ3InLCBw cm9yZXR0eXBlID0+ICd0aW1lc3RhbXB0eicsIHByb2FyZ3R5cGVzID0+ICcnLAogICBwcm9z cmMgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9zdGF0X3Jlc2V0X3RpbWUnIH0sCit7 IG9pZCA9PiAnNjM1NScsCisgIGRlc2NyID0+ICdzdGF0aXN0aWNzOiBjaGVja3BvaW50L3Jl c3RhcnRwb2ludCB0aW1lIHNwZW50IHdyaXRpbmcgYnVmZmVycyB0byBkaXNrLCBpbiBtaWxs aXNlY29uZHMnLAorICBwcm9uYW1lID0+ICdwZ19zdGF0X2dldF9jaGVja3BvaW50ZXJfd3Jp dGVfdGltZScsIHByb3ZvbGF0aWxlID0+ICdzJywKKyAgcHJvcGFyYWxsZWwgPT4gJ3InLCBw cm9yZXR0eXBlID0+ICdmbG9hdDgnLCBwcm9hcmd0eXBlcyA9PiAnJywKKyAgcHJvc3JjID0+ ICdwZ19zdGF0X2dldF9jaGVja3BvaW50ZXJfd3JpdGVfdGltZScgfSwKK3sgb2lkID0+ICc2 MzU2JywKKyAgZGVzY3IgPT4gJ3N0YXRpc3RpY3M6IGNoZWNrcG9pbnQvcmVzdGFydHBvaW50 IHRpbWUgc3BlbnQgc3luY2hyb25pemluZyBidWZmZXJzIHRvIGRpc2ssIGluIG1pbGxpc2Vj b25kcycsCisgIHByb25hbWUgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9zeW5jX3Rp bWUnLCBwcm92b2xhdGlsZSA9PiAncycsCisgIHByb3BhcmFsbGVsID0+ICdyJywgcHJvcmV0 dHlwZSA9PiAnZmxvYXQ4JywgcHJvYXJndHlwZXMgPT4gJycsCisgIHByb3NyYyA9PiAncGdf c3RhdF9nZXRfY2hlY2twb2ludGVyX3N5bmNfdGltZScgfSwKIHsgb2lkID0+ICcyNzcyJywK ICAgZGVzY3IgPT4gJ3N0YXRpc3RpY3M6IG51bWJlciBvZiBidWZmZXJzIHdyaXR0ZW4gYnkg dGhlIGJnd3JpdGVyIGZvciBjbGVhbmluZyBkaXJ0eSBidWZmZXJzJywKICAgcHJvbmFtZSA9 PiAncGdfc3RhdF9nZXRfYmd3cml0ZXJfYnVmX3dyaXR0ZW5fY2xlYW4nLCBwcm92b2xhdGls ZSA9PiAncycsCkBAIC01ODU3LDE2ICs1ODczLDYgQEAKICAgcHJvbmFtZSA9PiAncGdfc3Rh dF9nZXRfYmd3cml0ZXJfc3RhdF9yZXNldF90aW1lJywgcHJvdm9sYXRpbGUgPT4gJ3MnLAog ICBwcm9wYXJhbGxlbCA9PiAncicsIHByb3JldHR5cGUgPT4gJ3RpbWVzdGFtcHR6JywgcHJv YXJndHlwZXMgPT4gJycsCiAgIHByb3NyYyA9PiAncGdfc3RhdF9nZXRfYmd3cml0ZXJfc3Rh dF9yZXNldF90aW1lJyB9LAoteyBvaWQgPT4gJzMxNjAnLAotICBkZXNjciA9PiAnc3RhdGlz dGljczogY2hlY2twb2ludC9yZXN0YXJ0cG9pbnQgdGltZSBzcGVudCB3cml0aW5nIGJ1ZmZl cnMgdG8gZGlzaywgaW4gbWlsbGlzZWNvbmRzJywKLSAgcHJvbmFtZSA9PiAncGdfc3RhdF9n ZXRfY2hlY2twb2ludGVyX3dyaXRlX3RpbWUnLCBwcm92b2xhdGlsZSA9PiAncycsCi0gIHBy b3BhcmFsbGVsID0+ICdyJywgcHJvcmV0dHlwZSA9PiAnZmxvYXQ4JywgcHJvYXJndHlwZXMg PT4gJycsCi0gIHByb3NyYyA9PiAncGdfc3RhdF9nZXRfY2hlY2twb2ludGVyX3dyaXRlX3Rp bWUnIH0sCi17IG9pZCA9PiAnMzE2MScsCi0gIGRlc2NyID0+ICdzdGF0aXN0aWNzOiBjaGVj a3BvaW50L3Jlc3RhcnRwb2ludCB0aW1lIHNwZW50IHN5bmNocm9uaXppbmcgYnVmZmVycyB0 byBkaXNrLCBpbiBtaWxsaXNlY29uZHMnLAotICBwcm9uYW1lID0+ICdwZ19zdGF0X2dldF9j aGVja3BvaW50ZXJfc3luY190aW1lJywgcHJvdm9sYXRpbGUgPT4gJ3MnLAotICBwcm9wYXJh bGxlbCA9PiAncicsIHByb3JldHR5cGUgPT4gJ2Zsb2F0OCcsIHByb2FyZ3R5cGVzID0+ICcn LAotICBwcm9zcmMgPT4gJ3BnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9zeW5jX3RpbWUnIH0s CiB7IG9pZCA9PiAnMjg1OScsIGRlc2NyID0+ICdzdGF0aXN0aWNzOiBudW1iZXIgb2YgYnVm ZmVyIGFsbG9jYXRpb25zJywKICAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRfYnVmX2FsbG9j JywgcHJvdm9sYXRpbGUgPT4gJ3MnLCBwcm9wYXJhbGxlbCA9PiAncicsCiAgIHByb3JldHR5 cGUgPT4gJ2ludDgnLCBwcm9hcmd0eXBlcyA9PiAnJywgcHJvc3JjID0+ICdwZ19zdGF0X2dl dF9idWZfYWxsb2MnIH0sCmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS9wZ3N0YXQuaCBiL3Ny Yy9pbmNsdWRlL3Bnc3RhdC5oCmluZGV4IGJlMmM5MTE2OGExLi4zOTg1MGQ4ZjE0ZiAxMDA2 NDQKLS0tIGEvc3JjL2luY2x1ZGUvcGdzdGF0LmgKKysrIGIvc3JjL2luY2x1ZGUvcGdzdGF0 LmgKQEAgLTI5NCw2ICsyOTQsNyBAQCB0eXBlZGVmIHN0cnVjdCBQZ1N0YXRfQ2hlY2twb2lu dGVyU3RhdHMKIHsKIAlQZ1N0YXRfQ291bnRlciBudW1fdGltZWQ7CiAJUGdTdGF0X0NvdW50 ZXIgbnVtX3JlcXVlc3RlZDsKKwlQZ1N0YXRfQ291bnRlciBudW1fcGVyZm9ybWVkOwogCVBn U3RhdF9Db3VudGVyIHJlc3RhcnRwb2ludHNfdGltZWQ7CiAJUGdTdGF0X0NvdW50ZXIgcmVz dGFydHBvaW50c19yZXF1ZXN0ZWQ7CiAJUGdTdGF0X0NvdW50ZXIgcmVzdGFydHBvaW50c19w ZXJmb3JtZWQ7CmRpZmYgLS1naXQgYS9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3J1bGVz Lm91dCBiL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvcnVsZXMub3V0CmluZGV4IGExNjI2 ZjNmYWU5Li5mNTQzNGQ4MzY1YyAxMDA2NDQKLS0tIGEvc3JjL3Rlc3QvcmVncmVzcy9leHBl Y3RlZC9ydWxlcy5vdXQKKysrIGIvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9ydWxlcy5v dXQKQEAgLTE4MjQsNiArMTgyNCw3IEBAIHBnX3N0YXRfYmd3cml0ZXJ8IFNFTEVDVCBwZ19z dGF0X2dldF9iZ3dyaXRlcl9idWZfd3JpdHRlbl9jbGVhbigpIEFTIGJ1ZmZlcnNfY2xlCiAg ICAgcGdfc3RhdF9nZXRfYmd3cml0ZXJfc3RhdF9yZXNldF90aW1lKCkgQVMgc3RhdHNfcmVz ZXQ7CiBwZ19zdGF0X2NoZWNrcG9pbnRlcnwgU0VMRUNUIHBnX3N0YXRfZ2V0X2NoZWNrcG9p bnRlcl9udW1fdGltZWQoKSBBUyBudW1fdGltZWQsCiAgICAgcGdfc3RhdF9nZXRfY2hlY2tw b2ludGVyX251bV9yZXF1ZXN0ZWQoKSBBUyBudW1fcmVxdWVzdGVkLAorICAgIHBnX3N0YXRf Z2V0X2NoZWNrcG9pbnRlcl9udW1fcGVyZm9ybWVkKCkgQVMgbnVtX2RvbmUsCiAgICAgcGdf c3RhdF9nZXRfY2hlY2twb2ludGVyX3Jlc3RhcnRwb2ludHNfdGltZWQoKSBBUyByZXN0YXJ0 cG9pbnRzX3RpbWVkLAogICAgIHBnX3N0YXRfZ2V0X2NoZWNrcG9pbnRlcl9yZXN0YXJ0cG9p bnRzX3JlcXVlc3RlZCgpIEFTIHJlc3RhcnRwb2ludHNfcmVxLAogICAgIHBnX3N0YXRfZ2V0 X2NoZWNrcG9pbnRlcl9yZXN0YXJ0cG9pbnRzX3BlcmZvcm1lZCgpIEFTIHJlc3RhcnRwb2lu dHNfZG9uZSwKLS0gCjIuNDYuMAoK --------------GBBr0owxEivUgVlvfueZfPYT--