Content-Type: multipart/mixed; boundary="------------jdaQYEogzXYdPmc7RiUux8Jb"
Message-ID: <84b732cb-2eef-1d4c-e9f3-48514d492f55@inbox.ru>
Date: Mon, 19 Sep 2022 01:29:21 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: May be BUG. Periodic burst growth of the checkpoint_req counter
 on replica.
Content-Language: en-US
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org
References: <99b2ccd1-a77a-962a-0837-191cdf56c2b9@inbox.ru>
 <20220907.163946.2214338139097492905.horikyota.ntt@gmail.com>
From: "Anton A. Melnikov" <aamelnikov@inbox.ru>
In-Reply-To: <20220907.163946.2214338139097492905.horikyota.ntt@gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/84b732cb-2eef-1d4c-e9f3-48514d492f55%40inbox.ru>
Precedence: bulk

This is a multi-part message in MIME format.
--------------jdaQYEogzXYdPmc7RiUux8Jb
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello!

Thank you very much for your feedback and essential remarks.

On 07.09.2022 10:39, Kyotaro Horiguchi wrote:
> 
> It lets XLogPageRead run the same check with what CreateRestartPoint
> does, so it basically works (it is forgetting a lock, though. The
> reason for omitting the lock in CreateRestartPoint is that it knows
> checkopinter is the only updator of the shared variable.). I'm not
> sure I like that for the code duplication.
> 
> I'm not sure we need to fix that but if we do that, I would impletent
> IsNewCheckPointWALRecs() using XLogCtl->RedoRecPtr and
> XLogCtl->lastCheckPoint.redo instead since they are protected by the
> same lock, and they work more correct way, that is, that can avoid
> restartpoint requests while the last checkpoint is running.  And I
> would rename it as RestartPointAvailable() or something like that.

Corrected patch is attached (v2-0001-Fix-burst-checkpoint_req-growth.patch).
The access to Controlfile was removed so lwlock seems to be not needed.
Some logic duplication is still present and and i'm not quite sure if
it's possible to get rid of it. Would be glad to any suggestions.

> Or I might want to add XLogRestartpointNeeded(readSegNo) to reduce the
> required number of info_lck by reading XLogCtl members at once.

If we place this check into the XLogCheckpointNeeded() this will lead to a double
take of info_lck in XLogPageRead() when the restartpoint request is forming.
As it's done now taking of info_lck will be more rarely.
It seems i probably didn't understand your idea, please clarify it for me.

> Depends on how we see the counter value. I think this can be annoying
> but not a bug. CreateRestartPoint already handles that case. 

Yes! It is in fact annoying as docs says that checkpoint_req counts
"the number of requested checkpoints that have been performed".
But really checkpoints_req counts both the number of checkpoints requests
and restartpoint ones which may not be performed and resources are not spent.
The second frightening factor is the several times faster growth
of the checkpoints_timed counter on the replica vs primary due to scheduling
replays in 15 second if an attempt to create the restartpoint failed.

Here is a patch that leaves all logic as is, but adds a stats about
restartpoints. (v1-0001-Add-restartpoint-stats.patch)
.
For instance, for the same period on primary with this patch:
# SELECT CURRENT_TIME; select * from pg_stat_bgwriter \gx
     current_time
--------------------
  00:19:15.794561+03
(1 row)

-[ RECORD 1 ]---------+-----------------------------
checkpoints_timed     | 4
checkpoints_req       | 10
restartpoints_timed   | 0
restartpoints_req     | 0
restartpoints_done    | 0

On replica:
# SELECT CURRENT_TIME; select * from pg_stat_bgwriter \gx
     current_time
--------------------
  00:19:11.363009+03
(1 row)

-[ RECORD 1 ]---------+------------------------------
checkpoints_timed     | 0
checkpoints_req       | 0
restartpoints_timed   | 42
restartpoints_req     | 67
restartpoints_done    | 10

Only the counters checkpoints_timed, checkpoints_req and restartpoints_done give
the indication of resource-intensive operations.
Without this patch, the user would see on the replica something like this:

checkpoints_timed     | 42
checkpoints_req       | 109


With best regards,

-- 
Anton A. Melnikov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--------------jdaQYEogzXYdPmc7RiUux8Jb
Content-Type: text/x-patch; charset=UTF-8;
 name="v2-0001-Fix-burst-checkpoint_req-growth.patch"
Content-Disposition: attachment;
 filename="v2-0001-Fix-burst-checkpoint_req-growth.patch"
Content-Transfer-Encoding: base64

Y29tbWl0IGQ1YTU4ZDg1ODU2OTJiZTBkMjRkYjk0MTQ4NTkwODhlM2U3ZjdkYWQKQXV0aG9y
OiBBbnRvbiBBLiBNZWxuaWtvdiA8YS5tZWxuaWtvdkBwb3N0Z3Jlc3Byby5ydT4KRGF0ZTog
ICBUdWUgU2VwIDYgMTI6MTg6NTYgMjAyMiArMDMwMAoKICAgIFJlbW92ZSBidXJzdCBncm93
dGggb2YgdGhlIGNoZWNrcG9pbnRfcmVxIG9uIHJlcGxpY2EuCiAgICAKICAgIHdpdGggY29y
cmVjdHRpb25zIGFjY29yZGluZyB0byBjb21tZW50OgogICAgIGh0dHBzOi8vd3d3LnBvc3Rn
cmVzcWwub3JnL21lc3NhZ2UtaWQvMjAyMjA5MDcuMTYzOTQ2LjIyMTQzMzgxMzkwOTc0OTI5
MDUuaG9yaWt5b3RhLm50dCU0MGdtYWlsLmNvbQoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5k
L2FjY2Vzcy90cmFuc2FtL3hsb2cuYyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hs
b2cuYwppbmRleCA4MWQzMzlkNTdkLi4zZWQxYTg3OTQzIDEwMDY0NAotLS0gYS9zcmMvYmFj
a2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMKKysrIGIvc3JjL2JhY2tlbmQvYWNjZXNzL3Ry
YW5zYW0veGxvZy5jCkBAIC05MDU0LDMgKzkwNTQsMjAgQEAgU2V0V2FsV3JpdGVyU2xlZXBp
bmcoYm9vbCBzbGVlcGluZykKIAlYTG9nQ3RsLT5XYWxXcml0ZXJTbGVlcGluZyA9IHNsZWVw
aW5nOwogCVNwaW5Mb2NrUmVsZWFzZSgmWExvZ0N0bC0+aW5mb19sY2spOwogfQorCisvKgor
ICogQ2hlY2sgaWYgYSBuZXcgY2hlY2twb2ludCBXQUwgcmVjb3JkIGhhcyBiZWVuIHJlY2Vp
dmVkIHNpbmNlIHRoZQorICogbGFzdCByZXN0YXJ0cG9pbnQuIFNvIGl0IGlzIHBvc3NpYmxl
IHRvIGNyZWF0ZSBhIG5ldyBvbmUuCisgKi8KK2Jvb2wgUmVzdGFydFBvaW50QXZhaWxhYmxl
KHZvaWQpCit7CisJYm9vbCByZXN1bHQgPSBmYWxzZTsKKworCVNwaW5Mb2NrQWNxdWlyZSgm
WExvZ0N0bC0+aW5mb19sY2spOworCWlmICghWExvZ1JlY1B0cklzSW52YWxpZChYTG9nQ3Rs
LT5sYXN0Q2hlY2tQb2ludFJlY1B0cikKKwkJJiYgWExvZ0N0bC0+bGFzdENoZWNrUG9pbnQu
cmVkbyA+IFhMb2dDdGwtPlJlZG9SZWNQdHIpCisJCQkJcmVzdWx0ID0gdHJ1ZTsKKwlTcGlu
TG9ja1JlbGVhc2UoJlhMb2dDdGwtPmluZm9fbGNrKTsKKworCXJldHVybiByZXN1bHQ7Cit9
CmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9ncmVjb3Zlcnku
YyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2dyZWNvdmVyeS5jCmluZGV4IGI0
MWU2ODI2NjQuLjcyMzZlMGYwYTQgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2FjY2Vzcy90
cmFuc2FtL3hsb2dyZWNvdmVyeS5jCisrKyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2Ft
L3hsb2dyZWNvdmVyeS5jCkBAIC0zMTk5LDcgKzMxOTksMTUgQEAgWExvZ1BhZ2VSZWFkKFhM
b2dSZWFkZXJTdGF0ZSAqeGxvZ3JlYWRlciwgWExvZ1JlY1B0ciB0YXJnZXRQYWdlUHRyLCBp
bnQgcmVxTGVuLAogCQkJewogCQkJCSh2b2lkKSBHZXRSZWRvUmVjUHRyKCk7CiAJCQkJaWYg
KFhMb2dDaGVja3BvaW50TmVlZGVkKHJlYWRTZWdObykpCi0JCQkJCVJlcXVlc3RDaGVja3Bv
aW50KENIRUNLUE9JTlRfQ0FVU0VfWExPRyk7CisJCQkJeworCQkJCQkvKgorCQkJCQkJKiBJ
ZiB0aGVyZSBpcyBubyBuZXcgY2hlY2twb2ludCBXQUwgcmVjb3JkcyBzaW5jZSB0aGUKKwkJ
CQkJCSogbGFzdCByZXN0YXJ0cG9pbnQgdGhlIGNyZWF0aW9uIG9mIG5ldyBvbmUKKwkJCQkJ
CSogd2lsbCBjZXJ0YWlubHkgZmFpbCwgc28gc2tpcCBpdC4KKwkJCQkJCSovCisJCQkJCWlm
IChSZXN0YXJ0UG9pbnRBdmFpbGFibGUoKSkKKwkJCQkJCVJlcXVlc3RDaGVja3BvaW50KENI
RUNLUE9JTlRfQ0FVU0VfWExPRyk7CisJCQkJfQogCQkJfQogCQl9CiAKZGlmZiAtLWdpdCBh
L3NyYy9pbmNsdWRlL2FjY2Vzcy94bG9ncmVjb3ZlcnkuaCBiL3NyYy9pbmNsdWRlL2FjY2Vz
cy94bG9ncmVjb3ZlcnkuaAppbmRleCAwYWE4NWQ5MGU4Li5hMjQ4ZjIyNTFlIDEwMDY0NAot
LS0gYS9zcmMvaW5jbHVkZS9hY2Nlc3MveGxvZ3JlY292ZXJ5LmgKKysrIGIvc3JjL2luY2x1
ZGUvYWNjZXNzL3hsb2dyZWNvdmVyeS5oCkBAIC04Miw2ICs4Miw3IEBAIGV4dGVybiB2b2lk
IFhMb2dSZWNvdmVyeVNobWVtSW5pdCh2b2lkKTsKIAogZXh0ZXJuIHZvaWQgSW5pdFdhbFJl
Y292ZXJ5KENvbnRyb2xGaWxlRGF0YSAqQ29udHJvbEZpbGUsIGJvb2wgKndhc1NodXRkb3du
UHRyLCBib29sICpoYXZlQmFja3VwTGFiZWwsIGJvb2wgKmhhdmVUYmxzcGNNYXApOwogZXh0
ZXJuIHZvaWQgUGVyZm9ybVdhbFJlY292ZXJ5KHZvaWQpOworZXh0ZXJuIGJvb2wgUmVzdGFy
dFBvaW50QXZhaWxhYmxlKHZvaWQpOwogCiAvKgogICogRmluaXNoV2FsUmVjb3ZlcnkoKSBy
ZXR1cm5zIHRoaXMuICBJdCBjb250YWlucyBpbmZvcm1hdGlvbiBhYm91dCB0aGUgcG9pbnQK

--------------jdaQYEogzXYdPmc7RiUux8Jb
Content-Type: text/x-patch; charset=UTF-8;
 name="v1-0001-Add-restartpoint-stats.patch"
Content-Disposition: attachment;
 filename="v1-0001-Add-restartpoint-stats.patch"
Content-Transfer-Encoding: base64

Y29tbWl0IDlhMTU1ZGQ1NDYzNGFhODNkYmFiMDg1Mjg0OWUwYTY5Mjg4YjBiOWYKQXV0aG9y
OiBBbnRvbiBBLiBNZWxuaWtvdiA8YS5tZWxuaWtvdkBwb3N0Z3Jlc3Byby5ydT4KRGF0ZTog
ICBUaHUgU2VwIDE1IDAxOjUzOjE3IDIwMjIgKzAzMDAKCiAgICBBZGQgc3RhdGlzdGljIGFi
b3V0IHJlc3RhcnRwb2ludCBpbnRvIHBnX3N0YXRfYmd3cml0ZXIKCmRpZmYgLS1naXQgYS9z
cmMvYmFja2VuZC9jYXRhbG9nL3N5c3RlbV92aWV3cy5zcWwgYi9zcmMvYmFja2VuZC9jYXRh
bG9nL3N5c3RlbV92aWV3cy5zcWwKaW5kZXggNTVmN2VjNzllMC4uNWVmMmNlMGM4OSAxMDA2
NDQKLS0tIGEvc3JjL2JhY2tlbmQvY2F0YWxvZy9zeXN0ZW1fdmlld3Muc3FsCisrKyBiL3Ny
Yy9iYWNrZW5kL2NhdGFsb2cvc3lzdGVtX3ZpZXdzLnNxbApAQCAtMTEwNCw2ICsxMTA0LDkg
QEAgQ1JFQVRFIFZJRVcgcGdfc3RhdF9iZ3dyaXRlciBBUwogICAgIFNFTEVDVAogICAgICAg
ICBwZ19zdGF0X2dldF9iZ3dyaXRlcl90aW1lZF9jaGVja3BvaW50cygpIEFTIGNoZWNrcG9p
bnRzX3RpbWVkLAogICAgICAgICBwZ19zdGF0X2dldF9iZ3dyaXRlcl9yZXF1ZXN0ZWRfY2hl
Y2twb2ludHMoKSBBUyBjaGVja3BvaW50c19yZXEsCisgICAgICAgIHBnX3N0YXRfZ2V0X2Jn
d3JpdGVyX3RpbWVkX3Jlc3RhcnRwb2ludHMoKSBBUyByZXN0YXJ0cG9pbnRzX3RpbWVkLAor
ICAgICAgICBwZ19zdGF0X2dldF9iZ3dyaXRlcl9yZXF1ZXN0ZWRfcmVzdGFydHBvaW50cygp
IEFTIHJlc3RhcnRwb2ludHNfcmVxLAorICAgICAgICBwZ19zdGF0X2dldF9iZ3dyaXRlcl9w
ZXJmb3JtZWRfcmVzdGFydHBvaW50cygpIEFTIHJlc3RhcnRwb2ludHNfZG9uZSwKICAgICAg
ICAgcGdfc3RhdF9nZXRfY2hlY2twb2ludF93cml0ZV90aW1lKCkgQVMgY2hlY2twb2ludF93
cml0ZV90aW1lLAogICAgICAgICBwZ19zdGF0X2dldF9jaGVja3BvaW50X3N5bmNfdGltZSgp
IEFTIGNoZWNrcG9pbnRfc3luY190aW1lLAogICAgICAgICBwZ19zdGF0X2dldF9iZ3dyaXRl
cl9idWZfd3JpdHRlbl9jaGVja3BvaW50cygpIEFTIGJ1ZmZlcnNfY2hlY2twb2ludCwKZGlm
ZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3Bvc3RtYXN0ZXIvY2hlY2twb2ludGVyLmMgYi9zcmMv
YmFja2VuZC9wb3N0bWFzdGVyL2NoZWNrcG9pbnRlci5jCmluZGV4IDVmYzA3NmZjMTQuLjIy
OTY3MDFkZGYgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3Bvc3RtYXN0ZXIvY2hlY2twb2lu
dGVyLmMKKysrIGIvc3JjL2JhY2tlbmQvcG9zdG1hc3Rlci9jaGVja3BvaW50ZXIuYwpAQCAt
MzUwLDYgKzM1MCw4IEBAIENoZWNrcG9pbnRlck1haW4odm9pZCkKIAkJcGdfdGltZV90CW5v
dzsKIAkJaW50CQkJZWxhcHNlZF9zZWNzOwogCQlpbnQJCQljdXJfdGltZW91dDsKKwkJYm9v
bAkJY2hrcHRfb3JfcnN0cHRfcmVxdWVzdGVkID0gZmFsc2U7CisJCWJvb2wJCWNoa3B0X29y
X3JzdHB0X3RpbWVkID0gZmFsc2U7CiAKIAkJLyogQ2xlYXIgYW55IGFscmVhZHktcGVuZGlu
ZyB3YWtldXBzICovCiAJCVJlc2V0TGF0Y2goTXlMYXRjaCk7CkBAIC0zNjgsNyArMzcwLDcg
QEAgQ2hlY2twb2ludGVyTWFpbih2b2lkKQogCQlpZiAoKCh2b2xhdGlsZSBDaGVja3BvaW50
ZXJTaG1lbVN0cnVjdCAqKSBDaGVja3BvaW50ZXJTaG1lbSktPmNrcHRfZmxhZ3MpCiAJCXsK
IAkJCWRvX2NoZWNrcG9pbnQgPSB0cnVlOwotCQkJUGVuZGluZ0NoZWNrcG9pbnRlclN0YXRz
LnJlcXVlc3RlZF9jaGVja3BvaW50cysrOworCQkJY2hrcHRfb3JfcnN0cHRfcmVxdWVzdGVk
ID0gdHJ1ZTsKIAkJfQogCiAJCS8qCkBAIC0zODIsNyArMzg0LDcgQEAgQ2hlY2twb2ludGVy
TWFpbih2b2lkKQogCQlpZiAoZWxhcHNlZF9zZWNzID49IENoZWNrUG9pbnRUaW1lb3V0KQog
CQl7CiAJCQlpZiAoIWRvX2NoZWNrcG9pbnQpCi0JCQkJUGVuZGluZ0NoZWNrcG9pbnRlclN0
YXRzLnRpbWVkX2NoZWNrcG9pbnRzKys7CisJCQkJY2hrcHRfb3JfcnN0cHRfdGltZWQgPSB0
cnVlOwogCQkJZG9fY2hlY2twb2ludCA9IHRydWU7CiAJCQlmbGFncyB8PSBDSEVDS1BPSU5U
X0NBVVNFX1RJTUU7CiAJCX0KQEAgLTQxOCw2ICs0MjAsMjQgQEAgQ2hlY2twb2ludGVyTWFp
bih2b2lkKQogCQkJaWYgKGZsYWdzICYgQ0hFQ0tQT0lOVF9FTkRfT0ZfUkVDT1ZFUlkpCiAJ
CQkJZG9fcmVzdGFydHBvaW50ID0gZmFsc2U7CiAKKwkJCWlmIChjaGtwdF9vcl9yc3RwdF9y
ZXF1ZXN0ZWQpCisJCQl7CisJCQkJY2hrcHRfb3JfcnN0cHRfcmVxdWVzdGVkID0gZmFsc2U7
CisJCQkJaWYgKGRvX3Jlc3RhcnRwb2ludCkKKwkJCQkJUGVuZGluZ0NoZWNrcG9pbnRlclN0
YXRzLnJlcXVlc3RlZF9yZXN0YXJ0cG9pbnRzKys7CisJCQkJZWxzZQorCQkJCQlQZW5kaW5n
Q2hlY2twb2ludGVyU3RhdHMucmVxdWVzdGVkX2NoZWNrcG9pbnRzKys7CisJCQl9CisKKwkJ
CWlmIChjaGtwdF9vcl9yc3RwdF90aW1lZCkKKwkJCXsKKwkJCQljaGtwdF9vcl9yc3RwdF90
aW1lZCA9IGZhbHNlOworCQkJCWlmIChkb19yZXN0YXJ0cG9pbnQpCisJCQkJCVBlbmRpbmdD
aGVja3BvaW50ZXJTdGF0cy50aW1lZF9yZXN0YXJ0cG9pbnRzKys7CisJCQkJZWxzZQorCQkJ
CQlQZW5kaW5nQ2hlY2twb2ludGVyU3RhdHMudGltZWRfY2hlY2twb2ludHMrKzsKKwkJCX0K
KwogCQkJLyoKIAkJCSAqIFdlIHdpbGwgd2FybiBpZiAoYSkgdG9vIHNvb24gc2luY2UgbGFz
dCBjaGVja3BvaW50ICh3aGF0ZXZlcgogCQkJICogY2F1c2VkIGl0KSBhbmQgKGIpIHNvbWVi
b2R5IHNldCB0aGUgQ0hFQ0tQT0lOVF9DQVVTRV9YTE9HIGZsYWcKQEAgLTQ4MSw2ICs1MDEs
OSBAQCBDaGVja3BvaW50ZXJNYWluKHZvaWQpCiAJCQkJICogY2hlY2twb2ludHMgaGFwcGVu
IGF0IGEgcHJlZGljdGFibGUgc3BhY2luZy4KIAkJCQkgKi8KIAkJCQlsYXN0X2NoZWNrcG9p
bnRfdGltZSA9IG5vdzsKKworCQkJCWlmIChkb19yZXN0YXJ0cG9pbnQpCisJCQkJCVBlbmRp
bmdDaGVja3BvaW50ZXJTdGF0cy5wZXJmb3JtZWRfcmVzdGFydHBvaW50cysrOwogCQkJfQog
CQkJZWxzZQogCQkJewpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvYWN0aXZpdHkv
cGdzdGF0X2NoZWNrcG9pbnRlci5jIGIvc3JjL2JhY2tlbmQvdXRpbHMvYWN0aXZpdHkvcGdz
dGF0X2NoZWNrcG9pbnRlci5jCmluZGV4IGFmOGQ1MTNlN2IuLjlkZTk5YTJjZjEgMTAwNjQ0
Ci0tLSBhL3NyYy9iYWNrZW5kL3V0aWxzL2FjdGl2aXR5L3Bnc3RhdF9jaGVja3BvaW50ZXIu
YworKysgYi9zcmMvYmFja2VuZC91dGlscy9hY3Rpdml0eS9wZ3N0YXRfY2hlY2twb2ludGVy
LmMKQEAgLTQ5LDYgKzQ5LDkgQEAgcGdzdGF0X3JlcG9ydF9jaGVja3BvaW50ZXIodm9pZCkK
ICNkZWZpbmUgQ0hFQ0tQT0lOVEVSX0FDQyhmbGQpIHN0YXRzX3NobWVtLT5zdGF0cy5mbGQg
Kz0gUGVuZGluZ0NoZWNrcG9pbnRlclN0YXRzLmZsZAogCUNIRUNLUE9JTlRFUl9BQ0ModGlt
ZWRfY2hlY2twb2ludHMpOwogCUNIRUNLUE9JTlRFUl9BQ0MocmVxdWVzdGVkX2NoZWNrcG9p
bnRzKTsKKwlDSEVDS1BPSU5URVJfQUNDKHRpbWVkX3Jlc3RhcnRwb2ludHMpOworCUNIRUNL
UE9JTlRFUl9BQ0MocmVxdWVzdGVkX3Jlc3RhcnRwb2ludHMpOworCUNIRUNLUE9JTlRFUl9B
Q0MocGVyZm9ybWVkX3Jlc3RhcnRwb2ludHMpOwogCUNIRUNLUE9JTlRFUl9BQ0MoY2hlY2tw
b2ludF93cml0ZV90aW1lKTsKIAlDSEVDS1BPSU5URVJfQUNDKGNoZWNrcG9pbnRfc3luY190
aW1lKTsKIAlDSEVDS1BPSU5URVJfQUNDKGJ1Zl93cml0dGVuX2NoZWNrcG9pbnRzKTsKQEAg
LTExMiw2ICsxMTUsOSBAQCBwZ3N0YXRfY2hlY2twb2ludGVyX3NuYXBzaG90X2NiKHZvaWQp
CiAjZGVmaW5lIENIRUNLUE9JTlRFUl9DT01QKGZsZCkgcGdTdGF0TG9jYWwuc25hcHNob3Qu
Y2hlY2twb2ludGVyLmZsZCAtPSByZXNldC5mbGQ7CiAJQ0hFQ0tQT0lOVEVSX0NPTVAodGlt
ZWRfY2hlY2twb2ludHMpOwogCUNIRUNLUE9JTlRFUl9DT01QKHJlcXVlc3RlZF9jaGVja3Bv
aW50cyk7CisJQ0hFQ0tQT0lOVEVSX0NPTVAodGltZWRfcmVzdGFydHBvaW50cyk7CisJQ0hF
Q0tQT0lOVEVSX0NPTVAocmVxdWVzdGVkX3Jlc3RhcnRwb2ludHMpOworCUNIRUNLUE9JTlRF
Ul9DT01QKHBlcmZvcm1lZF9yZXN0YXJ0cG9pbnRzKTsKIAlDSEVDS1BPSU5URVJfQ09NUChj
aGVja3BvaW50X3dyaXRlX3RpbWUpOwogCUNIRUNLUE9JTlRFUl9DT01QKGNoZWNrcG9pbnRf
c3luY190aW1lKTsKIAlDSEVDS1BPSU5URVJfQ09NUChidWZfd3JpdHRlbl9jaGVja3BvaW50
cyk7CmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC91dGlscy9hZHQvcGdzdGF0ZnVuY3MuYyBi
L3NyYy9iYWNrZW5kL3V0aWxzL2FkdC9wZ3N0YXRmdW5jcy5jCmluZGV4IGJlMTViNGIyZTUu
LmY3NjQ0OGViYTYgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3V0aWxzL2FkdC9wZ3N0YXRm
dW5jcy5jCisrKyBiL3NyYy9iYWNrZW5kL3V0aWxzL2FkdC9wZ3N0YXRmdW5jcy5jCkBAIC0x
NjY4LDYgKzE2NjgsMjQgQEAgcGdfc3RhdF9nZXRfYmd3cml0ZXJfcmVxdWVzdGVkX2NoZWNr
cG9pbnRzKFBHX0ZVTkNUSU9OX0FSR1MpCiAJUEdfUkVUVVJOX0lOVDY0KHBnc3RhdF9mZXRj
aF9zdGF0X2NoZWNrcG9pbnRlcigpLT5yZXF1ZXN0ZWRfY2hlY2twb2ludHMpOwogfQogCitE
YXR1bQorcGdfc3RhdF9nZXRfYmd3cml0ZXJfdGltZWRfcmVzdGFydHBvaW50cyhQR19GVU5D
VElPTl9BUkdTKQoreworCVBHX1JFVFVSTl9JTlQ2NChwZ3N0YXRfZmV0Y2hfc3RhdF9jaGVj
a3BvaW50ZXIoKS0+dGltZWRfcmVzdGFydHBvaW50cyk7Cit9CisKK0RhdHVtCitwZ19zdGF0
X2dldF9iZ3dyaXRlcl9yZXF1ZXN0ZWRfcmVzdGFydHBvaW50cyhQR19GVU5DVElPTl9BUkdT
KQoreworCVBHX1JFVFVSTl9JTlQ2NChwZ3N0YXRfZmV0Y2hfc3RhdF9jaGVja3BvaW50ZXIo
KS0+cmVxdWVzdGVkX3Jlc3RhcnRwb2ludHMpOworfQorCitEYXR1bQorcGdfc3RhdF9nZXRf
Ymd3cml0ZXJfcGVyZm9ybWVkX3Jlc3RhcnRwb2ludHMoUEdfRlVOQ1RJT05fQVJHUykKK3sK
KwlQR19SRVRVUk5fSU5UNjQocGdzdGF0X2ZldGNoX3N0YXRfY2hlY2twb2ludGVyKCktPnBl
cmZvcm1lZF9yZXN0YXJ0cG9pbnRzKTsKK30KKwogRGF0dW0KIHBnX3N0YXRfZ2V0X2Jnd3Jp
dGVyX2J1Zl93cml0dGVuX2NoZWNrcG9pbnRzKFBHX0ZVTkNUSU9OX0FSR1MpCiB7CmRpZmYg
LS1naXQgYS9zcmMvaW5jbHVkZS9jYXRhbG9nL3BnX3Byb2MuZGF0IGIvc3JjL2luY2x1ZGUv
Y2F0YWxvZy9wZ19wcm9jLmRhdAppbmRleCBhMDdlNzM3YTMzLi5hMWQxZmU2Nzk2IDEwMDY0
NAotLS0gYS9zcmMvaW5jbHVkZS9jYXRhbG9nL3BnX3Byb2MuZGF0CisrKyBiL3NyYy9pbmNs
dWRlL2NhdGFsb2cvcGdfcHJvYy5kYXQKQEAgLTU2MDQsNiArNTYwNCwyMSBAQAogICBwcm9u
YW1lID0+ICdwZ19zdGF0X2dldF9iZ3dyaXRlcl9yZXF1ZXN0ZWRfY2hlY2twb2ludHMnLCBw
cm92b2xhdGlsZSA9PiAncycsCiAgIHByb3BhcmFsbGVsID0+ICdyJywgcHJvcmV0dHlwZSA9
PiAnaW50OCcsIHByb2FyZ3R5cGVzID0+ICcnLAogICBwcm9zcmMgPT4gJ3BnX3N0YXRfZ2V0
X2Jnd3JpdGVyX3JlcXVlc3RlZF9jaGVja3BvaW50cycgfSwKK3sgb2lkID0+ICc5OTM4JywK
KyAgZGVzY3IgPT4gJ3N0YXRpc3RpY3M6IG51bWJlciBvZiB0aW1lZCByZXN0YXJ0cG9pbnRz
IHN0YXJ0ZWQgYnkgdGhlIGJnd3JpdGVyJywKKyAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRf
Ymd3cml0ZXJfdGltZWRfcmVzdGFydHBvaW50cycsIHByb3ZvbGF0aWxlID0+ICdzJywKKyAg
cHJvcGFyYWxsZWwgPT4gJ3InLCBwcm9yZXR0eXBlID0+ICdpbnQ4JywgcHJvYXJndHlwZXMg
PT4gJycsCisgIHByb3NyYyA9PiAncGdfc3RhdF9nZXRfYmd3cml0ZXJfdGltZWRfcmVzdGFy
dHBvaW50cycgfSwKK3sgb2lkID0+ICc5OTM5JywKKyAgZGVzY3IgPT4gJ3N0YXRpc3RpY3M6
IG51bWJlciBvZiBiYWNrZW5kIHJlcXVlc3RlZCByZXN0YXJ0cG9pbnRzIHN0YXJ0ZWQgYnkg
dGhlIGJnd3JpdGVyJywKKyAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRfYmd3cml0ZXJfcmVx
dWVzdGVkX3Jlc3RhcnRwb2ludHMnLCBwcm92b2xhdGlsZSA9PiAncycsCisgIHByb3BhcmFs
bGVsID0+ICdyJywgcHJvcmV0dHlwZSA9PiAnaW50OCcsIHByb2FyZ3R5cGVzID0+ICcnLAor
ICBwcm9zcmMgPT4gJ3BnX3N0YXRfZ2V0X2Jnd3JpdGVyX3JlcXVlc3RlZF9yZXN0YXJ0cG9p
bnRzJyB9LAoreyBvaWQgPT4gJzk5NDAnLAorICBkZXNjciA9PiAnc3RhdGlzdGljczogbnVt
YmVyIG9mIGJhY2tlbmQgcGVyZm9ybWVkIHJlc3RhcnRwb2ludHMgc3RhcnRlZCBieSB0aGUg
Ymd3cml0ZXInLAorICBwcm9uYW1lID0+ICdwZ19zdGF0X2dldF9iZ3dyaXRlcl9wZXJmb3Jt
ZWRfcmVzdGFydHBvaW50cycsIHByb3ZvbGF0aWxlID0+ICdzJywKKyAgcHJvcGFyYWxsZWwg
PT4gJ3InLCBwcm9yZXR0eXBlID0+ICdpbnQ4JywgcHJvYXJndHlwZXMgPT4gJycsCisgIHBy
b3NyYyA9PiAncGdfc3RhdF9nZXRfYmd3cml0ZXJfcGVyZm9ybWVkX3Jlc3RhcnRwb2ludHMn
IH0sCiB7IG9pZCA9PiAnMjc3MScsCiAgIGRlc2NyID0+ICdzdGF0aXN0aWNzOiBudW1iZXIg
b2YgYnVmZmVycyB3cml0dGVuIGJ5IHRoZSBiZ3dyaXRlciBkdXJpbmcgY2hlY2twb2ludHMn
LAogICBwcm9uYW1lID0+ICdwZ19zdGF0X2dldF9iZ3dyaXRlcl9idWZfd3JpdHRlbl9jaGVj
a3BvaW50cycsIHByb3ZvbGF0aWxlID0+ICdzJywKZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRl
L3Bnc3RhdC5oIGIvc3JjL2luY2x1ZGUvcGdzdGF0LmgKaW5kZXggYWMyOGY4MTNiNC4uNjgw
ZmY2ZjM5YiAxMDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUvcGdzdGF0LmgKKysrIGIvc3JjL2lu
Y2x1ZGUvcGdzdGF0LmgKQEAgLTI2OSw2ICsyNjksOSBAQCB0eXBlZGVmIHN0cnVjdCBQZ1N0
YXRfQ2hlY2twb2ludGVyU3RhdHMKIHsKIAlQZ1N0YXRfQ291bnRlciB0aW1lZF9jaGVja3Bv
aW50czsKIAlQZ1N0YXRfQ291bnRlciByZXF1ZXN0ZWRfY2hlY2twb2ludHM7CisJUGdTdGF0
X0NvdW50ZXIgdGltZWRfcmVzdGFydHBvaW50czsKKwlQZ1N0YXRfQ291bnRlciByZXF1ZXN0
ZWRfcmVzdGFydHBvaW50czsKKwlQZ1N0YXRfQ291bnRlciBwZXJmb3JtZWRfcmVzdGFydHBv
aW50czsKIAlQZ1N0YXRfQ291bnRlciBjaGVja3BvaW50X3dyaXRlX3RpbWU7CS8qIHRpbWVz
IGluIG1pbGxpc2Vjb25kcyAqLwogCVBnU3RhdF9Db3VudGVyIGNoZWNrcG9pbnRfc3luY190
aW1lOwogCVBnU3RhdF9Db3VudGVyIGJ1Zl93cml0dGVuX2NoZWNrcG9pbnRzOwo=

--------------jdaQYEogzXYdPmc7RiUux8Jb--