MIME-Version: 1.0
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 25 Apr 2022 19:51:03 +0530
Message-ID: <CALj2ACUrOB59QaE6=jF2cFAyv1MR7fzD8tr4YM5+OwEYG1SNzA@mail.gmail.com>
Subject: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions
 in synchronous replication
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com>
Content-Type: multipart/mixed; boundary="000000000000f0b4f505dd7b494c"
Archived-At: <https://www.postgresql.org/message-id/CALj2ACUrOB59QaE6%3DjF2cFAyv1MR7fzD8tr4YM5%2BOwEYG1SNzA%40mail.gmail.com>
Precedence: bulk

--000000000000f0b4f505dd7b494c
Content-Type: text/plain; charset="UTF-8"

Hi,

With synchronous replication typically all the transactions (txns)
first locally get committed, then streamed to the sync standbys and
the backend that generated the transaction will wait for ack from sync
standbys. While waiting for ack, it may happen that the query or the
txn gets canceled (QueryCancelPending is true) or the waiting backend
is asked to exit (ProcDiePending is true). In either of these cases,
the wait for ack gets canceled and leaves the txn in an inconsistent
state (as in the client thinks that the txn would have replicated to
sync standbys) - "The transaction has already committed locally, but
might not have been replicated to the standby.". Upon restart after
the crash or in the next txn after the old locally committed txn was
canceled, the client will be able to see the txns that weren't
actually streamed to sync standbys. Also, if the client fails over to
one of the sync standbys after the crash (either by choice or because
of automatic failover management after crash), the locally committed
txns on the crashed primary would be lost which isn't good in a true
HA solution.

Here's a proposal (mentioned previously by Satya [1]) to avoid the
above problems:
1) Wait a configurable amount of time before canceling the sync
replication by the backends i.e. delay processing of
QueryCancelPending and ProcDiePending in Introduced a new timeout GUC
synchronous_replication_naptime_before_cancel, when set, it will let
the backends wait for the ack before canceling the synchronous
replication so that the transaction can be available in sync standbys
as well. If the ack isn't received even within this time frame, the
backend cancels the wait and goes ahead as it does today. In
production HA environments, the GUC can be set to a reasonable value
to avoid missing transactions during failovers.
2) Wait for sync standbys to catch up upon restart after the crash or
in the next txn after the old locally committed txn was canceled. One
way to achieve this is to let the backend, that's making the first
connection, wait for sync standbys to catch up in ClientAuthentication
right after successful authentication. However, I'm not sure this is
the best way to do it at this point.

Thoughts?

Here's a WIP patch implementing the (1), I'm yet to code for (2). I
haven't added tests, I'm yet to figure out how to add one as there's
no way we can delay the WAL sender so that we can reliably hit this
code. I will think more about this.

[1] https://www.postgresql.org/message-id/CAHg%2BQDdTdPsqtu0QLG8rMg3Xo%3D6Xo23TwHPYsUgGNEK13wTY5g%40mail.gmail.com

Regards,
Bharath Rupireddy.

--000000000000f0b4f505dd7b494c
Content-Type: application/x-patch; 
	name="v1-0001-Wait-specified-amount-of-time-before-cancelling-s.patch"
Content-Disposition: attachment; 
	filename="v1-0001-Wait-specified-amount-of-time-before-cancelling-s.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_l2creov00>
X-Attachment-Id: f_l2creov00

RnJvbSBkNWZlMDdiYmQ4MGI3MmRmYmYwNmU5YjAzOWI5ZTRhOTNhN2Y3YTA2IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBCaGFyYXRoIFJ1cGlyZWRkeSA8YmhhcmF0aC5ydXBpcmVkZHlm
b3Jwb3N0Z3Jlc0BnbWFpbC5jb20+CkRhdGU6IFN1biwgMjQgQXByIDIwMjIgMDM6NDI6NTkgKzAw
MDAKU3ViamVjdDogW1BBVENIIHYxXSBXYWl0IHNwZWNpZmllZCBhbW91bnQgb2YgdGltZSBiZWZv
cmUgY2FuY2VsbGluZyBzeW5jCiByZXBsaWNhdGlvbgoKSW4gUG9zdGdyZVNRTCBoaWdoIGF2YWls
YWJpbGl0eSBzZXR1cCB3aXRoIHN5bmNocm9ub3VzIHJlcGxpY2F0aW9uLAp0eXBpY2FsbHkgYWxs
IHRoZSB0cmFuc2FjdGlvbnMgZmlyc3QgbG9jYWxseSBnZXQgY29tbWl0dGVkLCB0aGVuCnN0cmVh
bWVkIHRvIHRoZSBzeW5jaHJvbm91cyBzdGFuZGJ5cyBhbmQgdGhlIGJhY2tlbmRzIHRoYXQgZ2Vu
ZXJhdGVkCnRoZSB0cmFuc2FjdGlvbiB3aWxsIHdhaXQgZm9yIGFja25vd2xlZGdlbWVudCBmcm9t
IHN5bmNocm9ub3VzCnN0YW5kYnlzLiBXaGlsZSB3YWl0aW5nIGZvciBhY2tub3dsZWRnZW1lbnQs
IGl0IG1heSBoYXBwZW4gdGhhdCB0aGUKcXVlcnkgb3IgdGhlIHRyYW5zYWN0aW9uIGdldHMgY2Fu
Y2VsZWQgb3IgdGhlIGJhY2tlbmQgdGhhdCdzIHdhaXRpbmcKZm9yIGFja25vd2xlZGdlbWVudCBp
cyBhc2tlZCB0byBleGl0LiBJbiBlaXRoZXIgb2YgdGhlc2UgY2FzZXMsIHRoZQp3YWl0IGZvciBh
Y2tub3dsZWRnZW1lbnQgZ2V0cyBjYW5jZWxlZCBhbmQgbGVhdmVzIHRyYW5zYWN0aW9uIGluIGFu
CmluY29uc2lzdGVudCBzdGF0ZSBhcyBpdCBnb3QgY29tbWl0dGVkIGxvY2FsbHkgYnV0IG5vdCBv
biB0aGUKc3RhbmRieXMuIFdoZW4gc2V0IHRoZSBHVUMgc3luY2hyb25vdXNfcmVwbGljYXRpb25f
bmFwdGltZV9iZWZvcmVfY2FuY2VsCmludHJvZHVjZWQgaW4gdGhpcyBwYXRjaCwgaXQgd2lsbCBs
ZXQgdGhlIGJhY2tlbmRzIHdhaXQgZm9yIHRoZQphY2tub3dsZWRnZW1lbnQgYmVmb3JlIGNhbmNl
bGluZyB0aGUgd2FpdCBmb3IgYWNrbm93bGVkZ2VtZW50IHNvCnRoYXQgdGhlIHRyYW5zYWN0aW9u
IGNhbiBiZSBhdmFpbGFibGUgaW4gc3luY2hyb25vdXMgc3RhbmRieXMgYXMKd2VsbC4KLS0tCiBk
b2Mvc3JjL3NnbWwvY29uZmlnLnNnbWwgICAgICAgICAgICAgICAgICAgICAgfCAzMCArKysrKysr
KysrKwogc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vc3luY3JlcC5jICAgICAgICAgICAgIHwgNTAg
KysrKysrKysrKysrKysrKysrKwogc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9ndWMuYyAgICAgICAg
ICAgICAgICAgIHwgMTIgKysrKysKIHNyYy9iYWNrZW5kL3V0aWxzL21pc2MvcG9zdGdyZXNxbC5j
b25mLnNhbXBsZSB8ICAyICsKIHNyYy9pbmNsdWRlL3JlcGxpY2F0aW9uL3N5bmNyZXAuaCAgICAg
ICAgICAgICB8ICAzICsrCiA1IGZpbGVzIGNoYW5nZWQsIDk3IGluc2VydGlvbnMoKykKCmRpZmYg
LS1naXQgYS9kb2Mvc3JjL3NnbWwvY29uZmlnLnNnbWwgYi9kb2Mvc3JjL3NnbWwvY29uZmlnLnNn
bWwKaW5kZXggMDM5ODY5NDZhOC4uMTY4MWVhMTczZiAxMDA2NDQKLS0tIGEvZG9jL3NyYy9zZ21s
L2NvbmZpZy5zZ21sCisrKyBiL2RvYy9zcmMvc2dtbC9jb25maWcuc2dtbApAQCAtNDUwNyw2ICs0
NTA3LDM2IEBAIEFOWSA8cmVwbGFjZWFibGUgY2xhc3M9InBhcmFtZXRlciI+bnVtX3N5bmM8L3Jl
cGxhY2VhYmxlPiAoIDxyZXBsYWNlYWJsZSBjbGFzcz0iCiAgICAgICA8L2xpc3RpdGVtPgogICAg
ICA8L3Zhcmxpc3RlbnRyeT4KIAorICAgICA8dmFybGlzdGVudHJ5IGlkPSJndWMtc3luY2hyb25v
dXMtcmVwbGljYXRpb24tbmFwdGltZS1iZWZvcmUtY2FuY2VsIiB4cmVmbGFiZWw9InN5bmNocm9u
b3VzX3JlcGxpY2F0aW9uX25hcHRpbWVfYmVmb3JlX2NhbmNlbCI+CisgICAgICA8dGVybT48dmFy
bmFtZT5zeW5jaHJvbm91c19yZXBsaWNhdGlvbl9uYXB0aW1lX2JlZm9yZV9jYW5jZWw8L3Zhcm5h
bWU+ICg8dHlwZT5pbnRlZ2VyPC90eXBlPikKKyAgICAgIDxpbmRleHRlcm0+CisgICAgICAgPHBy
aW1hcnk+PHZhcm5hbWU+c3luY2hyb25vdXNfcmVwbGljYXRpb25fbmFwdGltZV9iZWZvcmVfY2Fu
Y2VsPC92YXJuYW1lPiBjb25maWd1cmF0aW9uIHBhcmFtZXRlcjwvcHJpbWFyeT4KKyAgICAgIDwv
aW5kZXh0ZXJtPgorICAgICAgPC90ZXJtPgorICAgICAgPGxpc3RpdGVtPgorICAgICAgIDxwYXJh
PgorICAgICAgICBTcGVjaWZpZXMgdGhlIGFtb3VudCBvZiB0aW1lIGluIG1pbGxpc2Vjb25kcyB0
byB3YWl0IGZvciBzeW5jaHJvbm91cworICAgICAgICByZXBsaWNhdGlvbiBiZWZvcmUgY2FuY2Vs
bGluZy4gRGVmYXVsdCB2YWx1ZSBpcyAwLCBhIHZhbHVlIG9mIC0xIG9yIDAKKyAgICAgICAgZGlz
YWJsZXMgdGhpcyBmZWF0dXJlLiBJbiA8cHJvZHVjdG5hbWU+UG9zdGdyZVNRTDwvcHJvZHVjdG5h
bWU+IGhpZ2gKKyAgICAgICAgYXZhaWxhYmlsaXR5IHNldHVwIHdpdGggc3luY2hyb25vdXMgcmVw
bGljYXRpb24sIHR5cGljYWxseSBhbGwgdGhlCisgICAgICAgIHRyYW5zYWN0aW9ucyBmaXJzdCBs
b2NhbGx5IGdldCBjb21taXR0ZWQsIHRoZW4gc3RyZWFtZWQgdG8gdGhlCisgICAgICAgIHN5bmNo
cm9ub3VzIHN0YW5kYnlzIGFuZCB0aGUgYmFja2VuZHMgdGhhdCBnZW5lcmF0ZWQgdGhlIHRyYW5z
YWN0aW9uCisgICAgICAgIHdpbGwgd2FpdCBmb3IgYWNrbm93bGVkZ2VtZW50IGZyb20gc3luY2hy
b25vdXMgc3RhbmRieXMuIFdoaWxlIHdhaXRpbmcKKyAgICAgICAgZm9yIGFja25vd2xlZGdlbWVu
dCwgaXQgbWF5IGhhcHBlbiB0aGF0IHRoZSBxdWVyeSBvciB0aGUgdHJhbnNhY3Rpb24KKyAgICAg
ICAgZ2V0cyBjYW5jZWxlZCBvciB0aGUgYmFja2VuZCB0aGF0J3Mgd2FpdGluZyBmb3IgYWNrbm93
bGVkZ2VtZW50IGlzCisgICAgICAgIGFza2VkIHRvIGV4aXQuIEluIGVpdGhlciBvZiB0aGVzZSBj
YXNlcywgdGhlIHdhaXQgZm9yIGFja25vd2xlZGdlbWVudAorICAgICAgICBnZXRzIGNhbmNlbGVk
IGFuZCBsZWF2ZXMgdHJhbnNhY3Rpb24gaW4gYW4gaW5jb25zaXN0ZW50IHN0YXRlIGFzIGl0IGdv
dAorICAgICAgICBjb21taXR0ZWQgbG9jYWxseSBidXQgbm90IG9uIHRoZSBzdGFuZGJ5cy4gV2hl
biBzZXQgdGhlCisgICAgICAgIDx2YXJuYW1lPnN5bmNocm9ub3VzX3JlcGxpY2F0aW9uX25hcHRp
bWVfYmVmb3JlX2NhbmNlbDwvdmFybmFtZT4KKyAgICAgICAgcGFyYW1ldGVyLCBpdCB3aWxsIGxl
dCB0aGUgYmFja2VuZHMgd2FpdCBmb3IgdGhlIGFja25vd2xlZGdlbWVudAorICAgICAgICBiZWZv
cmUgY2FuY2VsaW5nIHRoZSB3YWl0IGZvciBhY2tub3dsZWRnZW1lbnQgc28gdGhhdCB0aGUgdHJh
bnNhY3Rpb24KKyAgICAgICAgY2FuIGJlIGF2YWlsYWJsZSBpbiBzeW5jaHJvbm91cyBzdGFuZGJ5
cyBhcyB3ZWxsLiBUaGlzIHBhcmFtZXRlciBjYW4KKyAgICAgICAgb25seSBiZSBzZXQgaW4gdGhl
IDxmaWxlbmFtZT5wb3N0Z3Jlc3FsLmNvbmY8L2ZpbGVuYW1lPiBmaWxlIG9yIG9uIHRoZQorICAg
ICAgICBzZXJ2ZXIgY29tbWFuZCBsaW5lLgorICAgICAgICA8L3BhcmE+CisgICAgICA8L2xpc3Rp
dGVtPgorICAgICA8L3Zhcmxpc3RlbnRyeT4KKwogICAgICA8L3ZhcmlhYmxlbGlzdD4KICAgICA8
L3NlY3QyPgogCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9zeW5jcmVwLmMg
Yi9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9zeW5jcmVwLmMKaW5kZXggY2UxNjNiOTllOS4uMGY1
NGQ4MWYyYiAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vc3luY3JlcC5jCisr
KyBiL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL3N5bmNyZXAuYwpAQCAtODgsNiArODgsNyBAQAog
CiAvKiBVc2VyLXNldHRhYmxlIHBhcmFtZXRlcnMgZm9yIHN5bmMgcmVwICovCiBjaGFyCSAgICpT
eW5jUmVwU3RhbmRieU5hbWVzOworaW50CQkJU3luY1JlcE5hcFRpbWVCZWZvcmVDYW5jZWwgPSAw
OwogCiAjZGVmaW5lIFN5bmNTdGFuZGJ5c0RlZmluZWQoKSBcCiAJKFN5bmNSZXBTdGFuZGJ5TmFt
ZXMgIT0gTlVMTCAmJiBTeW5jUmVwU3RhbmRieU5hbWVzWzBdICE9ICdcMCcpCkBAIC0xMTksNiAr
MTIwLDcgQEAgc3RhdGljIHZvaWQgU3luY1JlcEdldE50aExhdGVzdFN5bmNSZWNQdHIoWExvZ1Jl
Y1B0ciAqd3JpdGVQdHIsCiBzdGF0aWMgaW50CVN5bmNSZXBHZXRTdGFuZGJ5UHJpb3JpdHkodm9p
ZCk7CiBzdGF0aWMgaW50CXN0YW5kYnlfcHJpb3JpdHlfY29tcGFyYXRvcihjb25zdCB2b2lkICph
LCBjb25zdCB2b2lkICpiKTsKIHN0YXRpYyBpbnQJY21wX2xzbihjb25zdCB2b2lkICphLCBjb25z
dCB2b2lkICpiKTsKK3N0YXRpYyBib29sIFN5bmNSZXBOYXBCZWZvcmVDYW5jZWwodm9pZCk7CiAK
ICNpZmRlZiBVU0VfQVNTRVJUX0NIRUNLSU5HCiBzdGF0aWMgYm9vbCBTeW5jUmVwUXVldWVJc09y
ZGVyZWRCeUxTTihpbnQgbW9kZSk7CkBAIC0xMzAsNiArMTMyLDQyIEBAIHN0YXRpYyBib29sIFN5
bmNSZXBRdWV1ZUlzT3JkZXJlZEJ5TFNOKGludCBtb2RlKTsKICAqID09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09CiAgKi8KIAorLyoKKyAq
IFdhaXQgZm9yIHN5bmNocm9ub3VzIHJlcGxpY2F0aW9uIGJlZm9yZSBjYW5jZWxsaW5nLCBpZiBy
ZXF1ZXN0ZWQgYnkgdXNlci4KKyAqLworc3RhdGljIGJvb2wKK1N5bmNSZXBOYXBCZWZvcmVDYW5j
ZWwodm9pZCkKK3sKKwlpbnQgd2FpdF90aW1lOworCisJaWYgKFN5bmNSZXBOYXBUaW1lQmVmb3Jl
Q2FuY2VsIDw9IDApCisJCXJldHVybiBmYWxzZTsKKworCWVyZXBvcnQoV0FSTklORywKKwkJCShl
cnJtc2dfcGx1cmFsKCJ3YWl0aW5nICVkIG1pbGxpc2Vjb25kIGZvciBzeW5jaHJvbm91cyByZXBs
aWNhdGlvbiBiZWZvcmUgY2FuY2VsbGluZyIsCisJCQkJCQkgICAid2FpdGluZyAlZCBtaWxsaXNl
Y29uZHMgZm9yIHN5bmNocm9ub3VzIHJlcGxpY2F0aW9uIGJlZm9yZSBjYW5jZWxsaW5nIiwKKwkJ
CQkJCQlTeW5jUmVwTmFwVGltZUJlZm9yZUNhbmNlbCwKKwkJCQkJCQlTeW5jUmVwTmFwVGltZUJl
Zm9yZUNhbmNlbCkpKTsKKworCXdhaXRfdGltZSA9IFN5bmNSZXBOYXBUaW1lQmVmb3JlQ2FuY2Vs
OworCisJd2hpbGUgKHdhaXRfdGltZSA+IDApCisJeworCQkvKgorCQkgKiBXYWl0IGluIGludGVy
dmFscyBvZiAxIG1pbGxpc2Vjb25kIHNvIHRoYXQgd2UgY2FuIGZyZXF1ZW50bHkgY2hlY2sKKwkJ
ICogZm9yIHRoZSBhY2tub3dsZWRnZW1lbnQuCisJCSAqLworCQlwZ191c2xlZXAoMSAqIDEwMDBM
KTsKKworCQl3YWl0X3RpbWUtLTsKKworCQlpZiAoTXlQcm9jLT5zeW5jUmVwU3RhdGUgPT0gU1lO
Q19SRVBfV0FJVF9DT01QTEVURSkKKwkJCXJldHVybiB0cnVlOworCX0KKworCXJldHVybiB0cnVl
OworfQorCiAvKgogICogV2FpdCBmb3Igc3luY2hyb25vdXMgcmVwbGljYXRpb24sIGlmIHJlcXVl
c3RlZCBieSB1c2VyLgogICoKQEAgLTI2Myw2ICszMDEsMTIgQEAgU3luY1JlcFdhaXRGb3JMU04o
WExvZ1JlY1B0ciBsc24sIGJvb2wgY29tbWl0KQogCQkgKi8KIAkJaWYgKFByb2NEaWVQZW5kaW5n
KQogCQl7CisJCQlpZiAoU3luY1JlcE5hcEJlZm9yZUNhbmNlbCgpKQorCQkJeworCQkJCWlmIChN
eVByb2MtPnN5bmNSZXBTdGF0ZSA9PSBTWU5DX1JFUF9XQUlUX0NPTVBMRVRFKQorCQkJCQlicmVh
azsKKwkJCX0KKwogCQkJZXJlcG9ydChXQVJOSU5HLAogCQkJCQkoZXJyY29kZShFUlJDT0RFX0FE
TUlOX1NIVVRET1dOKSwKIAkJCQkJIGVycm1zZygiY2FuY2VsaW5nIHRoZSB3YWl0IGZvciBzeW5j
aHJvbm91cyByZXBsaWNhdGlvbiBhbmQgdGVybWluYXRpbmcgY29ubmVjdGlvbiBkdWUgdG8gYWRt
aW5pc3RyYXRvciBjb21tYW5kIiksCkBAIC0yODAsNiArMzI0LDEyIEBAIFN5bmNSZXBXYWl0Rm9y
TFNOKFhMb2dSZWNQdHIgbHNuLCBib29sIGNvbW1pdCkKIAkJICovCiAJCWlmIChRdWVyeUNhbmNl
bFBlbmRpbmcpCiAJCXsKKwkJCWlmIChTeW5jUmVwTmFwQmVmb3JlQ2FuY2VsKCkpCisJCQl7CisJ
CQkJaWYgKE15UHJvYy0+c3luY1JlcFN0YXRlID09IFNZTkNfUkVQX1dBSVRfQ09NUExFVEUpCisJ
CQkJCWJyZWFrOworCQkJfQorCiAJCQlRdWVyeUNhbmNlbFBlbmRpbmcgPSBmYWxzZTsKIAkJCWVy
ZXBvcnQoV0FSTklORywKIAkJCQkJKGVycm1zZygiY2FuY2VsaW5nIHdhaXQgZm9yIHN5bmNocm9u
b3VzIHJlcGxpY2F0aW9uIGR1ZSB0byB1c2VyIHJlcXVlc3QiKSwKZGlmZiAtLWdpdCBhL3NyYy9i
YWNrZW5kL3V0aWxzL21pc2MvZ3VjLmMgYi9zcmMvYmFja2VuZC91dGlscy9taXNjL2d1Yy5jCmlu
ZGV4IDhlOWI3MTM3NWMuLjU0N2JjMjcyN2YgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3V0aWxz
L21pc2MvZ3VjLmMKKysrIGIvc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9ndWMuYwpAQCAtMjc0Myw2
ICsyNzQzLDE4IEBAIHN0YXRpYyBzdHJ1Y3QgY29uZmlnX2ludCBDb25maWd1cmVOYW1lc0ludFtd
ID0KIAkJMCwgMCwgMTAwMDAwMCwJCQkvKiBzZWUgQ29tcHV0ZVhpZEhvcml6b25zICovCiAJCU5V
TEwsIE5VTEwsIE5VTEwKIAl9LAorCisJeworCQl7InN5bmNocm9ub3VzX3JlcGxpY2F0aW9uX25h
cHRpbWVfYmVmb3JlX2NhbmNlbCIsIFBHQ19TSUdIVVAsIFJFUExJQ0FUSU9OX1BSSU1BUlksCisJ
CQlnZXR0ZXh0X25vb3AoIlNldHMgdGhlIGFtb3VudCBvZiB0aW1lIHRvIHdhaXQgZm9yIHN5bmNo
cm9ub3VzIHJlcGxpY3RhaW9uIGJlZm9yZSBjYW5jZWxsaW5nLiIpLAorCQkJZ2V0dGV4dF9ub29w
KCJBIHZhbHVlIG9mIC0xIG9yIDAgZGlzYWJsZXMgdGhpcyBmZWF0dXJlLiIpLAorCQkJR1VDX1VO
SVRfTVMKKwkJfSwKKwkJJlN5bmNSZXBOYXBUaW1lQmVmb3JlQ2FuY2VsLAorCQkwLCAwLCBJTlRf
TUFYLAorCQlOVUxMLCBOVUxMLCBOVUxMCisJfSwKKwogCXsKIAkJeyJ2YWN1dW1fZmFpbHNhZmVf
YWdlIiwgUEdDX1VTRVJTRVQsIENMSUVOVF9DT05OX1NUQVRFTUVOVCwKIAkJCWdldHRleHRfbm9v
cCgiQWdlIGF0IHdoaWNoIFZBQ1VVTSBzaG91bGQgdHJpZ2dlciBmYWlsc2FmZSB0byBhdm9pZCBh
IHdyYXBhcm91bmQgb3V0YWdlLiIpLApkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvbWlz
Yy9wb3N0Z3Jlc3FsLmNvbmYuc2FtcGxlIGIvc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9wb3N0Z3Jl
c3FsLmNvbmYuc2FtcGxlCmluZGV4IDk0MjcwZWIwZWMuLjRmZDRkMDQ4MDQgMTAwNjQ0Ci0tLSBh
L3NyYy9iYWNrZW5kL3V0aWxzL21pc2MvcG9zdGdyZXNxbC5jb25mLnNhbXBsZQorKysgYi9zcmMv
YmFja2VuZC91dGlscy9taXNjL3Bvc3RncmVzcWwuY29uZi5zYW1wbGUKQEAgLTMyNCw2ICszMjQs
OCBAQAogCQkJCSMgYW5kIGNvbW1hLXNlcGFyYXRlZCBsaXN0IG9mIGFwcGxpY2F0aW9uX25hbWUK
IAkJCQkjIGZyb20gc3RhbmRieShzKTsgJyonID0gYWxsCiAjdmFjdXVtX2RlZmVyX2NsZWFudXBf
YWdlID0gMAkjIG51bWJlciBvZiB4YWN0cyBieSB3aGljaCBjbGVhbnVwIGlzIGRlbGF5ZWQKKyNz
eW5jaHJvbm91c19yZXBsaWNhdGlvbl9uYXB0aW1lX2JlZm9yZV9jYW5jZWwgPSAwICMgYW1vdW50
IG9mIHRpbWUgdG8gd2FpdCBmb3IKKwkJCQkjIHN5bmNocm9ub3VzIHJlcGxpY3RhaW9uIGJlZm9y
ZSBjYW5jZWxsaW5nOyAwIG9yIC0xIGRpc2FibGVzCiAKICMgLSBTdGFuZGJ5IFNlcnZlcnMgLQog
CmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS9yZXBsaWNhdGlvbi9zeW5jcmVwLmggYi9zcmMvaW5j
bHVkZS9yZXBsaWNhdGlvbi9zeW5jcmVwLmgKaW5kZXggNGQ3YzkwYjlmMC4uNjY3OGYxNGI5MyAx
MDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUvcmVwbGljYXRpb24vc3luY3JlcC5oCisrKyBiL3NyYy9p
bmNsdWRlL3JlcGxpY2F0aW9uL3N5bmNyZXAuaApAQCAtODEsNiArODEsOSBAQCBleHRlcm4gUEdE
TExJTVBPUlQgY2hhciAqc3luY3JlcF9wYXJzZV9lcnJvcl9tc2c7CiAvKiB1c2VyLXNldHRhYmxl
IHBhcmFtZXRlcnMgZm9yIHN5bmNocm9ub3VzIHJlcGxpY2F0aW9uICovCiBleHRlcm4gUEdETExJ
TVBPUlQgY2hhciAqU3luY1JlcFN0YW5kYnlOYW1lczsKIAorLyogdXNlci1zZXR0YWJsZSBuYXAg
dGltZSBmb3Igc3luY2hyb25vdXMgcmVwbGljdGFpb24gYmVmb3JlIGNhbmNlbGxpbmcgKi8KK2V4
dGVybiBQR0RMTElNUE9SVCBpbnQgU3luY1JlcE5hcFRpbWVCZWZvcmVDYW5jZWw7CisKIC8qIGNh
bGxlZCBieSB1c2VyIGJhY2tlbmQgKi8KIGV4dGVybiB2b2lkIFN5bmNSZXBXYWl0Rm9yTFNOKFhM
b2dSZWNQdHIgbHNuLCBib29sIGNvbW1pdCk7CiAKLS0gCjIuMjUuMQoK
--000000000000f0b4f505dd7b494c--