IronPort-SDR: 
 1oMAYG8GmQ5W5HOl+xAwpblhxu3QM0ZB0LurfEdmMEVlmpBsrvznMnP+j5YXqKQ83wOdkhVyKn
 ozAgW712imCQ==
Subject: Re: Add Information during standby recovery conflicts
To: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
CC: PostgreSQL-development <pgsql-hackers@postgresql.org>
References: <9a60178c-a853-1440-2cdc-c3af916cff59@amazon.com>
 <CA+fd4k6oBCoF+4sAgWqsC5+B5+EWQfT=FyPg9zVo4wh+zTXhUg@mail.gmail.com>
 <daeecc9d-567a-cad8-298a-6f12ed7cb665@amazon.com>
 <CA+fd4k45sOD63XV-fcpi-ACS12PQa4G19_qf+cCVUcKK2nfKJw@mail.gmail.com>
From: "Drouvot, Bertrand" <bdrouvot@amazon.com>
Message-ID: <3aededb3-8eb7-bfd6-6fd4-b6487f8f1d74@amazon.com>
Date: Mon, 6 Jul 2020 08:41:49 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:68.0)
 Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+fd4k45sOD63XV-fcpi-ACS12PQa4G19_qf+cCVUcKK2nfKJw@mail.gmail.com>
Content-Type: multipart/mixed;
	boundary="------------6F976B3B5C883C1735931861"
Content-Language: en-US
Precedence: bulk

--------------6F976B3B5C883C1735931861
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit


On 7/3/20 4:20 AM, Masahiko Sawada wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> On Wed, 1 Jul 2020 at 21:52, Drouvot, Bertrand <bdrouvot@amazon.com> wrote:
>> Hey,
>>
>> On 6/29/20 10:55 AM, Masahiko Sawada wrote:
>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>
>>>
>>>
>>> On Thu, 18 Jun 2020 at 16:28, Drouvot, Bertrand <bdrouvot@amazon.com> wrote:
>>>> Hi hackers,
>>>>
>>>> I've attached a patch to add information during standby recovery conflicts.
>>>> The motive behind is to be able to get some information:
>>>>
>>>> On the apply side
>>>> On the blocker(s) side
>>>>
>>>> Motivation:
>>>>
>>>> When a standby recovery conflict occurs it could be useful to get more information to be able to dive deep on the root cause and find a way to avoid/mitigate new occurrences.
>>> I think this is a good feature. Like log_lock_waits, it will help the
>>> users to investigate recovery conflict issues.
>> exactly, thanks for looking at it.
>>>> Adding this information would make the investigations easier, it could help answering questions like:
>>>>
>>>> On which LSN was the WAL apply blocked?
>>>> What was the purpose of the bocked WAL record?
>>>> On which relation (if any) was the blocked WAL record related to?
>>>> What was the blocker(s) doing?
>>>> When did the blocker(s) started their queries (if any)?
>>>> What was the blocker(s) waiting for? on which wait event?
>>>>
>>>> Technical context and proposal:
>>>>
>>>> There is 2 points in this patch:
>>>>
>>>> Add the information about the blocked WAL record. This is done in standby.c (ResolveRecoveryConflictWithVirtualXIDs, ResolveRecoveryConflictWithDatabase, StandbyTimeoutHandler)
>>> I think we already have the information about the WAL record being
>>> applied in errcontext.
>> right, but it won’t be displayed in case log_error_verbosity is set to
>> terse.
> Yes. I think in this case errcontext or errdetail is more appropriate
> for this information in this case. Otherwise, we will end up emitting
> same WAL record information twice in log_error_verbosity = verbose.
>
> For instance, here is the server logs when I tested a recovery
> conflict on buffer pin but it looks redundant:
>
> 2020-07-03 11:01:15.339 JST [60585] LOG:  wal record apply is blocked
> by 1 connection(s), reason: User query might have needed to see row
> versions that must be removed.
> 2020-07-03 11:01:15.339 JST [60585] CONTEXT:  WAL redo at 0/30025E0
> for Heap2/CLEAN: remxid 505
> 2020-07-03 11:01:15.339 JST [60585] LOG:  blocked wal record rmgr:
> Heap2, lsn: 0/030025E0, received at: 2020-07-03 11:01:15.338997+09,
> desc: CLEAN, relation: rel 1663/12930/16384 fork main blk 0
> 2020-07-03 11:01:15.339 JST [60585] CONTEXT:  WAL redo at 0/30025E0
> for Heap2/CLEAN: remxid 505
> 2020-07-03 11:01:15.347 JST [60604] LOG:  about to be interrupted:
> backend_type: client backend, state: active, wait_event: PgSleep,
> query_start: 2020-07-03 11:01:14.337708+09
> 2020-07-03 11:01:15.347 JST [60604] STATEMENT:  select pg_sleep(30);
> 2020-07-03 11:01:15.347 JST [60604] ERROR:  canceling statement due to
> conflict with recovery
> 2020-07-03 11:01:15.347 JST [60604] DETAIL:  User query might have
> needed to see row versions that must be removed.
> 2020-07-03 11:01:15.347 JST [60604] STATEMENT:  select pg_sleep(30);
>
> There are the information WAL record three times and the reason for
> interruption twice.


Fully makes sense, the new patch version attached is now producing:

2020-07-06 06:10:36.022 UTC [14035] LOG:  waiting for recovery conflict 
on snapshot
2020-07-06 06:10:36.022 UTC [14035] DETAIL:  WAL record received at 
2020-07-06 06:10:36.021963+00.
         Tablespace/database/relation are 1663/13586/16672, fork is main 
and block is 0.
         There is 1 blocking connection(s).
2020-07-06 06:10:36.022 UTC [14035] CONTEXT:  WAL redo at 0/3A05708 for 
Heap2/CLEAN: remxid 972

It does not provide the pid(s) of the blocking processes as they'll 
appear during the interruption(s).

>> Or did you had in mind to try to avoid using the new
>> “current_replayed_xlog” in xlog.c?
> Regarding LogBlockedWalRecordInfo(), it seems to me that the main
> message of this logging is to let users know both that the process got
> stuck due to recovery conflict and its reason (lock, database,
> bufferpin etc). Other information such as the details of blocked WAL,
> how many processes are blocking seems like additional information. So
> I think this information would be better to be shown errcontext or
> errdetails and we don’t need to create a similar feature as we already
> show the WAL record being replayed in errcontext.


I got rid of current_replayed_xlog in the new patch attached and make 
use of the errcontext argument instead.

>
> Also, this function emits two LOG messages related to each other but
> I'm concerned that it can be hard for users to associate these
> separate log lines especially when server logs are written at a high
> rate. And I think these two log lines don't follow our error message
> style[1].
>
>>> I wonder if we can show the recovery conflict information in the main
>>> LOG message, the blocker information in errdetail, and use errcontext
>>> with regard to WAL record information. For example:
>>>
>>> LOG:  process 500 waiting for recovery conflict on snapshot
>>> DETAIL:  conflicting transition id: 100, 200, 300
>>> CONTEXT:  WAL redo at 0/3001970 for Heap2/CLEAN: remxid 506
>> Not sure at all if it would be possible to put all the information at
>> the same time.
>> For example in case of shared buffer pin conflict, the blocker is
>> currently known “only” during the RecoveryConflictInterrupt call (so
>> still not known yet when we can “already” report the blocked LSN
>> information).
>> It might also happen that the blocker(s) will never get interrupted (was
>> not blocking anymore before max_standby_streaming_delay has been
>> reached): then it would not be possible to display all the information
>> here (aka when it is interrupted) while we still want to be warned that
>> the WAL replay has been blocked.
> I think it's okay with showing different information for different
> types of recovery conflict. In buffer pin conflict case, I think we
> can let the user know the process is waiting for recovery conflict on
> buffer pin in the main message and the WAL record being replayed in
> errdetails.
>
>>>> Outcome Example:
>>>>
>>>> 2020-06-15 06:48:23.774 UTC [6971] LOG:  wal record apply is blocked by 2 connection(s), reason: User query might have needed to see row versions that must be removed.
>>>> 2020-06-15 06:48:23.774 UTC [6971] LOG:  blocked wal record rmgr: Heap2, lsn: 0/038E2678, received at: 2020-06-15 06:48:23.774098+00, desc: CLEAN, relation: rel 1663/13586/16652 fork main blk 0
>>>> 2020-06-15 06:48:54.773 UTC [7088] LOG:  about to interrupt pid: 7088, backend_type: client backend, state: active, wait_event_type: Timeout, wait_event: PgSleep, query_start: 2020-06-15 06:48:14.87672+00
>>> I'm concerned that the above information on the process who is about
>>> to be interrupted is very detailed but I'm not sure it will be helpful
>>> for the users. If the blocker is waiting on something lock, the
>>> information should be logged by log_lock_waits.
>> The blocker could also be in “idle in transaction”, say for example, on
>> the standby (with hot_standby_feedback set to off):
>>
>> standby> begin;
>> standby> SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
>> standby> select * from bdt;
>>
>> on the master:
>>
>> master> update bdt set generate_series = 15;
>> master> vacuum verbose bdt;
>>
>> would produce:
>>
>> 2020-07-01 09:18:55.256 UTC [32751] LOG:  about to interrupt pid: 32751,
>> backend_type: client backend, state: idle in transaction,
>> wait_event_type: Client, wait_event: ClientRead, query_start: 2020-07-01
>> 09:18:17.390572+00
>> 2020-07-01 09:18:55.256 UTC [32751] DETAIL:  statement: select * from bdt;
>>
>> I think those information are useful to have (might get rid of
>> wait_event_type though: done in the new attached patch).
> Since the backend cancels query at a convenient time
> (CHECK_FOR_INTERRUPT()), I'm concerned that the wait event information
> displayed in that log might not be helpful. For example, even if the
> blocker is getting stuck on disk I/O (WAIT_EVENT_BUFFILE_READ etc)
> while holding a lock that the recovery process is waiting for, when
> the blocker is able to cancel the query it's no longer waiting for
> disk I/O. Also, regarding displaying the backend type, we already show
> the backend type by setting %d in log_line_prefix.
>
> I still think we should show this additional information (wait event,
> query start etc) in errdetails even if we want to show this
> information in the server logs. Perhaps we can improve
> errdetail_recovery_conflict()?
>
> Aside from the above comments from the perspective of high-level
> design, I think we can split this patch into two patches: logging the
> recovery process is waiting (adding log_recovery_conficts) and logging
> the details of blockers who is about to be interrupted. I think it'll
> make review easy.


Ok. Let's keep this thread for the new attached patch that focus on the 
recovery process waiting.

I'll create a new one with a dedicated thread for the blockers 
information once done.

Thanks

Bertrand

> Regards,
>
> [1] https://www.postgresql.org/docs/devel/error-style-guide.html
>
> --
> Masahiko Sawada            http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--------------6F976B3B5C883C1735931861
Content-Type: text/plain; charset="UTF-8"; x-mac-type=0; x-mac-creator=0;
	name="v1-0001-Add-Standby-Recovery-Conflicts-BlockedOnly-Info.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename*0="v1-0001-Add-Standby-Recovery-Conflicts-BlockedOnly-Info.patc";
	filename*1="h"

IGRvYy9zcmMvc2dtbC9jb25maWcuc2dtbCAgICAgICAgICAgICAgICAgICAgICB8IDE1ICsr
KysrKwogc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvcHJvY3NpZ25hbC5jICAgICAgICAgIHwg
MzYgKysrKysrKysrKysrKwogc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvc3RhbmRieS5jICAg
ICAgICAgICAgIHwgNzcgKysrKysrKysrKysrKysrKysrKysrKysrKysrCiBzcmMvYmFja2Vu
ZC91dGlscy9taXNjL2d1Yy5jICAgICAgICAgICAgICAgICAgfCAxMCArKystCiBzcmMvYmFj
a2VuZC91dGlscy9taXNjL3Bvc3RncmVzcWwuY29uZi5zYW1wbGUgfCAgMSArCiBzcmMvaW5j
bHVkZS9zdG9yYWdlL3Byb2NzaWduYWwuaCAgICAgICAgICAgICAgfCAgMSArCiBzcmMvaW5j
bHVkZS9zdG9yYWdlL3N0YW5kYnkuaCAgICAgICAgICAgICAgICAgfCAgMSArCiA3IGZpbGVz
IGNoYW5nZWQsIDE0MCBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9uKC0pCgpkaWZmIC0tZ2l0
IGEvZG9jL3NyYy9zZ21sL2NvbmZpZy5zZ21sIGIvZG9jL3NyYy9zZ21sL2NvbmZpZy5zZ21s
CmluZGV4IGFjYThmNzNhNTAuLjU1ODBlYTY5NzggMTAwNjQ0Ci0tLSBhL2RvYy9zcmMvc2dt
bC9jb25maWcuc2dtbAorKysgYi9kb2Mvc3JjL3NnbWwvY29uZmlnLnNnbWwKQEAgLTY3NjMs
NiArNjc2MywyMSBAQCBsb2dfbGluZV9wcmVmaXggPSAnJW0gWyVwXSAlcSV1QCVkLyVhICcK
ICAgICAgIDwvbGlzdGl0ZW0+CiAgICAgIDwvdmFybGlzdGVudHJ5PgogCisgICAgIDx2YXJs
aXN0ZW50cnkgaWQ9Imd1Yy1sb2ctcmVjb3ZlcnktY29uZmxpY3RzIiB4cmVmbGFiZWw9Imxv
Z19yZWNvdmVyeV9jb25mbGljdHMiPgorICAgICAgPHRlcm0+PHZhcm5hbWU+bG9nX3JlY292
ZXJ5X2NvbmZsaWN0czwvdmFybmFtZT4gKDx0eXBlPmJvb2xlYW48L3R5cGU+KQorICAgICAg
PGluZGV4dGVybT4KKyAgICAgICA8cHJpbWFyeT48dmFybmFtZT5sb2dfcmVjb3ZlcnlfY29u
ZmxpY3RzPC92YXJuYW1lPiBjb25maWd1cmF0aW9uIHBhcmFtZXRlcjwvcHJpbWFyeT4KKyAg
ICAgIDwvaW5kZXh0ZXJtPgorICAgICAgPC90ZXJtPgorICAgICAgPGxpc3RpdGVtPgorICAg
ICAgIDxwYXJhPgorICAgICAgICBDb250cm9scyB3aGV0aGVyIGRldGFpbGVkIGluZm9ybWF0
aW9uIGlzIHByb2R1Y2VkIHdoZW4gYSBjb25mbGljdAorICAgICAgICBvY2N1cnMgZHVyaW5n
IHN0YW5kYnkgcmVjb3ZlcnkuIFRoZSBkZWZhdWx0IGlzIDxsaXRlcmFsPm9uPC9saXRlcmFs
Pi4KKyAgICAgICAgT25seSBzdXBlcnVzZXJzIGNhbiBjaGFuZ2UgdGhpcyBzZXR0aW5nLgor
ICAgICAgIDwvcGFyYT4KKyAgICAgIDwvbGlzdGl0ZW0+CisgICAgIDwvdmFybGlzdGVudHJ5
PgorCiAgICAgIDx2YXJsaXN0ZW50cnkgaWQ9Imd1Yy1sb2ctcGFyYW1ldGVyLW1heC1sZW5n
dGgiIHhyZWZsYWJlbD0ibG9nX3BhcmFtZXRlcl9tYXhfbGVuZ3RoIj4KICAgICAgIDx0ZXJt
Pjx2YXJuYW1lPmxvZ19wYXJhbWV0ZXJfbWF4X2xlbmd0aDwvdmFybmFtZT4gKDx0eXBlPmlu
dGVnZXI8L3R5cGU+KQogICAgICAgPGluZGV4dGVybT4KZGlmZiAtLWdpdCBhL3NyYy9iYWNr
ZW5kL3N0b3JhZ2UvaXBjL3Byb2NzaWduYWwuYyBiL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBj
L3Byb2NzaWduYWwuYwppbmRleCBjODA5MTk2ZDA2Li5mODMxNzZjZmE3IDEwMDY0NAotLS0g
YS9zcmMvYmFja2VuZC9zdG9yYWdlL2lwYy9wcm9jc2lnbmFsLmMKKysrIGIvc3JjL2JhY2tl
bmQvc3RvcmFnZS9pcGMvcHJvY3NpZ25hbC5jCkBAIC01NzYsMyArNTc2LDM5IEBAIHByb2Nz
aWduYWxfc2lndXNyMV9oYW5kbGVyKFNJR05BTF9BUkdTKQogCiAJZXJybm8gPSBzYXZlX2Vy
cm5vOwogfQorCitleHRlcm4gY29uc3QgY2hhciAqZ2V0X3Byb2NzaWduYWxfcmVhc29uX2Rl
c2MoUHJvY1NpZ25hbFJlYXNvbiByZWFzb24pCisJeworCQljb25zdCBjaGFyICpyZWFzb25E
ZXNjID0gInVua25vd24gcmVhc29uIjsKKworCQlzd2l0Y2ggKHJlYXNvbikKKwkJeworCQkJ
Y2FzZSBQUk9DU0lHX1JFQ09WRVJZX0NPTkZMSUNUX0JVRkZFUlBJTjoKKwkJCQlyZWFzb25E
ZXNjID0gImJ1ZmZlciBwaW4iOworCQkJCWJyZWFrOworCQkJY2FzZSBQUk9DU0lHX1JFQ09W
RVJZX0NPTkZMSUNUX0xPQ0s6CisJCQkJcmVhc29uRGVzYyA9ICJsb2NrIjsKKwkJCQlicmVh
azsKKwkJCWNhc2UgUFJPQ1NJR19SRUNPVkVSWV9DT05GTElDVF9UQUJMRVNQQUNFOgorCQkJ
CXJlYXNvbkRlc2MgPSAidGFibGVzcGFjZSI7CisJCQkJYnJlYWs7CisJCQljYXNlIFBST0NT
SUdfUkVDT1ZFUllfQ09ORkxJQ1RfU05BUFNIT1Q6CisJCQkJcmVhc29uRGVzYyA9ICJzbmFw
c2hvdCI7CisJCQkJYnJlYWs7CisJCQljYXNlIFBST0NTSUdfUkVDT1ZFUllfQ09ORkxJQ1Rf
U1RBUlRVUF9ERUFETE9DSzoKKwkJCQlyZWFzb25EZXNjID0gImRlYWRsb2NrIjsKKwkJCQli
cmVhazsKKwkJCWNhc2UgUFJPQ1NJR19SRUNPVkVSWV9DT05GTElDVF9EQVRBQkFTRToKKwkJ
CQlyZWFzb25EZXNjID0gImRhdGFiYXNlIjsKKwkJCQlicmVhazsKKwkJCWNhc2UgUFJPQ1NJ
R19DQVRDSFVQX0lOVEVSUlVQVDoKKwkJCWNhc2UgUFJPQ1NJR19OT1RJRllfSU5URVJSVVBU
OgorCQkJY2FzZSBQUk9DU0lHX1BBUkFMTEVMX01FU1NBR0U6CisJCQljYXNlIFBST0NTSUdf
V0FMU05EX0lOSVRfU1RPUFBJTkc6CisJCQljYXNlIE5VTV9QUk9DU0lHTkFMUzoKKwkJCQli
cmVhazsKKwkJCWRlZmF1bHQ6CisJCQkJYnJlYWs7CisJCX0KKwlyZXR1cm4gcmVhc29uRGVz
YzsKK30KZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL3N0YW5kYnkuYyBi
L3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL3N0YW5kYnkuYwppbmRleCA5ZTBkNWVjMjU3Li4x
MjY4NThhMzYzIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9zdG9yYWdlL2lwYy9zdGFuZGJ5
LmMKKysrIGIvc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvc3RhbmRieS5jCkBAIC0zOSw2ICsz
OSw3IEBACiBpbnQJCQl2YWN1dW1fZGVmZXJfY2xlYW51cF9hZ2U7CiBpbnQJCQltYXhfc3Rh
bmRieV9hcmNoaXZlX2RlbGF5ID0gMzAgKiAxMDAwOwogaW50CQkJbWF4X3N0YW5kYnlfc3Ry
ZWFtaW5nX2RlbGF5ID0gMzAgKiAxMDAwOworYm9vbAkJCWxvZ19yZWNvdmVyeV9jb25mbGlj
dHMgPSB0cnVlOwogCiBzdGF0aWMgSFRBQiAqUmVjb3ZlcnlMb2NrTGlzdHM7CiAKQEAgLTIx
NSw2ICsyMTYsNTkgQEAgV2FpdEV4Y2VlZHNNYXhTdGFuZGJ5RGVsYXkodWludDMyIHdhaXRf
ZXZlbnRfaW5mbykKIAlyZXR1cm4gZmFsc2U7CiB9CiAKKy8qCisgKiBEaXNwbGF5IGluZm9y
bWF0aW9uIGFib3V0IHRoZSB3YWwgcmVjb3JkCisgKiBhcHBseSBiZWluZyBibG9ja2VkCisg
Ki8KK3N0YXRpYyB2b2lkCitMb2dCbG9ja2VkV2FsUmVjb3JkSW5mbyhpbnQgbnVtX3dhaXRs
aXN0X2VudHJpZXMsIFByb2NTaWduYWxSZWFzb24gcmVhc29uKQoreworCVJlbEZpbGVOb2Rl
IHJub2RlOworCUZvcmtOdW1iZXIgZm9ya251bTsKKwlCbG9ja051bWJlciBibGtudW07CisJ
VGltZXN0YW1wVHogcnRpbWU7CisJYm9vbCBmcm9tU3RyZWFtOworCWNoYXIgKnJlY2VpcHRf
dGltZV9zdHI7CisJY2hhciAqd2FsX3JlY29yZF9kZXRhaWxfc3RyOworCWNoYXIgKnJlY292
ZXJ5X2NvbmZsaWN0X21haW5fbWVzc2FnZTsKKwlFcnJvckNvbnRleHRDYWxsYmFjayAqZWNv
bnRleHQ7CisJTWVtb3J5Q29udGV4dCBvbGRjb250ZXh0OworCisJcmVjb3ZlcnlfY29uZmxp
Y3RfbWFpbl9tZXNzYWdlID0gcHNwcmludGYoIndhaXRpbmcgZm9yIHJlY292ZXJ5IGNvbmZs
aWN0IG9uICVzIiwKKwkJCQkJCQlnZXRfcHJvY3NpZ25hbF9yZWFzb25fZGVzYyhyZWFzb24p
KTsKKworCUdldFhMb2dSZWNlaXB0VGltZSgmcnRpbWUsICZmcm9tU3RyZWFtKTsKKwlyZWNl
aXB0X3RpbWVfc3RyID0gcHN0cmR1cCh0aW1lc3RhbXB0el90b19zdHIocnRpbWUpKTsKKwor
CisJb2xkY29udGV4dCA9IE1lbW9yeUNvbnRleHRTd2l0Y2hUbyhFcnJvckNvbnRleHQpOwor
CWVjb250ZXh0ID0gZXJyb3JfY29udGV4dF9zdGFjazsKKworCWlmIChYTG9nUmVjR2V0Qmxv
Y2tUYWcoZWNvbnRleHQtPmFyZywgMCwgJnJub2RlLCAmZm9ya251bSwgJmJsa251bSkpCisJ
CXdhbF9yZWNvcmRfZGV0YWlsX3N0ciA9IHBzcHJpbnRmKCJXQUwgcmVjb3JkIHJlY2VpdmVk
IGF0ICVzLlxuVGFibGVzcGFjZS9kYXRhYmFzZS9yZWxhdGlvbiBhcmUgJXUvJXUvJXUsIGZv
cmsgaXMgJXMgYW5kIGJsb2NrIGlzICV1LiIsCisJCQkJCQlyZWNlaXB0X3RpbWVfc3RyLCBy
bm9kZS5zcGNOb2RlLCBybm9kZS5kYk5vZGUsIHJub2RlLnJlbE5vZGUsCisJCQkJCQlmb3Jr
TmFtZXNbZm9ya251bV0sCisJCQkJCQlibGtudW0pOworCWVsc2UKKwkJd2FsX3JlY29yZF9k
ZXRhaWxfc3RyID0gcHNwcmludGYoIldBTCByZWNvcmQgcmVjZWl2ZWQgYXQgJXMuIiwgcmVj
ZWlwdF90aW1lX3N0cik7CisKKwlNZW1vcnlDb250ZXh0U3dpdGNoVG8ob2xkY29udGV4dCk7
CisKKwlpZiAobnVtX3dhaXRsaXN0X2VudHJpZXMgPiAwKQorCQllcmVwb3J0KExPRywKKwkJ
CQkoZXJybXNnKCIlcyIsIHJlY292ZXJ5X2NvbmZsaWN0X21haW5fbWVzc2FnZSksCisJCQkJ
ZXJyZGV0YWlsKCIlc1xuIiAiVGhlcmUgaXMgJWQgYmxvY2tpbmcgY29ubmVjdGlvbihzKS4i
LCB3YWxfcmVjb3JkX2RldGFpbF9zdHIsIG51bV93YWl0bGlzdF9lbnRyaWVzKSkpOworCWVs
c2UKKwkJZXJlcG9ydChMT0csCisJCQkJKGVycm1zZygiJXMiLCByZWNvdmVyeV9jb25mbGlj
dF9tYWluX21lc3NhZ2UpLAorCQkJCWVycmRldGFpbCgiJXMiLCB3YWxfcmVjb3JkX2RldGFp
bF9zdHIpKSk7CisKKworCXBmcmVlKHJlY292ZXJ5X2NvbmZsaWN0X21haW5fbWVzc2FnZSk7
CisJcGZyZWUod2FsX3JlY29yZF9kZXRhaWxfc3RyKTsKKwlwZnJlZShyZWNlaXB0X3RpbWVf
c3RyKTsKK30KKwogLyoKICAqIFRoaXMgaXMgdGhlIG1haW4gZXhlY3V0aW9uZXIgZm9yIGFu
eSBxdWVyeSBiYWNrZW5kIHRoYXQgY29uZmxpY3RzIHdpdGgKICAqIHJlY292ZXJ5IHByb2Nl
c3NpbmcuIEp1ZGdlbWVudCBoYXMgYWxyZWFkeSBiZWVuIHBhc3NlZCBvbiBpdCB3aXRoaW4K
QEAgLTIzMiw2ICsyODYsOCBAQCBSZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdpdGhWaXJ0dWFs
WElEcyhWaXJ0dWFsVHJhbnNhY3Rpb25JZCAqd2FpdGxpc3QsCiB7CiAJVGltZXN0YW1wVHog
d2FpdFN0YXJ0ID0gMDsKIAljaGFyCSAgICpuZXdfc3RhdHVzOworCVZpcnR1YWxUcmFuc2Fj
dGlvbklkICp0bXBXYWl0bGlzdDsKKwlpbnQgbnVtX3dhaXRsaXN0X2VudHJpZXM7CiAKIAkv
KiBGYXN0IGV4aXQsIHRvIGF2b2lkIGEga2VybmVsIGNhbGwgaWYgdGhlcmUncyBubyB3b3Jr
IHRvIGJlIGRvbmUuICovCiAJaWYgKCFWaXJ0dWFsVHJhbnNhY3Rpb25JZElzVmFsaWQoKndh
aXRsaXN0KSkKQEAgLTI0MSw2ICsyOTcsMTggQEAgUmVzb2x2ZVJlY292ZXJ5Q29uZmxpY3RX
aXRoVmlydHVhbFhJRHMoVmlydHVhbFRyYW5zYWN0aW9uSWQgKndhaXRsaXN0LAogCQl3YWl0
U3RhcnQgPSBHZXRDdXJyZW50VGltZXN0YW1wKCk7CiAJbmV3X3N0YXR1cyA9IE5VTEw7CQkJ
Lyogd2UgaGF2ZW4ndCBjaGFuZ2VkIHRoZSBwcyBkaXNwbGF5ICovCiAKKwl0bXBXYWl0bGlz
dCA9IHdhaXRsaXN0OworCXdoaWxlIChWaXJ0dWFsVHJhbnNhY3Rpb25JZElzVmFsaWQoKnRt
cFdhaXRsaXN0KSkKKwl7CisJCXRtcFdhaXRsaXN0Kys7CisJfQorCisJbnVtX3dhaXRsaXN0
X2VudHJpZXMgPSAodG1wV2FpdGxpc3QgLSB3YWl0bGlzdCk7CisKKwkvKiBkaXNwbGF5IHdh
bCByZWNvcmQgaW5mb3JtYXRpb24gKi8KKwlpZiAobG9nX3JlY292ZXJ5X2NvbmZsaWN0cykK
KwkJTG9nQmxvY2tlZFdhbFJlY29yZEluZm8obnVtX3dhaXRsaXN0X2VudHJpZXMsIHJlYXNv
bik7CisKIAl3aGlsZSAoVmlydHVhbFRyYW5zYWN0aW9uSWRJc1ZhbGlkKCp3YWl0bGlzdCkp
CiAJewogCQkvKiByZXNldCBzdGFuZGJ5V2FpdF91cyBmb3IgZWFjaCB4YWN0IHdlIHdhaXQg
Zm9yICovCkBAIC0zNzAsNiArNDM4LDExIEBAIFJlc29sdmVSZWNvdmVyeUNvbmZsaWN0V2l0
aERhdGFiYXNlKE9pZCBkYmlkKQogCSAqIGJsb2NrIGR1cmluZyBJbml0UG9zdGdyZXMoKSBh
bmQgdGhlbiBkaXNjb25uZWN0IHdoZW4gdGhleSBzZWUgdGhlCiAJICogZGF0YWJhc2UgaGFz
IGJlZW4gcmVtb3ZlZC4KIAkgKi8KKworCS8qIGRpc3BsYXkgd2FsIHJlY29yZCBpbmZvcm1h
dGlvbiAqLworCWlmIChsb2dfcmVjb3ZlcnlfY29uZmxpY3RzKQorCQlMb2dCbG9ja2VkV2Fs
UmVjb3JkSW5mbyhDb3VudERCQmFja2VuZHMoZGJpZCksIFBST0NTSUdfUkVDT1ZFUllfQ09O
RkxJQ1RfREFUQUJBU0UpOworCiAJd2hpbGUgKENvdW50REJCYWNrZW5kcyhkYmlkKSA+IDAp
CiAJewogCQlDYW5jZWxEQkJhY2tlbmRzKGRiaWQsIFBST0NTSUdfUkVDT1ZFUllfQ09ORkxJ
Q1RfREFUQUJBU0UsIHRydWUpOwpAQCAtNjA5LDYgKzY4MiwxMCBAQCBTdGFuZGJ5VGltZW91
dEhhbmRsZXIodm9pZCkKIAkvKiBmb3JnZXQgYW55IHBlbmRpbmcgU1RBTkRCWV9ERUFETE9D
S19USU1FT1VUIHJlcXVlc3QgKi8KIAlkaXNhYmxlX3RpbWVvdXQoU1RBTkRCWV9ERUFETE9D
S19USU1FT1VULCBmYWxzZSk7CiAKKwkvKiBkaXNwbGF5IHdhbCByZWNvcmQgaW5mb3JtYXRp
b24gKi8KKwlpZiAobG9nX3JlY292ZXJ5X2NvbmZsaWN0cykKKwkJTG9nQmxvY2tlZFdhbFJl
Y29yZEluZm8oLTEsIFBST0NTSUdfUkVDT1ZFUllfQ09ORkxJQ1RfQlVGRkVSUElOKTsKKwog
CVNlbmRSZWNvdmVyeUNvbmZsaWN0V2l0aEJ1ZmZlclBpbihQUk9DU0lHX1JFQ09WRVJZX0NP
TkZMSUNUX0JVRkZFUlBJTik7CiB9CiAKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3V0aWxz
L21pc2MvZ3VjLmMgYi9zcmMvYmFja2VuZC91dGlscy9taXNjL2d1Yy5jCmluZGV4IDJmM2Uw
YTcwZTAuLjdhYTU1ZGI1YWYgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3V0aWxzL21pc2Mv
Z3VjLmMKKysrIGIvc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9ndWMuYwpAQCAtMTU5NCw3ICsx
NTk0LDE1IEBAIHN0YXRpYyBzdHJ1Y3QgY29uZmlnX2Jvb2wgQ29uZmlndXJlTmFtZXNCb29s
W10gPQogCQlmYWxzZSwKIAkJTlVMTCwgTlVMTCwgTlVMTAogCX0sCi0KKwl7CisJCXsibG9n
X3JlY292ZXJ5X2NvbmZsaWN0cyIsIFBHQ19TVVNFVCwgTE9HR0lOR19XSEFULAorCQkJZ2V0
dGV4dF9ub29wKCJMb2dzIHN0YW5kYnkgcmVjb3ZlcnkgY29uZmxpY3RzLiIpLAorCQkJTlVM
TAorCQl9LAorCQkmbG9nX3JlY292ZXJ5X2NvbmZsaWN0cywKKwkJdHJ1ZSwKKwkJTlVMTCwg
TlVMTCwgTlVMTAorCX0sCiAJewogCQl7ImxvZ19ob3N0bmFtZSIsIFBHQ19TSUdIVVAsIExP
R0dJTkdfV0hBVCwKIAkJCWdldHRleHRfbm9vcCgiTG9ncyB0aGUgaG9zdCBuYW1lIGluIHRo
ZSBjb25uZWN0aW9uIGxvZ3MuIiksCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC91dGlscy9t
aXNjL3Bvc3RncmVzcWwuY29uZi5zYW1wbGUgYi9zcmMvYmFja2VuZC91dGlscy9taXNjL3Bv
c3RncmVzcWwuY29uZi5zYW1wbGUKaW5kZXggYWMwMmJkMGMwMC4uMTAwOTUwNmVmZiAxMDA2
NDQKLS0tIGEvc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9wb3N0Z3Jlc3FsLmNvbmYuc2FtcGxl
CisrKyBiL3NyYy9iYWNrZW5kL3V0aWxzL21pc2MvcG9zdGdyZXNxbC5jb25mLnNhbXBsZQpA
QCAtNTQ4LDYgKzU0OCw3IEBACiAJCQkJCSMgICAlJSA9ICclJwogCQkJCQkjIGUuZy4gJzwl
dSUlJWQ+ICcKICNsb2dfbG9ja193YWl0cyA9IG9mZgkJCSMgbG9nIGxvY2sgd2FpdHMgPj0g
ZGVhZGxvY2tfdGltZW91dAorI2xvZ19yZWNvdmVyeV9jb25mbGljdHMgPSBvbgkJIyBsb2cg
c3RhbmRieSByZWNvdmVyeSBjb25mbGljdHMKICNsb2dfcGFyYW1ldGVyX21heF9sZW5ndGgg
PSAtMQkJIyB3aGVuIGxvZ2dpbmcgc3RhdGVtZW50cywgbGltaXQgbG9nZ2VkCiAJCQkJCSMg
YmluZC1wYXJhbWV0ZXIgdmFsdWVzIHRvIE4gYnl0ZXM7CiAJCQkJCSMgLTEgbWVhbnMgcHJp
bnQgaW4gZnVsbCwgMCBkaXNhYmxlcwpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvc3RvcmFn
ZS9wcm9jc2lnbmFsLmggYi9zcmMvaW5jbHVkZS9zdG9yYWdlL3Byb2NzaWduYWwuaAppbmRl
eCBhMGMwYmMzY2U1Li5jNTc0MTNiMDZkIDEwMDY0NAotLS0gYS9zcmMvaW5jbHVkZS9zdG9y
YWdlL3Byb2NzaWduYWwuaAorKysgYi9zcmMvaW5jbHVkZS9zdG9yYWdlL3Byb2NzaWduYWwu
aApAQCAtNzAsNSArNzAsNiBAQCBleHRlcm4gdm9pZCBXYWl0Rm9yUHJvY1NpZ25hbEJhcnJp
ZXIodWludDY0IGdlbmVyYXRpb24pOwogZXh0ZXJuIHZvaWQgUHJvY2Vzc1Byb2NTaWduYWxC
YXJyaWVyKHZvaWQpOwogCiBleHRlcm4gdm9pZCBwcm9jc2lnbmFsX3NpZ3VzcjFfaGFuZGxl
cihTSUdOQUxfQVJHUyk7CitleHRlcm4gY29uc3QgY2hhciAqZ2V0X3Byb2NzaWduYWxfcmVh
c29uX2Rlc2MoUHJvY1NpZ25hbFJlYXNvbiByZWFzb24pOwogCiAjZW5kaWYJCQkJCQkJLyog
UFJPQ1NJR05BTF9IICovCmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS9zdG9yYWdlL3N0YW5k
YnkuaCBiL3NyYy9pbmNsdWRlL3N0b3JhZ2Uvc3RhbmRieS5oCmluZGV4IGNmYmU0MjZlNWEu
LjBhOTlhZDNmNDAgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL3N0b3JhZ2Uvc3RhbmRieS5o
CisrKyBiL3NyYy9pbmNsdWRlL3N0b3JhZ2Uvc3RhbmRieS5oCkBAIC0yMyw2ICsyMyw3IEBA
CiBleHRlcm4gaW50CXZhY3V1bV9kZWZlcl9jbGVhbnVwX2FnZTsKIGV4dGVybiBpbnQJbWF4
X3N0YW5kYnlfYXJjaGl2ZV9kZWxheTsKIGV4dGVybiBpbnQJbWF4X3N0YW5kYnlfc3RyZWFt
aW5nX2RlbGF5OworZXh0ZXJuIGJvb2wJbG9nX3JlY292ZXJ5X2NvbmZsaWN0czsKIAogZXh0
ZXJuIHZvaWQgSW5pdFJlY292ZXJ5VHJhbnNhY3Rpb25FbnZpcm9ubWVudCh2b2lkKTsKIGV4
dGVybiB2b2lkIFNodXRkb3duUmVjb3ZlcnlUcmFuc2FjdGlvbkVudmlyb25tZW50KHZvaWQp
Owo=
--------------6F976B3B5C883C1735931861--