IronPort-SDR: 
 1bqhf33mWwi+EHE6iN4nc3M8dMyoplr8OjtymsQYctb2sLGlWWe1n+2HV51C5HLviC9b+kU+GL
 O0ZZ648y6l/g==
Subject: Re: Add Information during standby recovery conflicts
To: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
CC: PostgreSQL-development <pgsql-hackers@postgresql.org>
References: <9a60178c-a853-1440-2cdc-c3af916cff59@amazon.com>
 <CA+fd4k6oBCoF+4sAgWqsC5+B5+EWQfT=FyPg9zVo4wh+zTXhUg@mail.gmail.com>
 <daeecc9d-567a-cad8-298a-6f12ed7cb665@amazon.com>
 <CA+fd4k45sOD63XV-fcpi-ACS12PQa4G19_qf+cCVUcKK2nfKJw@mail.gmail.com>
 <3aededb3-8eb7-bfd6-6fd4-b6487f8f1d74@amazon.com>
 <CA+fd4k5ftD-OZPA2K0gPJxDv12mxAknAM2LVTjMRFbjZCC2FhQ@mail.gmail.com>
From: "Drouvot, Bertrand" <bdrouvot@amazon.com>
Message-ID: <85c21e8e-4b12-9ff6-92bf-24fac2166632@amazon.com>
Date: Sat, 11 Jul 2020 14:55:57 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:68.0)
 Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+fd4k5ftD-OZPA2K0gPJxDv12mxAknAM2LVTjMRFbjZCC2FhQ@mail.gmail.com>
Content-Type: multipart/mixed;
	boundary="------------C16C6C96B3132FD69C03B2BF"
Content-Language: en-US
Precedence: bulk

--------------C16C6C96B3132FD69C03B2BF
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit


On 7/7/20 9:43 AM, Masahiko Sawada wrote:
> Fully makes sense, the new patch version attached is now producing:
>> 2020-07-06 06:10:36.022 UTC [14035] LOG:  waiting for recovery conflict
>> on snapshot
> How about adding the subject? that is, "recovery is waiting for
> recovery conflict on %s" or "recovery process <pid> is waiting for
> conflict on %s".


The subject is now added in the new attached patch (I did not include 
the pid as it is part of the log prefix).

It now looks like:

2020-07-11 12:00:41.092 UTC [23217] LOG:  recovery is waiting for 
recovery conflict on snapshot
2020-07-11 12:00:41.092 UTC [23217] DETAIL:  There is 1 blocking 
connection(s).
2020-07-11 12:00:41.092 UTC [23217] CONTEXT:  WAL redo at 0/4A0A6BF0 for 
Heap2/CLEAN: remxid 1128
         WAL record received at 2020-07-11 12:00:41.092231+00
         tbs 1663 db 13586 rel 16805, fork main, blkno 0
>
>> 2020-07-06 06:10:36.022 UTC [14035] DETAIL:  WAL record received at
>> 2020-07-06 06:10:36.021963+00.
>>           Tablespace/database/relation are 1663/13586/16672, fork is main
>> and block is 0.
>>           There is 1 blocking connection(s).
> To follow the existing log message, perhaps this can be something like
> "WAL record received at %s, rel %u/%u/%u, fork %s, blkno %u. %d
> processes". But I'm not sure the errdetail is the best place to
> display the WAL information as I mentioned in the latter part of this
> email.

moved to the context and formatted the same way as the current 
Standby/LOCK context.


> Ok. Let's keep this thread for the new attached patch that focus on the
>> recovery process waiting.
> Thank you for updating the patch!
>
> I've tested the latest patch.


Thank you for testing and reviewing!


> On recovery conflict on lock and on
> bufferpin, if max_standby_streaming_delay is disabled (set to -1), the
> logs don't appear even if log_recovery_conflicts is true.


Nice catch! it is fixed in the new attached patch (the log reporting has 
been moved out of StandbyTimeoutHandler()).

>
> Here is random comments on the code:
>
> +       recovery_conflict_main_message = psprintf("waiting for
> recovery conflict on %s",
> +
> get_procsignal_reason_desc(reason));
> :
> +               ereport(LOG,
> +                               (errmsg("%s", recovery_conflict_main_message),
> +                               errdetail("%s\n" "There is %d blocking
> connection(s).", wal_record_detail_str, num_waitlist_entries)));
>
> It's not translation-support-friendly. I think the message "waiting
> for recovery conflict on %s" should be surrounded by _(). Or we can
> just put it to ereport as follows:
>
> ereport(LOG,
>      (errmsg("waiting for recovery conflicts on %s",
> get_procsignal_reason_desc(reason))
>      ...


changed in the new attached patch.


>
> ---
> +       oldcontext = MemoryContextSwitchTo(ErrorContext);
> +       econtext = error_context_stack;
> +
> +       if (XLogRecGetBlockTag(econtext->arg, 0, &rnode, &forknum, &blknum))
>
> I don't think it's a good idea to rely on error_context_stack because
> other codes might set another error context before reaching here in
> the future.


right, changed in the new attached patch: this is now done in 
rm_redo_error_callback() and using the XLogReaderState passed as argument.


>
> ---
> +       if (XLogRecGetBlockTag(econtext->arg, 0, &rnode, &forknum, &blknum))
> +               wal_record_detail_str = psprintf("WAL record received
> at %s.\nTablespace/database/relation are %u/%u/%u, fork is %s and
> block is %u.",
> +                                               receipt_time_str,
> rnode.spcNode, rnode.dbNode, rnode.relNode,
> +                                               forkNames[forknum],
> +                                               blknum);
>
> There might be a block tag in block ids other than 0.


right, fixed in the new attached patch.


> I'm not sure the
> errdetail is the best place where we display WAL information.


moved to context in the new attached patch.


> For
> instance, we display both the relation oid and block number depending
> on RM as follows:
>
> 2020-07-07 15:50:30.733 JST [13344] LOG:  waiting for recovery conflict on lock
> 2020-07-07 15:50:30.733 JST [13344] DETAIL:  WAL record received at
> 2020-07-07 15:50:27.73378+09.
>          There is 1 blocking connection(s).
> 2020-07-07 15:50:30.733 JST [13344] CONTEXT:  WAL redo at 0/3000028
> for Standby/LOCK: xid 506 db 13586 rel 16384
>
> I wonder if we can display the details of redo WAL information by
> improving xlog_outdesc() or rm_redo_error_callback() so that it
> displays relfilenode, forknum, and block number. What do you think?


I think that fully make sense to move this to rm_redo_error_callback().

This is where the information is now been displayed in the new attached 
patch.


>
> ---
> +       /* display wal record information */
> +       if (log_recovery_conflicts)
> +               LogBlockedWalRecordInfo(num_waitlist_entries, reason);
>
> I'm concerned that we should log the recovery conflict information
> every time when a recovery conflict happens from the perspective of
> overheads and the amount of the logs. Can we logs that information
> after waiting for deadlock_timeouts secs like log_lock_waits or
> waiting for the fixed duration?


The new attached patch is now waiting for deadlock_timeout duration.

>
> ---
> @@ -609,6 +682,10 @@ StandbyTimeoutHandler(void)
>          /* forget any pending STANDBY_DEADLOCK_TIMEOUT request */
>          disable_timeout(STANDBY_DEADLOCK_TIMEOUT, false);
>
> +       /* display wal record information */
> +       if (log_recovery_conflicts)
> +               LogBlockedWalRecordInfo(-1,
> PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
> +
>          SendRecoveryConflictWithBufferPin(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
>   }
>
> ResolveRecoveryConflictWithBufferPin() which sets a timer to call
> StandbyTimeoutHandler() can be called multiple times even if the
> recovery is waiting for one buffer pin. I think we should avoid
> logging the same contents multiple times.


I do agree, only the first pass is now been logged.


>
> ---
> -
> +       {
> +               {"log_recovery_conflicts", PGC_SUSET, LOGGING_WHAT,
> +                       gettext_noop("Logs standby recovery conflicts."),
> +                       NULL
> +               },
> +               &log_recovery_conflicts,
> +               true,
> +               NULL, NULL, NULL
> +       },
>
> Other logging parameters such as log_lock_waits is false by default. I
> think this parameter can also be false by default but is there any
> reason to enable it by default?


now set to false by default.


>
> Since this parameter applies only to the startup process, I think it
> should be PGC_SIGHUP.


changed that way.


>
> ---
> +       /* display wal record information */
> +       if (log_recovery_conflicts)
> +               LogBlockedWalRecordInfo(CountDBBackends(dbid),
> PROCSIG_RECOVERY_CONFLICT_DATABASE);
>
> We log the recovery conflict into the server log but we don't update
> the process title to append "waiting". While discussing the process
> title update on recovery conflict, I got the review comment[1] that we
> don't need to update the process title because no wait occurs when
> recovery conflict with database happens. As the comment says, recovery
> is canceling the existing processes on the database being removed, but
> not waiting for something.


Ok, I keep reporting the conflict but changed the wording for this 
particular case.

Thanks

Bertrand


--------------C16C6C96B3132FD69C03B2BF
Content-Type: text/plain; charset="UTF-8"; x-mac-type=0; x-mac-creator=0;
	name="v1-0002-Add-Standby-Recovery-Conflicts-BlockedOnly-Info.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename*0="v1-0002-Add-Standby-Recovery-Conflicts-BlockedOnly-Info.patc";
	filename*1="h"

IGRvYy9zcmMvc2dtbC9jb25maWcuc2dtbCAgICAgICAgICAgICAgICAgICAgICB8IDE1ICsr
KysrKysKIHNyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYyAgICAgICAgICAgICB8
IDI5ICsrKysrKysrKysrKystCiBzcmMvYmFja2VuZC9zdG9yYWdlL2J1ZmZlci9idWZtZ3Iu
YyAgICAgICAgICAgfCAgNCArLQogc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvcHJvY3NpZ25h
bC5jICAgICAgICAgIHwgMzYgKysrKysrKysrKysrKysrKysKIHNyYy9iYWNrZW5kL3N0b3Jh
Z2UvaXBjL3N0YW5kYnkuYyAgICAgICAgICAgICB8IDU3ICsrKysrKysrKysrKysrKysrKysr
KysrKysrLQogc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9ndWMuYyAgICAgICAgICAgICAgICAg
IHwgMTAgKysrKy0KIHNyYy9iYWNrZW5kL3V0aWxzL21pc2MvcG9zdGdyZXNxbC5jb25mLnNh
bXBsZSB8ICAxICsKIHNyYy9pbmNsdWRlL3N0b3JhZ2UvcHJvY3NpZ25hbC5oICAgICAgICAg
ICAgICB8ICAxICsKIHNyYy9pbmNsdWRlL3N0b3JhZ2Uvc3RhbmRieS5oICAgICAgICAgICAg
ICAgICB8ICAzICstCiA5IGZpbGVzIGNoYW5nZWQsIDE1MCBpbnNlcnRpb25zKCspLCA2IGRl
bGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9jb25maWcuc2dtbCBiL2Rv
Yy9zcmMvc2dtbC9jb25maWcuc2dtbAppbmRleCBhY2E4ZjczYTUwLi41NTgwZWE2OTc4IDEw
MDY0NAotLS0gYS9kb2Mvc3JjL3NnbWwvY29uZmlnLnNnbWwKKysrIGIvZG9jL3NyYy9zZ21s
L2NvbmZpZy5zZ21sCkBAIC02NzYzLDYgKzY3NjMsMjEgQEAgbG9nX2xpbmVfcHJlZml4ID0g
JyVtIFslcF0gJXEldUAlZC8lYSAnCiAgICAgICA8L2xpc3RpdGVtPgogICAgICA8L3Zhcmxp
c3RlbnRyeT4KIAorICAgICA8dmFybGlzdGVudHJ5IGlkPSJndWMtbG9nLXJlY292ZXJ5LWNv
bmZsaWN0cyIgeHJlZmxhYmVsPSJsb2dfcmVjb3ZlcnlfY29uZmxpY3RzIj4KKyAgICAgIDx0
ZXJtPjx2YXJuYW1lPmxvZ19yZWNvdmVyeV9jb25mbGljdHM8L3Zhcm5hbWU+ICg8dHlwZT5i
b29sZWFuPC90eXBlPikKKyAgICAgIDxpbmRleHRlcm0+CisgICAgICAgPHByaW1hcnk+PHZh
cm5hbWU+bG9nX3JlY292ZXJ5X2NvbmZsaWN0czwvdmFybmFtZT4gY29uZmlndXJhdGlvbiBw
YXJhbWV0ZXI8L3ByaW1hcnk+CisgICAgICA8L2luZGV4dGVybT4KKyAgICAgIDwvdGVybT4K
KyAgICAgIDxsaXN0aXRlbT4KKyAgICAgICA8cGFyYT4KKyAgICAgICAgQ29udHJvbHMgd2hl
dGhlciBkZXRhaWxlZCBpbmZvcm1hdGlvbiBpcyBwcm9kdWNlZCB3aGVuIGEgY29uZmxpY3QK
KyAgICAgICAgb2NjdXJzIGR1cmluZyBzdGFuZGJ5IHJlY292ZXJ5LiBUaGUgZGVmYXVsdCBp
cyA8bGl0ZXJhbD5vbjwvbGl0ZXJhbD4uCisgICAgICAgIE9ubHkgc3VwZXJ1c2VycyBjYW4g
Y2hhbmdlIHRoaXMgc2V0dGluZy4KKyAgICAgICA8L3BhcmE+CisgICAgICA8L2xpc3RpdGVt
PgorICAgICA8L3Zhcmxpc3RlbnRyeT4KKwogICAgICA8dmFybGlzdGVudHJ5IGlkPSJndWMt
bG9nLXBhcmFtZXRlci1tYXgtbGVuZ3RoIiB4cmVmbGFiZWw9ImxvZ19wYXJhbWV0ZXJfbWF4
X2xlbmd0aCI+CiAgICAgICA8dGVybT48dmFybmFtZT5sb2dfcGFyYW1ldGVyX21heF9sZW5n
dGg8L3Zhcm5hbWU+ICg8dHlwZT5pbnRlZ2VyPC90eXBlPikKICAgICAgIDxpbmRleHRlcm0+
CmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMgYi9zcmMv
YmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMKaW5kZXggNTVjYWMxODZkYy4uOTVlNWIw
NDU3YiAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jCisr
KyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYwpAQCAtMTE3NDUsMTAgKzEx
NzQ1LDM3IEBAIHJtX3JlZG9fZXJyb3JfY2FsbGJhY2sodm9pZCAqYXJnKQogewogCVhMb2dS
ZWFkZXJTdGF0ZSAqcmVjb3JkID0gKFhMb2dSZWFkZXJTdGF0ZSAqKSBhcmc7CiAJU3RyaW5n
SW5mb0RhdGEgYnVmOworCWludCBibG9ja19pZDsKKwlSZWxGaWxlTm9kZSBybm9kZTsKKwlG
b3JrTnVtYmVyIGZvcmtudW07CisJQmxvY2tOdW1iZXIgYmxrbnVtOworCVRpbWVzdGFtcFR6
IHJ0aW1lOworCWJvb2wgZnJvbVN0cmVhbTsKKwljaGFyICpyZWNlaXB0X3RpbWVfc3RyOwog
CiAJaW5pdFN0cmluZ0luZm8oJmJ1Zik7CiAJeGxvZ19vdXRkZXNjKCZidWYsIHJlY29yZCk7
Ci0KKwkvKgorCSAqIEVuc3VyZSB3ZSBhcmUgaW4gdGhlIHN0YXJ0dXAgcHJvY2VzcworCSAq
IGlmIHdlIHdhbnQgdG8gbG9nIHN0YW5kYnkgcmVjb3ZlcnkgY29uZmxpY3RzCisJICovCisJ
aWYgKEFtU3RhcnR1cFByb2Nlc3MoKSAmJiBsb2dfcmVjb3ZlcnlfY29uZmxpY3RzKQorCXsK
KwkJR2V0WExvZ1JlY2VpcHRUaW1lKCZydGltZSwgJmZyb21TdHJlYW0pOworCQlpZiAoZnJv
bVN0cmVhbSkKKwkJeworCQkJcmVjZWlwdF90aW1lX3N0ciA9IHBzdHJkdXAodGltZXN0YW1w
dHpfdG9fc3RyKHJ0aW1lKSk7CisJCQlhcHBlbmRTdHJpbmdJbmZvKCZidWYsIlxuV0FMIHJl
Y29yZCByZWNlaXZlZCBhdCAlcyIsIHJlY2VpcHRfdGltZV9zdHIpOworCQkJZm9yIChibG9j
a19pZCA9IDA7IGJsb2NrX2lkIDw9IHJlY29yZC0+bWF4X2Jsb2NrX2lkOyBibG9ja19pZCsr
KQorCQkJeworCQkJCWlmIChYTG9nUmVjR2V0QmxvY2tUYWcocmVjb3JkLCBibG9ja19pZCwg
JnJub2RlLCAmZm9ya251bSwgJmJsa251bSkpCisJCQkJCWFwcGVuZFN0cmluZ0luZm8oJmJ1
ZiwiXG50YnMgJXUgZGIgJXUgcmVsICV1LCBmb3JrICVzLCBibGtubyAldSIsCisJCQkJCQkJ
CXJub2RlLnNwY05vZGUsIHJub2RlLmRiTm9kZSwgcm5vZGUucmVsTm9kZSwKKwkJCQkJCQkJ
Zm9ya05hbWVzW2ZvcmtudW1dLAorCQkJCQkJCQlibGtudW0pOworCQkJfQorCQl9CisJfQog
CS8qIHRyYW5zbGF0b3I6ICVzIGlzIGEgV0FMIHJlY29yZCBkZXNjcmlwdGlvbiAqLwogCWVy
cmNvbnRleHQoIldBTCByZWRvIGF0ICVYLyVYIGZvciAlcyIsCiAJCQkgICAodWludDMyKSAo
cmVjb3JkLT5SZWFkUmVjUHRyID4+IDMyKSwKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3N0
b3JhZ2UvYnVmZmVyL2J1Zm1nci5jIGIvc3JjL2JhY2tlbmQvc3RvcmFnZS9idWZmZXIvYnVm
bWdyLmMKaW5kZXggMjljOTIwODAwYS4uNTBjNjA3ZTBlOCAxMDA2NDQKLS0tIGEvc3JjL2Jh
Y2tlbmQvc3RvcmFnZS9idWZmZXIvYnVmbWdyLmMKKysrIGIvc3JjL2JhY2tlbmQvc3RvcmFn
ZS9idWZmZXIvYnVmbWdyLmMKQEAgLTM3ODAsNiArMzc4MCw3IEBAIExvY2tCdWZmZXJGb3JD
bGVhbnVwKEJ1ZmZlciBidWZmZXIpCiB7CiAJQnVmZmVyRGVzYyAqYnVmSGRyOwogCWNoYXIJ
ICAgKm5ld19zdGF0dXMgPSBOVUxMOworCWJvb2wgCSAgIGZpcnN0X3Bhc3MgPSB0cnVlOwog
CiAJQXNzZXJ0KEJ1ZmZlcklzVmFsaWQoYnVmZmVyKSk7CiAJQXNzZXJ0KFBpbkNvdW50V2Fp
dEJ1ZiA9PSBOVUxMKTsKQEAgLTM4NTYsNyArMzg1Nyw4IEBAIExvY2tCdWZmZXJGb3JDbGVh
bnVwKEJ1ZmZlciBidWZmZXIpCiAJCQkvKiBQdWJsaXNoIHRoZSBidWZpZCB0aGF0IFN0YXJ0
dXAgcHJvY2VzcyB3YWl0cyBvbiAqLwogCQkJU2V0U3RhcnR1cEJ1ZmZlclBpbldhaXRCdWZJ
ZChidWZmZXIgLSAxKTsKIAkJCS8qIFNldCBhbGFybSBhbmQgdGhlbiB3YWl0IHRvIGJlIHNp
Z25hbGVkIGJ5IFVucGluQnVmZmVyKCkgKi8KLQkJCVJlc29sdmVSZWNvdmVyeUNvbmZsaWN0
V2l0aEJ1ZmZlclBpbigpOworCQkJUmVzb2x2ZVJlY292ZXJ5Q29uZmxpY3RXaXRoQnVmZmVy
UGluKGZpcnN0X3Bhc3MpOworCQkJZmlyc3RfcGFzcyA9IGZhbHNlOwogCQkJLyogUmVzZXQg
dGhlIHB1Ymxpc2hlZCBidWZpZCAqLwogCQkJU2V0U3RhcnR1cEJ1ZmZlclBpbldhaXRCdWZJ
ZCgtMSk7CiAJCX0KZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL3Byb2Nz
aWduYWwuYyBiL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL3Byb2NzaWduYWwuYwppbmRleCBj
ODA5MTk2ZDA2Li5mODMxNzZjZmE3IDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9zdG9yYWdl
L2lwYy9wcm9jc2lnbmFsLmMKKysrIGIvc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvcHJvY3Np
Z25hbC5jCkBAIC01NzYsMyArNTc2LDM5IEBAIHByb2NzaWduYWxfc2lndXNyMV9oYW5kbGVy
KFNJR05BTF9BUkdTKQogCiAJZXJybm8gPSBzYXZlX2Vycm5vOwogfQorCitleHRlcm4gY29u
c3QgY2hhciAqZ2V0X3Byb2NzaWduYWxfcmVhc29uX2Rlc2MoUHJvY1NpZ25hbFJlYXNvbiBy
ZWFzb24pCisJeworCQljb25zdCBjaGFyICpyZWFzb25EZXNjID0gInVua25vd24gcmVhc29u
IjsKKworCQlzd2l0Y2ggKHJlYXNvbikKKwkJeworCQkJY2FzZSBQUk9DU0lHX1JFQ09WRVJZ
X0NPTkZMSUNUX0JVRkZFUlBJTjoKKwkJCQlyZWFzb25EZXNjID0gImJ1ZmZlciBwaW4iOwor
CQkJCWJyZWFrOworCQkJY2FzZSBQUk9DU0lHX1JFQ09WRVJZX0NPTkZMSUNUX0xPQ0s6CisJ
CQkJcmVhc29uRGVzYyA9ICJsb2NrIjsKKwkJCQlicmVhazsKKwkJCWNhc2UgUFJPQ1NJR19S
RUNPVkVSWV9DT05GTElDVF9UQUJMRVNQQUNFOgorCQkJCXJlYXNvbkRlc2MgPSAidGFibGVz
cGFjZSI7CisJCQkJYnJlYWs7CisJCQljYXNlIFBST0NTSUdfUkVDT1ZFUllfQ09ORkxJQ1Rf
U05BUFNIT1Q6CisJCQkJcmVhc29uRGVzYyA9ICJzbmFwc2hvdCI7CisJCQkJYnJlYWs7CisJ
CQljYXNlIFBST0NTSUdfUkVDT1ZFUllfQ09ORkxJQ1RfU1RBUlRVUF9ERUFETE9DSzoKKwkJ
CQlyZWFzb25EZXNjID0gImRlYWRsb2NrIjsKKwkJCQlicmVhazsKKwkJCWNhc2UgUFJPQ1NJ
R19SRUNPVkVSWV9DT05GTElDVF9EQVRBQkFTRToKKwkJCQlyZWFzb25EZXNjID0gImRhdGFi
YXNlIjsKKwkJCQlicmVhazsKKwkJCWNhc2UgUFJPQ1NJR19DQVRDSFVQX0lOVEVSUlVQVDoK
KwkJCWNhc2UgUFJPQ1NJR19OT1RJRllfSU5URVJSVVBUOgorCQkJY2FzZSBQUk9DU0lHX1BB
UkFMTEVMX01FU1NBR0U6CisJCQljYXNlIFBST0NTSUdfV0FMU05EX0lOSVRfU1RPUFBJTkc6
CisJCQljYXNlIE5VTV9QUk9DU0lHTkFMUzoKKwkJCQlicmVhazsKKwkJCWRlZmF1bHQ6CisJ
CQkJYnJlYWs7CisJCX0KKwlyZXR1cm4gcmVhc29uRGVzYzsKK30KZGlmZiAtLWdpdCBhL3Ny
Yy9iYWNrZW5kL3N0b3JhZ2UvaXBjL3N0YW5kYnkuYyBiL3NyYy9iYWNrZW5kL3N0b3JhZ2Uv
aXBjL3N0YW5kYnkuYwppbmRleCA5ZTBkNWVjMjU3Li4zMTUyNDAyNjYwIDEwMDY0NAotLS0g
YS9zcmMvYmFja2VuZC9zdG9yYWdlL2lwYy9zdGFuZGJ5LmMKKysrIGIvc3JjL2JhY2tlbmQv
c3RvcmFnZS9pcGMvc3RhbmRieS5jCkBAIC0zOSw2ICszOSw5IEBACiBpbnQJCQl2YWN1dW1f
ZGVmZXJfY2xlYW51cF9hZ2U7CiBpbnQJCQltYXhfc3RhbmRieV9hcmNoaXZlX2RlbGF5ID0g
MzAgKiAxMDAwOwogaW50CQkJbWF4X3N0YW5kYnlfc3RyZWFtaW5nX2RlbGF5ID0gMzAgKiAx
MDAwOworYm9vbAkJCWxvZ19yZWNvdmVyeV9jb25mbGljdHMgPSBmYWxzZTsKKworc3RhdGlj
IFRpbWVzdGFtcFR6IAlyZWNvdmVyeV9jb25mbGljdHNfbG9nX3RpbWU7CiAKIHN0YXRpYyBI
VEFCICpSZWNvdmVyeUxvY2tMaXN0czsKIApAQCAtMjE1LDYgKzIxOCwyNyBAQCBXYWl0RXhj
ZWVkc01heFN0YW5kYnlEZWxheSh1aW50MzIgd2FpdF9ldmVudF9pbmZvKQogCXJldHVybiBm
YWxzZTsKIH0KIAorLyoKKyAqIERpc3BsYXkgaW5mb3JtYXRpb24gYWJvdXQgdGhlIHdhbCBy
ZWNvcmQKKyAqIGFwcGx5IGJlaW5nIGJsb2NrZWQKKyAqLworc3RhdGljIHZvaWQKK0xvZ0Js
b2NrZWRXYWxSZWNvcmRJbmZvKGludCBudW1fd2FpdGxpc3RfZW50cmllcywgUHJvY1NpZ25h
bFJlYXNvbiByZWFzb24pCit7CisJaWYgKG51bV93YWl0bGlzdF9lbnRyaWVzID4gMCkKKwkJ
aWYgKHJlYXNvbiA9PSBQUk9DU0lHX1JFQ09WRVJZX0NPTkZMSUNUX0RBVEFCQVNFKQorCQkJ
ZXJlcG9ydChMT0csCisJCQkJKGVycm1zZygicmVjb3ZlcnkgaXMgZXhwZXJpZW5jaW5nIHJl
Y292ZXJ5IGNvbmZsaWN0IG9uICVzIiwgZ2V0X3Byb2NzaWduYWxfcmVhc29uX2Rlc2MocmVh
c29uKSksCisJCQkJZXJyZGV0YWlsKCJUaGVyZSBpcyAlZCBjb25mbGljdGluZyBjb25uZWN0
aW9uKHMpLiIsIG51bV93YWl0bGlzdF9lbnRyaWVzKSkpOworCQllbHNlCisJCQllcmVwb3J0
KExPRywKKwkJCQkoZXJybXNnKCJyZWNvdmVyeSBpcyB3YWl0aW5nIGZvciByZWNvdmVyeSBj
b25mbGljdCBvbiAlcyIsIGdldF9wcm9jc2lnbmFsX3JlYXNvbl9kZXNjKHJlYXNvbikpLAor
CQkJCWVycmRldGFpbCgiVGhlcmUgaXMgJWQgYmxvY2tpbmcgY29ubmVjdGlvbihzKS4iLCBu
dW1fd2FpdGxpc3RfZW50cmllcykpKTsKKwllbHNlCisJCWVyZXBvcnQoTE9HLAorCQkJKGVy
cm1zZygicmVjb3ZlcnkgaXMgd2FpdGluZyBmb3IgcmVjb3ZlcnkgY29uZmxpY3Qgb24gJXMi
LCBnZXRfcHJvY3NpZ25hbF9yZWFzb25fZGVzYyhyZWFzb24pKSkpOworfQorCiAvKgogICog
VGhpcyBpcyB0aGUgbWFpbiBleGVjdXRpb25lciBmb3IgYW55IHF1ZXJ5IGJhY2tlbmQgdGhh
dCBjb25mbGljdHMgd2l0aAogICogcmVjb3ZlcnkgcHJvY2Vzc2luZy4gSnVkZ2VtZW50IGhh
cyBhbHJlYWR5IGJlZW4gcGFzc2VkIG9uIGl0IHdpdGhpbgpAQCAtMjMyLDYgKzI1Niw4IEBA
IFJlc29sdmVSZWNvdmVyeUNvbmZsaWN0V2l0aFZpcnR1YWxYSURzKFZpcnR1YWxUcmFuc2Fj
dGlvbklkICp3YWl0bGlzdCwKIHsKIAlUaW1lc3RhbXBUeiB3YWl0U3RhcnQgPSAwOwogCWNo
YXIJICAgKm5ld19zdGF0dXM7CisJVmlydHVhbFRyYW5zYWN0aW9uSWQgKnRtcFdhaXRsaXN0
OworCWludCBudW1fd2FpdGxpc3RfZW50cmllczsKIAogCS8qIEZhc3QgZXhpdCwgdG8gYXZv
aWQgYSBrZXJuZWwgY2FsbCBpZiB0aGVyZSdzIG5vIHdvcmsgdG8gYmUgZG9uZS4gKi8KIAlp
ZiAoIVZpcnR1YWxUcmFuc2FjdGlvbklkSXNWYWxpZCgqd2FpdGxpc3QpKQpAQCAtMjQxLDYg
KzI2NywyMSBAQCBSZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdpdGhWaXJ0dWFsWElEcyhWaXJ0
dWFsVHJhbnNhY3Rpb25JZCAqd2FpdGxpc3QsCiAJCXdhaXRTdGFydCA9IEdldEN1cnJlbnRU
aW1lc3RhbXAoKTsKIAluZXdfc3RhdHVzID0gTlVMTDsJCQkvKiB3ZSBoYXZlbid0IGNoYW5n
ZWQgdGhlIHBzIGRpc3BsYXkgKi8KIAorCXRtcFdhaXRsaXN0ID0gd2FpdGxpc3Q7CisJd2hp
bGUgKFZpcnR1YWxUcmFuc2FjdGlvbklkSXNWYWxpZCgqdG1wV2FpdGxpc3QpKQorCXsKKwkJ
dG1wV2FpdGxpc3QrKzsKKwl9CisKKwludW1fd2FpdGxpc3RfZW50cmllcyA9ICh0bXBXYWl0
bGlzdCAtIHdhaXRsaXN0KTsKKworCS8qIGRpc3BsYXkgd2FsIHJlY29yZCBpbmZvcm1hdGlv
biAqLworCWlmIChsb2dfcmVjb3ZlcnlfY29uZmxpY3RzICYmIChUaW1lc3RhbXBEaWZmZXJl
bmNlRXhjZWVkcyhyZWNvdmVyeV9jb25mbGljdHNfbG9nX3RpbWUsIEdldEN1cnJlbnRUaW1l
c3RhbXAoKSwKKwkJCQkJCQkJCURlYWRsb2NrVGltZW91dCkpKSB7CisJCUxvZ0Jsb2NrZWRX
YWxSZWNvcmRJbmZvKG51bV93YWl0bGlzdF9lbnRyaWVzLCByZWFzb24pOworCQlyZWNvdmVy
eV9jb25mbGljdHNfbG9nX3RpbWUgPSBHZXRDdXJyZW50VGltZXN0YW1wKCk7CisJfQorCiAJ
d2hpbGUgKFZpcnR1YWxUcmFuc2FjdGlvbklkSXNWYWxpZCgqd2FpdGxpc3QpKQogCXsKIAkJ
LyogcmVzZXQgc3RhbmRieVdhaXRfdXMgZm9yIGVhY2ggeGFjdCB3ZSB3YWl0IGZvciAqLwpA
QCAtMzcwLDYgKzQxMSwxMSBAQCBSZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdpdGhEYXRhYmFz
ZShPaWQgZGJpZCkKIAkgKiBibG9jayBkdXJpbmcgSW5pdFBvc3RncmVzKCkgYW5kIHRoZW4g
ZGlzY29ubmVjdCB3aGVuIHRoZXkgc2VlIHRoZQogCSAqIGRhdGFiYXNlIGhhcyBiZWVuIHJl
bW92ZWQuCiAJICovCisKKwkvKiBkaXNwbGF5IHdhbCByZWNvcmQgaW5mb3JtYXRpb24gKi8K
KwlpZiAobG9nX3JlY292ZXJ5X2NvbmZsaWN0cykKKwkJTG9nQmxvY2tlZFdhbFJlY29yZElu
Zm8oQ291bnREQkJhY2tlbmRzKGRiaWQpLCBQUk9DU0lHX1JFQ09WRVJZX0NPTkZMSUNUX0RB
VEFCQVNFKTsKKwogCXdoaWxlIChDb3VudERCQmFja2VuZHMoZGJpZCkgPiAwKQogCXsKIAkJ
Q2FuY2VsREJCYWNrZW5kcyhkYmlkLCBQUk9DU0lHX1JFQ09WRVJZX0NPTkZMSUNUX0RBVEFC
QVNFLCB0cnVlKTsKQEAgLTQ4MCw3ICs1MjYsNyBAQCBSZXNvbHZlUmVjb3ZlcnlDb25mbGlj
dFdpdGhMb2NrKExPQ0tUQUcgbG9ja3RhZykKICAqIGF0IGxlYXN0IGRlYWRsb2NrX3RpbWVv
dXQuCiAgKi8KIHZvaWQKLVJlc29sdmVSZWNvdmVyeUNvbmZsaWN0V2l0aEJ1ZmZlclBpbih2
b2lkKQorUmVzb2x2ZVJlY292ZXJ5Q29uZmxpY3RXaXRoQnVmZmVyUGluKGJvb2wgZmlyc3Rf
cGFzcykKIHsKIAlUaW1lc3RhbXBUeiBsdGltZTsKIApAQCAtNDg4LDYgKzUzNCwxNCBAQCBS
ZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdpdGhCdWZmZXJQaW4odm9pZCkKIAogCWx0aW1lID0g
R2V0U3RhbmRieUxpbWl0VGltZSgpOwogCisJLyogZGlzcGxheSB3YWwgcmVjb3JkIGluZm9y
bWF0aW9uICovCisJaWYgKGxvZ19yZWNvdmVyeV9jb25mbGljdHMgJiYgZmlyc3RfcGFzcwor
CQkJCQkmJiAoVGltZXN0YW1wRGlmZmVyZW5jZUV4Y2VlZHMocmVjb3ZlcnlfY29uZmxpY3Rz
X2xvZ190aW1lLCBHZXRDdXJyZW50VGltZXN0YW1wKCksCisJCQkJCQlEZWFkbG9ja1RpbWVv
dXQpKSkgeworCQlMb2dCbG9ja2VkV2FsUmVjb3JkSW5mbygtMSwgUFJPQ1NJR19SRUNPVkVS
WV9DT05GTElDVF9CVUZGRVJQSU4pOworCQlyZWNvdmVyeV9jb25mbGljdHNfbG9nX3RpbWUg
PSBHZXRDdXJyZW50VGltZXN0YW1wKCk7OworCX0KKwogCWlmIChsdGltZSA9PSAwKQogCXsK
IAkJLyoKQEAgLTUyMiw3ICs1NzYsNiBAQCBSZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdpdGhC
dWZmZXJQaW4odm9pZCkKIAogCS8qIFdhaXQgdG8gYmUgc2lnbmFsZWQgYnkgVW5waW5CdWZm
ZXIoKSAqLwogCVByb2NXYWl0Rm9yU2lnbmFsKFBHX1dBSVRfQlVGRkVSX1BJTik7Ci0KIAkv
KgogCSAqIENsZWFyIGFueSB0aW1lb3V0IHJlcXVlc3RzIGVzdGFibGlzaGVkIGFib3ZlLiAg
V2UgYXNzdW1lIGhlcmUgdGhhdCB0aGUKIAkgKiBTdGFydHVwIHByb2Nlc3MgZG9lc24ndCBo
YXZlIGFueSBvdGhlciB0aW1lb3V0cyB0aGFuIHdoYXQgdGhpcyBmdW5jdGlvbgpkaWZmIC0t
Z2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9ndWMuYyBiL3NyYy9iYWNrZW5kL3V0aWxz
L21pc2MvZ3VjLmMKaW5kZXggMmYzZTBhNzBlMC4uYWEzYzNjMjMyNSAxMDA2NDQKLS0tIGEv
c3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9ndWMuYworKysgYi9zcmMvYmFja2VuZC91dGlscy9t
aXNjL2d1Yy5jCkBAIC0xNTk0LDcgKzE1OTQsMTUgQEAgc3RhdGljIHN0cnVjdCBjb25maWdf
Ym9vbCBDb25maWd1cmVOYW1lc0Jvb2xbXSA9CiAJCWZhbHNlLAogCQlOVUxMLCBOVUxMLCBO
VUxMCiAJfSwKLQorCXsKKwkJeyJsb2dfcmVjb3ZlcnlfY29uZmxpY3RzIiwgUEdDX1NJR0hV
UCwgTE9HR0lOR19XSEFULAorCQkJZ2V0dGV4dF9ub29wKCJMb2dzIHN0YW5kYnkgcmVjb3Zl
cnkgY29uZmxpY3RzLiIpLAorCQkJTlVMTAorCQl9LAorCQkmbG9nX3JlY292ZXJ5X2NvbmZs
aWN0cywKKwkJZmFsc2UsCisJCU5VTEwsIE5VTEwsIE5VTEwKKwl9LAogCXsKIAkJeyJsb2df
aG9zdG5hbWUiLCBQR0NfU0lHSFVQLCBMT0dHSU5HX1dIQVQsCiAJCQlnZXR0ZXh0X25vb3Ao
IkxvZ3MgdGhlIGhvc3QgbmFtZSBpbiB0aGUgY29ubmVjdGlvbiBsb2dzLiIpLApkaWZmIC0t
Z2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9wb3N0Z3Jlc3FsLmNvbmYuc2FtcGxlIGIv
c3JjL2JhY2tlbmQvdXRpbHMvbWlzYy9wb3N0Z3Jlc3FsLmNvbmYuc2FtcGxlCmluZGV4IGFj
MDJiZDBjMDAuLjQwZGE2OWMwN2MgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3V0aWxzL21p
c2MvcG9zdGdyZXNxbC5jb25mLnNhbXBsZQorKysgYi9zcmMvYmFja2VuZC91dGlscy9taXNj
L3Bvc3RncmVzcWwuY29uZi5zYW1wbGUKQEAgLTU0OCw2ICs1NDgsNyBAQAogCQkJCQkjICAg
JSUgPSAnJScKIAkJCQkJIyBlLmcuICc8JXUlJSVkPiAnCiAjbG9nX2xvY2tfd2FpdHMgPSBv
ZmYJCQkjIGxvZyBsb2NrIHdhaXRzID49IGRlYWRsb2NrX3RpbWVvdXQKKyNsb2dfcmVjb3Zl
cnlfY29uZmxpY3RzID0gb2ZmCQkjIGxvZyBzdGFuZGJ5IHJlY292ZXJ5IGNvbmZsaWN0cwog
I2xvZ19wYXJhbWV0ZXJfbWF4X2xlbmd0aCA9IC0xCQkjIHdoZW4gbG9nZ2luZyBzdGF0ZW1l
bnRzLCBsaW1pdCBsb2dnZWQKIAkJCQkJIyBiaW5kLXBhcmFtZXRlciB2YWx1ZXMgdG8gTiBi
eXRlczsKIAkJCQkJIyAtMSBtZWFucyBwcmludCBpbiBmdWxsLCAwIGRpc2FibGVzCmRpZmYg
LS1naXQgYS9zcmMvaW5jbHVkZS9zdG9yYWdlL3Byb2NzaWduYWwuaCBiL3NyYy9pbmNsdWRl
L3N0b3JhZ2UvcHJvY3NpZ25hbC5oCmluZGV4IGEwYzBiYzNjZTUuLmM1NzQxM2IwNmQgMTAw
NjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL3N0b3JhZ2UvcHJvY3NpZ25hbC5oCisrKyBiL3NyYy9p
bmNsdWRlL3N0b3JhZ2UvcHJvY3NpZ25hbC5oCkBAIC03MCw1ICs3MCw2IEBAIGV4dGVybiB2
b2lkIFdhaXRGb3JQcm9jU2lnbmFsQmFycmllcih1aW50NjQgZ2VuZXJhdGlvbik7CiBleHRl
cm4gdm9pZCBQcm9jZXNzUHJvY1NpZ25hbEJhcnJpZXIodm9pZCk7CiAKIGV4dGVybiB2b2lk
IHByb2NzaWduYWxfc2lndXNyMV9oYW5kbGVyKFNJR05BTF9BUkdTKTsKK2V4dGVybiBjb25z
dCBjaGFyICpnZXRfcHJvY3NpZ25hbF9yZWFzb25fZGVzYyhQcm9jU2lnbmFsUmVhc29uIHJl
YXNvbik7CiAKICNlbmRpZgkJCQkJCQkvKiBQUk9DU0lHTkFMX0ggKi8KZGlmZiAtLWdpdCBh
L3NyYy9pbmNsdWRlL3N0b3JhZ2Uvc3RhbmRieS5oIGIvc3JjL2luY2x1ZGUvc3RvcmFnZS9z
dGFuZGJ5LmgKaW5kZXggY2ZiZTQyNmU1YS4uY2QyYzczYTJiNSAxMDA2NDQKLS0tIGEvc3Jj
L2luY2x1ZGUvc3RvcmFnZS9zdGFuZGJ5LmgKKysrIGIvc3JjL2luY2x1ZGUvc3RvcmFnZS9z
dGFuZGJ5LmgKQEAgLTIzLDYgKzIzLDcgQEAKIGV4dGVybiBpbnQJdmFjdXVtX2RlZmVyX2Ns
ZWFudXBfYWdlOwogZXh0ZXJuIGludAltYXhfc3RhbmRieV9hcmNoaXZlX2RlbGF5OwogZXh0
ZXJuIGludAltYXhfc3RhbmRieV9zdHJlYW1pbmdfZGVsYXk7CitleHRlcm4gYm9vbAlsb2df
cmVjb3ZlcnlfY29uZmxpY3RzOwogCiBleHRlcm4gdm9pZCBJbml0UmVjb3ZlcnlUcmFuc2Fj
dGlvbkVudmlyb25tZW50KHZvaWQpOwogZXh0ZXJuIHZvaWQgU2h1dGRvd25SZWNvdmVyeVRy
YW5zYWN0aW9uRW52aXJvbm1lbnQodm9pZCk7CkBAIC0zMyw3ICszNCw3IEBAIGV4dGVybiB2
b2lkIFJlc29sdmVSZWNvdmVyeUNvbmZsaWN0V2l0aFRhYmxlc3BhY2UoT2lkIHRzaWQpOwog
ZXh0ZXJuIHZvaWQgUmVzb2x2ZVJlY292ZXJ5Q29uZmxpY3RXaXRoRGF0YWJhc2UoT2lkIGRi
aWQpOwogCiBleHRlcm4gdm9pZCBSZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdpdGhMb2NrKExP
Q0tUQUcgbG9ja3RhZyk7Ci1leHRlcm4gdm9pZCBSZXNvbHZlUmVjb3ZlcnlDb25mbGljdFdp
dGhCdWZmZXJQaW4odm9pZCk7CitleHRlcm4gdm9pZCBSZXNvbHZlUmVjb3ZlcnlDb25mbGlj
dFdpdGhCdWZmZXJQaW4oYm9vbCBmaXJzdF9wYXNzKTsKIGV4dGVybiB2b2lkIENoZWNrUmVj
b3ZlcnlDb25mbGljdERlYWRsb2NrKHZvaWQpOwogZXh0ZXJuIHZvaWQgU3RhbmRieURlYWRM
b2NrSGFuZGxlcih2b2lkKTsKIGV4dGVybiB2b2lkIFN0YW5kYnlUaW1lb3V0SGFuZGxlcih2
b2lkKTsK
--------------C16C6C96B3132FD69C03B2BF--