Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pJxzu-0007HZ-D9 for pgsql-hackers@arkaria.postgresql.org; Mon, 23 Jan 2023 14:40:34 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pJxzs-0004Cd-AT for pgsql-hackers@arkaria.postgresql.org; Mon, 23 Jan 2023 14:40:32 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pJxzr-0004CT-T2 for pgsql-hackers@lists.postgresql.org; Mon, 23 Jan 2023 14:40:32 +0000 Received: from mail-pj1-x1032.google.com ([2607:f8b0:4864:20::1032]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1pJxzp-0001Yn-LN for pgsql-hackers@lists.postgresql.org; Mon, 23 Jan 2023 14:40:31 +0000 Received: by mail-pj1-x1032.google.com with SMTP id z1-20020a17090a66c100b00226f05b9595so11217148pjl.0 for ; Mon, 23 Jan 2023 06:40:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=jJc479OOI3FhThlbXBbjPSx/z0BUMYRsFug/2kchlS0=; b=HWvsZYDujLXiSEhuEAEs/YYn2ySeyfkNeE1vTlb86620tvL5fbartY/3diJ0ZVgJn1 Hqscb0RnFPVTrpAo1MUkoXHlPaxQY2e9NfEpkCDGivjFZonr5/ijaFncS6cICCW5DbKa bmvlzDSrwev4Zl3j978rUDbZWp+u4fuI/D7bxyu5qxxYVgx8PM1ytJNdE8xsG3W6iGqv jaRUMuUamhDgueel5MkENIU1kp+Ygqetp5ptgy9J4zeVHzPzLeiQFJy/RXThdWSNeDV0 8TWx47TubAIUpRFDMJGNEgvuNDD1yFOUSa4lOBpSOYQdCd/WbIfFk3WN0y6MeH1wlGud rKVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=jJc479OOI3FhThlbXBbjPSx/z0BUMYRsFug/2kchlS0=; b=JTZTaoBL8uP2AoKyOhERb2P5oJAYU+4tGQeh9y6GhkfuO/v/QOJKrMWysc8bmfaI0X P9dfyhmOkQMwlAR4qzFjHgezpEMf5WUpS3KOaZfe8e844hGJvdq1f+51eeebxTmebgD5 hPJBGEwvLhhwKp3znZ/uQt7ianGS1993sujwPsLPQP3L8JeYfsQ/j9haAt0t0CV1HjCk g+S3SALfoaP0L3XHZ6WeBFimhDVDKgHUfl0oEzpJjTpGu0RYWVhuii4+1vt3QmixOh+0 2RH1eKEZ3gn+k8r7FyFQiYE2FUifxtVm4fcH3hHAk5WDLsUEO7oS7vWRuUXCCvf5LCKQ 7YAA== X-Gm-Message-State: AFqh2krXYv+kG0az30bHi+X/3gwU6cYu4oGzxd/7x7iIh5/IeipDG3RO YrIFMxFjzeT+JGKqM9RO5fX4hRDsJs5LafTKUAcubrK3sNw= X-Google-Smtp-Source: AMrXdXvId3ZKqTaqAvCUzrYhrG+ak4sPKbh69SQWV5E1CdhLG4UqwTwVyNfp6Ibj2b7jVzHybNCiyjSggFjZfo+PlAU= X-Received: by 2002:a17:90a:31c2:b0:22b:b19e:9feb with SMTP id j2-20020a17090a31c200b0022bb19e9febmr1326080pjf.5.1674484827147; Mon, 23 Jan 2023 06:40:27 -0800 (PST) MIME-Version: 1.0 From: Maxim Orlov Date: Mon, 23 Jan 2023 17:40:15 +0300 Message-ID: Subject: old_snapshot_threshold bottleneck on replica To: Postgres hackers Content-Type: multipart/mixed; boundary="0000000000004ded9705f2ef6142" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000004ded9705f2ef6142 Content-Type: multipart/alternative; boundary="0000000000004ded9505f2ef6140" --0000000000004ded9505f2ef6140 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi! One of our customers stumble onto a significant performance degradation while running multiple OLAP-like queries on a replica. After some investigation, it became clear that the problem is in accessing old_snapshot_threshold parameter. Accessing old_snapshot_threshold parameter is guarded by mutex_threshold. This is not a problem on primary server, since we rarely call GetOldSnapshotThresholdTimestamp: 5028 void 5029 TestForOldSnapshot_impl(Snapshot snapshot, Relation relation) 5030 { 5031 =C2=B7=C2=B7=C2=B7=C2=B7if (RelationAllowsEarlyPruning(relation) 5032 =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7&& (snapshot)->whenTak= en < GetOldSnapshotThresholdTimestamp()) 5033 =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7ereport(ERROR, 5034 =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2= =B7=C2=B7=C2=B7=C2=B7=C2=B7(errcode(ERRCODE_SNAPSHOT_TOO_OLD), 5035 =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2= =B7=C2=B7=C2=B7=C2=B7=C2=B7 errmsg("snapshot too old"))); But in case of a replica, we have to call GetOldSnapshotThresholdTimestamp much often. So, this become a bottleneck. The customer solve this issue by setting old_snapshot_threshold to 0. But, I think, we can do something about it. Some more investigation: -- On primary -- $ ./bin/psql postgres -c "create database benchmark" CREATE DATABASE $ ./bin/pgbench -i -Uorlov -s300 benchmark dropping old tables... NOTICE: table "pgbench_accounts" does not exist, skipping ... creating tables... generating data (client-side)... 30000000 of 30000000 tuples (100%) done (elapsed 142.37 s, remaining 0.00 s= ) vacuuming... creating primary keys... done in 177.67 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 144.45 s, vacuum 0.59 s, primary keys 32.61 s). -- On secondary -- $ touch 1.sql $ vim 1.sql $ cat 1.sql \set bid random(1, 300) BEGIN; SELECT sum(aid) FROM pgbench_accounts where bid =3D :bid GROUP BY bid; END; $ ./bin/pgbench -f 1.sql -p5433 -Uorlov -j10 -c100 -T720 -P1 -n benchmark pgbench (16devel) progress: 1.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed ... progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed $ perf record -F 99 -a -g --call-graph=3Ddwarf sleep 5 $ perf script --header --fields comm,pid,tid,time,event,ip,sym,dso > file $ grep s_lock file | wc -l 3486 My proposal is to use atomic for threshold_timestamp and threshold_xid. PFA 0001 patch. With patch 0001 we got: $ grep s_lock file2 | wc -l 8 Maybe, we shall go farther and remove mutex_threshold here? This will lead to inconsistency of threshold_timestamp and threshold_xid, but is this really a problem? Thoughts? --=20 Best regards, Maxim Orlov. --0000000000004ded9505f2ef6140 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi!

One of our customers stumble onto a sig= nificant performance degradation while running multiple OLAP-like queries o= n a replica.
After s= ome investigation, it became clear that = the problem is in accessing old_snapshot_threshold parameter.=

Accessing = old_snapshot_threshold parameter is guarded by mutex_threshold. This is not= a problem on primary
server, since we= rarely call GetOldSnapshotThresholdTimestamp:<= /div>

5028 void
5029 TestForO= ldSnapshot_impl(Snapshot snapshot, Relation relation)
5030 {
5031 =C2= =B7=C2=B7=C2=B7=C2=B7if (RelationAllowsEarlyPruning(relation)
5032 =C2= =B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7&& (snapshot)->when= Taken < GetOldSnapshotThresholdTimestamp())
5033 =C2=B7=C2=B7=C2=B7= =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7ereport(ERROR,
5034 =C2=B7=C2=B7=C2=B7=C2= =B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7= (errcode(ERRCODE_SNAPSHOT_TOO_OLD),
5035 =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7= =C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7=C2=B7 errmsg(&= quot;snapshot too old")));

<= span class=3D"gmail-HwtZe" lang=3D"en">But in case of a replica, we have to call= G= etOldSnapshotThresholdTimestamp much often. So, this become a
bottleneck. The custo= mer solve this issue by setting old_snapshot_threshold to 0. But, I think, = we can
<= span class=3D"gmail-jCAhz gmail-ChMk0b">do som= ething about it.

<= div>Some more investigation:

-- On primary --
$ ./bin/psql postgres -c "create database benchmark&= quot;
CREATE DATABASE
$ ./bin/pgbench -i -Uorlov -s300 benchmark
d= ropping old tables...
NOTICE: =C2=A0table "pgbench_accounts" d= oes not exist, skipping
...
creating tables...
generating data (cl= ient-side)...
30000000 of 30000000 tuples (100%) done (elapsed 142.37 s,= remaining 0.00 s)
vacuuming...
creating primary keys...
done in 1= 77.67 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 144= .45 s, vacuum 0.59 s, primary keys 32.61 s).
<= span class=3D"gmail-HwtZe" lang=3D"en">
-- On secondary --
$ touch 1.sql
$ vim 1.sql
$ cat 1.sql
\set bid random(1= , 300)
BEGIN;
SELECT sum(aid) FROM pgbench_accounts where bid =3D :bi= d GROUP BY bid;
END;
$ ./bin/pgbench -f 1.sql -p5433 -Uorlov -j10 -c100 -T720 -P1 -n = benchmark
pgbench (16devel)
progress: 1.0 s, 0.0 tps, lat 0.000 ms st= ddev 0.000, 0 failed
...
progress: 20.0 s, 0.0 tps, lat 0.000 ms stdd= ev 0.000, 0 failed<= /div>

= $ = perf record -F 99 -a -g --call-graph=3Ddwarf sleep 5
$ perf script --header --fields com= m,pid,tid,time,event,ip,sym,dso > file
= $ grep s_lock file | wc -l
3486
<= span style=3D"font-family:monospace">
My proposal= is to use atomic for threshold_timestamp and threshold_xid. PFA 0001 patch= .
With patch = 0001 we got:
<= div>
$ grep s_lock file2 | wc -l=20
8

Maybe, we sha= ll go farther and remove mutex_threshold here? This will lead to inconsiste= ncy of
<= span class=3D"gmail-jCAhz gmail-ChMk0b">thresh= old_timestamp and threshold_xid, but is this really a problem?

Thoughts?

--
Best regards,
Maxim Orlov.
--0000000000004ded9505f2ef6140-- --0000000000004ded9705f2ef6142 Content-Type: application/octet-stream; name="0001-PGPRO-7624-use-atomic-old_snapshot_threshold.patch" Content-Disposition: attachment; filename="0001-PGPRO-7624-use-atomic-old_snapshot_threshold.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_ld8wo01o0 RnJvbSA5OGZhNzA3NTRhOGM2OGNkZmIyMGYyN2JkN2Y2ZTZlM2U1ZjhlZDkyIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXhpbSBPcmxvdiA8b3Jsb3ZtZ0BnbWFpbC5jb20+CkRhdGU6 IFdlZCwgMTEgSmFuIDIwMjMgMTg6MDc6MTAgKzAzMDAKU3ViamVjdDogW1BBVENIXSBbUEdQUk8t NzYyNF0gdXNlIGF0b21pYyBvbGRfc25hcHNob3RfdGhyZXNob2xkCgpVc2luZyBzcGlubG9jayB0 byBhY2Nlc3Mgb2xkX3NuYXBzaG90X3RocmVzaG9sZCBsZWFkIHRvIHRoZSBib3R0bGVuZWNrIG9u CnJlcGxpY2EsIHNpbmNlIEdldE9sZFNuYXBzaG90VGhyZXNob2xkVGltZXN0YW1wIGlzIGNhbGxl ZCB0b28gb2Z0ZW4uIFNvLCBzd2l0Y2gKdG8gYW4gYXRvbWljIHZhbHVlcy4KCnRhZ3M6IGNvbW1p dGZlc3RfaG90Zml4Ci0tLQogc3JjL2JhY2tlbmQvdXRpbHMvdGltZS9zbmFwbWdyLmMgfCAzMSAr KysrKysrKysrKysrKysrKy0tLS0tLS0tLS0tLS0tCiBzcmMvaW5jbHVkZS91dGlscy9vbGRfc25h cHNob3QuaCB8ICA1ICsrKy0tCiAyIGZpbGVzIGNoYW5nZWQsIDIwIGluc2VydGlvbnMoKyksIDE2 IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3V0aWxzL3RpbWUvc25hcG1n ci5jIGIvc3JjL2JhY2tlbmQvdXRpbHMvdGltZS9zbmFwbWdyLmMKaW5kZXggN2QxMWFlMzQ3ODEu LjA5MDJlMjlmMjI0IDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC91dGlscy90aW1lL3NuYXBtZ3Iu YworKysgYi9zcmMvYmFja2VuZC91dGlscy90aW1lL3NuYXBtZ3IuYwpAQCAtMjMwLDggKzIzMCw4 IEBAIFNuYXBNZ3JJbml0KHZvaWQpCiAJCW9sZFNuYXBzaG90Q29udHJvbC0+bGF0ZXN0X3htaW4g PSBJbnZhbGlkVHJhbnNhY3Rpb25JZDsKIAkJb2xkU25hcHNob3RDb250cm9sLT5uZXh0X21hcF91 cGRhdGUgPSAwOwogCQlTcGluTG9ja0luaXQoJm9sZFNuYXBzaG90Q29udHJvbC0+bXV0ZXhfdGhy ZXNob2xkKTsKLQkJb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRfdGltZXN0YW1wID0gMDsK LQkJb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRfeGlkID0gSW52YWxpZFRyYW5zYWN0aW9u SWQ7CisJCXBnX2F0b21pY19pbml0X3U2NCgmb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRf dGltZXN0YW1wLCAwKTsKKwkJcGdfYXRvbWljX2luaXRfdTMyKCZvbGRTbmFwc2hvdENvbnRyb2wt PnRocmVzaG9sZF94aWQsIEludmFsaWRUcmFuc2FjdGlvbklkKTsKIAkJb2xkU25hcHNob3RDb250 cm9sLT5oZWFkX29mZnNldCA9IDA7CiAJCW9sZFNuYXBzaG90Q29udHJvbC0+aGVhZF90aW1lc3Rh bXAgPSAwOwogCQlvbGRTbmFwc2hvdENvbnRyb2wtPmNvdW50X3VzZWQgPSAwOwpAQCAtMTcwNiw5 ICsxNzA2LDcgQEAgR2V0T2xkU25hcHNob3RUaHJlc2hvbGRUaW1lc3RhbXAodm9pZCkKIHsKIAlU aW1lc3RhbXBUeiB0aHJlc2hvbGRfdGltZXN0YW1wOwogCi0JU3BpbkxvY2tBY3F1aXJlKCZvbGRT bmFwc2hvdENvbnRyb2wtPm11dGV4X3RocmVzaG9sZCk7Ci0JdGhyZXNob2xkX3RpbWVzdGFtcCA9 IG9sZFNuYXBzaG90Q29udHJvbC0+dGhyZXNob2xkX3RpbWVzdGFtcDsKLQlTcGluTG9ja1JlbGVh c2UoJm9sZFNuYXBzaG90Q29udHJvbC0+bXV0ZXhfdGhyZXNob2xkKTsKKwl0aHJlc2hvbGRfdGlt ZXN0YW1wID0gcGdfYXRvbWljX3JlYWRfdTY0KCZvbGRTbmFwc2hvdENvbnRyb2wtPnRocmVzaG9s ZF90aW1lc3RhbXApOwogCiAJcmV0dXJuIHRocmVzaG9sZF90aW1lc3RhbXA7CiB9CkBAIC0xNzE2 LDExICsxNzE0LDE4IEBAIEdldE9sZFNuYXBzaG90VGhyZXNob2xkVGltZXN0YW1wKHZvaWQpCiB2 b2lkCiBTZXRPbGRTbmFwc2hvdFRocmVzaG9sZFRpbWVzdGFtcChUaW1lc3RhbXBUeiB0cywgVHJh bnNhY3Rpb25JZCB4bGltaXQpCiB7CisJVGltZXN0YW1wVHogdGhyZXNob2xkX3RpbWVzdGFtcDsK KwlUcmFuc2FjdGlvbklkIHRocmVzaG9sZF94aWQ7CisKIAlTcGluTG9ja0FjcXVpcmUoJm9sZFNu YXBzaG90Q29udHJvbC0+bXV0ZXhfdGhyZXNob2xkKTsKLQlBc3NlcnQob2xkU25hcHNob3RDb250 cm9sLT50aHJlc2hvbGRfdGltZXN0YW1wIDw9IHRzKTsKLQlBc3NlcnQoVHJhbnNhY3Rpb25JZFBy ZWNlZGVzT3JFcXVhbHMob2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRfeGlkLCB4bGltaXQp KTsKLQlvbGRTbmFwc2hvdENvbnRyb2wtPnRocmVzaG9sZF90aW1lc3RhbXAgPSB0czsKLQlvbGRT bmFwc2hvdENvbnRyb2wtPnRocmVzaG9sZF94aWQgPSB4bGltaXQ7CisJdGhyZXNob2xkX3RpbWVz dGFtcCA9IHBnX2F0b21pY19yZWFkX3U2NCgmb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRf dGltZXN0YW1wKTsKKwl0aHJlc2hvbGRfeGlkID0gcGdfYXRvbWljX3JlYWRfdTMyKCZvbGRTbmFw c2hvdENvbnRyb2wtPnRocmVzaG9sZF94aWQpOworCisJQXNzZXJ0KHRocmVzaG9sZF90aW1lc3Rh bXAgPD0gdHMpOworCUFzc2VydChUcmFuc2FjdGlvbklkUHJlY2VkZXNPckVxdWFscyh0aHJlc2hv bGRfeGlkLCB4bGltaXQpKTsKKworCXBnX2F0b21pY193cml0ZV91NjQoJm9sZFNuYXBzaG90Q29u dHJvbC0+dGhyZXNob2xkX3RpbWVzdGFtcCwgdHMpOworCXBnX2F0b21pY193cml0ZV91MzIoJm9s ZFNuYXBzaG90Q29udHJvbC0+dGhyZXNob2xkX3hpZCwgeGxpbWl0KTsKIAlTcGluTG9ja1JlbGVh c2UoJm9sZFNuYXBzaG90Q29udHJvbC0+bXV0ZXhfdGhyZXNob2xkKTsKIH0KIApAQCAtMTczOSw5 ICsxNzQ0LDcgQEAgU25hcHNob3RUb29PbGRNYWdpY0ZvclRlc3Qodm9pZCkKIAogCXRzIC09IDUg KiBVU0VDU19QRVJfU0VDOwogCi0JU3BpbkxvY2tBY3F1aXJlKCZvbGRTbmFwc2hvdENvbnRyb2wt Pm11dGV4X3RocmVzaG9sZCk7Ci0Jb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRfdGltZXN0 YW1wID0gdHM7Ci0JU3BpbkxvY2tSZWxlYXNlKCZvbGRTbmFwc2hvdENvbnRyb2wtPm11dGV4X3Ro cmVzaG9sZCk7CisJcGdfYXRvbWljX3dyaXRlX3U2NCgmb2xkU25hcHNob3RDb250cm9sLT50aHJl c2hvbGRfdGltZXN0YW1wLCB0cyk7CiB9CiAKIC8qCkBAIC0xODQ2LDggKzE4NDksOCBAQCBUcmFu c2FjdGlvbklkTGltaXRlZEZvck9sZFNuYXBzaG90cyhUcmFuc2FjdGlvbklkIHJlY2VudFhtaW4s CiAKIAkJLyogQ2hlY2sgZm9yIGZhc3QgZXhpdCB3aXRob3V0IExXIGxvY2tpbmcuICovCiAJCVNw aW5Mb2NrQWNxdWlyZSgmb2xkU25hcHNob3RDb250cm9sLT5tdXRleF90aHJlc2hvbGQpOwotCQl0 aHJlc2hvbGRfdGltZXN0YW1wID0gb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRfdGltZXN0 YW1wOwotCQl0aHJlc2hvbGRfeGlkID0gb2xkU25hcHNob3RDb250cm9sLT50aHJlc2hvbGRfeGlk OworCQl0aHJlc2hvbGRfdGltZXN0YW1wID0gcGdfYXRvbWljX3JlYWRfdTY0KCZvbGRTbmFwc2hv dENvbnRyb2wtPnRocmVzaG9sZF90aW1lc3RhbXApOworCQl0aHJlc2hvbGRfeGlkID0gcGdfYXRv bWljX3JlYWRfdTMyKCZvbGRTbmFwc2hvdENvbnRyb2wtPnRocmVzaG9sZF94aWQpOwogCQlTcGlu TG9ja1JlbGVhc2UoJm9sZFNuYXBzaG90Q29udHJvbC0+bXV0ZXhfdGhyZXNob2xkKTsKIAogCQlp ZiAodHMgPT0gdGhyZXNob2xkX3RpbWVzdGFtcCkKZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRlL3V0 aWxzL29sZF9zbmFwc2hvdC5oIGIvc3JjL2luY2x1ZGUvdXRpbHMvb2xkX3NuYXBzaG90LmgKaW5k ZXggZjE5NzhhMjhlMWMuLjNkZDMxZDcyMWM3IDEwMDY0NAotLS0gYS9zcmMvaW5jbHVkZS91dGls cy9vbGRfc25hcHNob3QuaAorKysgYi9zcmMvaW5jbHVkZS91dGlscy9vbGRfc25hcHNob3QuaApA QCAtMTYsNiArMTYsNyBAQAogI2RlZmluZSBPTERfU05BUFNIT1RfSAogCiAjaW5jbHVkZSAiZGF0 YXR5cGUvdGltZXN0YW1wLmgiCisjaW5jbHVkZSAicG9ydC9hdG9taWNzLmgiCiAjaW5jbHVkZSAi c3RvcmFnZS9zX2xvY2suaCIKIAogLyoKQEAgLTMzLDggKzM0LDggQEAgdHlwZWRlZiBzdHJ1Y3Qg T2xkU25hcHNob3RDb250cm9sRGF0YQogCVRyYW5zYWN0aW9uSWQgbGF0ZXN0X3htaW47CS8qIGxh dGVzdCBzbmFwc2hvdCB4bWluICovCiAJVGltZXN0YW1wVHogbmV4dF9tYXBfdXBkYXRlOwkvKiBs YXRlc3Qgc25hcHNob3QgdmFsaWQgdXAgdG8gKi8KIAlzbG9ja190CQltdXRleF90aHJlc2hvbGQ7 CS8qIHByb3RlY3QgdGhyZXNob2xkIGZpZWxkcyAqLwotCVRpbWVzdGFtcFR6IHRocmVzaG9sZF90 aW1lc3RhbXA7CS8qIGVhcmxpZXIgc25hcHNob3QgaXMgb2xkICovCi0JVHJhbnNhY3Rpb25JZCB0 aHJlc2hvbGRfeGlkOwkvKiBlYXJsaWVyIHhpZCBtYXkgYmUgZ29uZSAqLworCXBnX2F0b21pY191 aW50NjQgdGhyZXNob2xkX3RpbWVzdGFtcDsJLyogZWFybGllciBzbmFwc2hvdCBpcyBvbGQgKi8K KwlwZ19hdG9taWNfdWludDMyIHRocmVzaG9sZF94aWQ7CS8qIGVhcmxpZXIgeGlkIG1heSBiZSBn b25lICovCiAKIAkvKgogCSAqIEtlZXAgb25lIHhpZCBwZXIgbWludXRlIGZvciBvbGQgc25hcHNo b3QgZXJyb3IgaGFuZGxpbmcuCi0tIAoyLjM0LjEKCg== --0000000000004ded9705f2ef6142--