Content-Type: multipart/mixed; boundary="------------oZUg2WB9mTKvHr0ZKO0djdrr"
Message-ID: <320a31e6-4c91-438b-ab17-8a1d72384727@tantorlabs.com>
Date: Tue, 26 Nov 2024 01:15:25 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Sample rate added to pg_stat_statements
To: Alexander Korotkov <aekorotkov@gmail.com>,
 Michael Paquier <michael@paquier.xyz>
Cc: Greg Sabino Mullane <htamfids@gmail.com>,
 "Andrey M. Borodin" <x4mmm@yandex-team.ru>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
References: <fe99e0ca-e564-480e-b865-5f0cee30bc60@tantorlabs.com>
 <c7ce658a-e0eb-4c5f-92a8-511314d89bbb@tantorlabs.com>
 <6707FCA9-FD1A-4609-A1A6-142456C14E0C@yandex-team.ru>
 <CAKAnmmLD8C-Y6CVF74Ccup8h9YpzBYzSvWWRG_Qf8+5YEMh4-g@mail.gmail.com>
 <Zz0MFPq1s4WFxxpL@paquier.xyz>
 <CAPpHfdsTKAQqC3A48-MGQhrhfEamXZPb64w=utk7thQcOMNr7Q@mail.gmail.com>
Content-Language: en-US
From: Ilia Evdokimov <ilya.evdokimov@tantorlabs.com>
In-Reply-To: <CAPpHfdsTKAQqC3A48-MGQhrhfEamXZPb64w=utk7thQcOMNr7Q@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/320a31e6-4c91-438b-ab17-8a1d72384727%40tantorlabs.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------oZUg2WB9mTKvHr0ZKO0djdrr
Content-Type: multipart/alternative;
 boundary="------------ULVwZFN0AI1qURhBmfQW0zeF"

--------------ULVwZFN0AI1qURhBmfQW0zeF
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 22.11.2024 09:08, Alexander Korotkov wrote:
> On Wed, Nov 20, 2024 at 12:07 AM Michael Paquier<michael@paquier.xyz>  wrote:
>> On Tue, Nov 19, 2024 at 09:39:21AM -0500, Greg Sabino Mullane wrote:
>>> Oh, and a +1 in general to the patch, OP, although it would also be nice to
>>> start finding the bottlenecks that cause such performance issues.
>> FWIW, I'm not eager to integrate this proposal without looking at this
>> exact argument in depth.
>>
>> One piece of it would be to see how much of such "bottlenecks" we
>> would be able to get rid of by integrating pg_stat_statements into
>> the central pgstats with the custom APIs, without pushing the module
>> into core.  This means that we would combine the existing hash of pgss
>> to shrink to 8 bytes for objid rather than 13 bytes now as the current
>> code relies on (toplevel, userid, queryid) for the entry lookup (entry
>> removal is sniped with these three values as well, or dshash seq
>> scans).  The odds of conflicts still still play in our favor even if
>> we have a few million entries, or even ten times that.
> If you run "pgbench -S -M prepared" on a pretty large machine with
> high concurrency, then spin lock in pgss_store() could become pretty
> much of a bottleneck.  And I'm not sure switching all counters to
> atomics could somehow improve the situation given we already have
> pretty many counters.
>
> I'm generally +1 for the approach taken in this thread.  But I would
> suggest introducing a threshold value for a query execution time, and
> sample just everything below that threshold.  The slower query
> shouldn't be sampled, because it can't be too frequent, and also it
> could be more valuable to be counter individually (while very fast
> queries probably only matter "in average").
>
> ------
> Regards,
> Alexander Korotkov
> Supabase

I really liked your idea, and I’d like to propose an enhancement that I 
believe improves it further.

Yes, if a query’s execution time exceeds the threshold, it should always 
be tracked without sampling. However, for queries with execution times 
below the threshold, the sampling logic should prioritize shorter 
queries over those closer to the threshold. In my view, the ideal 
approach is for shorter queries to have the highest probability of being 
sampled, while queries closer to the threshold are less likely to be 
sampled.

This behavior can be achieved with the following logic:

pg_stat_statements.sample_exectime_threshold * random(0, 1) < total_time

Here’s how it works:

  * As a query’s execution time approaches zero, the probability of it
    being sampled approaches one.
  * Conversely, as a query’s execution time approaches the threshold,
    the probability of it being sampled approaches zero.

In other words, the sampling probability decreases linearly from 1 to 0 
as the execution time gets closer to the threshold.

I believe this approach offers an ideal user experience. I have attached 
a new patch implementing this logic. Please let me know if you have any 
feedback regarding the comments in the code, the naming of variables or 
documentation. I’m always open to discussion.

--
Best regards,
Ilia Evdokimov,
Tantor Labs LLC.

--------------ULVwZFN0AI1qURhBmfQW0zeF
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 22.11.2024 09:08, Alexander Korotkov
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAPpHfdsTKAQqC3A48-MGQhrhfEamXZPb64w=utk7thQcOMNr7Q@mail.gmail.com">
      <pre class="moz-quote-pre" wrap="">On Wed, Nov 20, 2024 at 12:07 AM Michael Paquier <a class="moz-txt-link-rfc2396E" href="mailto:michael@paquier.xyz">&lt;michael@paquier.xyz&gt;</a> wrote:
</pre>
      <blockquote type="cite">
        <pre class="moz-quote-pre" wrap="">On Tue, Nov 19, 2024 at 09:39:21AM -0500, Greg Sabino Mullane wrote:
</pre>
        <blockquote type="cite">
          <pre class="moz-quote-pre" wrap="">Oh, and a +1 in general to the patch, OP, although it would also be nice to
start finding the bottlenecks that cause such performance issues.
</pre>
        </blockquote>
        <pre class="moz-quote-pre" wrap="">
FWIW, I'm not eager to integrate this proposal without looking at this
exact argument in depth.

One piece of it would be to see how much of such "bottlenecks" we
would be able to get rid of by integrating pg_stat_statements into
the central pgstats with the custom APIs, without pushing the module
into core.  This means that we would combine the existing hash of pgss
to shrink to 8 bytes for objid rather than 13 bytes now as the current
code relies on (toplevel, userid, queryid) for the entry lookup (entry
removal is sniped with these three values as well, or dshash seq
scans).  The odds of conflicts still still play in our favor even if
we have a few million entries, or even ten times that.
</pre>
      </blockquote>
      <pre class="moz-quote-pre" wrap="">
If you run "pgbench -S -M prepared" on a pretty large machine with
high concurrency, then spin lock in pgss_store() could become pretty
much of a bottleneck.  And I'm not sure switching all counters to
atomics could somehow improve the situation given we already have
pretty many counters.

I'm generally +1 for the approach taken in this thread.  But I would
suggest introducing a threshold value for a query execution time, and
sample just everything below that threshold.  The slower query
shouldn't be sampled, because it can't be too frequent, and also it
could be more valuable to be counter individually (while very fast
queries probably only matter "in average").

------
Regards,
Alexander Korotkov
Supabase
</pre>
    </blockquote>
    <p><span style="white-space: pre-wrap">
</span></p>
    <p>I really liked your idea, and I’d like to propose an enhancement
      that I believe improves it further.</p>
    <p>Yes, if a query’s execution time exceeds the threshold, it should
      always be tracked without sampling. However, for queries with
      execution times below the threshold, the sampling logic should
      prioritize shorter queries over those closer to the threshold. In
      my view, the ideal approach is for shorter queries to have the
      highest probability of being sampled, while queries closer to the
      threshold are less likely to be sampled.</p>
    <p>This behavior can be achieved with the following logic:</p>
    <p>pg_stat_statements.sample_exectime_threshold * random(0, 1) &lt;
      total_time</p>
    <p>Here’s how it works:</p>
    <ul>
      <li>As a query’s execution time approaches zero, the probability
        of it being sampled approaches one.</li>
      <li>Conversely, as a query’s execution time approaches the
        threshold, the probability of it being sampled approaches zero.</li>
    </ul>
    <p>In other words, the sampling probability decreases linearly from
      1 to 0 as the execution time gets closer to the threshold.</p>
    <p>I believe this approach offers an ideal user experience. I have
      attached a new patch implementing this logic. Please let me know
      if you have any feedback regarding the comments in the code, the
      naming of variables or documentation. I’m always open to
      discussion.</p>
    <p>--<br>
      Best regards,<br>
      Ilia Evdokimov,<br>
      Tantor Labs LLC.<br>
    </p>
  </body>
</html>

--------------ULVwZFN0AI1qURhBmfQW0zeF--
--------------oZUg2WB9mTKvHr0ZKO0djdrr
Content-Type: text/x-patch; charset=UTF-8;
 name="v5-0001-Add-time-based-sampling-to-pg_stat_statements.patch"
Content-Disposition: attachment;
 filename*0="v5-0001-Add-time-based-sampling-to-pg_stat_statements.patch"
Content-Transfer-Encoding: base64

RnJvbSBlZGJhMWJjZWZhODVlOTk1ZWMzZTlkZjZjOWU4ZDMwYWRjZDk0MGI5IE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBJbGlhIEV2ZG9raW1vdiA8aWx5YS5ldmRva2ltb3ZA
dGFudG9ybGFicy5jb20+CkRhdGU6IFR1ZSwgMjYgTm92IDIwMjQgMDA6NTg6MjQgKzAzMDAK
U3ViamVjdDogW1BBVENIXSBBZGQgdGltZS1iYXNlZCBzYW1wbGluZyB0byBwZ19zdGF0X3N0
YXRlbWVudHMKCk5ldyBjb25maWd1cmF0aW9uIHBhcmFtZXRlciBwZ19zdGF0X3N0YXRlbWVu
dHMuc2FtcGxlX2V4ZWN0aW1lX3RocmVzaG9sZCwKd2hpY2ggYWxsb3dzIHRyYWNraW5nIG9u
bHkgcXVlcmllcyB0aGF0IG1lZXQgYSBzcGVjaWZpZWQgZXhlY3V0aW9uIHRpbWUgdGhyZXNo
b2xkLgpRdWVyaWVzIHdpdGggZXhlY3V0aW9uIHRpbWVzIGFib3ZlIHRoZSB0aHJlc2hvbGQg
YXJlIGFsd2F5cyB0cmFja2VkLAp3aGlsZSBzaG9ydGVyIHF1ZXJpZXMgYXJlIHNhbXBsZWQg
cHJvYmFiaWxpc3RpY2FsbHkuClRoaXMgaGVscHMgcmVkdWNlIHRoZSBvdmVyaGVhZCBvZiB0
cmFja2luZyBmcmVxdWVudCwgZmFzdCBxdWVyaWVzCndoaWxlIHByZXNlcnZpbmcgZGF0YSBm
b3IgbG9uZ2VyLXJ1bm5pbmcgb25lcy4KLS0tCiAuLi4vcGdfc3RhdF9zdGF0ZW1lbnRzL3Bn
X3N0YXRfc3RhdGVtZW50cy5jICAgfCA1NyArKysrKysrKysrKysrKy0tLS0tCiBkb2Mvc3Jj
L3NnbWwvcGdzdGF0c3RhdGVtZW50cy5zZ21sICAgICAgICAgICAgfCAyMCArKysrKysrCiAy
IGZpbGVzIGNoYW5nZWQsIDY0IGluc2VydGlvbnMoKyksIDEzIGRlbGV0aW9ucygtKQoKZGlm
ZiAtLWdpdCBhL2NvbnRyaWIvcGdfc3RhdF9zdGF0ZW1lbnRzL3BnX3N0YXRfc3RhdGVtZW50
cy5jIGIvY29udHJpYi9wZ19zdGF0X3N0YXRlbWVudHMvcGdfc3RhdF9zdGF0ZW1lbnRzLmMK
aW5kZXggNDljNjU3YjNlMC4uNmVjMWUzZTdlZCAxMDA2NDQKLS0tIGEvY29udHJpYi9wZ19z
dGF0X3N0YXRlbWVudHMvcGdfc3RhdF9zdGF0ZW1lbnRzLmMKKysrIGIvY29udHJpYi9wZ19z
dGF0X3N0YXRlbWVudHMvcGdfc3RhdF9zdGF0ZW1lbnRzLmMKQEAgLTQ5LDYgKzQ5LDcgQEAK
IAogI2luY2x1ZGUgImFjY2Vzcy9wYXJhbGxlbC5oIgogI2luY2x1ZGUgImNhdGFsb2cvcGdf
YXV0aGlkLmgiCisjaW5jbHVkZSAiY29tbW9uL3BnX3BybmcuaCIKICNpbmNsdWRlICJjb21t
b24vaW50LmgiCiAjaW5jbHVkZSAiZXhlY3V0b3IvaW5zdHJ1bWVudC5oIgogI2luY2x1ZGUg
ImZ1bmNhcGkuaCIKQEAgLTI4OSw2ICsyOTAsNyBAQCBzdGF0aWMgY29uc3Qgc3RydWN0IGNv
bmZpZ19lbnVtX2VudHJ5IHRyYWNrX29wdGlvbnNbXSA9CiB9OwogCiBzdGF0aWMgaW50CXBn
c3NfbWF4ID0gNTAwMDsJLyogbWF4ICMgc3RhdGVtZW50cyB0byB0cmFjayAqLworc3RhdGlj
IGludAlwZ3NzX3NhbXBsZV9leGVjdGltZV90aHJlc2hvbGQgPSAwOyAvKiBUaHJlc2hvbGQg
Zm9yIHF1ZXJ5IGV4ZWN1dGlvbiBzYW1wbGluZyAobXNlYykgKi8KIHN0YXRpYyBpbnQJcGdz
c190cmFjayA9IFBHU1NfVFJBQ0tfVE9QOwkvKiB0cmFja2luZyBsZXZlbCAqLwogc3RhdGlj
IGJvb2wgcGdzc190cmFja191dGlsaXR5ID0gdHJ1ZTsJLyogd2hldGhlciB0byB0cmFjayB1
dGlsaXR5IGNvbW1hbmRzICovCiBzdGF0aWMgYm9vbCBwZ3NzX3RyYWNrX3BsYW5uaW5nID0g
ZmFsc2U7CS8qIHdoZXRoZXIgdG8gdHJhY2sgcGxhbm5pbmcKQEAgLTQxNCw2ICs0MTYsMTkg
QEAgX1BHX2luaXQodm9pZCkKIAkJCQkJCQlOVUxMLAogCQkJCQkJCU5VTEwpOwogCisJRGVm
aW5lQ3VzdG9tSW50VmFyaWFibGUoInBnX3N0YXRfc3RhdGVtZW50cy5zYW1wbGVfZXhlY3Rp
bWVfdGhyZXNob2xkIiwKKwkJCQkJCQkiU2V0cyB0aGUgdGhyZXNob2xkIChpbiBtc2VjKSBm
b3IgcXVlcnkgZXhlY3V0aW9uIHRpbWUgc2FtcGxpbmcuIiwKKwkJCQkJCQlOVUxMLAorCQkJ
CQkJCSZwZ3NzX3NhbXBsZV9leGVjdGltZV90aHJlc2hvbGQsCisJCQkJCQkJMCwKKwkJCQkJ
CQkwLAorCQkJCQkJCUlOVF9NQVggLyAyLAorCQkJCQkJCVBHQ19TVVNFVCwKKwkJCQkJCQkw
LAorCQkJCQkJCU5VTEwsCisJCQkJCQkJTlVMTCwKKwkJCQkJCQlOVUxMKTsKKwogCURlZmlu
ZUN1c3RvbUVudW1WYXJpYWJsZSgicGdfc3RhdF9zdGF0ZW1lbnRzLnRyYWNrIiwKIAkJCQkJ
CQkgIlNlbGVjdHMgd2hpY2ggc3RhdGVtZW50cyBhcmUgdHJhY2tlZCBieSBwZ19zdGF0X3N0
YXRlbWVudHMuIiwKIAkJCQkJCQkgTlVMTCwKQEAgLTEwNzEsMjUgKzEwODYsNDEgQEAgcGdz
c19FeGVjdXRvckVuZChRdWVyeURlc2MgKnF1ZXJ5RGVzYykKIAlpZiAocXVlcnlJZCAhPSBV
SU5UNjRDT05TVCgwKSAmJiBxdWVyeURlc2MtPnRvdGFsdGltZSAmJgogCQlwZ3NzX2VuYWJs
ZWQobmVzdGluZ19sZXZlbCkpCiAJeworCQlkb3VibGUgdG90YWxfdGltZSA9IDAuMDsKIAkJ
LyoKIAkJICogTWFrZSBzdXJlIHN0YXRzIGFjY3VtdWxhdGlvbiBpcyBkb25lLiAgKE5vdGU6
IGl0J3Mgb2theSBpZiBzZXZlcmFsCiAJCSAqIGxldmVscyBvZiBob29rIGFsbCBkbyB0aGlz
LikKIAkJICovCiAJCUluc3RyRW5kTG9vcChxdWVyeURlc2MtPnRvdGFsdGltZSk7CiAKLQkJ
cGdzc19zdG9yZShxdWVyeURlc2MtPnNvdXJjZVRleHQsCi0JCQkJICAgcXVlcnlJZCwKLQkJ
CQkgICBxdWVyeURlc2MtPnBsYW5uZWRzdG10LT5zdG10X2xvY2F0aW9uLAotCQkJCSAgIHF1
ZXJ5RGVzYy0+cGxhbm5lZHN0bXQtPnN0bXRfbGVuLAotCQkJCSAgIFBHU1NfRVhFQywKLQkJ
CQkgICBxdWVyeURlc2MtPnRvdGFsdGltZS0+dG90YWwgKiAxMDAwLjAsCS8qIGNvbnZlcnQg
dG8gbXNlYyAqLwotCQkJCSAgIHF1ZXJ5RGVzYy0+ZXN0YXRlLT5lc190b3RhbF9wcm9jZXNz
ZWQsCi0JCQkJICAgJnF1ZXJ5RGVzYy0+dG90YWx0aW1lLT5idWZ1c2FnZSwKLQkJCQkgICAm
cXVlcnlEZXNjLT50b3RhbHRpbWUtPndhbHVzYWdlLAotCQkJCSAgIHF1ZXJ5RGVzYy0+ZXN0
YXRlLT5lc19qaXQgPyAmcXVlcnlEZXNjLT5lc3RhdGUtPmVzX2ppdC0+aW5zdHIgOiBOVUxM
LAotCQkJCSAgIE5VTEwsCi0JCQkJICAgcXVlcnlEZXNjLT5lc3RhdGUtPmVzX3BhcmFsbGVs
X3dvcmtlcnNfdG9fbGF1bmNoLAotCQkJCSAgIHF1ZXJ5RGVzYy0+ZXN0YXRlLT5lc19wYXJh
bGxlbF93b3JrZXJzX2xhdW5jaGVkKTsKKwkJLyogY29udmVydCB0byBtc2VjICovCisJCXRv
dGFsX3RpbWUgPSBxdWVyeURlc2MtPnRvdGFsdGltZS0+dG90YWwgKiAxMDAwLjA7CisKKwkJ
LyoKKwkJKiBTYW1wbGluZyBpcyBpbXBsZW1lbnRlZCBieSBjb21wYXJpbmcgdGhlIGV4ZWN1
dGlvbiB0aW1lIG9mIHRoZSBxdWVyeQorCQkqIHRvIGEgcmFuZG9tIHRocmVzaG9sZCBkZXJp
dmVkIGZyb20gdGhlIGNvbmZpZ3VyZWQgc2FtcGxlX2V4ZWN1dGlvbl90aW1lLgorCQkqIFRo
aXMgbWV0aG9kIGVuc3VyZXMgcHJvcG9ydGlvbmFsIHNhbXBsaW5nOiBxdWVyaWVzIHdpdGgg
ZXhlY3V0aW9uIHRpbWVzCisJCSogY2xvc2VyIHRvIHRoZSB0aHJlc2hvbGQgYXJlIG1vcmUg
bGlrZWx5IHRvIGJlIHRyYWNrZWRlZCwgd2hpbGUgdmVyeSBzaG9ydAorCQkqIHF1ZXJpZXMg
YXJlIG1vcmUgbGlrZWx5IHRvIGJlIHNhbXBsZWQuCisJCSovCisJCWlmICh0b3RhbF90aW1l
ID49IHBnc3Nfc2FtcGxlX2V4ZWN0aW1lX3RocmVzaG9sZCB8fAorCQkgICAodG90YWxfdGlt
ZSA8IHBnc3Nfc2FtcGxlX2V4ZWN0aW1lX3RocmVzaG9sZCAmJgorCQkgICAocGdzc19zYW1w
bGVfZXhlY3RpbWVfdGhyZXNob2xkICogcGdfcHJuZ19kb3VibGUoJnBnX2dsb2JhbF9wcm5n
X3N0YXRlKSA8IHRvdGFsX3RpbWUpKSkKKwkJeworCQkJcGdzc19zdG9yZShxdWVyeURlc2Mt
PnNvdXJjZVRleHQsCisJCQkJICAgcXVlcnlJZCwKKwkJCQkgICBxdWVyeURlc2MtPnBsYW5u
ZWRzdG10LT5zdG10X2xvY2F0aW9uLAorCQkJCSAgIHF1ZXJ5RGVzYy0+cGxhbm5lZHN0bXQt
PnN0bXRfbGVuLAorCQkJCSAgIFBHU1NfRVhFQywKKwkJCQkgICB0b3RhbF90aW1lLAorCQkJ
CSAgIHF1ZXJ5RGVzYy0+ZXN0YXRlLT5lc190b3RhbF9wcm9jZXNzZWQsCisJCQkJICAgJnF1
ZXJ5RGVzYy0+dG90YWx0aW1lLT5idWZ1c2FnZSwKKwkJCQkgICAmcXVlcnlEZXNjLT50b3Rh
bHRpbWUtPndhbHVzYWdlLAorCQkJCSAgIHF1ZXJ5RGVzYy0+ZXN0YXRlLT5lc19qaXQgPyAm
cXVlcnlEZXNjLT5lc3RhdGUtPmVzX2ppdC0+aW5zdHIgOiBOVUxMLAorCQkJCSAgIE5VTEws
CisJCQkJICAgcXVlcnlEZXNjLT5lc3RhdGUtPmVzX3BhcmFsbGVsX3dvcmtlcnNfdG9fbGF1
bmNoLAorCQkJCSAgIHF1ZXJ5RGVzYy0+ZXN0YXRlLT5lc19wYXJhbGxlbF93b3JrZXJzX2xh
dW5jaGVkKTsKKwkJfQogCX0KIAogCWlmIChwcmV2X0V4ZWN1dG9yRW5kKQpkaWZmIC0tZ2l0
IGEvZG9jL3NyYy9zZ21sL3Bnc3RhdHN0YXRlbWVudHMuc2dtbCBiL2RvYy9zcmMvc2dtbC9w
Z3N0YXRzdGF0ZW1lbnRzLnNnbWwKaW5kZXggNTAxYjQ2OGU5YS4uN2I5OThhNzdmMyAxMDA2
NDQKLS0tIGEvZG9jL3NyYy9zZ21sL3Bnc3RhdHN0YXRlbWVudHMuc2dtbAorKysgYi9kb2Mv
c3JjL3NnbWwvcGdzdGF0c3RhdGVtZW50cy5zZ21sCkBAIC04NzIsNiArODcyLDI2IEBACiAg
ICAgPC9saXN0aXRlbT4KICAgIDwvdmFybGlzdGVudHJ5PgogCisgIDx2YXJpYWJsZWxpc3Q+
CisgICA8dmFybGlzdGVudHJ5PgorICAgIDx0ZXJtPgorICAgICA8dmFybmFtZT5wZ19zdGF0
X3N0YXRlbWVudHMuc2FtcGxlX2V4ZWN0aW1lX3RocmVzaG9sZDwvdmFybmFtZT4gKDx0eXBl
PmludGVnZXI8L3R5cGU+KQorICAgICA8aW5kZXh0ZXJtPgorICAgICAgPHByaW1hcnk+PHZh
cm5hbWU+cGdfc3RhdF9zdGF0ZW1lbnRzLnNhbXBsZV9leGVjdGltZV90aHJlc2hvbGQ8L3Zh
cm5hbWU+IGNvbmZpZ3VyYXRpb24gcGFyYW1ldGVyPC9wcmltYXJ5PgorICAgICA8L2luZGV4
dGVybT4KKyAgICA8L3Rlcm0+CisKKyAgICA8bGlzdGl0ZW0+CisgICAgIDxwYXJhPgorICAg
ICAgPHZhcm5hbWU+cGdfc3RhdF9zdGF0ZW1lbnRzLnNhbXBsZV9leGVjdGltZV90aHJlc2hv
bGQ8L3Zhcm5hbWU+IGlzIHRoZSB0aHJlc2hvbGQgKGluIG1zZWMpCisgICAgICBmb3IgcXVl
cnkgZXhlY3V0aW9uIHRpbWUgc2FtcGxpbmcuIFF1ZXJpZXMgd2l0aCBleGVjdXRpb24gdGlt
ZXMgYWJvdmUgdGhpcyB0aHJlc2hvbGQgYXJlCisgICAgICBhcmUgbmV2ZXIgc2FtcGxlZC4g
RXhlY3V0aW9uIHRpbWUgYmVsb3cgdGhyZXNob2xkIGFyZSBzYW1wbGVkIHRvIHJlZHVjZSBv
dmVyaGVhZCBmcm9tIGZyZXF1ZW50IHNob3J0IHF1ZXJpZXMuCisgICAgICBUaGUgZGVmYXVs
dCB2YWx1ZSBpcyAwLgorICAgICAgT25seSBzdXBlcnVzZXJzIGNhbiBjaGFuZ2UgdGhpcyBz
ZXR0aW5nLgorICAgICA8L3BhcmE+CisgICAgPC9saXN0aXRlbT4KKyAgIDwvdmFybGlzdGVu
dHJ5PgorCiAgICA8dmFybGlzdGVudHJ5PgogICAgIDx0ZXJtPgogICAgICA8dmFybmFtZT5w
Z19zdGF0X3N0YXRlbWVudHMudHJhY2s8L3Zhcm5hbWU+ICg8dHlwZT5lbnVtPC90eXBlPikK
LS0gCjIuMzQuMQoK

--------------oZUg2WB9mTKvHr0ZKO0djdrr--