Content-Type: multipart/mixed; boundary="------------3mxxZinpeDglB15YlXniAwGl"
Message-ID: <fe0391a8-dfa9-41c3-bf1c-7ea058e40f30@gmail.com>
Date: Mon, 6 Jan 2025 09:00:00 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
From: Alexander Lakhin <exclusion@gmail.com>
Subject: stats.sql might fail due to shared buffers also used by parallel
 tests
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Andres Freund <andres@anarazel.de>,
 Melanie Plageman <melanieplageman@gmail.com>
Archived-At: <https://www.postgresql.org/message-id/fe0391a8-dfa9-41c3-bf1c-7ea058e40f30%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------3mxxZinpeDglB15YlXniAwGl
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello hackers,

I'd like to share my investigation of one mysterious stats.sql failure
occurred in December: [1].
The difference of the failure is:
--- C:/prog/bf/root/HEAD/pgsql/src/test/regress/expected/stats.out 2024-09-18 19:31:14.665516500 +0000
+++ C:/prog/bf/root/HEAD/pgsql.build/testrun/recovery/027_stream_regress/data/results/stats.out 2024-12-17 
09:57:08.944588500 +0000
@@ -1291,7 +1291,7 @@
  SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
   ?column?
  ----------
- t
+ f
  (1 row)

027_stream_regress_primary.log contains:
2024-12-17 09:57:06.778 UTC [8568:109] pg_regress/stats LOG: statement: SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
       FROM pg_stat_io
       WHERE object = 'relation'
2024-12-17 09:57:06.779 UTC [8568:110] pg_regress/stats LOG: statement: CREATE TABLE test_io_shared(a int);
2024-12-17 09:57:06.780 UTC [8568:111] pg_regress/stats LOG: statement: INSERT INTO test_io_shared SELECT i FROM 
generate_series(1,100)i;
...
2024-12-17 09:57:06.782 UTC [8568:115] pg_regress/stats LOG: statement: CHECKPOINT;
2024-12-17 09:57:06.794 UTC [3664:40] LOG:  checkpoint starting: immediate force wait
2024-12-17 09:57:06.856 UTC [3664:41] LOG:  checkpoint complete: wrote 0 buffers (0.0%), wrote 1 SLRU buffers; 0 WAL 
file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.062 s; sync files=0, longest=0.000 s, 
average=0.000 s; distance=1875 kB, estimate=52682 kB; lsn=0/14A2F410, redo lsn=0/14A2F3B8
2024-12-17 09:57:06.857 UTC [8568:116] pg_regress/stats LOG: statement: CHECKPOINT;
2024-12-17 09:57:06.857 UTC [3664:42] LOG:  checkpoint starting: immediate force wait
2024-12-17 09:57:06.859 UTC [3664:43] LOG:  checkpoint complete: wrote 0 buffers (0.0%), wrote 0 SLRU buffers; 0 WAL 
file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.002 s; sync files=0, longest=0.000 s, 
average=0.000 s; distance=0 kB, estimate=47414 kB; lsn=0/14A2F4E0, redo lsn=0/14A2F488
2024-12-17 09:57:06.859 UTC [8568:117] pg_regress/stats LOG: statement: SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
       FROM pg_stat_io
       WHERE object = 'relation'
2024-12-17 09:57:06.860 UTC [8568:118] pg_regress/stats LOG: statement: SELECT 77693 > 77693;

The corresponding test case (added by 10a082bf7, dated 2023-02-11) is:
SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
   FROM pg_stat_io
   WHERE object = 'relation' \gset io_sum_shared_before_
...
CREATE TABLE test_io_shared(a int);
INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
...
CHECKPOINT;
CHECKPOINT;
...
SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
   FROM pg_stat_get_backend_io(pg_backend_pid())
   WHERE object = 'relation' \gset my_io_sum_shared_after_
SELECT :my_io_sum_shared_after_writes >= :my_io_sum_shared_before_writes;

So, as we can see from the log above, both checkpoints wrote 0 buffers,
moreover, no other process/backend wrote any buffers.

After series of experiments, I found that this situation is possible when
other backend steals shared buffers from checkpointer, without making new
buffers dirty, and then delays flushing it's statistics.

Please find attached a reproducer for this issue: I've added two sleeps
to stats.sql and reduced parallel_schedule, but I believe it's a legal
change, which doesn't affect the test case essentially.

With bufchurn.sql running concurrently. and TEMP_CONFIG containing
shared_buffers = 1MB (as in the 027_stream_regress test in question), I can
see exactly the same failure:
$ echo 'shared_buffers = 1MB' >/tmp/extra.config; TEMP_CONFIG=/tmp/extra.config make -s check
# +++ regress check in src/test/regress +++
# initializing database system by copying initdb template
# using temp instance on port 65312 with PID 129700
ok 1         - test_setup                                356 ms
ok 2         - create_index                             1002 ms
# parallel group (2 tests):  stats bufchurn
not ok 3     + stats                                   21176 ms
...

diff -U3 .../src/test/regress/expected/stats.out .../src/test/regress/results/stats.out
--- .../src/test/regress/expected/stats.out  2025-01-06 08:03:07.720275980 +0200
+++ .../src/test/regress/results/stats.out   2025-01-06 08:15:34.602170721 +0200
@@ -1321,7 +1321,7 @@
  SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
   ?column?
  ----------
- t
+ f
  (1 row)
...

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-12-17%2008%3A59%3A44

Best regards,
Alexander
--------------3mxxZinpeDglB15YlXniAwGl
Content-Type: text/x-patch; charset=UTF-8; name="stats-bufs-write-repro.patch"
Content-Disposition: attachment; filename="stats-bufs-write-repro.patch"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvYnVmY2h1cm4ub3V0IGIv
c3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9idWZjaHVybi5vdXQKbmV3IGZpbGUgbW9kZSAx
MDA2NDQKaW5kZXggMDAwMDAwMDAwMC4uZTY5ZGUyOWJiMgpkaWZmIC0tZ2l0IGEvc3JjL3Rl
c3QvcmVncmVzcy9leHBlY3RlZC9zdGF0cy5vdXQgYi9zcmMvdGVzdC9yZWdyZXNzL2V4cGVj
dGVkL3N0YXRzLm91dAppbmRleCBhMDMxN2I3MjA4Li42NTE4OThiOWY0IDEwMDY0NAotLS0g
YS9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3N0YXRzLm91dAorKysgYi9zcmMvdGVzdC9y
ZWdyZXNzL2V4cGVjdGVkL3N0YXRzLm91dApAQCAtODM0LDYgKzgzNCwxMiBAQCBTRUxFQ1Qg
bnVtX3JlcXVlc3RlZCBBUyBycXN0X2NrcHRzX2JlZm9yZSBGUk9NIHBnX3N0YXRfY2hlY2tw
b2ludGVyIFxnc2V0CiBTRUxFQ1Qgd2FsX2J5dGVzIEFTIHdhbF9ieXRlc19iZWZvcmUgRlJP
TSBwZ19zdGF0X3dhbCBcZ3NldAogQ1JFQVRFIFRFTVAgVEFCTEUgdGVzdF9zdGF0c190ZW1w
IEFTIFNFTEVDVCAxNzsKIERST1AgVEFCTEUgdGVzdF9zdGF0c190ZW1wOworU0VMRUNUIHBn
X3NsZWVwKDEwKTsKKyBwZ19zbGVlcCAKKy0tLS0tLS0tLS0KKyAKKygxIHJvdykKKwogLS0g
Q2hlY2twb2ludCB0d2ljZTogVGhlIGNoZWNrcG9pbnRlciByZXBvcnRzIHN0YXRzIGFmdGVy
IHJlcG9ydGluZyBjb21wbGV0aW9uCiAtLSBvZiB0aGUgY2hlY2twb2ludC4gQnV0IGFmdGVy
IGEgc2Vjb25kIGNoZWNrcG9pbnQgd2UnbGwgc2VlIGF0IGxlYXN0IHRoZQogLS0gcmVzdWx0
cyBvZiB0aGUgZmlyc3QuCkBAIC0xMjk4LDYgKzEzMDQsMTIgQEAgU0VMRUNUIDpteV9pb19z
dW1fc2hhcmVkX2FmdGVyX2V4dGVuZHMgPiA6bXlfaW9fc3VtX3NoYXJlZF9iZWZvcmVfZXh0
ZW5kczsKICB0CiAoMSByb3cpCiAKK1NFTEVDVCBwZ19zbGVlcCgxMCk7CisgcGdfc2xlZXAg
CistLS0tLS0tLS0tCisgCisoMSByb3cpCisKIC0tIEFmdGVyIGEgY2hlY2twb2ludCwgdGhl
cmUgc2hvdWxkIGJlIHNvbWUgYWRkaXRpb25hbCBJT0NPTlRFWFRfTk9STUFMIHdyaXRlcwog
LS0gYW5kIGZzeW5jcyBpbiB0aGUgZ2xvYmFsIHN0YXRzICh1c3VhbGx5IG5vdCBmb3IgdGhl
IGJhY2tlbmQpLgogLS0gU2VlIGNvbW1lbnQgYWJvdmUgZm9yIHJhdGlvbmFsZSBmb3IgdHdv
IGV4cGxpY2l0IENIRUNLUE9JTlRzLgpkaWZmIC0tZ2l0IGEvc3JjL3Rlc3QvcmVncmVzcy9w
YXJhbGxlbF9zY2hlZHVsZSBiL3NyYy90ZXN0L3JlZ3Jlc3MvcGFyYWxsZWxfc2NoZWR1bGUK
aW5kZXggMWVkZDllNDVlYi4uMjE0Mjg0MmEzMSAxMDA2NDQKLS0tIGEvc3JjL3Rlc3QvcmVn
cmVzcy9wYXJhbGxlbF9zY2hlZHVsZQorKysgYi9zcmMvdGVzdC9yZWdyZXNzL3BhcmFsbGVs
X3NjaGVkdWxlCkBAIC0xLDEzOCArMSw0IEBACi0jIC0tLS0tLS0tLS0KLSMgc3JjL3Rlc3Qv
cmVncmVzcy9wYXJhbGxlbF9zY2hlZHVsZQotIwotIyBNb3N0IHRlc3Qgc2NyaXB0cyBjYW4g
YmUgcnVuIGFmdGVyIHJ1bm5pbmcganVzdCB0ZXN0X3NldHVwIGFuZCBwb3NzaWJseQotIyBj
cmVhdGVfaW5kZXguICBFeGNlcHRpb25zIHRvIHRoaXMgcnVsZSBhcmUgZG9jdW1lbnRlZCBi
ZWxvdy4KLSMKLSMgQnkgY29udmVudGlvbiwgd2UgcHV0IG5vIG1vcmUgdGhhbiB0d2VudHkg
dGVzdHMgaW4gYW55IG9uZSBwYXJhbGxlbCBncm91cDsKLSMgdGhpcyBsaW1pdHMgdGhlIG51
bWJlciBvZiBjb25uZWN0aW9ucyBuZWVkZWQgdG8gcnVuIHRoZSB0ZXN0cy4KLSMgLS0tLS0t
LS0tLQotCi0jIHJlcXVpcmVkIHNldHVwIHN0ZXBzCiB0ZXN0OiB0ZXN0X3NldHVwCi0KLSMg
LS0tLS0tLS0tLQotIyBUaGUgZmlyc3QgZ3JvdXAgb2YgcGFyYWxsZWwgdGVzdHMKLSMgLS0t
LS0tLS0tLQotdGVzdDogYm9vbGVhbiBjaGFyIG5hbWUgdmFyY2hhciB0ZXh0IGludDIgaW50
NCBpbnQ4IG9pZCBmbG9hdDQgZmxvYXQ4IGJpdCBudW1lcmljIHR4aWQgdXVpZCBlbnVtIG1v
bmV5IHJhbmdldHlwZXMgcGdfbHNuIHJlZ3Byb2MKLQotIyAtLS0tLS0tLS0tCi0jIFRoZSBz
ZWNvbmQgZ3JvdXAgb2YgcGFyYWxsZWwgdGVzdHMKLSMgbXVsdGlyYW5nZXR5cGVzIGRlcGVu
ZHMgb24gcmFuZ2V0eXBlcwotIyBtdWx0aXJhbmdldHlwZXMgc2hvdWxkbid0IHJ1biBjb25j
dXJyZW50bHkgd2l0aCB0eXBlX3Nhbml0eQotIyAtLS0tLS0tLS0tCi10ZXN0OiBzdHJpbmdz
IG1kNSBudW1lcm9sb2d5IHBvaW50IGxzZWcgbGluZSBib3ggcGF0aCBwb2x5Z29uIGNpcmNs
ZSBkYXRlIHRpbWUgdGltZXR6IHRpbWVzdGFtcCB0aW1lc3RhbXB0eiBpbnRlcnZhbCBpbmV0
IG1hY2FkZHIgbWFjYWRkcjggbXVsdGlyYW5nZXR5cGVzCi0KLSMgLS0tLS0tLS0tLQotIyBB
bm90aGVyIGdyb3VwIG9mIHBhcmFsbGVsIHRlc3RzCi0jIGdlb21ldHJ5IGRlcGVuZHMgb24g
cG9pbnQsIGxzZWcsIGxpbmUsIGJveCwgcGF0aCwgcG9seWdvbiwgY2lyY2xlCi0jIGhvcm9s
b2d5IGRlcGVuZHMgb24gZGF0ZSwgdGltZSwgdGltZXR6LCB0aW1lc3RhbXAsIHRpbWVzdGFt
cHR6LCBpbnRlcnZhbAotIyAtLS0tLS0tLS0tCi10ZXN0OiBnZW9tZXRyeSBob3JvbG9neSB0
c3R5cGVzIHJlZ2V4IHR5cGVfc2FuaXR5IG9wcl9zYW5pdHkgbWlzY19zYW5pdHkgY29tbWVu
dHMgZXhwcmVzc2lvbnMgdW5pY29kZSB4aWQgbXZjYyBkYXRhYmFzZSBzdGF0c19pbXBvcnQK
LQotIyAtLS0tLS0tLS0tCi0jIExvYWQgaHVnZSBhbW91bnRzIG9mIGRhdGEKLSMgV2Ugc2hv
dWxkIHNwbGl0IHRoZSBkYXRhIGZpbGVzIGludG8gc2luZ2xlIGZpbGVzIGFuZCB0aGVuCi0j
IGV4ZWN1dGUgdHdvIGNvcHkgdGVzdHMgaW4gcGFyYWxsZWwsIHRvIGNoZWNrIHRoYXQgY29w
eSBpdHNlbGYKLSMgaXMgY29uY3VycmVudCBzYWZlLgotIyAtLS0tLS0tLS0tCi10ZXN0OiBj
b3B5IGNvcHlzZWxlY3QgY29weWRtbCBjb3B5ZW5jb2RpbmcgaW5zZXJ0IGluc2VydF9jb25m
bGljdAotCi0jIC0tLS0tLS0tLS0KLSMgTW9yZSBncm91cHMgb2YgcGFyYWxsZWwgdGVzdHMK
LSMgTm90ZTogbWFueSBvZiB0aGUgdGVzdHMgaW4gbGF0ZXIgZ3JvdXBzIGRlcGVuZCBvbiBj
cmVhdGVfaW5kZXgKLSMgLS0tLS0tLS0tLQotdGVzdDogY3JlYXRlX2Z1bmN0aW9uX2MgY3Jl
YXRlX21pc2MgY3JlYXRlX29wZXJhdG9yIGNyZWF0ZV9wcm9jZWR1cmUgY3JlYXRlX3RhYmxl
IGNyZWF0ZV90eXBlIGNyZWF0ZV9zY2hlbWEKLXRlc3Q6IGNyZWF0ZV9pbmRleCBjcmVhdGVf
aW5kZXhfc3BnaXN0IGNyZWF0ZV92aWV3IGluZGV4X2luY2x1ZGluZyBpbmRleF9pbmNsdWRp
bmdfZ2lzdAotCi0jIC0tLS0tLS0tLS0KLSMgQW5vdGhlciBncm91cCBvZiBwYXJhbGxlbCB0
ZXN0cwotIyAtLS0tLS0tLS0tCi10ZXN0OiBjcmVhdGVfYWdncmVnYXRlIGNyZWF0ZV9mdW5j
dGlvbl9zcWwgY3JlYXRlX2Nhc3QgY29uc3RyYWludHMgdHJpZ2dlcnMgc2VsZWN0IGluaGVy
aXQgdHlwZWRfdGFibGUgdmFjdXVtIGRyb3BfaWZfZXhpc3RzIHVwZGF0YWJsZV92aWV3cyBy
b2xlYXR0cmlidXRlcyBjcmVhdGVfYW0gaGFzaF9mdW5jIGVycm9ycyBpbmZpbml0ZV9yZWN1
cnNlCi0KLSMgLS0tLS0tLS0tLQotIyBzYW5pdHlfY2hlY2sgZG9lcyBhIHZhY3V1bSwgYWZm
ZWN0aW5nIHRoZSBzb3J0IG9yZGVyIG9mIFNFTEVDVCAqCi0jIHJlc3VsdHMuIFNvIGl0IHNo
b3VsZCBub3QgcnVuIHBhcmFsbGVsIHRvIG90aGVyIHRlc3RzLgotIyAtLS0tLS0tLS0tCi10
ZXN0OiBzYW5pdHlfY2hlY2sKLQotIyAtLS0tLS0tLS0tCi0jIEFub3RoZXIgZ3JvdXAgb2Yg
cGFyYWxsZWwgdGVzdHMKLSMgYWdncmVnYXRlcyBkZXBlbmRzIG9uIGNyZWF0ZV9hZ2dyZWdh
dGUKLSMgam9pbiBkZXBlbmRzIG9uIGNyZWF0ZV9taXNjCi0jIC0tLS0tLS0tLS0KLXRlc3Q6
IHNlbGVjdF9pbnRvIHNlbGVjdF9kaXN0aW5jdCBzZWxlY3RfZGlzdGluY3Rfb24gc2VsZWN0
X2ltcGxpY2l0IHNlbGVjdF9oYXZpbmcgc3Vic2VsZWN0IHVuaW9uIGNhc2Ugam9pbiBhZ2dy
ZWdhdGVzIHRyYW5zYWN0aW9ucyByYW5kb20gcG9ydGFscyBhcnJheXMgYnRyZWVfaW5kZXgg
aGFzaF9pbmRleCB1cGRhdGUgZGVsZXRlIG5hbWVzcGFjZSBwcmVwYXJlZF94YWN0cwotCi0j
IC0tLS0tLS0tLS0KLSMgQW5vdGhlciBncm91cCBvZiBwYXJhbGxlbCB0ZXN0cwotIyAtLS0t
LS0tLS0tCi10ZXN0OiBicmluIGdpbiBnaXN0IHNwZ2lzdCBwcml2aWxlZ2VzIGluaXRfcHJp
dnMgc2VjdXJpdHlfbGFiZWwgY29sbGF0ZSBtYXR2aWV3IGxvY2sgcmVwbGljYV9pZGVudGl0
eSByb3dzZWN1cml0eSBvYmplY3RfYWRkcmVzcyB0YWJsZXNhbXBsZSBncm91cGluZ3NldHMg
ZHJvcF9vcGVyYXRvciBwYXNzd29yZCBpZGVudGl0eSBnZW5lcmF0ZWRfc3RvcmVkIGpvaW5f
aGFzaAotCi0jIC0tLS0tLS0tLS0KLSMgQWRkaXRpb25hbCBCUklOIHRlc3RzCi0jIC0tLS0t
LS0tLS0KLXRlc3Q6IGJyaW5fYmxvb20gYnJpbl9tdWx0aQotCi0jIC0tLS0tLS0tLS0KLSMg
QW5vdGhlciBncm91cCBvZiBwYXJhbGxlbCB0ZXN0cwotIyBwc3FsIGRlcGVuZHMgb24gY3Jl
YXRlX2FtCi0jIGFtdXRpbHMgZGVwZW5kcyBvbiBnZW9tZXRyeSwgY3JlYXRlX2luZGV4X3Nw
Z2lzdCwgaGFzaF9pbmRleCwgYnJpbgotIyAtLS0tLS0tLS0tCi10ZXN0OiBjcmVhdGVfdGFi
bGVfbGlrZSBhbHRlcl9nZW5lcmljIGFsdGVyX29wZXJhdG9yIG1pc2MgYXN5bmMgZGJzaXpl
IG1lcmdlIG1pc2NfZnVuY3Rpb25zIHN5c3ZpZXdzIHRzcmYgdGlkIHRpZHNjYW4gdGlkcmFu
Z2VzY2FuIGNvbGxhdGUudXRmOCBjb2xsYXRlLmljdS51dGY4IGluY3JlbWVudGFsX3NvcnQg
Y3JlYXRlX3JvbGUgd2l0aG91dF9vdmVybGFwcwotCi0jIGNvbGxhdGUubGludXgudXRmOCBh
bmQgY29sbGF0ZS5pY3UudXRmOCB0ZXN0cyBjYW5ub3QgYmUgcnVuIGluIHBhcmFsbGVsIHdp
dGggZWFjaCBvdGhlcgotdGVzdDogcnVsZXMgcHNxbCBwc3FsX2Nyb3NzdGFiIGFtdXRpbHMg
c3RhdHNfZXh0IGNvbGxhdGUubGludXgudXRmOCBjb2xsYXRlLndpbmRvd3Mud2luMTI1Mgot
Ci0jIC0tLS0tLS0tLS0KLSMgUnVuIHRoZXNlIGFsb25lIHNvIHRoZXkgZG9uJ3QgcnVuIG91
dCBvZiBwYXJhbGxlbCB3b3JrZXJzCi0jIHNlbGVjdF9wYXJhbGxlbCBkZXBlbmRzIG9uIGNy
ZWF0ZV9taXNjCi0jIC0tLS0tLS0tLS0KLXRlc3Q6IHNlbGVjdF9wYXJhbGxlbAotdGVzdDog
d3JpdGVfcGFyYWxsZWwKLXRlc3Q6IHZhY3V1bV9wYXJhbGxlbAotCi0jIG5vIHJlbGF0aW9u
IHJlbGF0ZWQgdGVzdHMgY2FuIGJlIHB1dCBpbiB0aGlzIGdyb3VwCi10ZXN0OiBwdWJsaWNh
dGlvbiBzdWJzY3JpcHRpb24KLQotIyAtLS0tLS0tLS0tCi0jIEFub3RoZXIgZ3JvdXAgb2Yg
cGFyYWxsZWwgdGVzdHMKLSMgc2VsZWN0X3ZpZXdzIGRlcGVuZHMgb24gY3JlYXRlX3ZpZXcK
LSMgLS0tLS0tLS0tLQotdGVzdDogc2VsZWN0X3ZpZXdzIHBvcnRhbHNfcDIgZm9yZWlnbl9r
ZXkgY2x1c3RlciBkZXBlbmRlbmN5IGd1YyBiaXRtYXBvcHMgY29tYm9jaWQgdHNlYXJjaCB0
c2RpY3RzIGZvcmVpZ25fZGF0YSB3aW5kb3cgeG1sbWFwIGZ1bmN0aW9uYWxfZGVwcyBhZHZp
c29yeV9sb2NrIGluZGlyZWN0X3RvYXN0IGVxdWl2Y2xhc3MKLQotIyAtLS0tLS0tLS0tCi0j
IEFub3RoZXIgZ3JvdXAgb2YgcGFyYWxsZWwgdGVzdHMgKEpTT04gcmVsYXRlZCkKLSMgLS0t
LS0tLS0tLQotdGVzdDoganNvbiBqc29uYiBqc29uX2VuY29kaW5nIGpzb25wYXRoIGpzb25w
YXRoX2VuY29kaW5nIGpzb25iX2pzb25wYXRoIHNxbGpzb24gc3FsanNvbl9xdWVyeWZ1bmNz
IHNxbGpzb25fanNvbnRhYmxlCi0KLSMgLS0tLS0tLS0tLQotIyBBbm90aGVyIGdyb3VwIG9m
IHBhcmFsbGVsIHRlc3RzCi0jIHdpdGggZGVwZW5kcyBvbiBjcmVhdGVfbWlzYwotIyBOQjog
dGVtcC5zcWwgZG9lcyBhIHJlY29ubmVjdCB3aGljaCB0cmFuc2llbnRseSB1c2VzIDIgY29u
bmVjdGlvbnMsCi0jIHNvIGtlZXAgdGhpcyBwYXJhbGxlbCBncm91cCB0byBhdCBtb3N0IDE5
IHRlc3RzCi0jIC0tLS0tLS0tLS0KLXRlc3Q6IHBsYW5jYWNoZSBsaW1pdCBwbHBnc3FsIGNv
cHkyIHRlbXAgZG9tYWluIHJhbmdlZnVuY3MgcHJlcGFyZSBjb252ZXJzaW9uIHRydW5jYXRl
IGFsdGVyX3RhYmxlIHNlcXVlbmNlIHBvbHltb3JwaGlzbSByb3d0eXBlcyByZXR1cm5pbmcg
bGFyZ2VvYmplY3Qgd2l0aCB4bWwKLQotIyAtLS0tLS0tLS0tCi0jIEFub3RoZXIgZ3JvdXAg
b2YgcGFyYWxsZWwgdGVzdHMKLSMKLSMgVGhlIHN0YXRzIHRlc3QgcmVzZXRzIHN0YXRzLCBz
byBub3RoaW5nIGVsc2UgbmVlZGluZyBzdGF0cyBhY2Nlc3MgY2FuIGJlIGluCi0jIHRoaXMg
Z3JvdXAuCi0jIC0tLS0tLS0tLS0KLXRlc3Q6IHBhcnRpdGlvbl9qb2luIHBhcnRpdGlvbl9w
cnVuZSByZWxvcHRpb25zIGhhc2hfcGFydCBpbmRleGluZyBwYXJ0aXRpb25fYWdncmVnYXRl
IHBhcnRpdGlvbl9pbmZvIHR1cGxlc29ydCBleHBsYWluIGNvbXByZXNzaW9uIG1lbW9pemUg
c3RhdHMgcHJlZGljYXRlCi0KLSMgZXZlbnRfdHJpZ2dlciBkZXBlbmRzIG9uIGNyZWF0ZV9h
bSBhbmQgY2Fubm90IHJ1biBjb25jdXJyZW50bHkgd2l0aAotIyBhbnkgdGVzdCB0aGF0IHJ1
bnMgRERMCi0jIG9pZGpvaW5zIGlzIHJlYWQtb25seSwgdGhvdWdoLCBhbmQgc2hvdWxkIHJ1
biBsYXRlIGZvciBiZXN0IGNvdmVyYWdlCi10ZXN0OiBvaWRqb2lucyBldmVudF90cmlnZ2Vy
Ci0KLSMgZXZlbnRfdHJpZ2dlcl9sb2dpbiBjYW5ub3QgcnVuIGNvbmN1cnJlbnRseSB3aXRo
IGFueSBvdGhlciB0ZXN0cyBiZWNhdXNlCi0jIG9uLWxvZ2luIGV2ZW50IGhhbmRsaW5nIGNv
dWxkIGNhdGNoIGNvbm5lY3Rpb24gb2YgYSBjb25jdXJyZW50IHRlc3QuCi10ZXN0OiBldmVu
dF90cmlnZ2VyX2xvZ2luCi0KLSMgdGhpcyB0ZXN0IGFsc28gdXNlcyBldmVudCB0cmlnZ2Vy
cywgc28gbGlrZXdpc2UgcnVuIGl0IGJ5IGl0c2VsZgotdGVzdDogZmFzdF9kZWZhdWx0Ci0K
LSMgcnVuIHRhYmxlc3BhY2UgdGVzdCBhdCB0aGUgZW5kIGJlY2F1c2UgaXQgZHJvcHMgdGhl
IHRhYmxlc3BhY2UgY3JlYXRlZCBkdXJpbmcKLSMgc2V0dXAgdGhhdCBvdGhlciB0ZXN0cyBt
YXkgdXNlLgotdGVzdDogdGFibGVzcGFjZQordGVzdDogY3JlYXRlX2luZGV4Cit0ZXN0OiBz
dGF0cyBidWZjaHVybgorIyB0ZXN0OiBtaW5pc3RhdHMgYnVmY2h1cm4KZGlmZiAtLWdpdCBh
L3NyYy90ZXN0L3JlZ3Jlc3Mvc3FsL2J1ZmNodXJuLnNxbCBiL3NyYy90ZXN0L3JlZ3Jlc3Mv
c3FsL2J1ZmNodXJuLnNxbApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAw
Li5jMjlhYzRhY2Y5Ci0tLSAvZGV2L251bGwKKysrIGIvc3JjL3Rlc3QvcmVncmVzcy9zcWwv
YnVmY2h1cm4uc3FsCkBAIC0wLDAgKzEsMjIgQEAKK1NFTEVDVCBwZ19zbGVlcCgyKTsKK0NS
RUFURSBUQUJMRSB0MChpIGludCwgdCB0ZXh0KTsKK0FMVEVSIFRBQkxFIHQwIEFMVEVSIHQg
U0VUIFNUT1JBR0UgUExBSU47CitJTlNFUlQgSU5UTyB0MCBTRUxFQ1QgZywgZyB8fCByZXBl
YXQoJ3gnLCA4MDAwKSBGUk9NIGdlbmVyYXRlX3NlcmllcygxLCAyNTYpIGc7CisKK0NSRUFU
RSBUQUJMRSB0MShpIGludCwgdCB0ZXh0KTsKK0FMVEVSIFRBQkxFIHRsIEFMVEVSIHQgU0VU
IFNUT1JBR0UgUExBSU47CitJTlNFUlQgSU5UTyB0MSBTRUxFQ1QgZywgZyB8fCByZXBlYXQo
J3gnLCA4MDAwKSBGUk9NIGdlbmVyYXRlX3NlcmllcygxLCAyNTYpIGc7CisKK0VYUExBSU4g
QU5BTFlaRSBTRUxFQ1QgU1VNKGkpIEZST00gdDAgR1JPVVAgQlkgdDsKK0VYUExBSU4gQU5B
TFlaRSBTRUxFQ1QgU1VNKGkpIEZST00gdDEgR1JPVVAgQlkgdDsKKworXGMgLQorU0VMRUNU
ICdTRUxFQ1QgcGdfc2xlZXAoMSk7JyBGUk9NIGdlbmVyYXRlX3NlcmllcygxLCAxMCkKK1xn
ZXhlYworCitCRUdJTjsKK1NFTEVDVCAnRVhQTEFJTiBBTkFMWVpFIFNFTEVDVCBTVU0oaSkg
RlJPTSB0MDsgRVhQTEFJTiBBTkFMWVpFIFNFTEVDVCBTVU0oaSkgRlJPTSB0MTsnIEZST00g
Z2VuZXJhdGVfc2VyaWVzKDEsIDIwKQorXGdleGVjCisKK1NFTEVDVCBwZ19zbGVlcCgxMCk7
CitST0xMQkFDSzsKZGlmZiAtLWdpdCBhL3NyYy90ZXN0L3JlZ3Jlc3Mvc3FsL3N0YXRzLnNx
bCBiL3NyYy90ZXN0L3JlZ3Jlc3Mvc3FsL3N0YXRzLnNxbAppbmRleCAzOTljNzJiYmNmLi5m
Y2VjMWJlMzc3IDEwMDY0NAotLS0gYS9zcmMvdGVzdC9yZWdyZXNzL3NxbC9zdGF0cy5zcWwK
KysrIGIvc3JjL3Rlc3QvcmVncmVzcy9zcWwvc3RhdHMuc3FsCkBAIC00MjYsNiArNDI2LDcg
QEAgU0VMRUNUIHdhbF9ieXRlcyBBUyB3YWxfYnl0ZXNfYmVmb3JlIEZST00gcGdfc3RhdF93
YWwgXGdzZXQKIENSRUFURSBURU1QIFRBQkxFIHRlc3Rfc3RhdHNfdGVtcCBBUyBTRUxFQ1Qg
MTc7CiBEUk9QIFRBQkxFIHRlc3Rfc3RhdHNfdGVtcDsKIAorU0VMRUNUIHBnX3NsZWVwKDEw
KTsKIC0tIENoZWNrcG9pbnQgdHdpY2U6IFRoZSBjaGVja3BvaW50ZXIgcmVwb3J0cyBzdGF0
cyBhZnRlciByZXBvcnRpbmcgY29tcGxldGlvbgogLS0gb2YgdGhlIGNoZWNrcG9pbnQuIEJ1
dCBhZnRlciBhIHNlY29uZCBjaGVja3BvaW50IHdlJ2xsIHNlZSBhdCBsZWFzdCB0aGUKIC0t
IHJlc3VsdHMgb2YgdGhlIGZpcnN0LgpAQCAtNjMyLDYgKzYzMyw3IEBAIFNFTEVDVCBzdW0o
ZXh0ZW5kcykgQVMgbXlfaW9fc3VtX3NoYXJlZF9hZnRlcl9leHRlbmRzCiAgIFdIRVJFIGNv
bnRleHQgPSAnbm9ybWFsJyBBTkQgb2JqZWN0ID0gJ3JlbGF0aW9uJyBcZ3NldAogU0VMRUNU
IDpteV9pb19zdW1fc2hhcmVkX2FmdGVyX2V4dGVuZHMgPiA6bXlfaW9fc3VtX3NoYXJlZF9i
ZWZvcmVfZXh0ZW5kczsKIAorU0VMRUNUIHBnX3NsZWVwKDEwKTsKIC0tIEFmdGVyIGEgY2hl
Y2twb2ludCwgdGhlcmUgc2hvdWxkIGJlIHNvbWUgYWRkaXRpb25hbCBJT0NPTlRFWFRfTk9S
TUFMIHdyaXRlcwogLS0gYW5kIGZzeW5jcyBpbiB0aGUgZ2xvYmFsIHN0YXRzICh1c3VhbGx5
IG5vdCBmb3IgdGhlIGJhY2tlbmQpLgogLS0gU2VlIGNvbW1lbnQgYWJvdmUgZm9yIHJhdGlv
bmFsZSBmb3IgdHdvIGV4cGxpY2l0IENIRUNLUE9JTlRzLgo=

--------------3mxxZinpeDglB15YlXniAwGl--