Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tZsdZ-00DxhL-I7 for pgsql-hackers@arkaria.postgresql.org; Mon, 20 Jan 2025 14:20:22 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tZsdX-002b73-9f for pgsql-hackers@arkaria.postgresql.org; Mon, 20 Jan 2025 14:20:19 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tZsdW-002b6u-SS for pgsql-hackers@lists.postgresql.org; Mon, 20 Jan 2025 14:20:19 +0000 Received: from forward103b.mail.yandex.net ([2a02:6b8:c02:900:1:45:d181:d103]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1tZsdU-000Xwk-07 for pgsql-hackers@postgresql.org; Mon, 20 Jan 2025 14:20:18 +0000 Received: from mail-nwsmtp-smtp-production-main-91.sas.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-91.sas.yp-c.yandex.net [IPv6:2a02:6b8:c10:2d9f:0:640:f6ce:0]) by forward103b.mail.yandex.net (Yandex) with ESMTPS id 2282F60D59; Mon, 20 Jan 2025 17:20:15 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-91.sas.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id BKWATefOj8c0-iZR8jZiL; Mon, 20 Jan 2025 17:20:14 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tantorlabs.com; s=mail; t=1737382814; bh=BPK5iyCTExomOHRQPJS2gXusn/yIQkexIyxiUTo3khs=; h=In-Reply-To:Cc:Date:References:To:Subject:Message-ID:From; b=a03B9EvbZ8pzfaEDxN7Noat2za3QZfFWk2H8GrOKuDmCj8nHjB3M+sZr9tyvcdxW2 UlsNsy3UZBM/7RtCcTP4LLXMIyT8i2Q++1SiAyw9brq3IJyVtegv04z7/1lCwqumjY IIutzglJtlf5RJgkO3I+Bxw5K39IFgxM8jFh26X0= Authentication-Results: mail-nwsmtp-smtp-production-main-91.sas.yp-c.yandex.net; dkim=pass header.i=@tantorlabs.com Content-Type: multipart/mixed; boundary="------------tIrVj4iKOkX9Kk5WVkzHHVhB" Message-ID: <7da368b6-e44f-47bd-891e-236bfb51bd26@tantorlabs.com> Date: Mon, 20 Jan 2025 17:20:10 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Sample rate added to pg_stat_statements To: Sami Imseih Cc: "Andrey M. Borodin" , Alena Rybakina , Alexander Korotkov , Michael Paquier , Greg Sabino Mullane , pgsql-hackers References: <320a31e6-4c91-438b-ab17-8a1d72384727@tantorlabs.com> <52e5e3aa-066a-4daf-8d65-77c34a2f0d98@tantorlabs.com> <6482A576-4465-4E46-A42C-F1767E8DDE30@yandex-team.ru> <20994d10-ab44-4c6b-91bc-c4675cd2e58e@tantorlabs.com> <9D8A5BC8-A3A5-4DA9-9382-F30D343247D1@yandex-team.ru> <6cf6270b-e8fa-4d99-b74a-17e781cdf309@tantorlabs.com> <3a27c830-b81f-4560-9774-a87d9b922fbd@tantorlabs.com> Content-Language: en-US From: Ilia Evdokimov In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------tIrVj4iKOkX9Kk5WVkzHHVhB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 15.01.2025 20:16, Sami Imseih wrote: >> Probably, but first I suggest benchmarking with sampling applied to all queries. If the results are good, we can later filter certain queries based on different characteristics. > Absolutely. The benchmark numbers to justify this feature are > the next step. Thanks for your work on this! > > Regards, > > Sami > Hi hackers, I’d like to share the results of my benchmark. To stress the spinlock in pg_stat_statements while incrementing counters for popular entries, it's important to use the same query repeatedly. To avoid overloading pgss with normalization, the queries should not contain constants. I found that using the query 'SELECT now()' works best for this purpose. I ran the benchmark on a machine with 48 CPUs, which may not be sufficient to fully test sampling, but I was able to achieve around 1.5 million TPS using 'SELECT now()'. To load the CPUs to about 85-90%, I ran pgbench with -c 45 -j 45, using a custom 'SELECT now()' in pgbench_script.sql file containing 'SELECT now()'. The benchmark was conducted on a newly created 'pgbench' database, , with processes like autovacuum, fsync, and checkpoints disabled(e.g., checkpoint _timeout = '24h'). I tested various 'sample_rate' values and compared the performance with pgss both enabled and disabled. The detailed results are attached in the 'sample_rate.txt' file, along with pgbench progress reports every 10 seconds. pgbench -c 45 -j 45 -T 200 -f pgbench_script.sql --progress=10 pgbench sample_rate | num of transactions | lat | stddev | tps         1.0 |      1207127.572580 |0.037|  0.030 | 241400836         0.8 |      1403551.516338 |0.032|  0.031 | 280673286         0.5 |      1658596.614064 |0.027|  0.012 | 331679344         0.2 |      1757747.509476 |0.025|  0.008 | 351507156         0.0 |      1760055.986259 |0.025|  0.008 | 351965559    pgss off |      1828743.791205 |0.024|  0.008 | 365703762 If anyone has the capability to run this benchmark on machines with more CPUs or with different queries, it would be nice. I’d appreciate any suggestions or feedback. --. Best regards, Ilia Evdokimov, Tantor Labs LLC. --------------tIrVj4iKOkX9Kk5WVkzHHVhB Content-Type: text/plain; charset=UTF-8; name="sample_rate.txt" Content-Disposition: attachment; filename="sample_rate.txt" Content-Transfer-Encoding: base64 cGdfc3RhdF9zdGF0ZW1lbnRzLnNhbXBsZV9yYXRlID0gMS4wCgpjcHUgPSA4NSUKCnBnYmVu Y2ggLWMgNDUgLWogNDUgLVQgMjAwIC1mIHBnYmVuY2hfc2NyaXB0LnNxbCAtLXByb2dyZXNz PTEwIHBnYmVuY2gKcGdiZW5jaCAoMTYuNikKc3RhcnRpbmcgdmFjdXVtLi4uZW5kLgpwcm9n cmVzczogMTAuMCBzLCAxMjAwNDQ1LjEgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDM4 LCAwIGZhaWxlZApwcm9ncmVzczogMjAuMCBzLCAxMjMyNjM0LjUgdHBzLCBsYXQgMC4wMzYg bXMgc3RkZGV2IDAuMDMwLCAwIGZhaWxlZApwcm9ncmVzczogMzAuMCBzLCAxMjA5NzYwLjcg dHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI3LCAwIGZhaWxlZApwcm9ncmVzczogNDAu MCBzLCAxMjA0NjI1LjUgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI2LCAwIGZhaWxl ZApwcm9ncmVzczogNTAuMCBzLCAxMjA2NjA4LjcgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2 IDAuMDI3LCAwIGZhaWxlZApwcm9ncmVzczogNjAuMCBzLCAxMjE0MDQ3LjkgdHBzLCBsYXQg MC4wMzcgbXMgc3RkZGV2IDAuMDM0LCAwIGZhaWxlZApwcm9ncmVzczogNzAuMCBzLCAxMjIy MTI2LjUgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDM2LCAwIGZhaWxlZApwcm9ncmVz czogODAuMCBzLCAxMTk4NDUyLjkgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDMwLCAw IGZhaWxlZApwcm9ncmVzczogOTAuMCBzLCAxMTk4ODcxLjcgdHBzLCBsYXQgMC4wMzcgbXMg c3RkZGV2IDAuMDI3LCAwIGZhaWxlZApwcm9ncmVzczogMTAwLjAgcywgMTIwMDc4Ni45IHRw cywgbGF0IDAuMDM3IG1zIHN0ZGRldiAwLjAyOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDExMC4w IHMsIDEyMDA0MTkuNSB0cHMsIGxhdCAwLjAzNyBtcyBzdGRkZXYgMC4wMzAsIDAgZmFpbGVk CnByb2dyZXNzOiAxMjAuMCBzLCAxMjA4Mzk0LjQgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2 IDAuMDMwLCAwIGZhaWxlZApwcm9ncmVzczogMTMwLjAgcywgMTE5OTIwNC4zIHRwcywgbGF0 IDAuMDM3IG1zIHN0ZGRldiAwLjAyOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE0MC4wIHMsIDEy MTM5MTkuNiB0cHMsIGxhdCAwLjAzNyBtcyBzdGRkZXYgMC4wMzcsIDAgZmFpbGVkCnByb2dy ZXNzOiAxNTAuMCBzLCAxMTk1NjcwLjEgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI4 LCAwIGZhaWxlZApwcm9ncmVzczogMTYwLjAgcywgMTE5MDUxOS4yIHRwcywgbGF0IDAuMDM4 IG1zIHN0ZGRldiAwLjAyOSwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE3MC4wIHMsIDEyMTM0NTAu MSB0cHMsIGxhdCAwLjAzNyBtcyBzdGRkZXYgMC4wMjcsIDAgZmFpbGVkCnByb2dyZXNzOiAx ODAuMCBzLCAxMjIwNDMzLjAgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI5LCAwIGZh aWxlZApwcm9ncmVzczogMTkwLjAgcywgMTE5OTYyOC43IHRwcywgbGF0IDAuMDM3IG1zIHN0 ZGRldiAwLjAyOCwgMCBmYWlsZWQKdHJhbnNhY3Rpb24gdHlwZTogcGdiZW5jaF9zY3JpcHQu c3FsCnNjYWxpbmcgZmFjdG9yOiAxCnF1ZXJ5IG1vZGU6IHNpbXBsZQpudW1iZXIgb2YgY2xp ZW50czogNDUKbnVtYmVyIG9mIHRocmVhZHM6IDQ1Cm1heGltdW0gbnVtYmVyIG9mIHRyaWVz OiAxCmR1cmF0aW9uOiAyMDAgcwpudW1iZXIgb2YgdHJhbnNhY3Rpb25zIGFjdHVhbGx5IHBy b2Nlc3NlZDogMjQxNDAwODM2Cm51bWJlciBvZiBmYWlsZWQgdHJhbnNhY3Rpb25zOiAwICgw LjAwMCUpCmxhdGVuY3kgYXZlcmFnZSA9IDAuMDM3IG1zCmxhdGVuY3kgc3RkZGV2ID0gMC4w MzAgbXMKaW5pdGlhbCBjb25uZWN0aW9uIHRpbWUgPSAyNS4wNTEgbXMKdHBzID0gMTIwNzEy Ny41NzI1ODAgKHdpdGhvdXQgaW5pdGlhbCBjb25uZWN0aW9uIHRpbWUpCgoKPT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PQoKcGdfc3RhdF9zdGF0ZW1lbnRzLnNhbXBsZV9yYXRlID0gMC44CgpjcHUgPSA5 MCUKCnBnYmVuY2ggLWMgNDUgLWogNDUgLVQgMjAwIC1mIHBnYmVuY2hfc2NyaXB0LnNxbCAt LXByb2dyZXNzPTEwIHBnYmVuY2gKcGdiZW5jaCAoMTYuNikKc3RhcnRpbmcgdmFjdXVtLi4u ZW5kLgpwcm9ncmVzczogMTAuMCBzLCAxNDI5OTIyLjggdHBzLCBsYXQgMC4wMzEgbXMgc3Rk ZGV2IDAuMDI4LCAwIGZhaWxlZApwcm9ncmVzczogMjAuMCBzLCAxMjAwMzgwLjIgdHBzLCBs YXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI5LCAwIGZhaWxlZApwcm9ncmVzczogMzAuMCBzLCAx MTk4MTAyLjYgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI3LCAwIGZhaWxlZApwcm9n cmVzczogNDAuMCBzLCAxMjI1OTEyLjkgdHBzLCBsYXQgMC4wMzcgbXMgc3RkZGV2IDAuMDI4 LCAwIGZhaWxlZApwcm9ncmVzczogNTAuMCBzLCAxMjM0MDU2LjIgdHBzLCBsYXQgMC4wMzYg bXMgc3RkZGV2IDAuMDI3LCAwIGZhaWxlZApwcm9ncmVzczogNjAuMCBzLCAxMjM0ODk0LjYg dHBzLCBsYXQgMC4wMzYgbXMgc3RkZGV2IDAuMDI3LCAwIGZhaWxlZApwcm9ncmVzczogNzAu MCBzLCAxMjM1MDMyLjkgdHBzLCBsYXQgMC4wMzYgbXMgc3RkZGV2IDAuMDI3LCAwIGZhaWxl ZApwcm9ncmVzczogODAuMCBzLCAxNTQ4NjA3LjkgdHBzLCBsYXQgMC4wMjkgbXMgc3RkZGV2 IDAuMDM1LCAwIGZhaWxlZApwcm9ncmVzczogOTAuMCBzLCAxNTUzOTY2LjkgdHBzLCBsYXQg MC4wMjkgbXMgc3RkZGV2IDAuMDMxLCAwIGZhaWxlZApwcm9ncmVzczogMTAwLjAgcywgMTM1 NDY5NS4yIHRwcywgbGF0IDAuMDMzIG1zIHN0ZGRldiAwLjAzMCwgMCBmYWlsZWQKcHJvZ3Jl c3M6IDExMC4wIHMsIDEyMzg5OTAuNCB0cHMsIGxhdCAwLjAzNiBtcyBzdGRkZXYgMC4wMjcs IDAgZmFpbGVkCnByb2dyZXNzOiAxMjAuMCBzLCAxMjQ2MTY5LjcgdHBzLCBsYXQgMC4wMzYg bXMgc3RkZGV2IDAuMDI3LCAwIGZhaWxlZApwcm9ncmVzczogMTMwLjAgcywgMTIzNDU2Mi41 IHRwcywgbGF0IDAuMDM2IG1zIHN0ZGRldiAwLjAyOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE0 MC4wIHMsIDE1NDQ0MjMuMiB0cHMsIGxhdCAwLjAyOSBtcyBzdGRkZXYgMC4wMzMsIDAgZmFp bGVkCnByb2dyZXNzOiAxNTAuMCBzLCAxNTk0MzIxLjcgdHBzLCBsYXQgMC4wMjggbXMgc3Rk ZGV2IDAuMDM0LCAwIGZhaWxlZApwcm9ncmVzczogMTYwLjAgcywgMTU5NzcxMS4yIHRwcywg bGF0IDAuMDI4IG1zIHN0ZGRldiAwLjAzNSwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE3MC4wIHMs IDE2MDU2NjkuNiB0cHMsIGxhdCAwLjAyOCBtcyBzdGRkZXYgMC4wMzQsIDAgZmFpbGVkCnBy b2dyZXNzOiAxODAuMCBzLCAxNTk1Mjc2LjUgdHBzLCBsYXQgMC4wMjggbXMgc3RkZGV2IDAu MDMzLCAwIGZhaWxlZApwcm9ncmVzczogMTkwLjAgcywgMTYwMDIzMy4wIHRwcywgbGF0IDAu MDI4IG1zIHN0ZGRldiAwLjAzMywgMCBmYWlsZWQKdHJhbnNhY3Rpb24gdHlwZTogcGdiZW5j aF9zY3JpcHQuc3FsCnNjYWxpbmcgZmFjdG9yOiAxCnF1ZXJ5IG1vZGU6IHNpbXBsZQpudW1i ZXIgb2YgY2xpZW50czogNDUKbnVtYmVyIG9mIHRocmVhZHM6IDQ1Cm1heGltdW0gbnVtYmVy IG9mIHRyaWVzOiAxCmR1cmF0aW9uOiAyMDAgcwpudW1iZXIgb2YgdHJhbnNhY3Rpb25zIGFj dHVhbGx5IHByb2Nlc3NlZDogMjgwNjczMjg2Cm51bWJlciBvZiBmYWlsZWQgdHJhbnNhY3Rp b25zOiAwICgwLjAwMCUpCmxhdGVuY3kgYXZlcmFnZSA9IDAuMDMyIG1zCmxhdGVuY3kgc3Rk ZGV2ID0gMC4wMzEgbXMKaW5pdGlhbCBjb25uZWN0aW9uIHRpbWUgPSAyNy40ODIgbXMKdHBz ID0gMTQwMzU1MS41MTYzMzggKHdpdGhvdXQgaW5pdGlhbCBjb25uZWN0aW9uIHRpbWUpCgo9 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09CgpwZ19zdGF0X3N0YXRlbWVudHMuc2FtcGxlX3JhdGUgPSAwLjUK CmNwdSA9IDkzJQoKcGdiZW5jaCAtYyA0NSAtaiA0NSAtVCAyMDAgLWYgcGdiZW5jaF9zY3Jp cHQuc3FsIC0tcHJvZ3Jlc3M9MTAgcGdiZW5jaApwZ2JlbmNoICgxNi42KQpzdGFydGluZyB2 YWN1dW0uLi5lbmQuCnByb2dyZXNzOiAxMC4wIHMsIDE2Nzg2OTMuNCB0cHMsIGxhdCAwLjAy NyBtcyBzdGRkZXYgMC4wMTIsIDAgZmFpbGVkCnByb2dyZXNzOiAyMC4wIHMsIDE2ODgzMjcu MCB0cHMsIGxhdCAwLjAyNiBtcyBzdGRkZXYgMC4wMTAsIDAgZmFpbGVkCnByb2dyZXNzOiAz MC4wIHMsIDE2OTU5NTMuMCB0cHMsIGxhdCAwLjAyNiBtcyBzdGRkZXYgMC4wMTAsIDAgZmFp bGVkCnByb2dyZXNzOiA0MC4wIHMsIDE3MTEyNjUuOCB0cHMsIGxhdCAwLjAyNiBtcyBzdGRk ZXYgMC4wMTEsIDAgZmFpbGVkCnByb2dyZXNzOiA1MC4wIHMsIDE2ODc4NjUuMyB0cHMsIGxh dCAwLjAyNiBtcyBzdGRkZXYgMC4wMTEsIDAgZmFpbGVkCnByb2dyZXNzOiA2MC4wIHMsIDE2 ODI4NzMuMyB0cHMsIGxhdCAwLjAyNyBtcyBzdGRkZXYgMC4wMTAsIDAgZmFpbGVkCnByb2dy ZXNzOiA3MC4wIHMsIDE2ODMyMjEuNyB0cHMsIGxhdCAwLjAyNyBtcyBzdGRkZXYgMC4wMTEs IDAgZmFpbGVkCnByb2dyZXNzOiA4MC4wIHMsIDE2ODg4NzEuMiB0cHMsIGxhdCAwLjAyNiBt cyBzdGRkZXYgMC4wMTAsIDAgZmFpbGVkCnByb2dyZXNzOiA5MC4wIHMsIDE2OTc2MzMuMyB0 cHMsIGxhdCAwLjAyNiBtcyBzdGRkZXYgMC4wMTAsIDAgZmFpbGVkCnByb2dyZXNzOiAxMDAu MCBzLCAxNjkwMDA0LjUgdHBzLCBsYXQgMC4wMjYgbXMgc3RkZGV2IDAuMDEyLCAwIGZhaWxl ZApwcm9ncmVzczogMTEwLjAgcywgMTY3Njk0MS4xIHRwcywgbGF0IDAuMDI3IG1zIHN0ZGRl diAwLjAxMSwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDEyMC4wIHMsIDE2OTI2NTkuMCB0cHMsIGxh dCAwLjAyNiBtcyBzdGRkZXYgMC4wMTAsIDAgZmFpbGVkCnByb2dyZXNzOiAxMzAuMCBzLCAx NjkwNjYwLjAgdHBzLCBsYXQgMC4wMjYgbXMgc3RkZGV2IDAuMDExLCAwIGZhaWxlZApwcm9n cmVzczogMTQwLjAgcywgMTcxMDk4NC45IHRwcywgbGF0IDAuMDI2IG1zIHN0ZGRldiAwLjAw OSwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE1MC4wIHMsIDE2ODYyODUuMCB0cHMsIGxhdCAwLjAy NiBtcyBzdGRkZXYgMC4wMTAsIDAgZmFpbGVkCnByb2dyZXNzOiAxNjAuMCBzLCAxNjkxNjk0 LjMgdHBzLCBsYXQgMC4wMjYgbXMgc3RkZGV2IDAuMDEyLCAwIGZhaWxlZApwcm9ncmVzczog MTcwLjAgcywgMTcxODQzMS4xIHRwcywgbGF0IDAuMDI2IG1zIHN0ZGRldiAwLjAxMSwgMCBm YWlsZWQKcHJvZ3Jlc3M6IDE4MC4wIHMsIDE2NzE2MzQuMiB0cHMsIGxhdCAwLjAyNyBtcyBz dGRkZXYgMC4wMTEsIDAgZmFpbGVkCnByb2dyZXNzOiAxOTAuMCBzLCAxNDgxNTY0LjYgdHBz LCBsYXQgMC4wMzAgbXMgc3RkZGV2IDAuMDE5LCAwIGZhaWxlZAp0cmFuc2FjdGlvbiB0eXBl OiBwZ2JlbmNoX3NjcmlwdC5zcWwKc2NhbGluZyBmYWN0b3I6IDEKcXVlcnkgbW9kZTogc2lt cGxlCm51bWJlciBvZiBjbGllbnRzOiA0NQpudW1iZXIgb2YgdGhyZWFkczogNDUKbWF4aW11 bSBudW1iZXIgb2YgdHJpZXM6IDEKZHVyYXRpb246IDIwMCBzCm51bWJlciBvZiB0cmFuc2Fj dGlvbnMgYWN0dWFsbHkgcHJvY2Vzc2VkOiAzMzE2NzkzNDQKbnVtYmVyIG9mIGZhaWxlZCB0 cmFuc2FjdGlvbnM6IDAgKDAuMDAwJSkKbGF0ZW5jeSBhdmVyYWdlID0gMC4wMjcgbXMKbGF0 ZW5jeSBzdGRkZXYgPSAwLjAxMiBtcwppbml0aWFsIGNvbm5lY3Rpb24gdGltZSA9IDI1LjEx NyBtcwp0cHMgPSAxNjU4NTk2LjYxNDA2NCAod2l0aG91dCBpbml0aWFsIGNvbm5lY3Rpb24g dGltZSkKCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT0KCnBnX3N0YXRfc3RhdGVtZW50cy5zYW1wbGVfcmF0 ZSA9IDAuMgoKY3B1ID0gOTQlCgpwZ2JlbmNoIC1jIDQ1IC1qIDQ1IC1UIDIwMCAtZiBwZ2Jl bmNoX3NjcmlwdC5zcWwgLS1wcm9ncmVzcz0xMCBwZ2JlbmNoCnBnYmVuY2ggKDE2LjYpCnN0 YXJ0aW5nIHZhY3V1bS4uLmVuZC4KcHJvZ3Jlc3M6IDEwLjAgcywgMTY5Mzk1Ny4xIHRwcywg bGF0IDAuMDI2IG1zIHN0ZGRldiAwLjAxMywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDIwLjAgcywg MTc2ODMyOC41IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAwNywgMCBmYWlsZWQKcHJv Z3Jlc3M6IDMwLjAgcywgMTc3ODU3Ny44IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAw NywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDQwLjAgcywgMTc4NzUzNC4wIHRwcywgbGF0IDAuMDI1 IG1zIHN0ZGRldiAwLjAwNywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDUwLjAgcywgMTc4MTk4My41 IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAwNywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDYw LjAgcywgMTc2NzI1Mi4xIHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAwOCwgMCBmYWls ZWQKcHJvZ3Jlc3M6IDcwLjAgcywgMTc2MTIwMi42IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRl diAwLjAwNywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDgwLjAgcywgMTc3OTMxOC4wIHRwcywgbGF0 IDAuMDI1IG1zIHN0ZGRldiAwLjAwOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDkwLjAgcywgMTc2 OTY0NC43IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAxMCwgMCBmYWlsZWQKcHJvZ3Jl c3M6IDEwMC4wIHMsIDE3NDA4NzIuOCB0cHMsIGxhdCAwLjAyNiBtcyBzdGRkZXYgMC4wMDks IDAgZmFpbGVkCnByb2dyZXNzOiAxMTAuMCBzLCAxNzYxOTg0LjggdHBzLCBsYXQgMC4wMjUg bXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogMTIwLjAgcywgMTcyOTUzMi4x IHRwcywgbGF0IDAuMDI2IG1zIHN0ZGRldiAwLjAwOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDEz MC4wIHMsIDE3NTY4MTkuNSB0cHMsIGxhdCAwLjAyNSBtcyBzdGRkZXYgMC4wMDcsIDAgZmFp bGVkCnByb2dyZXNzOiAxNDAuMCBzLCAxNzI3ODc0LjMgdHBzLCBsYXQgMC4wMjYgbXMgc3Rk ZGV2IDAuMDA5LCAwIGZhaWxlZApwcm9ncmVzczogMTUwLjAgcywgMTczNDE3OC43IHRwcywg bGF0IDAuMDI2IG1zIHN0ZGRldiAwLjAxMCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE2MC4wIHMs IDE3NjU2MDEuOCB0cHMsIGxhdCAwLjAyNSBtcyBzdGRkZXYgMC4wMDcsIDAgZmFpbGVkCnBy b2dyZXNzOiAxNzAuMCBzLCAxNzY0OTk2LjQgdHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAu MDA5LCAwIGZhaWxlZApwcm9ncmVzczogMTgwLjAgcywgMTc4MzA0Mi42IHRwcywgbGF0IDAu MDI1IG1zIHN0ZGRldiAwLjAwOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE5MC4wIHMsIDE3NTA3 MTkuNiB0cHMsIGxhdCAwLjAyNiBtcyBzdGRkZXYgMC4wMDgsIDAgZmFpbGVkCnRyYW5zYWN0 aW9uIHR5cGU6IHBnYmVuY2hfc2NyaXB0LnNxbApzY2FsaW5nIGZhY3RvcjogMQpxdWVyeSBt b2RlOiBzaW1wbGUKbnVtYmVyIG9mIGNsaWVudHM6IDQ1Cm51bWJlciBvZiB0aHJlYWRzOiA0 NQptYXhpbXVtIG51bWJlciBvZiB0cmllczogMQpkdXJhdGlvbjogMjAwIHMKbnVtYmVyIG9m IHRyYW5zYWN0aW9ucyBhY3R1YWxseSBwcm9jZXNzZWQ6IDM1MTUwNzE1NgpudW1iZXIgb2Yg ZmFpbGVkIHRyYW5zYWN0aW9uczogMCAoMC4wMDAlKQpsYXRlbmN5IGF2ZXJhZ2UgPSAwLjAy NSBtcwpsYXRlbmN5IHN0ZGRldiA9IDAuMDA4IG1zCmluaXRpYWwgY29ubmVjdGlvbiB0aW1l ID0gMjUuMTY2IG1zCnRwcyA9IDE3NTc3NDcuNTA5NDc2ICh3aXRob3V0IGluaXRpYWwgY29u bmVjdGlvbiB0aW1lKQoKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQoKcGdfc3RhdF9zdGF0ZW1lbnRzLnNh bXBsZV9yYXRlID0gMC4wCgpjcHUgPSA5NCUKCnBnYmVuY2ggLWMgNDUgLWogNDUgLVQgMjAw IC1mIHBnYmVuY2hfc2NyaXB0LnNxbCAtLXByb2dyZXNzPTEwIHBnYmVuY2gKcGdiZW5jaCAo MTYuNikKc3RhcnRpbmcgdmFjdXVtLi4uZW5kLgpwcm9ncmVzczogMTAuMCBzLCAxNzMwODYw LjkgdHBzLCBsYXQgMC4wMjYgbXMgc3RkZGV2IDAuMDExLCAwIGZhaWxlZApwcm9ncmVzczog MjAuMCBzLCAxNzg5MDA2LjkgdHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZh aWxlZApwcm9ncmVzczogMzAuMCBzLCAxNzQ2NDIxLjcgdHBzLCBsYXQgMC4wMjYgbXMgc3Rk ZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogNDAuMCBzLCAxNzYxNDIwLjEgdHBzLCBs YXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogNTAuMCBzLCAx NzQ5NjU0LjUgdHBzLCBsYXQgMC4wMjYgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9n cmVzczogNjAuMCBzLCAxNzUzNjUyLjYgdHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA4 LCAwIGZhaWxlZApwcm9ncmVzczogNzAuMCBzLCAxNzM3NTMxLjUgdHBzLCBsYXQgMC4wMjYg bXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogODAuMCBzLCAxNzcyMzYyLjIg dHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogOTAu MCBzLCAxNzQzMDI1LjAgdHBzLCBsYXQgMC4wMjYgbXMgc3RkZGV2IDAuMDA4LCAwIGZhaWxl ZApwcm9ncmVzczogMTAwLjAgcywgMTc2ODExNS43IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRl diAwLjAwOCwgMCBmYWlsZWQKcHJvZ3Jlc3M6IDExMC4wIHMsIDE3NjgxNDkuOSB0cHMsIGxh dCAwLjAyNSBtcyBzdGRkZXYgMC4wMDcsIDAgZmFpbGVkCnByb2dyZXNzOiAxMjAuMCBzLCAx NzY3Mjc0LjIgdHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9n cmVzczogMTMwLjAgcywgMTczNzQwOS4xIHRwcywgbGF0IDAuMDI2IG1zIHN0ZGRldiAwLjAw NywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDE0MC4wIHMsIDE3NjQ4ODkuOCB0cHMsIGxhdCAwLjAy NSBtcyBzdGRkZXYgMC4wMDgsIDAgZmFpbGVkCnByb2dyZXNzOiAxNTAuMCBzLCAxNzU5MDE0 LjAgdHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA5LCAwIGZhaWxlZApwcm9ncmVzczog MTYwLjAgcywgMTc2NTI5Ni45IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAxMCwgMCBm YWlsZWQKcHJvZ3Jlc3M6IDE3MC4wIHMsIDE3NjI5NTkuMiB0cHMsIGxhdCAwLjAyNSBtcyBz dGRkZXYgMC4wMDgsIDAgZmFpbGVkCnByb2dyZXNzOiAxODAuMCBzLCAxNzY3MTQ1LjMgdHBz LCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogMTkwLjAg cywgMTc3MjUxOS44IHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAwOCwgMCBmYWlsZWQK dHJhbnNhY3Rpb24gdHlwZTogcGdiZW5jaF9zY3JpcHQuc3FsCnNjYWxpbmcgZmFjdG9yOiAx CnF1ZXJ5IG1vZGU6IHNpbXBsZQpudW1iZXIgb2YgY2xpZW50czogNDUKbnVtYmVyIG9mIHRo cmVhZHM6IDQ1Cm1heGltdW0gbnVtYmVyIG9mIHRyaWVzOiAxCmR1cmF0aW9uOiAyMDAgcwpu dW1iZXIgb2YgdHJhbnNhY3Rpb25zIGFjdHVhbGx5IHByb2Nlc3NlZDogMzUxOTY1NTU5Cm51 bWJlciBvZiBmYWlsZWQgdHJhbnNhY3Rpb25zOiAwICgwLjAwMCUpCmxhdGVuY3kgYXZlcmFn ZSA9IDAuMDI1IG1zCmxhdGVuY3kgc3RkZGV2ID0gMC4wMDggbXMKaW5pdGlhbCBjb25uZWN0 aW9uIHRpbWUgPSAyNi45NDQgbXMKdHBzID0gMTc2MDA1NS45ODYyNTkgKHdpdGhvdXQgaW5p dGlhbCBjb25uZWN0aW9uIHRpbWUpCgo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09CgpwZ19zdGF0X3N0YXRl bWVudHMgaXMgb2ZmCgpjcHUgPSA5NSUKCnBnYmVuY2ggLWMgNDUgLWogNDUgLVQgMjAwIC1m IHBnYmVuY2hfc2NyaXB0LnNxbCAtLXByb2dyZXNzPTEwIHBnYmVuY2gKcGdiZW5jaCAoMTYu NikKc3RhcnRpbmcgdmFjdXVtLi4uZW5kLgpwcm9ncmVzczogMTAuMCBzLCAxNzcxMjI2LjQg dHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDEzLCAwIGZhaWxlZApwcm9ncmVzczogMjAu MCBzLCAxODczMDAzLjkgdHBzLCBsYXQgMC4wMjQgbXMgc3RkZGV2IDAuMDA2LCAwIGZhaWxl ZApwcm9ncmVzczogMzAuMCBzLCAxODUyODQ5LjMgdHBzLCBsYXQgMC4wMjQgbXMgc3RkZGV2 IDAuMDA4LCAwIGZhaWxlZApwcm9ncmVzczogNDAuMCBzLCAxODYzNDExLjEgdHBzLCBsYXQg MC4wMjQgbXMgc3RkZGV2IDAuMDA2LCAwIGZhaWxlZApwcm9ncmVzczogNTAuMCBzLCAxODc3 MjE5LjIgdHBzLCBsYXQgMC4wMjQgbXMgc3RkZGV2IDAuMDA2LCAwIGZhaWxlZApwcm9ncmVz czogNjAuMCBzLCAxODI2MzgxLjUgdHBzLCBsYXQgMC4wMjQgbXMgc3RkZGV2IDAuMDA3LCAw IGZhaWxlZApwcm9ncmVzczogNzAuMCBzLCAxODMwNTYzLjcgdHBzLCBsYXQgMC4wMjQgbXMg c3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogODAuMCBzLCAxODM2ODIyLjMgdHBz LCBsYXQgMC4wMjQgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogOTAuMCBz LCAxODQ0OTY5LjIgdHBzLCBsYXQgMC4wMjQgbXMgc3RkZGV2IDAuMDA4LCAwIGZhaWxlZApw cm9ncmVzczogMTAwLjAgcywgMTgzMjE2Ny44IHRwcywgbGF0IDAuMDI0IG1zIHN0ZGRldiAw LjAwNywgMCBmYWlsZWQKcHJvZ3Jlc3M6IDExMC4wIHMsIDE4MzA2MDMuOSB0cHMsIGxhdCAw LjAyNCBtcyBzdGRkZXYgMC4wMDgsIDAgZmFpbGVkCnByb2dyZXNzOiAxMjAuMCBzLCAxODAw OTg3LjIgdHBzLCBsYXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVz czogMTMwLjAgcywgMTgyMjg3Ny42IHRwcywgbGF0IDAuMDI0IG1zIHN0ZGRldiAwLjAwNywg MCBmYWlsZWQKcHJvZ3Jlc3M6IDE0MC4wIHMsIDE4NDM4OTEuMiB0cHMsIGxhdCAwLjAyNCBt cyBzdGRkZXYgMC4wMDcsIDAgZmFpbGVkCnByb2dyZXNzOiAxNTAuMCBzLCAxODIxODQ1LjIg dHBzLCBsYXQgMC4wMjQgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogMTYw LjAgcywgMTc5NDc0OC4zIHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAwOSwgMCBmYWls ZWQKcHJvZ3Jlc3M6IDE3MC4wIHMsIDE4MTMxOTEuMSB0cHMsIGxhdCAwLjAyNSBtcyBzdGRk ZXYgMC4wMDcsIDAgZmFpbGVkCnByb2dyZXNzOiAxODAuMCBzLCAxODE0Njc1LjAgdHBzLCBs YXQgMC4wMjUgbXMgc3RkZGV2IDAuMDA3LCAwIGZhaWxlZApwcm9ncmVzczogMTkwLjAgcywg MTgyMDY2Ni4wIHRwcywgbGF0IDAuMDI1IG1zIHN0ZGRldiAwLjAwNywgMCBmYWlsZWQKdHJh bnNhY3Rpb24gdHlwZTogcGdiZW5jaF9zY3JpcHQuc3FsCnNjYWxpbmcgZmFjdG9yOiAxCnF1 ZXJ5IG1vZGU6IHNpbXBsZQpudW1iZXIgb2YgY2xpZW50czogNDUKbnVtYmVyIG9mIHRocmVh ZHM6IDQ1Cm1heGltdW0gbnVtYmVyIG9mIHRyaWVzOiAxCmR1cmF0aW9uOiAyMDAgcwpudW1i ZXIgb2YgdHJhbnNhY3Rpb25zIGFjdHVhbGx5IHByb2Nlc3NlZDogMzY1NzAzNzYyCm51bWJl ciBvZiBmYWlsZWQgdHJhbnNhY3Rpb25zOiAwICgwLjAwMCUpCmxhdGVuY3kgYXZlcmFnZSA9 IDAuMDI0IG1zCmxhdGVuY3kgc3RkZGV2ID0gMC4wMDggbXMKaW5pdGlhbCBjb25uZWN0aW9u IHRpbWUgPSAyNS42MjQgbXMKdHBzID0gMTgyODc0My43OTEyMDUgKHdpdGhvdXQgaW5pdGlh bCBjb25uZWN0aW9uIHRpbWUp --------------tIrVj4iKOkX9Kk5WVkzHHVhB--