MIME-Version: 1.0
References: 
 <CAA5RZ0upTpKqgrdNfMSX7UJdjx=+=CsQ6Xct+vcCZPvUVhdZvw@mail.gmail.com>
 <CAApHDvp1=FOs6GneTzLSCHnCmC7z1_80=U3M=CKd82-pwS3YHg@mail.gmail.com>
 <aPuWev3D9M4iGCUt@nathan>
 <CAApHDvoM5MEHHBc0TNdrzkpq39WdEHSZhdWrtnx9zOWNXTSFGw@mail.gmail.com>
 <aP-YgrcPi0EhgR9x@nathan>
 <CAApHDvpOq09uVq7aXcuSBPAhZBTfAL-m2c4FOF2PphFe-YcnRg@mail.gmail.com>
 <aQEvm40W3aVizp5Q@nathan>
 <CAA5RZ0sdhUjVVKYbBGs1qFsYC3-Mn+as=K5v8ydVGR5iziabFQ@mail.gmail.com>
 <aQI39ln_jZ8qorLE@nathan>
 <CA+Tgmoa_5aC0w1fG7pLhev+F-GtRhQ2OzePy7t059c9JTnvjow@mail.gmail.com>
 <aQPSYD11NDoREZsg@nathan>
 <CAA5RZ0uo-nU9KqgXV5Tcf8aWWQkxsb5BDpkb-2Qfwbw-UVVaUA@mail.gmail.com>
In-Reply-To: 
 <CAA5RZ0uo-nU9KqgXV5Tcf8aWWQkxsb5BDpkb-2Qfwbw-UVVaUA@mail.gmail.com>
From: Sami Imseih <samimseih@gmail.com>
Date: Thu, 30 Oct 2025 19:38:15 -0500
Message-ID: 
 <CAA5RZ0tpV3PRHejTGG5-LSsqNKKV0qP=SDvurs8wj7pTk7jYJw@mail.gmail.com>
Subject: Re: another autovacuum scheduling thread
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, David Rowley <dgrowleyml@gmail.com>,
	Jeremy Schneider <schneider@ardentperf.com>, pgsql-hackers@postgresql.org
Content-Type: multipart/mixed; boundary="0000000000008d5027064269955c"
Archived-At: 
 <https://www.postgresql.org/message-id/CAA5RZ0tpV3PRHejTGG5-LSsqNKKV0qP%3DSDvurs8wj7pTk7jYJw%40mail.gmail.com>
Precedence: bulk

--0000000000008d5027064269955c
Content-Type: text/plain; charset="UTF-8"

> FWIW, I've been putting some scripts together to test some workloads
> and I will share shortly what I have.

Here is my attempt to test the behavior with the new prioritization.
I wanted a way to run the same tests with different workloads, both with
and without the prioritization patch, and to see if anything stands out as
suspicious in terms of autovacuum or autoanalyze activity. For example,
certain tables showing too little or too much autovacuum activity.

The scripts I put together (attached) run a busy update workload (OLTP)
and a separate batch workload. They use pgbench to execute custom scripts
that are generated on the fly.

The results are summarized by the average number of autovacuum and
autoanalyze runs *per table*, along with some other DML activity stats to
ensure that the workloads being compared have similar DML activity.

Using the scripts:

Place the attached scripts in a specific directory, and modify the
section under "Caller should adjust these values" in run_workloads.sh
to adjust the workload. The scripts assume you have a running cluster with
your specific config file adjusted for the test.

Once ready, call run_workloads.sh and at the end a summary will show up
as you see below. Hopefully it works for you :)

The summary.sh script can also be run while the workloads are executing.

Here is a example of a test I wanted to run based on the discussion [0]:

This scenario is one that was mentioned, but there are others in which a
batch process performing inserts only is prioritized over the update
workload.

I ran this test for 10 minutes, using 200 clients for the update workload
and 5 clients for the batch workload, with the following configuration:

```
max_connections=1000;
autovacuum_naptime = '10s'
shared_buffers = '4GB'
autovacuum_max_workers = 6
```


-- HEAD

```
Total Activity
-[ RECORD 1 ]-------------+----------
total_n_dead_tup          | 985183
total_n_mod_since_analyze | 220294866
total_reltuples           | 247690373
total_autovacuum_count    | 137
total_autoanalyze_count   | 470
total_n_tup_upd           | 7720012
total_n_tup_ins           | 446683000
table_count               | 105

Activity By Workload Type
-[ RECORD 1 ]-----------------+----------------
table_group                   | batch_tables
**      avg_autovacuum_count  | 7.400
**      avg_autoanalyze_count | 8.000
avg_vacuum_count              | 0.000
avg_analyze_count             | 0.000
rows_inserted                 | 436683000
rows_updated                  | 0
rows_hot_updated              | 0
table_count                   | 5
-[ RECORD 2 ]-----------------+----------------
table_group                   | numbered_tables
**      avg_autovacuum_count  | 1.000
**      avg_autoanalyze_count | 4.300
avg_vacuum_count              | 1.000
avg_analyze_count             | 0.000
rows_inserted                 | 10000000
rows_updated                  | 7720012
rows_hot_updated              | 7094573
table_count                   | 100

```

-- with v7 applied

```
Total Activity
-[ RECORD 1 ]-------------+----------
total_n_dead_tup          | 1233045
total_n_mod_since_analyze | 137843507
total_reltuples           | 350704437
total_autovacuum_count    | 146
total_autoanalyze_count   | 605
total_n_tup_upd           | 7896354
total_n_tup_ins           | 487974000
table_count               | 105

Activity By Workload Type
-[ RECORD 1 ]-----------------+----------------
table_group                   | batch_tables
**      avg_autovacuum_count  | 11.000
**      avg_autoanalyze_count | 13.200
avg_vacuum_count              | 0.000
avg_analyze_count             | 0.000
rows_inserted                 | 477974000
rows_updated                  | 0
rows_hot_updated              | 0
table_count                   | 5
-[ RECORD 2 ]-----------------+----------------
table_group                   | numbered_tables
**      avg_autovacuum_count  | 0.910
**      avg_autoanalyze_count | 5.390
avg_vacuum_count              | 1.000
avg_analyze_count             | 0.000
rows_inserted                 | 10000000
rows_updated                  | 7896354
rows_hot_updated              | 7123134
table_count                   | 100
```

The results above show what I expected: the batch tables receive higher
priority, as seen from the averages of autovacuum and autoanalyze runs.
This behavior is expected, but it may catch some users by surprise after
an upgrade, since certain tables will now receive more attention than
others. Longer tests might also show more bloat accumulating on heavily
updated tables. In such cases, a user may need to adjust autovacuum
settings on a per-table basis to restore the previous behavior.

So, I am not quite sure what is the best way to test except for trying
to find these non steady state workloads and see the impact of the
prioritization change to (auto)vacuum/analyze activity .

Maybe there is a better way?

[0] https://www.postgresql.org/message-id/aQI7tGEs8IOPxG64%40nathan


--
Sami Imseih
Amazon Web Services (AWS)

--0000000000008d5027064269955c
Content-Type: application/x-sh; name="batch.sh"
Content-Disposition: attachment; filename="batch.sh"
Content-Transfer-Encoding: base64
Content-ID: <f_mhe4fuor2>
X-Attachment-Id: f_mhe4fuor2

IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKQkFUQ0hfU0laRT0kMQpUTVBf
RElSPSQyCk5VTV9UQUJMRVM9JDMKQ09OTkVDVElPTlM9IiQ0IgpUSU1FT1VUPSIkNSIKCiMgLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIFVzYWdlIGNoZWNrCiMgLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLQppZiBbICIkIyIgLW5lIDUgXTsgdGhlbgogICAgZWNobyAiVXNhZ2U6ICQw
IEJBVENIX1NJWkUgVE1QX0RJUiBOVU1fVEFCTEVTIENPTk5FQ1RJT05TIFRJTUVPVVQiID4mMgog
ICAgZXhpdCAxCmZpCgojIGNyZWF0ZSB0bXAgZGlyZWN0b3J5Cm1rZGlyIC1wICIkVE1QX0RJUiIK
CiMgY3JlYXRlIGJ0Y2ggdGFibGVzIGxpc3QKVEFCTEVTPSgpCmZvciBpIGluICQoc2VxIDEgIiRO
VU1fVEFCTEVTIik7IGRvCiAgICBUQUJMRVMrPSgidGFibGVfYmF0Y2hfJGkiKQpkb25lCmVjaG8g
IlRhYmxlczogJHtUQUJMRVNbQF19IgoKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMg
U3RlcCAxOiBDcmVhdGUgdGFibGVzCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQplY2hv
ICJDcmVhdGluZyB0YWJsZXMuLi4iCmZvciBUQUJMRSBpbiAiJHtUQUJMRVNbQF19IjsgZG8KICAg
IHBzcWwgLWMgIkNSRUFURSBVTkxPR0dFRCBUQUJMRSBJRiBOT1QgRVhJU1RTICRUQUJMRSAoCiAg
ICAgICAgaWQgQklHSU5ULCBpZDIgQklHSU5ULCBpZDMgQklHSU5ULCBpZDQgQklHSU5UCiAgICAp
OyIKZG9uZQoKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgU3RlcCAyOiBDcmVhdGUg
cGdiZW5jaCBTUUwgZmlsZXMKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmVjaG8gIkNy
ZWF0aW5nIHBnYmVuY2ggU1FMIGZpbGVzLi4uIgpQR19GSUxFUz0oKQpmb3IgVEFCTEUgaW4gIiR7
VEFCTEVTW0BdfSI7IGRvCiAgICBQR19GSUxFPSIkVE1QX0RJUi8ke1RBQkxFfS5zcWwiCiAgICBj
YXQgPiAiJFBHX0ZJTEUiIDw8RU9GCgpJTlNFUlQgSU5UTyAkVEFCTEUgKGlkLCBpZDIsIGlkMywg
aWQ0KQpTRUxFQ1QgbiwgbiwgbiwgbgpGUk9NIGdlbmVyYXRlX3NlcmllcygxLCAkQkFUQ0hfU0la
RSkgQVMgbjsKRU9GCiAgICBQR19GSUxFUys9KCItZiIgIiRQR19GSUxFQDEiKQpkb25lCgojIC0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIyBTdGVwIDM6IFJ1biBwZ2JlbmNoIHNlcmlhbGx5
CiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQplY2hvICJTdGFydGluZyBwZ2JlbmNoIGlu
c2VydHMgKDEwMDAgcm93cyBwZXIgdHJhbnNhY3Rpb24gcGVyIHRhYmxlKS4uLiIKcGdiZW5jaCAt
YzEgLW4gIiR7UEdfRklMRVNbQF19IiAtYyRDT05ORUNUSU9OUyAtVCRUSU1FT1VUCgplY2hvICJw
Z2JlbmNoIGluc2VydHMgY29tcGxldGVkISIK
--0000000000008d5027064269955c
Content-Type: application/x-sh; name="summary.sh"
Content-Disposition: attachment; filename="summary.sh"
Content-Transfer-Encoding: base64
Content-ID: <f_mhe4fuou3>
X-Attachment-Id: f_mhe4fuou3

IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKcHNxbCA8PEVPRgpceApcXGVj
aG8gVG90YWwgQWN0aXZpdHkKU0VMRUNUCiAgICBTVU0oYy5uX2RlYWRfdHVwKSBhcyB0b3RhbF9u
X2RlYWRfdHVwLAogICAgU1VNKGMubl9tb2Rfc2luY2VfYW5hbHl6ZSkgYXMgdG90YWxfbl9tb2Rf
c2luY2VfYW5hbHl6ZSwKICAgIFNVTShiLnJlbHR1cGxlczo6bnVtZXJpYykgYXMgdG90YWxfcmVs
dHVwbGVzLAogICAgU1VNKGMuYXV0b3ZhY3V1bV9jb3VudCkgYXMgdG90YWxfYXV0b3ZhY3V1bV9j
b3VudCwKICAgIFNVTShjLmF1dG9hbmFseXplX2NvdW50KSBhcyB0b3RhbF9hdXRvYW5hbHl6ZV9j
b3VudCwKICAgIFNVTShjLm5fdHVwX3VwZCkgYXMgdG90YWxfbl90dXBfdXBkLAogICAgU1VNKGMu
bl90dXBfaW5zKSBhcyB0b3RhbF9uX3R1cF9pbnMsCiAgICBDT1VOVCgqKSBhcyB0YWJsZV9jb3Vu
dApGUk9NIHBnX2NsYXNzIGIsCiAgICAgcGdfc3RhdF9hbGxfdGFibGVzIGMKV0hFUkUgYi5vaWQg
PSBjLnJlbGlkCiAgQU5EIChVUFBFUihiLnJlbG5hbWUpIExJS0UgJyVUQUJMJScKICBBTkQgYi5y
ZWxuYW1lIE5PVCBMSUtFICdwZ18lJyk7CgpcXGVjaG8gQWN0aXZpdHkgQnkgV29ya2xvYWQgVHlw
ZQpTRUxFQ1QKICAgIENBU0UKICAgICAgICBXSEVOIGIucmVsbmFtZSBMSUtFICd0YWJsZV9iYXRj
aCUnIFRIRU4gJ2JhdGNoX3RhYmxlcycKICAgICAgICBXSEVOIGIucmVsbmFtZSB+ICdedGFibGVf
WzAtOV0rJCcgVEhFTiAnbnVtYmVyZWRfdGFibGVzJwogICAgRU5EIGFzIHRhYmxlX2dyb3VwLAog
ICAgUk9VTkQoQVZHKGMuYXV0b3ZhY3V1bV9jb3VudCksIDMpIGFzICIqKiAgICAgIGF2Z19hdXRv
dmFjdXVtX2NvdW50IiwKICAgIFJPVU5EKEFWRyhjLmF1dG9hbmFseXplX2NvdW50KSwgMykgYXMg
IioqICAgICAgYXZnX2F1dG9hbmFseXplX2NvdW50IiwKICAgIFJPVU5EKEFWRyhjLnZhY3V1bV9j
b3VudCksIDMpIGFzIGF2Z192YWN1dW1fY291bnQsCiAgICBST1VORChBVkcoYy5hbmFseXplX2Nv
dW50KSwgMykgYXMgYXZnX2FuYWx5emVfY291bnQsCiAgICBST1VORChTVU0obl90dXBfaW5zKSwg
MCkgcm93c19pbnNlcnRlZCwKICAgIFJPVU5EKFNVTShuX3R1cF91cGQpLCAwKSByb3dzX3VwZGF0
ZWQsCiAgICBST1VORChTVU0obl90dXBfaG90X3VwZCksIDApIHJvd3NfaG90X3VwZGF0ZWQsCiAg
ICBDT1VOVCgqKSBhcyB0YWJsZV9jb3VudApGUk9NIHBnX2NsYXNzIGIsCiAgICAgcGdfc3RhdF9h
bGxfdGFibGVzIGMKV0hFUkUgYi5vaWQgPSBjLnJlbGlkCiAgQU5EIChVUFBFUihiLnJlbG5hbWUp
IExJS0UgJyVUQUJMJScKICBBTkQgYi5yZWxuYW1lIE5PVCBMSUtFICdwZ18lJykKICBBTkQgKGIu
cmVsbmFtZSBMSUtFICd0YWJsZV9iYXRjaCUnIE9SIGIucmVsbmFtZSB+ICdedGFibGVfWzAtOV0r
JCcpCkdST1VQIEJZIHRhYmxlX2dyb3VwOwoKXHgKU0VMRUNUIHF1ZXJ5IEZST00gcGdfc3RhdF9h
Y3Rpdml0eSBXSEVSRSBxdWVyeSBMSUtFICclYXV0b3YlJyBhbmQgcGlkIDw+IHBnX2JhY2tlbmRf
cGlkKCk7CgpFT0YK
--0000000000008d5027064269955c
Content-Type: application/x-sh; name="oltp.sh"
Content-Disposition: attachment; filename="oltp.sh"
Content-Transfer-Encoding: base64
Content-ID: <f_mhe4fuof0>
X-Attachment-Id: f_mhe4fuof0

IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKIyAtLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tCiMgVXNhZ2UgY2hlY2sKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
CmlmIFsgIiQjIiAtbmUgNiBdOyB0aGVuCiAgICBlY2hvICJVc2FnZTogJDAgUEdfRklMRSBOVU1f
VEFCTEVTIE5VTV9ST1dTIiA+JjIKICAgIGV4aXQgMQpmaQoKUEdfRklMRT0iJDEiCk5VTV9UQUJM
RVM9IiQyIgpOVU1fUk9XUz0iJDMiClJFQURZX0ZJTEU9IiQ0IgpDT05ORUNUSU9OUz0iJDUiClRJ
TUVPVVQ9IiQ2IgoKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgQ3JlYXRlIHRhYmxl
cwojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KcHNxbCA8PEVPRgpETwpcJFwkCkRFQ0xB
UkUKICAgIGkgSU5UOwogICAgdGJsX25hbWUgVEVYVDsKQkVHSU4KICAgIEZPUiBpIElOIDEuLiRO
VU1fVEFCTEVTIExPT1AKICAgICAgICB0YmxfbmFtZSA6PSBmb3JtYXQoJ3RhYmxlXyVzJywgaSk7
CiAgICAgICAgRVhFQ1VURSBmb3JtYXQoJwogICAgICAgICAgICBDUkVBVEUgVU5MT0dHRUQgVEFC
TEUgJUkgKAogICAgICAgICAgICAgICAgaWQgQklHU0VSSUFMIFBSSU1BUlkgS0VZLAogICAgICAg
ICAgICAgICAgY29sMSBURVhULAogICAgICAgICAgICAgICAgY29sMiBURVhULAogICAgICAgICAg
ICAgICAgY29sMyBURVhULAogICAgICAgICAgICAgICAgY29sNCBCSUdJTlQsCiAgICAgICAgICAg
ICAgICBjb2w1IEJJR0lOVCwKICAgICAgICAgICAgICAgIGNvbDYgQklHSU5ULAogICAgICAgICAg
ICAgICAgY29sNyBCSUdJTlQsCiAgICAgICAgICAgICAgICBjb2w4IEJJR0lOVCwKICAgICAgICAg
ICAgICAgIGNvbDkgQklHSU5ULAogICAgICAgICAgICAgICAgY29sMTAgQklHSU5ULAogICAgICAg
ICAgICAgICAgY29sMTEgVElNRVNUQU1QIERFRkFVTFQgTk9XKCksCiAgICAgICAgICAgICAgICBj
b2wxMiBUSU1FU1RBTVAsCiAgICAgICAgICAgICAgICBjb2wxMyBUSU1FU1RBTVAsCiAgICAgICAg
ICAgICAgICBjb2wxNCBURVhULAogICAgICAgICAgICAgICAgY29sMTUgQklHSU5UCiAgICAgICAg
ICAgICk7CiAgICAgICAgJywgdGJsX25hbWUpOwogICAgRU5EIExPT1A7CkVORApcJFwkOwpFT0YK
CiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIFBvcHVsYXRlIHRhYmxlcyAKIyAtLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmZvciBpIGluICQoc2VxIDEgIiROVU1fVEFCTEVTIik7
IGRvCiAgICBlY2hvICJJbnNlcnRpbmcgJE5VTV9ST1dTIHJvd3MgaW50byB0YWJsZV8kaS4uLiIK
ICAgIHBzcWwgLWMgIgogICAgSU5TRVJUIElOVE8gdGFibGVfJHtpfSAoY29sMSwgY29sMiwgY29s
MywgY29sNCwgY29sNSwgY29sNiwgY29sNywgY29sOCwgY29sOSwgY29sMTAsIGNvbDExLCBjb2wx
MiwgY29sMTMsIGNvbDE0LCBjb2wxNSkKICAgIFNFTEVDVAogICAgICAgIG1kNShyYW5kb20oKTo6
dGV4dCksCiAgICAgICAgbWQ1KHJhbmRvbSgpOjp0ZXh0KSwKICAgICAgICBtZDUocmFuZG9tKCk6
OnRleHQpLAogICAgICAgIChyYW5kb20oKSoxMDAwMDAwKTo6QklHSU5ULAogICAgICAgIChyYW5k
b20oKSoxMDAwMDAwKTo6QklHSU5ULAogICAgICAgIChyYW5kb20oKSoxMDAwMDAwKTo6QklHSU5U
LAogICAgICAgIChyYW5kb20oKSoxMDAwMDAwKTo6QklHSU5ULAogICAgICAgIChyYW5kb20oKSox
MDAwMDAwKTo6QklHSU5ULAogICAgICAgIChyYW5kb20oKSoxMDAwMDAwKTo6QklHSU5ULAogICAg
ICAgIChyYW5kb20oKSoxMDAwMDAwKTo6QklHSU5ULAogICAgICAgIE5PVygpIC0gKHJhbmRvbSgp
ICogaW50ZXJ2YWwgJzMwIGRheXMnKSwKICAgICAgICBOT1coKSAtIChyYW5kb20oKSAqIGludGVy
dmFsICczMCBkYXlzJyksCiAgICAgICAgTk9XKCkgLSAocmFuZG9tKCkgKiBpbnRlcnZhbCAnMzAg
ZGF5cycpLAogICAgICAgIG1kNShyYW5kb20oKTo6dGV4dCksCiAgICAgICAgKHJhbmRvbSgpKjEw
MDAwMDApOjpCSUdJTlQKICAgIEZST00gZ2VuZXJhdGVfc2VyaWVzKDEsICROVU1fUk9XUyk7CiAg
ICAiCmRvbmUKCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIFZBQ1VVTSB0YWJsZXMK
IyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmZvciBpIGluICQoc2VxIDEgIiROVU1fVEFC
TEVTIik7IGRvCiAgICBlY2hvICJWYWN1dW1pbmcgdGFibGVfJGkuLi4iCiAgICBwc3FsIC1jICJW
QUNVVU0gRlJFRVpFIHRhYmxlXyRpOyIKZG9uZQoKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tCiMgU3RvcCBhbmQgcmVzdGFydCBQb3N0Z3JlU1FMIAojIC0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0KZWNobyAiU3RvcHBpbmcgUG9zdGdyZVNRTC4uLiIKcGdfY3RsIHN0b3AKCmVjaG8g
IldhaXRpbmcgZm9yIFBvc3RncmVTUUwgdG8gZnVsbHkgc3RvcC4uLiIKd2hpbGUgcGdyZXAgLXgg
cG9zdGdyZXMgPi9kZXYvbnVsbDsgZG8KICAgIGVjaG8gIiAgU3RpbGwgc2h1dHRpbmcgZG93bi4u
LiIKICAgIHNsZWVwIDEKZG9uZQplY2hvICJQb3N0Z3JlU1FMIGlzIGZ1bGx5IHN0b3BwZWQuIgoK
ZWNobyAiU3RhcnRpbmcgUG9zdGdyZVNRTC4uLiIKcGdfY3RsIHN0YXJ0CgplY2hvICJXYWl0aW5n
IGZvciBQb3N0Z3JlU1FMIHRvIHN0YXJ0IHVwLi4uIgp1bnRpbCBwZ19pc3JlYWR5IC1xOyBkbwog
ICAgZWNobyAiICBTdGlsbCBzdGFydGluZy4uLiIKICAgIHNsZWVwIDEKZG9uZQplY2hvICJQb3N0
Z3JlU1FMIGlzIHVwIGFuZCBydW5uaW5nISIKCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LQojIFNpZ25hbCBPTFRQIHJlYWRpbmVzcwojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0K
dG91Y2ggJFJFQURZX0ZJTEUKCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIENyZWF0
ZSBwZ2JlbmNoIFNRTCBmaWxlCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpjYXQgPiIk
UEdfRklMRSIgPDxFT0YKXHNldCB1cGQxIHJhbmRvbSgxLCAkTlVNX1RBQkxFUykKXHNldCBpZCBy
YW5kb20oMSwgJE5VTV9ST1dTKQoKVVBEQVRFIHRhYmxlXzp1cGQxIFNFVCBjb2wxID0gY29sMSBX
SEVSRSBpZCA9IDppZDsKRU9GCgojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIyBSdW4g
cGdiZW5jaAojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KcGdiZW5jaCAtZiAiJFBHX0ZJ
TEUiIC1jJENPTk5FQ1RJT05TIC1UJFRJTUVPVVQK
--0000000000008d5027064269955c
Content-Type: application/x-sh; name="run_workloads.sh"
Content-Disposition: attachment; filename="run_workloads.sh"
Content-Transfer-Encoding: base64
Content-ID: <f_mhe4fuom1>
X-Attachment-Id: f_mhe4fuom1

IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKIyAtLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLQojIENhbGxlciBzaG91bGQgYWRqdXN0IHRoZXNlIHZhbHVlcwoj
IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgU2NyaXB0cyBwYXRoCkJBU0Vf
RElSPSIkSE9NRS9zY3JpcHRzL3Rlc3RfYXV0b3ZhY3V1bV9wcmlvcml0aXphdGlvbiIKIyBXb3Jr
bG9hZCBwYXJhbWV0ZXJzCk9MVFBfVEFCTEVTPTEwMApPTFRQX1JPV1NfUEVSX1RBQkxFPTEwMDAw
MApCQVRDSF9UQUJMRVM9NQpCQVRDSF9TSVpFPTEwMDAKQkFUQ0hfQ09OTkVDVElPTlM9MQpPTFRQ
X0NPTk5FQ1RJT05TPTMKVElNRU9VVD02MDAKCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LQojIERPIE5PVCBUT1VDSCEKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tClRNUF9ESVI9
IiRCQVNFX0RJUi90bXAiClRNUF9PTFRQX0ZJTEU9IiRCQVNFX0RJUi9vbHRwX3dvcmtsb2FkLnNx
bCIKVE1QX0JBVENIX0ZJTEVTPSIkQkFTRV9ESVIvYmF0Y2hfd29ya2xvYWQiCgpPTFRQX1NDUklQ
VD0iJEJBU0VfRElSL29sdHAuc2giCkJBVENIX1NDUklQVD0iJEJBU0VfRElSL2JhdGNoLnNoIgpT
VU1NQVJZX1NDUklQVD0iJEJBU0VfRElSL3N1bW1hcnkuc2giCgpPTFRQX0xPRz0iJFRNUF9ESVIv
b2x0cC5vdXQiCkJBVENIX0xPRz0iJFRNUF9ESVIvYmF0Y2gub3V0IgpSRUFEWV9GSUxFPSIkVE1Q
X0RJUi9vbHRwX3JlYWR5IgoKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgRW5zdXJl
IHRlbXAgZGlyZWN0b3J5IGV4aXN0cwojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KaWYg
WyAtbiAiJFRNUF9ESVIiIF0gJiYgWyAiJFRNUF9ESVIiICE9ICIvIiBdOyB0aGVuCiAgICBpZiBb
IC1kICIkVE1QX0RJUiIgXTsgdGhlbgogICAgICAgIGVjaG8gIlJlbW92aW5nIGV4aXN0aW5nIHRl
bXAgZGlyZWN0b3J5OiAkVE1QX0RJUiIKICAgICAgICBybSAtcmYgIiRUTVBfRElSIgogICAgZmkK
ICAgIG1rZGlyIC1wICIkVE1QX0RJUiIgfHwgeyBlY2hvICJFcnJvcjogZmFpbGVkIHRvIGNyZWF0
ZSAkVE1QX0RJUiIgPiYyOyBleGl0IDE7IH0KZWxzZQogICAgZWNobyAiRXJyb3I6IFRNUF9ESVIg
aXMgaW52YWxpZCEiID4mMgogICAgZXhpdCAxCmZpCgojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0KIyBTdGFydCBPTFRQIHdvcmtsb2FkIGluIGJhY2tncm91bmQKIyAtLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tCmVjaG8gIlN0YXJ0aW5nIE9MVFAgd29ya2xvYWQuLi4iCmJhc2ggIiRP
TFRQX1NDUklQVCIgIiRUTVBfT0xUUF9GSUxFIiAiJE9MVFBfVEFCTEVTIiAiJE9MVFBfUk9XU19Q
RVJfVEFCTEUiICIkUkVBRFlfRklMRSIgIiRPTFRQX0NPTk5FQ1RJT05TIiAiJFRJTUVPVVQiIHwg
dGVlICIkT0xUUF9MT0ciIDI+JjEgJgpPTFRQX1BJRD0kIQoKIyAtLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tCiMgV2FpdCBmb3IgT0xUUCByZWFkeSBmaWxlCiMgLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLQplY2hvICJXYWl0aW5nIGZvciBPTFRQIHJlYWR5IGZpbGUuLi4iCndoaWxlIFsg
ISAtZiAiJFJFQURZX0ZJTEUiIF07IGRvCiAgICBzbGVlcCAxCmRvbmUKZWNobyAiT0xUUCByZWFk
eSBmaWxlIGRldGVjdGVkLiBTdGFydGluZyBiYXRjaCB3b3JrbG9hZC4uLiIKCiMgLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLQojIFN0YXJ0IGJhdGNoIHdvcmtsb2FkIGluIGJhY2tncm91bmQK
IyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmJhc2ggIiRCQVRDSF9TQ1JJUFQiICIkQkFU
Q0hfU0laRSIgIiRUTVBfQkFUQ0hfRklMRVMiICIkQkFUQ0hfVEFCTEVTIiAiJEJBVENIX0NPTk5F
Q1RJT05TIiAiJFRJTUVPVVQiIHwgdGVlICIkQkFUQ0hfTE9HIiAyPiYxICYKQkFUQ0hfUElEPSQh
CgplY2hvICJPTFRQIFBJRDogJE9MVFBfUElEIHwgQmF0Y2ggUElEOiAkQkFUQ0hfUElEIgoKIyAt
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgTW9uaXRvciBib3RoIHdvcmtsb2FkcwojIC0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KU1RBUlRfVElNRT0kKGRhdGUgKyVzKQoKd2hpbGUg
dHJ1ZTsgZG8KICAgIENVUlJFTlRfVElNRT0kKGRhdGUgKyVzKQogICAgRUxBUFNFRD0kKChDVVJS
RU5UX1RJTUUgLSBTVEFSVF9USU1FKSkKCiAgICAjIElmIE9MVFAgZmluaXNoZWQsIHRlcm1pbmF0
ZSBiYXRjaAogICAgaWYgISBraWxsIC0wICIkT0xUUF9QSUQiIDI+L2Rldi9udWxsOyB0aGVuCiAg
ICAgICAgZWNobyAiT0xUUCB3b3JrbG9hZCBjb21wbGV0ZWQsIHRlcm1pbmF0aW5nIGJhdGNoLi4u
IgogICAgICAgIGtpbGwgLVRFUk0gIiRCQVRDSF9QSUQiIDI+L2Rldi9udWxsIHx8IHRydWUKICAg
ICAgICB3YWl0ICIkQkFUQ0hfUElEIiAyPi9kZXYvbnVsbCB8fCB0cnVlCiAgICAgICAgYnJlYWsK
ICAgIGZpCgogICAgIyBJZiBiYXRjaCBmaW5pc2hlZCwgdGVybWluYXRlIE9MVFAKICAgIGlmICEg
a2lsbCAtMCAiJEJBVENIX1BJRCIgMj4vZGV2L251bGw7IHRoZW4KICAgICAgICBlY2hvICJCYXRj
aCB3b3JrbG9hZCBjb21wbGV0ZWQsIHRlcm1pbmF0aW5nIE9MVFAuLi4iCiAgICAgICAga2lsbCAt
VEVSTSAiJE9MVFBfUElEIiAyPi9kZXYvbnVsbCB8fCB0cnVlCiAgICAgICAgd2FpdCAiJE9MVFBf
UElEIiAyPi9kZXYvbnVsbCB8fCB0cnVlCiAgICAgICAgYnJlYWsKICAgIGZpCgogICAgIyBDaGVj
ayB0aW1lb3V0CiAgICBpZiBbICIkRUxBUFNFRCIgLWdlICIkVElNRU9VVCIgXTsgdGhlbgogICAg
ICAgIGVjaG8gIuKPsSBUaW1lb3V0IHJlYWNoZWQgKCR7RUxBUFNFRH1zKSwgc3RvcHBpbmcgYm90
aCB3b3JrbG9hZHMuLi4iCiAgICAgICAga2lsbCAtVEVSTSAiJE9MVFBfUElEIiAiJEJBVENIX1BJ
RCIgMj4vZGV2L251bGwgfHwgdHJ1ZQogICAgICAgIHdhaXQgIiRPTFRQX1BJRCIgMj4vZGV2L251
bGwgfHwgdHJ1ZQogICAgICAgIHdhaXQgIiRCQVRDSF9QSUQiIDI+L2Rldi9udWxsIHx8IHRydWUK
ICAgICAgICBicmVhawogICAgZmkKCiAgICBzbGVlcCA1CmRvbmUKCmVjaG8gIkJvdGggd29ya2xv
YWRzIHN0b3BwZWQuIgoKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgUmVzdGFydCBQ
b3N0Z3JlU1FMCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQplY2hvICJSZXN0YXJ0aW5n
IFBvc3RncmVzLi4uIgpwZ19jdGwgcmVzdGFydAplY2hvICJQb3N0Z3JlcyByZXN0YXJ0ZWQuIgoK
IyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgUnVuIHN1bW1hcnkgc2NyaXB0CiMgLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQoiJFNVTU1BUllfU0NSSVBUIgo=
--0000000000008d5027064269955c--