Content-Type: multipart/mixed; boundary="------------tpO8apL2U5SEAvKbdSzGITgq"
Message-ID: <d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com>
Date: Tue, 21 Nov 2023 12:47:38 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Content-Language: en-US
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
Subject: undetected deadlock in ALTER SUBSCRIPTION ... REFRESH PUBLICATION
Archived-At: 
 <https://www.postgresql.org/message-id/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e%40enterprisedb.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------tpO8apL2U5SEAvKbdSzGITgq
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi,

I decided to do some stress-testing of the built-in logical replication,
as part of the sequence decoding work. And I soon ran into an undetected
deadlock related to ALTER SUBSCRIPTION ... REFRESH PUBLICATION :-(

The attached bash scripts triggers that in a couple seconds for me. The
script looks complicated, but most of the code is waiting for sync to
complete, catchup, and that sort of thing.

What the script does is pretty simple:

1) initialize two clusters, set them as publisher/subscriber pair

2) create some number of tables, add them to publication and wait for
   the sync to complete

3) start two pgbench runs in the background, modifying the publication
   (one removes+adds all tables in a single transaction, one does that
    with transaction per table)

4) run refresh.sh which does ALTER PUBLICATION ... REFRESH PUBLICATION
   in a loop (now that I think about it, could be another pgbench
   script, but well ...)

5) some consistency checks, but the lockup happens earlier so this does
   not really matter

After a small number of refresh cycles (for me it's usually a couple
dozen), we end up with a couple stuck locks (I shortened the backend
type string a bit, for formatting reasons):

  test=# select a.pid, classid, objid, backend_type, query
           from pg_locks l join pg_stat_activity a on (a.pid = l.pid)
          where not granted;

     pid   | classid | objid | backend_type     |         query
  ---------+---------+-------+------------------+----------------------
   2691941 |    6100 | 16785 | client backend   | ALTER SUBSCRIPTION s
                                                  REFRESH PUBLICATION
   2691837 |    6100 | 16785 | tablesync worker |
   2691936 |    6100 | 16785 | tablesync worker |
  (3 rows)

All these backends wait for 6100/16785, which is the subscription row in
pg_subscription. The tablesync workers are requesting AccessShareLock,
the client backend however asks for AccessExclusiveLock.

The entry is currently locked by:

  test=# select a.pid, mode, backend_type from pg_locks l
           join pg_stat_activity a on (a.pid = l.pid)
          where classid=6100 and objid=16785 and granted;

     pid   |      mode       |           backend_type
  ---------+-----------------+----------------------------------
   2690477 | AccessShareLock | logical replication apply worker
  (1 row)

But the apply worker is not waiting for any locks, so what's going on?

Well, the problem is the apply worker is waiting for notification from
the tablesync workers the relation is synced, which happens through
updating the pg_subscription_rel row. And that wait happens in
wait_for_relation_state_change, which simply checks the row in a loop,
with a sleep by WaitLatch().

Unfortunately, the tablesync workers can't update the row because the
client backend executing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
sneaked in, and waits for an AccessExclusiveLock. So the tablesync
workers are stuck in the queue and can't proceed.

The client backend can't proceed, because it's waiting for a lock held
by the apply worker.

The tablesync workers can't proceed because their lock request is stuck
behind the AccessExclusiveLock request.

And the apply worker can't proceed, because it's waiting for status
update from the tablesync workers.

And the deadlock is undetected, because the apply worker is not waiting
on a lock, but sleeping on a latch :-(


I don't know what's the right solution here. I wonder if the apply
worker might release the lock before waiting for the update, that'd
solve this whole issue.

Alternatively, ALTER PUBLICATION might wait for the lock only for a
limited amount of time, and try again (but then it'd be susceptible to
starving, of course).

Or maybe there's a way to make this work in a way that would be visible
to the deadlock detector? That'd mean we occasionally get processes
killed to resolve a deadlock, but that's still better than processes
stuck indefinitely ...


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--------------tpO8apL2U5SEAvKbdSzGITgq
Content-Type: application/x-shellscript; name="lockup-test.sh"
Content-Disposition: attachment; filename="lockup-test.sh"
Content-Transfer-Encoding: base64

IyEvdXNyL2Jpbi9iYXNoCiMKCk9VVERJUj0kMQpOVU1UQUJMRVM9JDIKUkVGUkVTSD0kMwpT
TEVFUD0kNAoKIyBob3cgbG9uZyB0byBydW4gdGhlIHZhcmlvdXMgdGFza3MKRFVSQVRJT05f
Q0hFQ0s9MTIwMApEVVJBVElPTl9BTFRFUj0xMDAwCkRVUkFUSU9OX1dPUktMT0FEPTExMDAK
RFVSQVRJT05fUkVGUkVTSD0xMjUwCgpTVEFSVF9USU1FU1RBTVA9YGRhdGUgKyVzYAoKZWNo
byBbYGRhdGVgXSBbYGRhdGUgKyVzYF0gIk5VTVRBQkxFUz0kTlVNVEFCTEVTICBTTEVFUD0k
U0xFRVAgIFJFRlJFU0g9JFJFRlJFU0giCgojIGNvbmZpZwpQVUJfUE9SVD01MDAxClBVQl9E
QVRBPSRPVVRESVIvZGF0YS1wdWIKClNVQl9QT1JUPTUwMDIKU1VCX0RBVEE9JE9VVERJUi9k
YXRhLXN1YgoKIyBpbml0aWFsaXplIGNsdXN0ZXJzCnBnX2N0bCAtRCAkUFVCX0RBVEEgaW5p
dApwZ19jdGwgLUQgJFNVQl9EQVRBIGluaXQKCmVjaG8gJ3dhbF9sZXZlbCA9IGxvZ2ljYWwn
ID4+ICRQVUJfREFUQS9wb3N0Z3Jlc3FsLmNvbmYKZWNobyAicG9ydCA9ICRQVUJfUE9SVCIg
Pj4gJFBVQl9EQVRBL3Bvc3RncmVzcWwuY29uZgplY2hvICJsb2dfbGluZV9wcmVmaXggPSAn
JW4gJW0gWyVwXSBbJWJdIFslYV0gWyV2LyV4XSAnIiA+PiAkUFVCX0RBVEEvcG9zdGdyZXNx
bC5jb25mCgplY2hvICJwb3J0ID0gJFNVQl9QT1JUIiA+PiAkU1VCX0RBVEEvcG9zdGdyZXNx
bC5jb25mCmVjaG8gImxvZ19saW5lX3ByZWZpeCA9ICclbiAlbSBbJXBdIFslYl0gWyVhXSBb
JXYvJXhdICciID4+ICRTVUJfREFUQS9wb3N0Z3Jlc3FsLmNvbmYKCnBnX2N0bCAtRCAkUFVC
X0RBVEEgLXcgLWwgJE9VVERJUi9wZy1wdWIubG9nIHN0YXJ0CnBnX2N0bCAtRCAkU1VCX0RB
VEEgLXcgLWwgJE9VVERJUi9wZy1zdWIubG9nIHN0YXJ0CgpjcmVhdGVkYiAtcCAkUFVCX1BP
UlQgdGVzdCA+ICRPVVRESVIvaW5pdC1wdWIubG9nIDI+JjEKY3JlYXRlZGIgLXAgJFNVQl9Q
T1JUIHRlc3QgPiAkT1VURElSL2luaXQtc3ViLmxvZyAyPiYxCgojIGNyZWF0ZSB0YWJsZXMg
b24gYm90aCBlbmRzCmZvciB0IGluIGBzZXEgMSAkTlVNVEFCTEVTYDsgZG8KCXBzcWwgLXAg
JFBVQl9QT1JUIHRlc3QgLWMgIkNSRUFURSBUQUJMRSB0XyR7dH0gKGlkIHNlcmlhbCBwcmlt
YXJ5IGtleSwgdmFsdWUgYmlnaW50KSIKCXBzcWwgLXAgJFNVQl9QT1JUIHRlc3QgLWMgIkNS
RUFURSBUQUJMRSB0XyR7dH0gKGlkIHNlcmlhbCBwcmltYXJ5IGtleSwgdmFsdWUgYmlnaW50
KSIKZG9uZQoKIyBnZW5lcmF0ZSBzY3JpcHRzIGZvciBwZ2JlbmNoCmVjaG8gPiBpbnNlcnQu
c3FsCmVjaG8gPiBhZGQuc3FsCmVjaG8gPiBkcm9wLnNxbAplY2hvID4gc2luZ2xlLnNxbAoK
Zm9yIHMgaW4gYHNlcSAxICROVU1UQUJMRVNgOyBkbwoJbmFtZT0idF8kcyIKCWVjaG8gIklO
U0VSVCBJTlRPICRuYW1lICh2YWx1ZSkgVkFMVUVTIChyYW5kb20oKSAqIDEwMDApOyIgPj4g
aW5zZXJ0LnNxbAoKCSMgd2UnbGwgbWl4IGFkZC9kcm9wIGluIGEgb25lIGh1Z2UgdHJhbnNh
Y3Rpb24KCWVjaG8gIkFMVEVSIFBVQkxJQ0FUSU9OIHAgRFJPUCBUQUJMRSAkbmFtZTsiID4+
IGRyb3Auc3FsOwoJZWNobyAiU0VMRUNUIHBnX3NsZWVwKHJhbmRvbSgpLzEwMCk7IiA+PiBk
cm9wLnNxbAoKCWVjaG8gIkFMVEVSIFBVQkxJQ0FUSU9OIHAgQUREIFRBQkxFICRuYW1lOyIg
Pj4gYWRkLnNxbDsKCWVjaG8gIlNFTEVDVCBwZ19zbGVlcChyYW5kb20oKS8xMDApOyIgPj4g
YWRkLnNxbDsKCgkjIGFuZCB0aGVuIGEgYnVuY2ggb2Ygc21hbGwgdHJhbnNhY3Rpb24gYWRk
aW5nL3JlbW92aW5nIHRoZSB0YWJsZQoJZWNobyAiQkVHSU47IiA+PiBzaW5nbGUuc3FsOwoJ
ZWNobyAiQUxURVIgUFVCTElDQVRJT04gcCBEUk9QIFRBQkxFICRuYW1lOyIgPj4gc2luZ2xl
LnNxbDsKCWVjaG8gIlNFTEVDVCBwZ19zbGVlcChyYW5kb20oKS8xMDApOyIgPj4gc2luZ2xl
LnNxbDsKCWVjaG8gIkFMVEVSIFBVQkxJQ0FUSU9OIHAgQUREIFRBQkxFICRuYW1lOyIgPj4g
c2luZ2xlLnNxbDsKCWVjaG8gIlNFTEVDVCBwZ19zbGVlcChyYW5kb20oKS8xMDApOyIgPj4g
c2luZ2xlLnNxbDsKCWVjaG8gIkNPTU1JVDsiID4+IHNpbmdsZS5zcWw7CgllY2hvICJTRUxF
Q1QgcGdfc2xlZXAocmFuZG9tKCkvMTApOyIgPj4gc2luZ2xlLnNxbDsKZG9uZQoKIyBzaHVm
ZmxlIHRoZSBpbnNlcnRzIGEgYml0LCB0byByYW5kb21pemUgdGhlIGluc2VydCBvcmRlcgpz
b3J0IC1SIGluc2VydC5zcWwgPiBpbnNlcnQudG1wICYmIG12IGluc2VydC50bXAgaW5zZXJ0
LnNxbAoKIyBzaHVmZmxlIHRoZSBBTFRFUiBQVUJMSUNBVElPTiAuLi4gQUREL0RST1AgdG9v
LCBidXQgbWFiZSBzdXJlIHRvIGtlZXAKIyBEUk9QIGJlZm9yZSBBREQgb2YgdGhlIHNhbWUg
dGFibGUKIwojIHVzZSBzaHVmZmxlIGJlY2F1c2UgInNvcnQgLVIiIGdyb3VwcyB0aGUgc2Ft
ZSBrZXkgKGUuZy4gcGdfc2xlZXAgY2FsbHMpCiMgYW5kIHdlIGRvbid0IHdhbnQgdGhhdApz
aHVmIGFkZC5zcWwgPiBhZGQudG1wICYmIG12IGFkZC50bXAgYWRkLnNxbApzaHVmIGRyb3Au
c3FsID4gZHJvcC50bXAgJiYgbXYgZHJvcC50bXAgZHJvcC5zcWwKCiMgZG8gdGhlIGRyb3Av
YWRkIGluIGEgc2luZ2xlIGxhcmdlIHRyYW5zYXRpb24KZWNobyAiQkVHSU47IiA+IG11bHRp
LnRtcApjYXQgZHJvcC5zcWwgPj4gbXVsdGkudG1wCmNhdCBhZGQuc3FsID4+IG11bHRpLnRt
cAplY2hvICJDT01NSVQ7IiA+PiBtdWx0aS50bXAKbXYgbXVsdGkudG1wIG11bHRpLnNxbAoK
IyBidXQgYWxzbyBoYXZlIGEgc2NyaXB0IHdpdGggZWFjaCBEUk9QL0FERCBpbiBhIHNlcGFy
YXRlIHRyYW5zYWN0aW9uCmVjaG8gIlNFTEVDVCBwZ19zbGVlcChyYW5kb20oKS8xMCk7IiA+
PiBzaW5nbGUuc3FsOwoKIyBtb3ZlIHRoYXQgdG8gdGhlIG91dHB1dCBkaXJlY3RvcnkKbXYg
aW5zZXJ0LnNxbCBzaW5nbGUuc3FsIG11bHRpLnNxbCAkT1VURElSLwoKIyBydW4gcGdiZW5j
aCBvbiB0aGUgc291cmNlIGluIHRoZSBiYWNrZ3JvdW5kICg1bSBpcyBvdmVya2lsbCwgd2Ug
d2lsbCB0ZXJtaW5hdGUpCnBnYmVuY2ggLW4gLWMgOCAtZiAkT1VURElSL2luc2VydC5zcWwg
LXAgJFBVQl9QT1JUIC1UICREVVJBVElPTl9XT1JLTE9BRCB0ZXN0ID4gJE9VVERJUi9wZ2Jl
bmNoLmxvZyAyPiYxICYKCiMgY3JlYXRlIHB1YmxpY2F0aW9uIG9uIDEyIGFuZCBhZGQgdGhl
IHRhYmxlIHRvIGl0CmVjaG8gW2BkYXRlYF0gW2BkYXRlICslc2BdICJjcmVhdGluZyBwdWJs
aWNhdGlvbiIKcHNxbCAtcCAkUFVCX1BPUlQgdGVzdCAtYyAiQ1JFQVRFIFBVQkxJQ0FUSU9O
IHAiCgojIHNsZWVwIGZvciBhIGZldyBzZWNvbmRzCmVjaG8gW2BkYXRlYF0gW2BkYXRlICsl
c2BdICJzbGVlcGluZyIKc2xlZXAgJFNMRUVQCgojIGFkZCBjcmVhdGUgc3Vic2NyaXB0aW9u
IG9uIDE0CmVjaG8gW2BkYXRlYF0gW2BkYXRlICslc2BdICJjcmVhdGluZyBzdWJzY3JpcHRp
b24iCnBzcWwgLXAgJFNVQl9QT1JUIHRlc3QgLWMgIkNSRUFURSBTVUJTQ1JJUFRJT04gcyBD
T05ORUNUSU9OICdhcHBsaWNhdGlvbl9uYW1lPXRlc3QgaG9zdD1sb2NhbGhvc3QgdXNlcj11
c2VyIGRibmFtZT10ZXN0IHBvcnQ9JFBVQl9QT1JUJyBQVUJMSUNBVElPTiBwIgoKIyBzbGVl
cCBmb3IgYSBmZXcgc2Vjb25kcwplY2hvIFtgZGF0ZWBdIFtgZGF0ZSArJXNgXSAic2xlZXBp
bmciCnNsZWVwICRTTEVFUAoKIyByZWZyZXNoIHRoZSBzdWJzY3JpcHRpb24gaW4gYSBsb29w
Ci4vcmVmcmVzaC5zaCAkU1VCX1BPUlQgJERVUkFUSU9OX1JFRlJFU0ggPiAkT1VURElSL3Jl
ZnJlc2gubG9nIDI+JjEgJgoKIyBhZGQgdGFibGVzLCBvY2Nhc2lvbmFsbHkgZG8gcmVmcmVz
aApmb3IgcyBpbiBgc2VxIDEgJE5VTVRBQkxFU2A7IGRvCgoJbHNuPWBwc3FsIC1wICRQVUJf
UE9SVCB0ZXN0IC10IC1BIC1jICJTRUxFQ1QgcGdfY3VycmVudF93YWxfbHNuKCkiYAoJZWNo
byBbYGRhdGVgXSBbYGRhdGUgKyVzYF0gImFkZGluZyB0YWJsZSAkcyB0byBwdWJsaWNhdGlv
biBbJGxzbl0iCgoJcHNxbCAtcCAkUFVCX1BPUlQgdGVzdCAtYyAiQUxURVIgUFVCTElDQVRJ
T04gcCBBREQgVEFCTEUgdF8ke3N9IgoKCW09JCgodCAlIFJFRlJFU0gpKQoKCWlmIFsgIiRt
IiA9PSAiMCIgXTsgdGhlbgoKCQlsc249YHBzcWwgLXAgJFBVQl9QT1JUIHRlc3QgLXQgLUEg
LWMgIlNFTEVDVCBwZ19jdXJyZW50X3dhbF9sc24oKSJgCgkJZWNobyBbYGRhdGVgXSBbYGRh
dGUgKyVzYF0gInJlZnJlc2hpbmcgcHVibGljYXRpb24gWyRsc25dIgoKCQlwc3FsIC1wICRT
VUJfUE9SVCB0ZXN0IC1jICJBTFRFUiBTVUJTQ1JJUFRJT04gcyBSRUZSRVNIIFBVQkxJQ0FU
SU9OIgoJCXBzcWwgLXAgJFBVQl9QT1JUIHRlc3QgLWMgIlNFTEVDVCBwZ19jdXJyZW50X3dh
bF9sc24oKSBBUyB3YWxfbHNuLCBwZ19jdXJyZW50X3dhbF9pbnNlcnRfbHNuKCkgQVMgaW5z
ZXJ0X2xzbiwgKFNFTEVDVCByZWxuYW1lIEZST00gcGdfY2xhc3MgQVMgYyBXSEVSRSBjLm9p
ZCA9IHAucHJyZWxpZCkgQVMgcmVsbmFtZSwgKiBGUk9NIHBnX3B1YmxpY2F0aW9uX3JlbCBw
IE9SREVSIEJZIHBycmVsaWQiID4+ICRPVVRESVIvcGdfcHVibGljYXRpb25fcmVsLmxvZyAy
PiYxCglmaQoKZG9uZQoKbHNuPWBwc3FsIC1wICRQVUJfUE9SVCB0ZXN0IC10IC1BIC1jICJT
RUxFQ1QgcGdfY3VycmVudF93YWxfbHNuKCkiYAplY2hvIFtgZGF0ZWBdIFtgZGF0ZSArJXNg
XSAicmVmcmVzaGluZyBwdWJsaWNhdGlvbiBbJGxzbl0iCgojIGJ1dCBtYWtlIHN1cmUgdG8g
ZG8gYSByZWZyZXNoIGF0IHRoZSBlbmQKcHNxbCAtcCAkU1VCX1BPUlQgdGVzdCAtYyAiQUxU
RVIgU1VCU0NSSVBUSU9OIHMgUkVGUkVTSCBQVUJMSUNBVElPTiIKCnBzcWwgLXAgJFBVQl9Q
T1JUIHRlc3QgLWMgIlNFTEVDVCBwZ19jdXJyZW50X3dhbF9sc24oKSBBUyB3YWxfbHNuLCBw
Z19jdXJyZW50X3dhbF9pbnNlcnRfbHNuKCkgQVMgaW5zZXJ0X2xzbiwgKFNFTEVDVCByZWxu
YW1lIEZST00gcGdfY2xhc3MgQVMgYyBXSEVSRSBjLm9pZCA9IHAucHJyZWxpZCkgQVMgcmVs
bmFtZSwgKiBGUk9NIHBnX3B1YmxpY2F0aW9uX3JlbCBwIE9SREVSIEJZIHBycmVsaWQiID4+
ICRPVVRESVIvcGdfcHVibGljYXRpb25fcmVsLmxvZyAyPiYxCgojIHJ1biBwZ2JlbmNoIG9u
IHRoZSBzb3VyY2UgaW4gdGhlIGJhY2tncm91bmQsIGFkZGluZy9yZW1vdmluZyB0YWJsZXMK
cGdiZW5jaCAtbiAtZiAkT1VURElSL3NpbmdsZS5zcWwgLXAgJFBVQl9QT1JUIC1UICREVVJB
VElPTl9BTFRFUiB0ZXN0ID4gJE9VVERJUi9zaW5nbGUubG9nIDI+JjEgJgpwZ2JlbmNoIC1u
IC1mICRPVVRESVIvbXVsdGkuc3FsIC1wICRQVUJfUE9SVCAtVCAkRFVSQVRJT05fQUxURVIg
dGVzdCA+ICRPVVRESVIvbXVsdGkubG9nIDI+JjEgJgoKIyB3YWl0IGZvciBhbGwgdGFibGVz
IHRvIGdldCBpbiBzeW5jIChlc3NlbnRpYWxseSB3aGF0IHdhaXRfZm9yX3N1YnNjcmlwdGlv
bl9zeW5jIGRvZXMpCndoaWxlIC9iaW4vdHJ1ZTsgZG8KCWVjaG8gW2BkYXRlYF0gW2BkYXRl
ICslc2BdICJ3YWl0aW5nIGZvciBzdWJzY3JpcHRpb24gc3luYyIKCWM9YHBzcWwgLXAgJFNV
Ql9QT1JUIHRlc3QgLXQgLUEgLWMgIlNFTEVDVCBjb3VudCgqKSBGUk9NIHBnX3N1YnNjcmlw
dGlvbl9yZWwgV0hFUkUgc3JzdWJzdGF0ZSA9ICdyJyJgCglpZiBbICIkYyIgPT0gIiROVU1U
QUJMRVMiIF07IHRoZW4KCQlicmVhawoJZmkKCWVjaG8gW2BkYXRlYF0gW2BkYXRlICslc2Bd
ICJzeW5jZWQgJGMgdGFibGVzIG91dCBvZiAkTlVNVEFCTEVTIgoJc2xlZXAgMQpkb25lCgpl
Y2hvIFtgZGF0ZWBdIFtgZGF0ZSArJXNgXSAic3Vic2NyaXB0aW9uIHN5bmMgZG9uZSIKCiMg
d2FpdCBmb3IgdGhlIHBnYmVuY2ggdG8gZmluaXNoIChpZiBpdCdzIHN0aWxsIHJ1bm5pbmcp
CmVjaG8gW2BkYXRlYF0gW2BkYXRlICslc2BdICJ3YWl0aW5nIGZvciBwZ2JlbmNoIHRvIGNv
bXBsZXRlIgp3YWl0CgojIGdldCBjdXJyZW50IExTTiBvbiBwdWJsaXNoZXIgYW5kIHdhaXQg
Zm9yIHN1YnNjcmliZXIgdG8gZnVsbHkgY2F0Y2h1cCAoc2FtZSBhcyB3YWl0X2Zvcl9jYXRj
aHVwKQpsc249YHBzcWwgLXAgJFBVQl9QT1JUIHRlc3QgLXQgLUEgLWMgIlNFTEVDVCBwZ19j
dXJyZW50X3dhbF9sc24oKSJgCgplY2hvIFtgZGF0ZWBdIFtgZGF0ZSArJXNgXSAid2FpdGlu
ZyBmb3IgcmVwbGF5IG9mIExTTiAkbHNuIgoKc3RhcnRfbHNuPWBwc3FsIC1wICRQVUJfUE9S
VCB0ZXN0IC10IC1BIC1jICJTRUxFQ1QgcmVwbGF5X2xzbiBGUk9NIHBnX2NhdGFsb2cucGdf
c3RhdF9yZXBsaWNhdGlvbiBXSEVSRSBhcHBsaWNhdGlvbl9uYW1lID0gJ3Rlc3QnImAKc3Rh
cnRfdGltZT1gZGF0ZSArJXNgCgp3aGlsZSAvYmluL3RydWU7IGRvCgllY2hvIFtgZGF0ZWBd
IFtgZGF0ZSArJXNgXSAid2FpdGluZyBmb3IgY2F0Y2h1cCAkbHNuIgoJcmVwbGF5X2xzbj1g
cHNxbCAtcCAkUFVCX1BPUlQgdGVzdCAtdCAtQSAtYyAiU0VMRUNUIHJlcGxheV9sc24gRlJP
TSBwZ19jYXRhbG9nLnBnX3N0YXRfcmVwbGljYXRpb24gV0hFUkUgYXBwbGljYXRpb25fbmFt
ZSA9ICd0ZXN0JyJgCglyZW1haW5pbmdfd2FsPWBwc3FsIC1wICRQVUJfUE9SVCB0ZXN0IC10
IC1BIC1jICJTRUxFQ1QgKCckbHNuJyAtIHJlcGxheV9sc24pIEZST00gcGdfY2F0YWxvZy5w
Z19zdGF0X3JlcGxpY2F0aW9uIFdIRVJFIGFwcGxpY2F0aW9uX25hbWUgPSAndGVzdCciYAoJ
cmVtYWluaW5nX2tiPSQoKHJlbWFpbmluZ193YWwvMTAyNCkpCgoJY3VycmVudF90aW1lPWBk
YXRlICslc2AKCWRlbHRhX3RpbWU9JCgoY3VycmVudF90aW1lIC0gc3RhcnRfdGltZSArIDEp
KQoJZGVsdGFfd2FsPWBwc3FsIC1wICRQVUJfUE9SVCB0ZXN0IC10IC1BIC1jICJTRUxFQ1Qg
KHJlcGxheV9sc24gLSAnJHN0YXJ0X2xzbicpIEZST00gcGdfY2F0YWxvZy5wZ19zdGF0X3Jl
cGxpY2F0aW9uIFdIRVJFIGFwcGxpY2F0aW9uX25hbWUgPSAndGVzdCciYAoJdG90YWxfd2Fs
PWBwc3FsIC1wICRQVUJfUE9SVCB0ZXN0IC10IC1BIC1jICJTRUxFQ1QgKCckbHNuJzo6cGdf
bHNuIC0gJyRzdGFydF9sc24nOjpwZ19sc24pIEZST00gcGdfY2F0YWxvZy5wZ19zdGF0X3Jl
cGxpY2F0aW9uIFdIRVJFIGFwcGxpY2F0aW9uX25hbWUgPSAndGVzdCciYAoJc3BlZWQ9JCgo
ZGVsdGFfd2FsIC8gZGVsdGFfdGltZSkpCgoJaWYgWyAiJHNwZWVkIiAhPSAiMCIgXTsgdGhl
bgoJCXJlbWFpbmluZ19zZWM9JCgocmVtYWluaW5nX3dhbCAvIHNwZWVkKSkKCWVsc2UKCQly
ZW1haW5pbmdfc2VjPSc/Pz8nCglmaQoKCXI9YHBzcWwgLXAgJFBVQl9QT1JUIHRlc3QgLXQg
LUEgLWMgIlNFTEVDVCByZXBsYXlfbHNuID49ICckbHNuJyBBTkQgc3RhdGUgPSAnc3RyZWFt
aW5nJyBGUk9NIHBnX2NhdGFsb2cucGdfc3RhdF9yZXBsaWNhdGlvbiBXSEVSRSBhcHBsaWNh
dGlvbl9uYW1lID0gJ3Rlc3QnImAKCWlmIFsgIiRyIiA9PSAidCIgXTsgdGhlbgoJCWJyZWFr
CglmaQoJZWNobyBbYGRhdGVgXSBbYGRhdGUgKyVzYF0gInJlcGxheSBMU04gJHJlcGxheV9s
c24sIHJlbWFpbmluZyAkcmVtYWluaW5nX2tiIGtCIC8gJHJlbWFpbmluZ19zZWMgc2Vjb25k
cyIKCXNsZWVwIDEKZG9uZQoKIyBkdW1wIGJhc2ljIGluZm8gYWJvdXQgc3Vic2NyaXB0aW9u
IGFuZCBzdWJzY3JpYmVyCgpwc3FsIC1wICRTVUJfUE9SVCB0ZXN0IC1jICJTRUxFQ1Qgb2lk
LCByZWxuYW1lIEZST00gcGdfY2xhc3MgQVMgYyBXSEVSRSByZWxuYW1lIGxpa2UgJ3NlcV8l
JyBPUkRFUiBCWSByZWxuYW1lIiA+ICRPVVRESVIvcGdfY2xhc3MubG9nIDI+JjEKCnBzcWwg
LXAgJFNVQl9QT1JUIHRlc3QgLWMgIlNFTEVDVCAoU0VMRUNUIHJlbG5hbWUgRlJPTSBwZ19j
bGFzcyBBUyBjIFdIRVJFIGMub2lkID0gci5zcnJlbGlkKSBBUyByZWxuYW1lLCAqIEZST00g
cGdfc3Vic2NyaXB0aW9uX3JlbCBBUyByIE9SREVSIEJZIDEiID4gJE9VVERJUi9wZ19zdWJz
Y3JpcHRpb25fcmVsLmxvZyAyPiYxCgpwc3FsIC1wICRQVUJfUE9SVCB0ZXN0IC1jICJTRUxF
Q1QgcGdfY3VycmVudF93YWxfbHNuKCkgQVMgd2FsX2xzbiwgcGdfY3VycmVudF93YWxfaW5z
ZXJ0X2xzbigpIEFTIGluc2VydF9sc24sICogRlJPTSBwZ19zdGF0X3JlcGxpY2F0aW9uIiA+
ICRPVVRESVIvcGdfc3RhdF9yZXBsaWNhdGlvbi5sb2cgMj4mMQoKIyBjaGVjayBjb25zaXN0
ZW5jeSAtIGNvbXBhcmUgbnVtYmVyIG9mIHJlY29yZHMsIHN1bSwgLi4uIGNvdWxkIGJlIG1h
ZGUgbW9yZSB0aG9yb3VnaCwgYnV0IGdvb2QgZW5vdWdoCmVjaG8gW2BkYXRlYF0gW2BkYXRl
ICslc2BdICJjYXVnaHQgdXAsIGNoZWNrIGNvbnNpc3RlbmN5IgpyPSJPSyIKYz0wCmZvciB0
IGluIGBzZXEgMSAkTlVNVEFCTEVTYDsgZG8KCWE9YHBzcWwgLXAgJFBVQl9QT1JUIHRlc3Qg
LXQgLUEgLWMgIlNFTEVDVCBpZCwgdmFsdWUgRlJPTSB0XyR7dH0iYAoJYj1gcHNxbCAtcCAk
U1VCX1BPUlQgdGVzdCAtdCAtQSAtYyAiU0VMRUNUIGlkLCB2YWx1ZSBGUk9NIHRfJHt0fSJg
CglpZiBbICIkYSIgPT0gIiRiIiBdOyB0aGVuCgkJZWNobyBbYGRhdGVgXSBbYGRhdGUgKyVz
YF0gIiR0IE9LICgkYSAkYikiCgllbHNlCgkJZWNobyBbYGRhdGVgXSBbYGRhdGUgKyVzYF0g
IiR0IEVSUk9SICgkYSAkYikiCgkJcj0iRVJST1IiCgkJYz0xCglmaQpkb25lCgojIGNsZWFu
dXAgdGFibGVzIG9uIGJvdGggZW5kcwpmb3IgdCBpbiBgc2VxIDEgJE5VTVRBQkxFU2A7IGRv
Cglwc3FsIC1wICRQVUJfUE9SVCB0ZXN0IC1jICJTRUxFQ1QgKiBGUk9NIHRfJHt0fSIgPiAk
T1VURElSL3Rlc3RfJHt0fV9wdWIuZGF0YQoJcHNxbCAtcCAkU1VCX1BPUlQgdGVzdCAtYyAi
U0VMRUNUICogRlJPTSB0XyR7dH0iID4gJE9VVERJUi90ZXN0XyR7dH1fc3ViLmRhdGEKZG9u
ZQoKcGdfY3RsIC1EICRQVUJfREFUQSAtdyAtbSBpbW1lZGlhdGUgc3RvcApwZ19jdGwgLUQg
JFNVQl9EQVRBIC13IC1tIGltbWVkaWF0ZSBzdG9wCgojIGlmIHdlbnQgT0ssIHJlbW92ZSB0
aGUgZGF0YSBkaXJlY3RvcmllcwppZiBbICIkciIgPT0gIk9LIiBdOyB0aGVuCglybSAtUmYg
JFBVQl9EQVRBCglybSAtUmYgJFNVQl9EQVRBCgkjIHJlbW92ZSB0aGUgbG9nIGZpbGVzIHRv
bywgbWF5IGJlIHByZXR0eSBsYXJnZSBkdWUgdG8gZGVidWdnaW5nCglybSAkT1VURElSL3Bn
LXB1Yi5sb2cgJE9VVERJUi9wZy1zdWIubG9nCmVsc2UKCSMgZm9yIGVycm9yLCBjb21wcmVz
cyB0aGUgZGF0YSBkaXJlY3RvcmllcwoJdGFyIC1jICRQVUJfREFUQSB8IGd6aXAgLS1mYXN0
ID4gJFBVQl9EQVRBLnRnegoJdGFyIC1jICRTVUJfREFUQSB8IGd6aXAgLS1mYXN0ID4gJFNV
Ql9EQVRBLnRnegoJIyBhbmQgYWxzbyB0aGUgZGF0YSBkaXJlY3RvcmllcwoJcGlneiAtLWZh
c3QgJE9VVERJUi9wZy1wdWIubG9nICRPVVRESVIvcGctc3ViLmxvZwpmaQoKZGF0ZQoKRU5E
X1RJTUVTVEFNUD1gZGF0ZSArJXNgCgplY2hvIFtgZGF0ZWBdIFtgZGF0ZSArJXNgXSAiUlVO
VElNRTogIiAkKChFTkRfVElNRVNUQU1QIC0gU1RBUlRfVElNRVNUQU1QKSkKZWNobyBbYGRh
dGVgXSBbYGRhdGUgKyVzYF0gIlJFU1VMVDogJHIiCgpleGl0ICRjCg==
--------------tpO8apL2U5SEAvKbdSzGITgq
Content-Type: application/x-shellscript; name="refresh.sh"
Content-Disposition: attachment; filename="refresh.sh"
Content-Transfer-Encoding: base64

IyEvdXNyL2Jpbi9iYXNoCgpQT1JUPSQxCkRVUkFUSU9OPSQyCgojIHJlY3JlYXRlIGRpcmVj
dG9yeSB3aXRoIHNlcXVlbmNlIHN0YXRlcwpybSAtUmYgc3RhdGVzCm1rZGlyIHN0YXRlcwoK
c3RhcnRfdGltZT1gZGF0ZSArJXNgCm5sb29wcz0wCm5lcnJvcnM9MAoKd2hpbGUgL2Jpbi90
cnVlOyBkbwoKCXBzcWwgLXAgJFBPUlQgdGVzdCAtYyAiQUxURVIgU1VCU0NSSVBUSU9OIHMg
UkVGUkVTSCBQVUJMSUNBVElPTiIKCglubG9vcHM9JCgobmxvb3BzKzEpKQoKCWVjaG8gImxv
b3AgJG5sb29wcyIKCgl0PWBkYXRlICslc2AKCWQ9JCgodC1zdGFydF90aW1lKSkKCglpZiBb
ICRkIC1ndCAkRFVSQVRJT04gXTsgdGhlbgoJCWJyZWFrCglmaQoKCSMgcHNxbCAtcCAkU1VC
X1BPUlQgdGVzdCAtYyAiU0VMRUNUIHBnX3NsZWVwKHJhbmRvbSgpKSIKCmRvbmUKCmVjaG8g
J2xvb3BzJyAkbmxvb3BzICdlcnJvcnMnICRuZXJyb3JzCg==

--------------tpO8apL2U5SEAvKbdSzGITgq--