Content-Type: multipart/mixed; boundary="------------HATO70W9anbTTpzMQbaQ89u3"
Message-ID: <163fcbec-900b-4b07-beaa-d2ead8634bec@postgrespro.ru>
Date: Wed, 26 Mar 2025 17:17:17 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
To: pgsql-hackers@lists.postgresql.org
Content-Language: en-US
From: "Maksim.Melnikov" <m.melnikov@postgrespro.ru>
Subject: sync_standbys_defined read/write race on startup
Archived-At: 
 <https://www.postgresql.org/message-id/163fcbec-900b-4b07-beaa-d2ead8634bec%40postgrespro.ru>
Precedence: bulk

This is a multi-part message in MIME format.
--------------HATO70W9anbTTpzMQbaQ89u3
Content-Type: multipart/alternative;
 boundary="------------CuOUBTqCMZq1wgPx9gzPMR0N"

--------------CuOUBTqCMZq1wgPx9gzPMR0N
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,
On my windows machine I've found unstable test, it fails very rarely, 
but anyway, sometimes it fails.

src/test/recovery/t/009_twophase.pl

#   Failed test 'Committed prepared transaction is visible to new 
snapshot in standby'
#   at t/009_twophase.pl line 360.
#          got: '0'
#     expected: '2'
# Looks like you failed 1 test of 30.
t/009_twophase.pl ....................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/30 subtests

cat src/test/recovery/t/009_twophase.pl

....
$cur_primary->psql(
     'postgres', "
     SET synchronous_commit='remote_apply'; -- To ensure the standby is 
caught up
     CREATE TABLE t_009_tbl_standby_mvcc (id int, msg text);
     BEGIN;
     INSERT INTO t_009_tbl_standby_mvcc VALUES (1, 'issued to 
${cur_primary_name}');
     SAVEPOINT s1;
     INSERT INTO t_009_tbl_standby_mvcc VALUES (2, 'issued to 
${cur_primary_name}');
     PREPARE TRANSACTION 'xact_009_standby_mvcc';
     ");
$cur_primary->stop;
$cur_standby->restart;

# Acquire a snapshot in standby, before we commit the prepared transaction
my $standby_session =
   $cur_standby->background_psql('postgres', on_error_die => 1);
$standby_session->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
$psql_out =
   $standby_session->query_safe("SELECT count(*) FROM 
t_009_tbl_standby_mvcc");
is($psql_out, '0',
     "Prepared transaction not visible in standby before commit");

# Commit the transaction in primary
$cur_primary->start;
$cur_primary->psql(
     'postgres', "
SET synchronous_commit='remote_apply'; -- To ensure the standby is caught up
COMMIT PREPARED 'xact_009_standby_mvcc';
");

# Still not visible to the old snapshot
$psql_out =
   $standby_session->query_safe("SELECT count(*) FROM 
t_009_tbl_standby_mvcc");
is($psql_out, '0',
     "Committed prepared transaction not visible to old snapshot in 
standby");

# Is visible to a new snapshot
$standby_session->query_safe("COMMIT");
$psql_out =
   $standby_session->query_safe("SELECT count(*) FROM 
t_009_tbl_standby_mvcc");
*is($psql_out, '2',
     "Committed prepared transaction is visible to new snapshot in 
standby");*
$standby_session->quit;
.....


Test check visibility of prepared transactions in standby after a 
restart while primary is down.
Failed assert check that changes, commited on transaction on primary 
node, were synchronously replicated to standby node.
In test flow we have primary server stop/start events.
Assert failure is hard to reproduce, but I found how to do it 
synthetically, can do it by injecting in code synthetical sleep call

--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -924,6 +924,7 @@ SyncRepUpdateSyncStandbysDefined(void)

         if (sync_standbys_defined != WalSndCtl->sync_standbys_defined)
         {
+               sleep(2);
                 LWLockAcquire(SyncRepLock, LW_EXCLUSIVE);

                 /*


The root cause is WalSndCtl->sync_standbys_defined, it is located in 
shared memory and can be updated by checkpointer process only,
in backends it can be read in case of sync replication. So we have race 
on postgres startup between backend, processing commit request, and 
checkpointer,
setting WalSndCtl->sync_standbys_defined.

Synchronous replication occurred in SyncRepWaitForLSN(syncrep.c)

void
SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
{
.....

     /*
      * Fast exit if user has not requested sync replication, or there 
are no
      * sync replication standby names defined.
      *
      * Since this routine gets called every commit time, it's important to
      * exit quickly if sync replication is not requested. So we check
      * WalSndCtl->sync_standbys_defined flag without the lock and exit
      * immediately if it's false. If it's true, we need to check it again
      * later while holding the lock, to check the flag and operate the sync
      * rep queue atomically. This is necessary to avoid the race condition
      * described in SyncRepUpdateSyncStandbysDefined(). On the other 
hand, if
      * it's false, the lock is not necessary because we don't touch the 
queue.
      */
     if (!SyncRepRequested() ||
         !((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
         return;

.....

     if (!WalSndCtl->sync_standbys_defined ||
         lsn <= WalSndCtl->lsn[mode])
     {
         LWLockRelease(SyncRepLock);
         return;
     }

...........


Checkpointer set WalSndCtl->sync_standbys_defined parameter in 
src/backend/replication/syncrep.c,
SyncRepUpdateSyncStandbysDefined method

void
SyncRepUpdateSyncStandbysDefined(void)
{
     bool        sync_standbys_defined = SyncStandbysDefined();

     if (sync_standbys_defined != WalSndCtl->sync_standbys_defined)
     {
         LWLockAcquire(SyncRepLock, LW_EXCLUSIVE);
..................
         WalSndCtl->sync_standbys_defined = sync_standbys_defined;

         LWLockRelease(SyncRepLock);
     }
}

So my question is
can we accept such behavior as bug of postgres codebase, or, on other 
way, it is bug of unstable test?
P.S.
For me it is looking strange, that postgres start processing workload 
before full config initializing.
Attached possible patch for codebase.

Best regards
Melnikov Maksim

--------------CuOUBTqCMZq1wgPx9gzPMR0N
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi, <br>
      On my windows machine I've found unstable test, it fails very
      rarely, but anyway, sometimes it fails.<br>
      <br>
      src/test/recovery/t/009_twophase.pl<br>
      <br>
      #   Failed test 'Committed prepared transaction is visible to new
      snapshot in standby'<br>
      #   at t/009_twophase.pl line 360.<br>
      #          got: '0'<br>
      #     expected: '2'<br>
      # Looks like you failed 1 test of 30.<br>
      t/009_twophase.pl .................... <br>
      Dubious, test returned 1 (wstat 256, 0x100)<br>
      Failed 1/30 subtests <br>
      <br>
      cat src/test/recovery/t/009_twophase.pl<br>
      <br>
      ....<br>
      $cur_primary-&gt;psql(<br>
          'postgres', "<br>
          SET synchronous_commit='remote_apply'; -- To ensure the
      standby is caught up<br>
          CREATE TABLE t_009_tbl_standby_mvcc (id int, msg text);<br>
          BEGIN;<br>
          INSERT INTO t_009_tbl_standby_mvcc VALUES (1, 'issued to
      ${cur_primary_name}');<br>
          SAVEPOINT s1;<br>
          INSERT INTO t_009_tbl_standby_mvcc VALUES (2, 'issued to
      ${cur_primary_name}');<br>
          PREPARE TRANSACTION 'xact_009_standby_mvcc';<br>
          ");<br>
      $cur_primary-&gt;stop;<br>
      $cur_standby-&gt;restart;<br>
      <br>
      # Acquire a snapshot in standby, before we commit the prepared
      transaction<br>
      my $standby_session =<br>
        $cur_standby-&gt;background_psql('postgres', on_error_die =&gt;
      1);<br>
      $standby_session-&gt;query_safe("BEGIN ISOLATION LEVEL REPEATABLE
      READ");<br>
      $psql_out =<br>
        $standby_session-&gt;query_safe("SELECT count(*) FROM
      t_009_tbl_standby_mvcc");<br>
      is($psql_out, '0',<br>
          "Prepared transaction not visible in standby before commit");<br>
      <br>
      # Commit the transaction in primary<br>
      $cur_primary-&gt;start;<br>
      $cur_primary-&gt;psql(<br>
          'postgres', "<br>
      SET synchronous_commit='remote_apply'; -- To ensure the standby is
      caught up<br>
      COMMIT PREPARED 'xact_009_standby_mvcc';<br>
      ");<br>
      <br>
      # Still not visible to the old snapshot<br>
      $psql_out =<br>
        $standby_session-&gt;query_safe("SELECT count(*) FROM
      t_009_tbl_standby_mvcc");<br>
      is($psql_out, '0',<br>
          "Committed prepared transaction not visible to old snapshot in
      standby");<br>
      <br>
      # Is visible to a new snapshot<br>
      $standby_session-&gt;query_safe("COMMIT");<br>
      $psql_out =<br>
        $standby_session-&gt;query_safe("SELECT count(*) FROM
      t_009_tbl_standby_mvcc");<br>
      <b>is($psql_out, '2',<br>
            "Committed prepared transaction is visible to new snapshot
        in standby");</b><br>
      $standby_session-&gt;quit;<br>
      .....<br>
      <br>
      <br>
      <br>
      Test check visibility of prepared transactions in standby after a
      restart while primary is down.<br>
      Failed assert check that changes, commited on transaction on
      primary node, were synchronously replicated to standby node.<br>
      In test flow we have primary server stop/start events.<br>
      Assert failure is hard to reproduce, but I found how to do it
      synthetically, can do it by injecting in code synthetical sleep
      call<br>
      <br>
      --- a/src/backend/replication/syncrep.c<br>
      +++ b/src/backend/replication/syncrep.c<br>
      @@ -924,6 +924,7 @@ SyncRepUpdateSyncStandbysDefined(void)<br>
       <br>
              if (sync_standbys_defined !=
      WalSndCtl-&gt;sync_standbys_defined)<br>
              {<br>
      +               sleep(2);<br>
                      LWLockAcquire(SyncRepLock, LW_EXCLUSIVE);<br>
       <br>
                      /*<br>
                      <br>
      <br>
      The root cause is WalSndCtl-&gt;sync_standbys_defined, it is
      located in shared memory and can be updated by checkpointer
      process only,<br>
      in backends it can be read in case of sync replication. So we have
      race on postgres startup between backend, processing commit
      request, and checkpointer,<br>
      setting WalSndCtl-&gt;sync_standbys_defined.<br>
      <br>
      Synchronous replication occurred in SyncRepWaitForLSN(syncrep.c)<br>
      <br>
      void<br>
      SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)<br>
      {<br>
      .....<br>
      <br>
          /*<br>
           * Fast exit if user has not requested sync replication, or
      there are no<br>
           * sync replication standby names defined.<br>
           *<br>
           * Since this routine gets called every commit time, it's
      important to<br>
           * exit quickly if sync replication is not requested. So we
      check<br>
           * WalSndCtl-&gt;sync_standbys_defined flag without the lock
      and exit<br>
           * immediately if it's false. If it's true, we need to check
      it again<br>
           * later while holding the lock, to check the flag and operate
      the sync<br>
           * rep queue atomically. This is necessary to avoid the race
      condition<br>
           * described in SyncRepUpdateSyncStandbysDefined(). On the
      other hand, if<br>
           * it's false, the lock is not necessary because we don't
      touch the queue.<br>
           */<br>
          if (!SyncRepRequested() ||<br>
              !((volatile WalSndCtlData *)
      WalSndCtl)-&gt;sync_standbys_defined)<br>
              return;<br>
      <br>
      .....<br>
      <br>
          if (!WalSndCtl-&gt;sync_standbys_defined ||<br>
              lsn &lt;= WalSndCtl-&gt;lsn[mode])<br>
          {<br>
              LWLockRelease(SyncRepLock);<br>
              return;<br>
          }<br>
          <br>
      ...........<br>
      <br>
      <br>
      Checkpointer set WalSndCtl-&gt;sync_standbys_defined parameter in
      src/backend/replication/syncrep.c,<br>
      SyncRepUpdateSyncStandbysDefined method<br>
      <br>
      void<br>
      SyncRepUpdateSyncStandbysDefined(void)<br>
      {<br>
          bool        sync_standbys_defined = SyncStandbysDefined();<br>
      <br>
          if (sync_standbys_defined !=
      WalSndCtl-&gt;sync_standbys_defined)<br>
          {<br>
              LWLockAcquire(SyncRepLock, LW_EXCLUSIVE);<br>
      ..................<br>
              WalSndCtl-&gt;sync_standbys_defined =
      sync_standbys_defined;<br>
      <br>
              LWLockRelease(SyncRepLock);<br>
          }<br>
      }<br>
      <br>
      So my question is <br>
      can we accept such behavior as bug of postgres codebase, or, on
      other way, it is bug of unstable test?<br>
      P.S.<br>
      For me it is looking strange, that postgres start processing
      workload before full config initializing.<br>
      Attached possible patch for codebase.<br>
      <br>
      Best regards<br>
      Melnikov Maksim<br>
    </p>
  </body>
</html>

--------------CuOUBTqCMZq1wgPx9gzPMR0N--

--------------HATO70W9anbTTpzMQbaQ89u3
Content-Type: text/x-patch; charset=UTF-8;
 name="0001-Fix-sync_standbys_defined-race-on-startup.patch"
Content-Disposition: attachment;
 filename="0001-Fix-sync_standbys_defined-race-on-startup.patch"
Content-Transfer-Encoding: base64

RnJvbSA4ZmEyOTVjODFhOGRiODg1NjhkNDVkYTNmNzc5ZDMwOTdhZWI2OTQyIE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBNYWtzaW0gTWVsbmlrb3YgPG0ubWVsbmlrb3ZAcG9z
dGdyZXNwcm8ucnU+CkRhdGU6IFdlZCwgMjYgTWFyIDIwMjUgMTY6NDQ6MDkgKzAzMDAKU3Vi
amVjdDogW1BBVENIXSBGaXggc3luY19zdGFuZGJ5c19kZWZpbmVkIHJhY2Ugb24gc3RhcnR1
cC4KCldhbFNuZEN0bC0+c3luY19zdGFuZGJ5c19kZWZpbmVkLCBjYW4gYmUgdXBkYXRlZApi
eSBjaGVja3BvaW50ZXIgcHJvY2VzcyBvbmx5IGFuZCBpbiBiYWNrZW5kcyBpdApjYW4gYmUg
cmVhZC4gU28sIG9uIHBvc3RncmVzIHN0YXJ0dXAsIHdlIGhhdmUgcmFjZQpiZXR3ZWVuIGNo
ZWNrcG9pbnRlciBwcm9jZXNzIGFuZCBiYWNrZW5kcywgdGhhdCBjaGVjawpzeW5jX3N0YW5k
YnlzX2RlZmluZWQgZm9yIHN5bmMgcmVwbGljYXRpb25zLgoKVG8gYXZvaWQgdGhpcywgbm93
IHdlIGFyZSB3b3JraW5nIHdpdGggc3luY19zdGFuZGJ5c19kZWZpbmVkCm5vdCBhcyBib29s
IGZpZWxkLCBidXQgYXMgYml0bWFzaywgd2hlcmUgN3RoIGJpdCBjaGVjawpzdGFuZGJ5c19k
ZWZpbmVkIG9yIG5vdCwgOHRoIGJpdCBjaGVjawpXYWxTbmRDdGxEYXRhLT5zeW5jX3N0YW5k
YnlzX2RlZmluZWQgZmllbGQgaGFzIGJlZW4KaW5pdGlhbGl6ZWQgYnkgY2hlY2twb2ludGVy
LiBTbyBpZiBpbiBzeW5jaHJvbm91cyByZXBsaWNhdGlvbiBmdW5jdGlvbnMKd2FzIGRldGVj
dGVkIHRoYXQgc3luY19zdGFuZGJ5c19kZWZpbmVkIGlzIG5vdCBpbml0aWFsaXplZCwKY2hl
Y2sgU3luY1N0YW5kYnlzRGVmaW5lZCgpIHZhbHVlLgotLS0KIHNyYy9iYWNrZW5kL3JlcGxp
Y2F0aW9uL3N5bmNyZXAuYyAgICAgICAgICAgfCAzMSArKysrKysrKysrKysrKysrKy0tLS0K
IHNyYy9pbmNsdWRlL3JlcGxpY2F0aW9uL3dhbHNlbmRlcl9wcml2YXRlLmggfCAxMCArKysr
KystCiAyIGZpbGVzIGNoYW5nZWQsIDM0IGluc2VydGlvbnMoKyksIDcgZGVsZXRpb25zKC0p
CgpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vc3luY3JlcC5jIGIvc3Jj
L2JhY2tlbmQvcmVwbGljYXRpb24vc3luY3JlcC5jCmluZGV4IGQ3NWUzOTY4MDM1Li5jZDI5
YmZiZGM5NCAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vc3luY3JlcC5j
CisrKyBiL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL3N5bmNyZXAuYwpAQCAtMTQ4LDYgKzE0
OCw3IEBAIHZvaWQKIFN5bmNSZXBXYWl0Rm9yTFNOKFhMb2dSZWNQdHIgbHNuLCBib29sIGNv
bW1pdCkKIHsKIAlpbnQJCQltb2RlOworCWJvb2wJCXN5bmNfc3RhbmRieXNfZGVmaW5lZDsK
IAogCS8qCiAJICogVGhpcyBzaG91bGQgYmUgY2FsbGVkIHdoaWxlIGhvbGRpbmcgaW50ZXJy
dXB0cyBkdXJpbmcgYSB0cmFuc2FjdGlvbgpAQCAtMTcwLDcgKzE3MSw4IEBAIFN5bmNSZXBX
YWl0Rm9yTFNOKFhMb2dSZWNQdHIgbHNuLCBib29sIGNvbW1pdCkKIAkgKiBpdCdzIGZhbHNl
LCB0aGUgbG9jayBpcyBub3QgbmVjZXNzYXJ5IGJlY2F1c2Ugd2UgZG9uJ3QgdG91Y2ggdGhl
IHF1ZXVlLgogCSAqLwogCWlmICghU3luY1JlcFJlcXVlc3RlZCgpIHx8Ci0JCSEoKHZvbGF0
aWxlIFdhbFNuZEN0bERhdGEgKikgV2FsU25kQ3RsKS0+c3luY19zdGFuZGJ5c19kZWZpbmVk
KQorCQkoKCh2b2xhdGlsZSBXYWxTbmRDdGxEYXRhICopIFdhbFNuZEN0bCktPnN5bmNfc3Rh
bmRieXNfZGVmaW5lZCAmIFNZTkNTVEFOREJZU0RFRklORURJTklUSUFMSVpFRCkKKwkJPT0g
U1lOQ1NUQU5EQllTSU5JVElBTElaRUQpCiAJCXJldHVybjsKIAogCS8qIENhcCB0aGUgbGV2
ZWwgZm9yIGFueXRoaW5nIG90aGVyIHRoYW4gY29tbWl0IHRvIHJlbW90ZSBmbHVzaCBvbmx5
LiAqLwpAQCAtMTg0LDYgKzE4Niw4IEBAIFN5bmNSZXBXYWl0Rm9yTFNOKFhMb2dSZWNQdHIg
bHNuLCBib29sIGNvbW1pdCkKIAogCUxXTG9ja0FjcXVpcmUoU3luY1JlcExvY2ssIExXX0VY
Q0xVU0lWRSk7CiAJQXNzZXJ0KE15UHJvYy0+c3luY1JlcFN0YXRlID09IFNZTkNfUkVQX05P
VF9XQUlUSU5HKTsKKwlzeW5jX3N0YW5kYnlzX2RlZmluZWQgPSBXYWxTbmRDdGwtPnN5bmNf
c3RhbmRieXNfZGVmaW5lZCAmIFNZTkNTVEFOREJZU0lOSVRJQUxJWkVECisJCT8gV2FsU25k
Q3RsLT5zeW5jX3N0YW5kYnlzX2RlZmluZWQgJiBTWU5DU1RBTkRCWVNERUZJTkVEIDogU3lu
Y1N0YW5kYnlzRGVmaW5lZCgpOwogCiAJLyoKIAkgKiBXZSBkb24ndCB3YWl0IGZvciBzeW5j
IHJlcCBpZiBXYWxTbmRDdGwtPnN5bmNfc3RhbmRieXNfZGVmaW5lZCBpcyBub3QKQEAgLTE5
Myw3ICsxOTcsNyBAQCBTeW5jUmVwV2FpdEZvckxTTihYTG9nUmVjUHRyIGxzbiwgYm9vbCBj
b21taXQpCiAJICogY29uZGl0aW9uIGJ1dCB3ZSdsbCBiZSBmZXRjaGluZyB0aGF0IGNhY2hl
IGxpbmUgYW55d2F5IHNvIGl0J3MgbGlrZWx5CiAJICogdG8gYmUgYSBsb3cgY29zdCBjaGVj
ay4KIAkgKi8KLQlpZiAoIVdhbFNuZEN0bC0+c3luY19zdGFuZGJ5c19kZWZpbmVkIHx8CisJ
aWYgKCFzeW5jX3N0YW5kYnlzX2RlZmluZWQgfHwKIAkJbHNuIDw9IFdhbFNuZEN0bC0+bHNu
W21vZGVdKQogCXsKIAkJTFdMb2NrUmVsZWFzZShTeW5jUmVwTG9jayk7CkBAIC05MjAsOSAr
OTI0LDEwIEBAIFN5bmNSZXBXYWtlUXVldWUoYm9vbCBhbGwsIGludCBtb2RlKQogdm9pZAog
U3luY1JlcFVwZGF0ZVN5bmNTdGFuZGJ5c0RlZmluZWQodm9pZCkKIHsKLQlib29sCQlzeW5j
X3N0YW5kYnlzX2RlZmluZWQgPSBTeW5jU3RhbmRieXNEZWZpbmVkKCk7CisJYm9vbAkJbmV3
X3N5bmNfc3RhbmRieXNfZGVmaW5lZCA9IFN5bmNTdGFuZGJ5c0RlZmluZWQoKTsKKwlib29s
CQlvbGRfc3luY19zdGFuZGJ5c19kZWZpbmVkID0gKFdhbFNuZEN0bC0+c3luY19zdGFuZGJ5
c19kZWZpbmVkICYgU1lOQ1NUQU5EQllTREVGSU5FRCk7CiAKLQlpZiAoc3luY19zdGFuZGJ5
c19kZWZpbmVkICE9IFdhbFNuZEN0bC0+c3luY19zdGFuZGJ5c19kZWZpbmVkKQorCWlmIChu
ZXdfc3luY19zdGFuZGJ5c19kZWZpbmVkICE9IG9sZF9zeW5jX3N0YW5kYnlzX2RlZmluZWQp
CiAJewogCQlMV0xvY2tBY3F1aXJlKFN5bmNSZXBMb2NrLCBMV19FWENMVVNJVkUpOwogCkBA
IC05MzEsNyArOTM2LDcgQEAgU3luY1JlcFVwZGF0ZVN5bmNTdGFuZGJ5c0RlZmluZWQodm9p
ZCkKIAkJICogZm9yIGJhY2tlbmRzIHRvIGNvbnRpbnVlIHdhaXRpbmcuICBTaW5jZSB0aGUg
dXNlciBubyBsb25nZXIgd2FudHMKIAkJICogc3luY2hyb25vdXMgcmVwbGljYXRpb24sIHdl
J2QgYmV0dGVyIHdha2UgdGhlbSB1cC4KIAkJICovCi0JCWlmICghc3luY19zdGFuZGJ5c19k
ZWZpbmVkKQorCQlpZiAoIW5ld19zeW5jX3N0YW5kYnlzX2RlZmluZWQpCiAJCXsKIAkJCWlu
dAkJCWk7CiAKQEAgLTk0NiwxMCArOTUxLDI0IEBAIFN5bmNSZXBVcGRhdGVTeW5jU3RhbmRi
eXNEZWZpbmVkKHZvaWQpCiAJCSAqIGJhY2tlbmQgdGhhdCBoYXNuJ3QgeWV0IHJlbG9hZGVk
IGl0cyBjb25maWcgbWlnaHQgZ28gdG8gc2xlZXAgb24KIAkJICogdGhlIHF1ZXVlIChhbmQg
bmV2ZXIgd2FrZSB1cCkuICBUaGlzIHByZXZlbnRzIHRoYXQuCiAJCSAqLwotCQlXYWxTbmRD
dGwtPnN5bmNfc3RhbmRieXNfZGVmaW5lZCA9IHN5bmNfc3RhbmRieXNfZGVmaW5lZDsKKwkJ
V2FsU25kQ3RsLT5zeW5jX3N0YW5kYnlzX2RlZmluZWQgfD0gbmV3X3N5bmNfc3RhbmRieXNf
ZGVmaW5lZAorCQkJPyAoU1lOQ1NUQU5EQllTREVGSU5FRElOSVRJQUxJWkVEKQorCQkJOiBT
WU5DU1RBTkRCWVNJTklUSUFMSVpFRDsKIAogCQlMV0xvY2tSZWxlYXNlKFN5bmNSZXBMb2Nr
KTsKIAl9CisJZWxzZSBpZiAoIShXYWxTbmRDdGwtPnN5bmNfc3RhbmRieXNfZGVmaW5lZCAm
IFNZTkNTVEFOREJZU0lOSVRJQUxJWkVEKSkKKwl7CisJCS8qCisJCSAqIEV2ZW4gaWYgbmV3
X3N5bmNfc3RhbmRieXNfZGVmaW5lZCA9PSBvbGRfc3luY19zdGFuZGJ5c19kZWZpbmVkLCB3
ZQorCQkgKiBzaG91bGQgY2hlY2sgV2FsU25kQ3RsLT5zeW5jX3N0YW5kYnlzX2RlZmluZWQg
aXMgaW5pdGlhbGl6ZWQgYW5kCisJCSAqIHNldCBpdHMgaW5pdGlhbGl6ZSBiaXQsIGlmIGl0
IHJlc2V0LiBBbnl3YXkgdGhpcyBiaXQgc2hvdWxkIGJlIHNldAorCQkgKiBvbmx5IG9uIHN0
YXJ0IGFuZCB0aGVuIGl0IHdpbGwgYmUgYWx3YXlzIHNldHRlZC4KKwkJICovCisJCUxXTG9j
a0FjcXVpcmUoU3luY1JlcExvY2ssIExXX0VYQ0xVU0lWRSk7CisJCVdhbFNuZEN0bC0+c3lu
Y19zdGFuZGJ5c19kZWZpbmVkIHw9IFNZTkNTVEFOREJZU0lOSVRJQUxJWkVEOworCQlMV0xv
Y2tSZWxlYXNlKFN5bmNSZXBMb2NrKTsKKwl9CiB9CiAKICNpZmRlZiBVU0VfQVNTRVJUX0NI
RUNLSU5HCmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS9yZXBsaWNhdGlvbi93YWxzZW5kZXJf
cHJpdmF0ZS5oIGIvc3JjL2luY2x1ZGUvcmVwbGljYXRpb24vd2Fsc2VuZGVyX3ByaXZhdGUu
aAppbmRleCAwZmM3N2YxYjRhZi4uNWIwOTZhMWRjOTAgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNs
dWRlL3JlcGxpY2F0aW9uL3dhbHNlbmRlcl9wcml2YXRlLmgKKysrIGIvc3JjL2luY2x1ZGUv
cmVwbGljYXRpb24vd2Fsc2VuZGVyX3ByaXZhdGUuaApAQCAtMTAwLDcgKzEwMCw3IEBAIHR5
cGVkZWYgc3RydWN0CiAJICogY29uZmlnIGZpbGUgc2FmZWx5LCBzbyBjaGVja3BvaW50ZXIg
dXBkYXRlcyB0aGlzIHZhbHVlIGFzIG5lZWRlZC4KIAkgKiBQcm90ZWN0ZWQgYnkgU3luY1Jl
cExvY2suCiAJICovCi0JYm9vbAkJc3luY19zdGFuZGJ5c19kZWZpbmVkOworCXVpbnQ4CQlz
eW5jX3N0YW5kYnlzX2RlZmluZWQ7CiAKIAkvKiB1c2VkIGFzIGEgcmVnaXN0cnkgb2YgcGh5
c2ljYWwgLyBsb2dpY2FsIHdhbHNlbmRlcnMgdG8gd2FrZSAqLwogCUNvbmRpdGlvblZhcmlh
YmxlIHdhbF9mbHVzaF9jdjsKQEAgLTExNiw2ICsxMTYsMTQgQEAgdHlwZWRlZiBzdHJ1Y3QK
IAlXYWxTbmQJCXdhbHNuZHNbRkxFWElCTEVfQVJSQVlfTUVNQkVSXTsKIH0gV2FsU25kQ3Rs
RGF0YTsKIAorLyoKKyAqIFdhbFNuZEN0bERhdGEtPnN5bmNfc3RhbmRieXNfZGVmaW5lZCBi
aXQgZmxhZ3MuIDd0aCBiaXQgY2hlY2sgc3RhbmRieXNfZGVmaW5lZCBvciBub3QsCisgKiA4
dGggYml0IGNoZWNrIFdhbFNuZEN0bERhdGEtPnN5bmNfc3RhbmRieXNfZGVmaW5lZCBmaWVs
ZCBoYXMgYmVlbiBpbml0aWFsaXplZC4KKyAqLworI2RlZmluZSBTWU5DU1RBTkRCWVNJTklU
SUFMSVpFRAkoMSA8PCAwKQorI2RlZmluZSBTWU5DU1RBTkRCWVNERUZJTkVECQkoMSA8PCAy
KQorI2RlZmluZSBTWU5DU1RBTkRCWVNERUZJTkVESU5JVElBTElaRUQJKFNZTkNTVEFOREJZ
U0lOSVRJQUxJWkVEICYgU1lOQ1NUQU5EQllTREVGSU5FRCkKKwogZXh0ZXJuIFBHRExMSU1Q
T1JUIFdhbFNuZEN0bERhdGEgKldhbFNuZEN0bDsKIAogCi0tIAoyLjQzLjAKCg==

--------------HATO70W9anbTTpzMQbaQ89u3--