From: Oleg Kharin <ok@uvadrev.ru>
Subject: Re: Incorrect choice of Nested Loop for a skewed distribution
To: Justin Pryzby <pryzby@telsasoft.com>
Cc: pgsql-performance@lists.postgresql.org
References: <9df40087-f396-143d-5e4a-da3c97b5e536@uvadrev.ru>
 <20190905001832.GJ7201@telsasoft.com>
Message-ID: <a5fb2f9d-a8ac-2c8a-22fc-6d5f62f6d6b3@uvadrev.ru>
Date: Fri, 6 Sep 2019 09:02:43 +0400
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <20190905001832.GJ7201@telsasoft.com>
Content-Type: multipart/mixed;
 boundary="------------A0612124DFB0F4900C74A16C"
Content-Language: en-GB
Precedence: bulk

This is a multi-part message in MIME format.
--------------A0612124DFB0F4900C74A16C
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Thank you Justin!

On Wed, 04 Sep 2019 17:18:36 -0700 (PDT), Justin Pryzby wrote:

> When you dropped the index, you necessarily refused the plan involving 
> index scan.
> So it did merge join instead (which it thinks of as expensive because it has to
> sort both sides).
>
> As you saw, the rowcount estimate of the join result is badly off.  But that's
> true of both plans.
>
> Choice of join type is affected by the size of its *inputs*, but doesn't and
> shouldn't have any effect on the result rowcount (or its) estimate.  The
> rowcount *output* of the join would only affect any *following* plan nodes (of
> which there are none in this case).
>
> You suggested using the MCV list, but I don't think that's possible, since the
> nested loop is evaluating its "inner" table multiple times, in a "loop":
>
>>     ->  Index Scan using t_f on t  (cost=0.29..0.36 rows=1 width=8) (actual time=0.565..1.125 rows=1 loops=20020)
> Hypothetically, one could plan the query 20k times, for each value of u.f and
> u.g, but that's tantamount to actually executing the query, which is why it
> uses (I gather) var_eq_non_const.

In fact the planner has information about the outer loop relation when 
it is estimating the number of inner loop rows for a nested-loop join. 
It ignores this information and considers the outer/inner loop relations 
as independent. So it uses for the rowcount estimate the number of 
distinct values only, not MCV lists of both tables.

For the test case above, the planner estimates the selectivity of the 
clause "t.f=u.f" as 1/10011. It hopes to scan 1/10011*20020=2 rows in a 
inner loop for each row of the outer loop. In fact it scans 10010 rows 
10010 times and 1 row 10010 times. The average number of rows scanned in 
a inner loop is (10010*10010+10010)/20020=5005. That is badly off from 2 
rows expected. The planner should use 5005/20020 as a more accurate 
value for the effective selectivity of "t.f=u.f".

I tried to make improvements to the function var_eq_non_const() so that 
it calculates the effective selectivity using MCV lists if possible. The 
patch for PostgreSQL 11.5 is attached to the letter.

The patched PostgreSQL chooses an optimal plan for the test case. As a 
result the query execution time  is reduced by more than 600 times.

Now if we force the planner to choose the nested-loop join

   set enable_mergejoin=off;

   set enable_hashjoin=off;

   explain analyze select * from t,u where t.f=u.f and t.g=u.g;

  Nested Loop  (cost=0.29..2261681.75 rows=25055030 width=16) (actual 
time=0.048..22519.232 rows=20020 loops=1)
    ->  Seq Scan on u  (cost=0.00..289.20 rows=20020 width=8) (actual 
time=0.012..2.970 rows=20020 loops=1)
    ->  Index Scan using t_f on t  (cost=0.29..100.44 rows=1252 width=8) 
(actual time=0.565..1.124 rows=1 loops=20020)
          Index Cond: (f = u.f)
          Filter: (u.g = g)
          Rows Removed by Filter: 5004
  Planning Time: 0.188 ms
  Execution Time: 22521.248 ms

we will see that the cost of the index scan is increased from 0.36 to 
100.44, which is much more realistic and reflects 2 to 5005 rows ratio.

Regards,

Oleg


--------------A0612124DFB0F4900C74A16C
Content-Type: text/plain; charset=UTF-8;
 name="nestloop.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="nestloop.patch"

LS0tIGEvc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3NlbGZ1bmNzLmMJMjAxOS0wOC0wNiAwMTox
NDo1OS4wMDAwMDAwMDAgKzA0MDAKKysrIGIvc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3NlbGZ1
bmNzLmMJMjAxOS0wOS0wNiAwODoyMzozNC45MzI5Nzg4NjMgKzA0MDAKQEAgLTE2MSw3ICsx
NjEsNyBAQAogc3RhdGljIGRvdWJsZSB2YXJfZXFfY29uc3QoVmFyaWFibGVTdGF0RGF0YSAq
dmFyZGF0YSwgT2lkIG9wZXJhdG9yLAogCQkJIERhdHVtIGNvbnN0dmFsLCBib29sIGNvbnN0
aXNudWxsLAogCQkJIGJvb2wgdmFyb25sZWZ0LCBib29sIG5lZ2F0ZSk7Ci1zdGF0aWMgZG91
YmxlIHZhcl9lcV9ub25fY29uc3QoVmFyaWFibGVTdGF0RGF0YSAqdmFyZGF0YSwgT2lkIG9w
ZXJhdG9yLAorc3RhdGljIGRvdWJsZSB2YXJfZXFfbm9uX2NvbnN0KFZhcmlhYmxlU3RhdERh
dGEgKnZhcmRhdGEsIFBsYW5uZXJJbmZvICpyb290LCBMaXN0ICphcmdzLCBPaWQgb3BlcmF0
b3IsIAogCQkJCSBOb2RlICpvdGhlciwKIAkJCQkgYm9vbCB2YXJvbmxlZnQsIGJvb2wgbmVn
YXRlKTsKIHN0YXRpYyBkb3VibGUgaW5lcV9oaXN0b2dyYW1fc2VsZWN0aXZpdHkoUGxhbm5l
ckluZm8gKnJvb3QsCkBAIC0yOTEsNyArMjkxLDcgQEAKIAkJCQkJCQkgKChDb25zdCAqKSBv
dGhlciktPmNvbnN0aXNudWxsLAogCQkJCQkJCSB2YXJvbmxlZnQsIG5lZ2F0ZSk7CiAJZWxz
ZQotCQlzZWxlYyA9IHZhcl9lcV9ub25fY29uc3QoJnZhcmRhdGEsIG9wZXJhdG9yLCBvdGhl
ciwKKwkJc2VsZWMgPSB2YXJfZXFfbm9uX2NvbnN0KCZ2YXJkYXRhLCByb290LCBhcmdzLCBv
cGVyYXRvciwgb3RoZXIsIC8qIHJvb3QsIGFyZ3MgZm9yIGRldGFpbGVkIHN0YXRpc3RpY3Mg
b2YgdmFyaWFibGUgb24gdGhlIG90aGVyIGhhbmQgKi8KIAkJCQkJCQkJIHZhcm9ubGVmdCwg
bmVnYXRlKTsKIAogCVJlbGVhc2VWYXJpYWJsZVN0YXRzKHZhcmRhdGEpOwpAQCAtNDUzLDEx
ICs0NTMsMTYxIEBACiAJcmV0dXJuIHNlbGVjOwogfQogCisvKiB2YXJfZXFfdmFyX21jdnMg
LSBzZWxlY3Rpdml0eSBvZiAidmFyID0gdmFyMiIgdXNpbmcgTUNWIGxpc3RzCisgKiBiYXNl
ZCBvbiBlcWpvaW5zZWxfaW5uZXIoKSBjb2RlCisgKi8KK3N0YXRpYyBib29sCit2YXJfZXFf
dmFyX21jdnMoT2lkIG9wZXJhdG9yLCBWYXJpYWJsZVN0YXREYXRhICp2YXJkYXRhMSwgVmFy
aWFibGVTdGF0RGF0YSAqdmFyZGF0YTIsIAorCQkJCUF0dFN0YXRzU2xvdCAqc3Nsb3QxLCBk
b3VibGUgbnVsbGZyYWMxLAorCQkJCWRvdWJsZSBuZDEsIGRvdWJsZSAqc2VsZWMpCit7CisJ
T2lkCQkJb3BmdW5jb2lkOworCUZvcm1fcGdfc3RhdGlzdGljIHN0YXRzMiA9IE5VTEw7CisJ
Ym9vbAkJaGF2ZV9tY3ZzMiA9IGZhbHNlOworCUF0dFN0YXRzU2xvdCBzc2xvdDI7CisJYm9v
bAkJcmVzdWx0ID0gZmFsc2U7IC8qIG1hbmFnZWQgdG8gdXNlIG9mIHRoZSBNQ1YgbGlzdHMg
Ki8KKworCW9wZnVuY29pZCA9IGdldF9vcGNvZGUob3BlcmF0b3IpOworCisJbWVtc2V0KCZz
c2xvdDIsIDAsIHNpemVvZihzc2xvdDIpKTsKKworCWlmIChIZWFwVHVwbGVJc1ZhbGlkKHZh
cmRhdGEyLT5zdGF0c1R1cGxlKSkKKwl7CisJCS8qIG5vdGUgd2UgYWxsb3cgdXNlIG9mIG51
bGxmcmFjIHJlZ2FyZGxlc3Mgb2Ygc2VjdXJpdHkgY2hlY2sgKi8KKwkJc3RhdHMyID0gKEZv
cm1fcGdfc3RhdGlzdGljKSBHRVRTVFJVQ1QodmFyZGF0YTItPnN0YXRzVHVwbGUpOworCQlp
ZiAoc3RhdGlzdGljX3Byb2Nfc2VjdXJpdHlfY2hlY2sodmFyZGF0YTIsIG9wZnVuY29pZCkp
CisJCQloYXZlX21jdnMyID0gZ2V0X2F0dHN0YXRzc2xvdCgmc3Nsb3QyLCB2YXJkYXRhMi0+
c3RhdHNUdXBsZSwKKwkJCQkJCQkJCQkgIFNUQVRJU1RJQ19LSU5EX01DViwgSW52YWxpZE9p
ZCwKKwkJCQkJCQkJCQkgIEFUVFNUQVRTU0xPVF9WQUxVRVMgfCBBVFRTVEFUU1NMT1RfTlVN
QkVSUyk7CisJfQorCQorCWlmIChoYXZlX21jdnMyICYmIHN0YXRzMi0+c3RhbnVsbGZyYWMg
PCAxLjAgJiYgc3Nsb3QyLm52YWx1ZXMgPiAwKSAvKiB3ZSB1c2UgTUNWIGxpc3QgZm9yIHNl
bGVjdGl2aXR5IGVzdGltYXRlIGlmIHZhcjIgaGFzIE1DViBzdGF0aXN0aWNzLCBoYXMgYW55
IG5vdCBudWxsIHZhbHVlIGFuZCBNQ1YgbGlzdCBpcyBub3QgZW1wdHkgKi8KKwl7CisJCS8q
CisJCSAqIFdlIGhhdmUgbW9zdC1jb21tb24tdmFsdWUgbGlzdHMgZm9yIGJvdGggcmVsYXRp
b25zLiAgUnVuIHRocm91Z2gKKwkJICogdGhlIGxpc3RzIHRvIHNlZSB3aGljaCBNQ1ZzIGFj
dHVhbGx5IGpvaW4gdG8gZWFjaCBvdGhlciB3aXRoIHRoZQorCQkgKiBnaXZlbiBvcGVyYXRv
ci4gIFRoaXMgYWxsb3dzIHVzIHRvIGRldGVybWluZSB0aGUgZXhhY3Qgam9pbgorCQkgKiBz
ZWxlY3Rpdml0eSBmb3IgdGhlIHBvcnRpb24gb2YgdGhlIHJlbGF0aW9ucyByZXByZXNlbnRl
ZCBieSB0aGUgTUNWCisJCSAqIGxpc3RzLiAgV2Ugc3RpbGwgaGF2ZSB0byBlc3RpbWF0ZSBm
b3IgdGhlIHJlbWFpbmluZyBwb3B1bGF0aW9uLCBidXQKKwkJICogaW4gYSBza2V3ZWQgZGlz
dHJpYnV0aW9uIHRoaXMgZ2l2ZXMgdXMgYSBiaWcgbGVnIHVwIGluIGFjY3VyYWN5LgorCQkg
KiBGb3IgbW90aXZhdGlvbiBzZWUgdGhlIGFuYWx5c2lzIGluIFkuIElvYW5uaWRpcyBhbmQg
Uy4KKwkJICogQ2hyaXN0b2RvdWxha2lzLCAiT24gdGhlIHByb3BhZ2F0aW9uIG9mIGVycm9y
cyBpbiB0aGUgc2l6ZSBvZiBqb2luCisJCSAqIHJlc3VsdHMiLCBUZWNobmljYWwgUmVwb3J0
IDEwMTgsIENvbXB1dGVyIFNjaWVuY2UgRGVwdC4sIFVuaXZlcnNpdHkKKwkJICogb2YgV2lz
Y29uc2luLCBNYWRpc29uLCBNYXJjaCAxOTkyIChhdmFpbGFibGUgZnJvbSBmdHAuY3Mud2lz
Yy5lZHUpLgorCQkgKi8KKwkJRm1nckluZm8JZXFwcm9jOworCQlib29sCSAgICpoYXNtYXRj
aDE7CisJCWJvb2wJICAgKmhhc21hdGNoMjsKKwkJLypkb3VibGUJCW51bGxmcmFjMSA9IHN0
YXRzMS0+c3RhbnVsbGZyYWM7Ki8KKwkJZG91YmxlCQludWxsZnJhYzIgPSBzdGF0czItPnN0
YW51bGxmcmFjOworCQlkb3VibGUJCW1hdGNocHJvZGZyZXEsCisJCQkJCW1hdGNoZnJlcTEs
CisJCQkJCW1hdGNoZnJlcTIsCisJCQkJCXVubWF0Y2hmcmVxMSwKKwkJCQkJdW5tYXRjaGZy
ZXEyLAorCQkJCQlvdGhlcmZyZXExLAorCQkJCQlvdGhlcmZyZXEyLAorCQkJCQl0b3RhbHNl
bDI7CisJCWludAkJCWksCisJCQkJCW5tYXRjaGVzOworCisJCWZtZ3JfaW5mbyhvcGZ1bmNv
aWQsICZlcXByb2MpOworCQloYXNtYXRjaDEgPSAoYm9vbCAqKSBwYWxsb2MwKHNzbG90MS0+
bnZhbHVlcyAqIHNpemVvZihib29sKSk7CisJCWhhc21hdGNoMiA9IChib29sICopIHBhbGxv
YzAoc3Nsb3QyLm52YWx1ZXMgKiBzaXplb2YoYm9vbCkpOworCisJCS8qCisJCSAqIE5vdGUg
d2UgYXNzdW1lIHRoYXQgZWFjaCBNQ1Ygd2lsbCBtYXRjaCBhdCBtb3N0IG9uZSBtZW1iZXIg
b2YgdGhlCisJCSAqIG90aGVyIE1DViBsaXN0LiAgSWYgdGhlIG9wZXJhdG9yIGlzbid0IHJl
YWxseSBlcXVhbGl0eSwgdGhlcmUgY291bGQKKwkJICogYmUgbXVsdGlwbGUgbWF0Y2hlcyAt
LS0gYnV0IHdlIGRvbid0IGxvb2sgZm9yIHRoZW0sIGJvdGggZm9yIHNwZWVkCisJCSAqIGFu
ZCBiZWNhdXNlIHRoZSBtYXRoIHdvdWxkbid0IGFkZCB1cC4uLgorCQkgKi8KKwkJbWF0Y2hw
cm9kZnJlcSA9IDAuMDsKKwkJbm1hdGNoZXMgPSAwOworCQlmb3IgKGkgPSAwOyBpIDwgc3Ns
b3QxLT5udmFsdWVzOyBpKyspCisJCXsKKwkJCWludAkJCWo7CisKKwkJCWZvciAoaiA9IDA7
IGogPCBzc2xvdDIubnZhbHVlczsgaisrKQorCQkJeworCQkJCWlmIChoYXNtYXRjaDJbal0p
CisJCQkJCWNvbnRpbnVlOworCQkJCWlmIChEYXR1bUdldEJvb2woRnVuY3Rpb25DYWxsMkNv
bGwoJmVxcHJvYywKKwkJCQkJCQkJCQkJCSAgIERFRkFVTFRfQ09MTEFUSU9OX09JRCwKKwkJ
CQkJCQkJCQkJCSAgIHNzbG90MS0+dmFsdWVzW2ldLAorCQkJCQkJCQkJCQkJICAgc3Nsb3Qy
LnZhbHVlc1tqXSkpKQorCQkJCXsKKwkJCQkJaGFzbWF0Y2gxW2ldID0gaGFzbWF0Y2gyW2pd
ID0gdHJ1ZTsKKwkJCQkJbWF0Y2hwcm9kZnJlcSArPSBzc2xvdDEtPm51bWJlcnNbaV0gKiBz
c2xvdDIubnVtYmVyc1tqXTsKKwkJCQkJbm1hdGNoZXMrKzsKKwkJCQkJYnJlYWs7CisJCQkJ
fQorCQkJfQorCQl9CisJCUNMQU1QX1BST0JBQklMSVRZKG1hdGNocHJvZGZyZXEpOworCQkv
KiBTdW0gdXAgZnJlcXVlbmNpZXMgb2YgbWF0Y2hlZCBhbmQgdW5tYXRjaGVkIE1DVnMgKi8K
KwkJbWF0Y2hmcmVxMSA9IHVubWF0Y2hmcmVxMSA9IDAuMDsKKwkJZm9yIChpID0gMDsgaSA8
IHNzbG90MS0+bnZhbHVlczsgaSsrKQorCQl7CisJCQlpZiAoaGFzbWF0Y2gxW2ldKQorCQkJ
CW1hdGNoZnJlcTEgKz0gc3Nsb3QxLT5udW1iZXJzW2ldOworCQkJZWxzZQorCQkJCXVubWF0
Y2hmcmVxMSArPSBzc2xvdDEtPm51bWJlcnNbaV07CisJCX0KKwkJQ0xBTVBfUFJPQkFCSUxJ
VFkobWF0Y2hmcmVxMSk7CisJCUNMQU1QX1BST0JBQklMSVRZKHVubWF0Y2hmcmVxMSk7CisJ
CW1hdGNoZnJlcTIgPSB1bm1hdGNoZnJlcTIgPSAwLjA7CisJCWZvciAoaSA9IDA7IGkgPCBz
c2xvdDIubnZhbHVlczsgaSsrKQorCQl7CisJCQlpZiAoaGFzbWF0Y2gyW2ldKQorCQkJCW1h
dGNoZnJlcTIgKz0gc3Nsb3QyLm51bWJlcnNbaV07CisJCQllbHNlCisJCQkJdW5tYXRjaGZy
ZXEyICs9IHNzbG90Mi5udW1iZXJzW2ldOworCQl9CisJCUNMQU1QX1BST0JBQklMSVRZKG1h
dGNoZnJlcTIpOworCQlDTEFNUF9QUk9CQUJJTElUWSh1bm1hdGNoZnJlcTIpOworCQlwZnJl
ZShoYXNtYXRjaDEpOworCQlwZnJlZShoYXNtYXRjaDIpOworCisJCS8qCisJCSAqIENvbXB1
dGUgdG90YWwgZnJlcXVlbmN5IG9mIG5vbi1udWxsIHZhbHVlcyB0aGF0IGFyZSBub3QgaW4g
dGhlIE1DVgorCQkgKiBsaXN0cy4KKwkJICovCisJCW90aGVyZnJlcTEgPSAxLjAgLSBudWxs
ZnJhYzEgLSBtYXRjaGZyZXExIC0gdW5tYXRjaGZyZXExOworCQlvdGhlcmZyZXEyID0gMS4w
IC0gbnVsbGZyYWMyIC0gbWF0Y2hmcmVxMiAtIHVubWF0Y2hmcmVxMjsKKwkJQ0xBTVBfUFJP
QkFCSUxJVFkob3RoZXJmcmVxMSk7CisJCUNMQU1QX1BST0JBQklMSVRZKG90aGVyZnJlcTIp
OworCisJCS8qCisJCSAqIFdlIGNhbiBlc3RpbWF0ZSB0aGUgdG90YWwgc2VsZWN0aXZpdHkg
ZnJvbSB0aGUgcG9pbnQgb2YgdmlldyBvZgorCQkgKiByZWxhdGlvbiAxIGFzOiB0aGUga25v
d24gc2VsZWN0aXZpdHkgZm9yIG1hdGNoZWQgTUNWcywgcGx1cworCQkgKiB1bm1hdGNoZWQg
TUNWcyB0aGF0IGFyZSBhc3N1bWVkIHRvIG1hdGNoIGFnYWluc3QgcmFuZG9tIG1lbWJlcnMg
b2YKKwkJICogcmVsYXRpb24gMidzIG5vbi1NQ1YgcG9wdWxhdGlvbiwgcGx1cyBub24tTUNW
IHZhbHVlcyB0aGF0IGFyZQorCQkgKiBhc3N1bWVkIHRvIG1hdGNoIGFnYWluc3QgcmFuZG9t
IG1lbWJlcnMgb2YgcmVsYXRpb24gMidzIHVubWF0Y2hlZAorCQkgKiBNQ1ZzIHBsdXMgbm9u
LU1DViB2YWx1ZXMuCisJCSAqLworCQkvKiBTYW1lIGVzdGltYXRlIGZyb20gdGhlIHBvaW50
IG9mIHZpZXcgb2YgcmVsYXRpb24gMi4gKi8KKwkJdG90YWxzZWwyID0gbWF0Y2hwcm9kZnJl
cTsKKwkJaWYgKG5kMSA+IHNzbG90MS0+bnZhbHVlcykKKwkJCXRvdGFsc2VsMiArPSB1bm1h
dGNoZnJlcTIgKiBvdGhlcmZyZXExIC8gKG5kMSAtIHNzbG90MS0+bnZhbHVlcyk7CisJCWlm
IChuZDEgPiBubWF0Y2hlcykKKwkJCXRvdGFsc2VsMiArPSBvdGhlcmZyZXEyICogKG90aGVy
ZnJlcTEgKyB1bm1hdGNoZnJlcTEpIC8KKwkJCQkobmQxIC0gbm1hdGNoZXMpOworCisJCSpz
ZWxlYyA9IHRvdGFsc2VsMi8oMS1udWxsZnJhYzIpOworCQlyZXN1bHQgPSB0cnVlOworCisJ
fQorCisJZnJlZV9hdHRzdGF0c3Nsb3QoJnNzbG90Mik7CisKKwlyZXR1cm4gcmVzdWx0Owor
fQorCiAvKgogICogdmFyX2VxX25vbl9jb25zdCAtLS0gZXFzZWwgZm9yIHZhciA9IHNvbWV0
aGluZy1vdGhlci10aGFuLWNvbnN0IGNhc2UKICAqLwogc3RhdGljIGRvdWJsZQotdmFyX2Vx
X25vbl9jb25zdChWYXJpYWJsZVN0YXREYXRhICp2YXJkYXRhLCBPaWQgb3BlcmF0b3IsCit2
YXJfZXFfbm9uX2NvbnN0KFZhcmlhYmxlU3RhdERhdGEgKnZhcmRhdGEsIFBsYW5uZXJJbmZv
ICpyb290LCBMaXN0ICphcmdzLCBPaWQgb3BlcmF0b3IsCiAJCQkJIE5vZGUgKm90aGVyLAog
CQkJCSBib29sIHZhcm9ubGVmdCwgYm9vbCBuZWdhdGUpCiB7CkBAIC01MTMsOCArNjYzLDIw
IEBACiAJCSAqLwogCQlpZiAoZ2V0X2F0dHN0YXRzc2xvdCgmc3Nsb3QsIHZhcmRhdGEtPnN0
YXRzVHVwbGUsCiAJCQkJCQkJIFNUQVRJU1RJQ19LSU5EX01DViwgSW52YWxpZE9pZCwKKwkJ
CQkJCQkgQVRUU1RBVFNTTE9UX1ZBTFVFUyB8CS8qIG5lZWQgTUNWIGxpc3QgKi8KIAkJCQkJ
CQkgQVRUU1RBVFNTTE9UX05VTUJFUlMpKQogCQl7CisJCQlpZiAoc3Nsb3QubnZhbHVlcyA+
IDApIC8qIG9ubHkgaWYgTUNWIGxpc3QgaXMgbm90IGVtcHR5ICovCisJCQl7CisJCQkJTm9k
ZQkqbm9kZTI7CisJCQkJVmFyaWFibGVTdGF0RGF0YSB2YXJkYXRhMjsKKworCQkJCW5vZGUy
ID0gdmFyb25sZWZ0ID8gbHNlY29uZChhcmdzKSA6IGxpbml0aWFsKGFyZ3MpOyAvKiB2YXJp
YWJsZTIgaXMgb24gdGhlIG90aGVyIHNpZGUgKi8KKwkJCQlleGFtaW5lX3ZhcmlhYmxlKHJv
b3QsIG5vZGUyLCAwLCAmdmFyZGF0YTIpOyAvKiBkYXRhIG9mIHRoZSBvdGhlciBzaWRlICov
CisJCQkJaWYgKHZhcmRhdGEyLnJlbCkgLyogcmVsIGZyb20gdGhlIG90aGVyIHNpZGUgKi8K
KwkJCQkJdmFyX2VxX3Zhcl9tY3ZzKG9wZXJhdG9yLCB2YXJkYXRhLCAmdmFyZGF0YTIsICZz
c2xvdCwgbnVsbGZyYWMsIG5kaXN0aW5jdCwgJnNlbGVjKTsgLyogdHJ5aW5nIHRvIGVzdGlt
YXRlIHRoZSBzZWxlY3Rpdml0eSBtb3JlIGFjY3VyYXRlIHVzaW5nIHRoZSBNQ1YgbGlzdHMg
Ki8KKwkJCQlSZWxlYXNlVmFyaWFibGVTdGF0cyh2YXJkYXRhMik7CisJCQl9CiAJCQlpZiAo
c3Nsb3Qubm51bWJlcnMgPiAwICYmIHNlbGVjID4gc3Nsb3QubnVtYmVyc1swXSkKIAkJCQlz
ZWxlYyA9IHNzbG90Lm51bWJlcnNbMF07CiAJCQlmcmVlX2F0dHN0YXRzc2xvdCgmc3Nsb3Qp
Owo=
--------------A0612124DFB0F4900C74A16C--