MIME-Version: 1.0
References: 
 <CAN55FZ2RPMxquXE6TH7dQkhtoiBcOOOZq8EOXj5COHv3ecP_cw@mail.gmail.com>
 <CA+K2Ru=fFTUVgEDr-fBed5aOMeDbH9vrOEhapXzHEpBeOxkucg@mail.gmail.com>
 <CAKWEB6pq7C0Wv1wT9Y1_c_1fn-+cR8pb210Pj3w2FcEOmNGxbQ@mail.gmail.com>
 <CAN55FZ2DT4-k06umn=7NYG+NoM6gnVJVQCCwRrr2qOraO+Jadw@mail.gmail.com>
 <aZikzQP6WPJ5Rq2S@nathan>
 <CAN55FZ3cBN_TncLVWyXAKm-KfewguN1AUjyRhoR6zL_QCxHh7A@mail.gmail.com>
 <CAKWEB6qzsZEQ4Czo9QBFiMXqdXVJknHUJwg6wjRwNzLn4+Jw0g@mail.gmail.com>
 <CAN55FZ2O2Ls==sdpROHqxWRx-PMBZ0riJ6eVKoHj8=vssTavxw@mail.gmail.com>
 <aZ3kYQnF9_u6sUQp@nathan>
 <CAN55FZ3+NYF1TkKyNtpRQuLiaauSYk9G5tA+fpruOA4-14Y_ZA@mail.gmail.com>
 <aaXrGSyq4u2d9qEC@nathan>
In-Reply-To: <aaXrGSyq4u2d9qEC@nathan>
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 4 Mar 2026 18:15:53 +0300
Message-ID: 
 <CAN55FZ2DNaKCK3Kf_kHizb2pAbQvULeDYtzaiz97B_xz7YbrkQ@mail.gmail.com>
Subject: Re: Speed up COPY FROM text/CSV parsing using SIMD
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Manni Wood <manni.wood@enterprisedb.com>, KAZAR Ayoub <ma_kazar@esi.dz>,
	Neil Conway <neil.conway@gmail.com>, Andrew Dunstan <andrew@dunslane.net>,
	Shinya Kato <shinya11.kato@gmail.com>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Type: multipart/mixed; boundary="0000000000008700c4064c344ca1"
Archived-At: 
 <https://www.postgresql.org/message-id/CAN55FZ2DNaKCK3Kf_kHizb2pAbQvULeDYtzaiz97B_xz7YbrkQ%40mail.gmail.com>
Precedence: bulk

--0000000000008700c4064c344ca1
Content-Type: text/plain; charset="UTF-8"

Hi,

On Mon, 2 Mar 2026 at 22:55, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Feb 25, 2026 at 05:24:27PM +0300, Nazir Bilal Yavuz wrote:
> > If anyone has any suggestions/ideas, please let me know!

I am able to fix the problem. My first assumption was that the
branching of SIMD code caused that problem, so I moved SIMD code to
the CopyReadLineTextSIMDHelper() function. Then I moved this
CopyReadLineTextSIMDHelper() to top of CopyReadLineText(), by doing
that we won't have any branching in the non-SIMD (scalar) code path.
This didn't solve the problem and then I realized that even though I
disable SIMD code path with 'if (false)', there is still regression
but if I comment all of the 'if (cstate->simd_enabled)' branch, then
there is no regression at all.

To find out more, I compared assembly outputs of both and found out
the possible reason. What I understood is that the compiler can't
promote a variable to register, instead these variables live in the
stack; which is slower. Please see the two different assembly outputs:

Slow code:

        c = copy_input_buf[input_buf_ptr++];
     db0:    48 8b 55 b8              mov    -0x48(%rbp),%rdx
     db4:    48 63 c6                 movslq %esi,%rax
     db7:    44 8d 66 01              lea    0x1(%rsi),%r12d
     dbb:    44 89 65 cc              mov    %r12d,-0x34(%rbp)
     dbf:    0f be 14 02              movsbl (%rdx,%rax,1),%edx

Fast code:

        c = copy_input_buf[input_buf_ptr++];
     d80:    49 63 c4                 movslq %r12d,%rax
     d83:    45 8d 5c 24 01           lea    0x1(%r12),%r11d
     d88:    41 0f be 04 06           movsbl (%r14,%rax,1),%eax

And the reason for that is sending the address of input_buf_ptr to a
CopyReadLineTextSIMDHelper(..., &input_buf_ptr). If I change it to
this:

int            temp_input_buf_ptr = input_buf_ptr;
CopyReadLineTextSIMDHelper(..., &temp_input_buf_ptr);

Then there is no regression. However, I am still not completely sure
if that is the same problem in the v10, I am planning to spend more
time debugging this.

> A couple of random ideas:
>
> * Additional inlining for callers.  I looked around a little bit and didn't
> see any great candidates, so I don't have much faith in this, but maybe
> you'll see something I don't.

I agree with you. CopyReadLineText() is already quite a big function.

> * Disable SIMD if we are consistently getting small rows.  That won't help
> your "wide & CSV 1/3" case in all likelihood, but perhaps it'll help with
> the regression for narrow rows described elsewhere.

I implemented this, two consecutive small rows disables SIMD.

> * Surround the variable initializations with "if (simd_enabled)".
> Presumably compilers are smart enough to remove those in the non-SIMD paths
> already, but it could be worth a try.

Done.

> * Add simd_enabled function parameter to CopyReadLine(),
> NextCopyFromRawFieldsInternal(), and CopyFromTextLikeOneRow(), and do the
> bool literal trick in CopyFrom{Text,CSV}OneRow().  That could encourage the
> compiler to do some additional optimizations to reduce branching.

I think we don't need this. At least the implementation with
CopyReadLineTextSIMDHelper() doesn't need this since branching will be
at the top and it will be once per line.

I think v11 looks better compared to v10. I liked the
CopyReadLineTextSIMDHelper() helper function. I also liked it being at
the top of CopyReadLineText(), not being in the scalar path. This
gives us more optimization options without affecting the scalar path.

Here are the new benchmark results, I benchmarked the changes with
both -O2 and -O3 and also both with and without 'changing
default_toast_compression to lz4' commit (65def42b1d5). Benchmark
results show that there is no regression and the performance
improvement is much bigger with 65def42b1d5, it is close to 2x for
text format and more than 2x for the csv format.

------------------------------

Benchmark results:

With 65def42b1d5:

+---------------------------------------------------------+
|                    Optimization: -O2                    |
+--------------------------+--------------+---------------+
|                          |     Text     |      CSV      |
+--------------------------+------+-------+-------+-------+
|           WIDE           | None |  1/3  |  None |  1/3  |
+--------------------------+------+-------+-------+-------+
|        Old Master        | 4220 |  4780 |  5930 |  8250 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 2520 |  4500 |  2520 |  7800 |
+--------------------------+------+-------+-------+-------+
|                          |      |       |       |       |
+--------------------------+------+-------+-------+-------+
|                          |     Text     |      CSV      |
+--------------------------+------+-------+-------+-------+
|          NARROW          | None |  1/3  |  None |  1/3  |
+--------------------------+------+-------+-------+-------+
|        Old Master        | 9920 | 10100 | 10200 | 10470 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9970 | 10000 | 10180 | 10350 |
+--------------------------+------+-------+-------+-------+
|                                                         |
+---------------------------------------------------------+
|                                                         |
+---------------------------------------------------------+
|                    Optimization: -O3                    |
+--------------------------+--------------+---------------+
|                          |     Text     |      CSV      |
+--------------------------+------+-------+-------+-------+
|           WIDE           | None |  1/3  |  None |  1/3  |
+--------------------------+------+-------+-------+-------+
|        Old Master        | 4100 |  4900 |  6200 |  8300 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 2470 |  4440 |  2570 |  7700 |
+--------------------------+------+-------+-------+-------+
|                          |      |       |       |       |
+--------------------------+------+-------+-------+-------+
|                          |     Text     |      CSV      |
+--------------------------+------+-------+-------+-------+
|          NARROW          | None |  1/3  |  None |  1/3  |
+--------------------------+------+-------+-------+-------+
|        Old Master        | 9530 |  9690 |  9800 | 10080 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9350 |  9450 |  9700 | 10000 |
+--------------------------+------+-------+-------+-------+

------------------------------

Without 65def42b1d5:

+----------------------------------------------------------+
|                     Optimization: -O2                    |
+--------------------------+---------------+---------------+
|                          |      Text     |      CSV      |
+--------------------------+-------+-------+-------+-------+
|           WIDE           |  None |  1/3  |  None |  1/3  |
+--------------------------+-------+-------+-------+-------+
|        Old Master        | 10550 | 11030 | 12250 | 14400 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 |  8890 | 10700 |  8870 | 14070 |
+--------------------------+-------+-------+-------+-------+
|                          |       |       |       |       |
+--------------------------+-------+-------+-------+-------+
|                          |      Text     |      CSV      |
+--------------------------+-------+-------+-------+-------+
|          NARROW          |  None |  1/3  |  None |  1/3  |
+--------------------------+-------+-------+-------+-------+
|        Old Master        |  9921 | 10205 | 10123 | 10420 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 |  9880 | 10070 | 10150 | 10400 |
+--------------------------+-------+-------+-------+-------+
|                                                          |
+----------------------------------------------------------+
|                                                          |
+----------------------------------------------------------+
|                     Optimization: -O3                    |
+--------------------------+---------------+---------------+
|                          |      Text     |      CSV      |
+--------------------------+-------+-------+-------+-------+
|           WIDE           |  None |  1/3  |  None |  1/3  |
+--------------------------+-------+-------+-------+-------+
|        Old Master        | 10500 | 11100 | 12600 | 14580 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 |  8900 | 10660 |  8860 | 13990 |
+--------------------------+-------+-------+-------+-------+
|                          |       |       |       |       |
+--------------------------+-------+-------+-------+-------+
|                          |      Text     |      CSV      |
+--------------------------+-------+-------+-------+-------+
|          NARROW          |  None |  1/3  |  None |  1/3  |
+--------------------------+-------+-------+-------+-------+
|        Old Master        |  9600 |  9700 |  9800 | 10150 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 |  9300 |  9470 |  9600 |  9880 |
+--------------------------+-------+-------+-------+-------+

--
Regards,
Nazir Bilal Yavuz
Microsoft

--0000000000008700c4064c344ca1
Content-Type: text/x-patch; charset="US-ASCII";
	name="v11-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch"
Content-Disposition: attachment;
	filename="v11-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mmc690y10>
X-Attachment-Id: f_mmc690y10

RnJvbSA3YWNhZWIzMjAxYWU0YWUyNzliZjhiMjU2NDFiZWE3ZjhjYjkyY2JlIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBOYXppciBCaWxhbCBZYXZ1eiA8YnlhdnV6ODFAZ21haWwuY29t
PgpEYXRlOiBXZWQsIDQgTWFyIDIwMjYgMTc6Mjg6NTQgKzAzMDAKU3ViamVjdDogW1BBVENIIHYx
MV0gU3BlZWQgdXAgQ09QWSBGUk9NIHRleHQvQ1NWIHBhcnNpbmcgdXNpbmcgU0lNRAoKVGhpcyBw
YXRjaCBkaXNhYmxlcyBTSU1EIHdoZW4gU0lNRCBlbmNvdW50ZXJzIGEgc3BlY2lhbCBjaGFyYWN0
ZXIgd2hpY2gKaXMgbmVpdGhlciBFT0Ygbm9yIEVPTC4KCkF1dGhvcjogU2hpbnlhIEthdG8gPHNo
aW55YTExLmthdG9AZ21haWwuY29tPgpBdXRob3I6IE5hemlyIEJpbGFsIFlhdnV6IDxieWF2dXo4
MUBnbWFpbC5jb20+ClJldmlld2VkLWJ5OiBLYXphciBBeW91YiA8bWFfa2F6YXJAZXNpLmR6PgpS
ZXZpZXdlZC1ieTogTmF0aGFuIEJvc3NhcnQgPG5hdGhhbmRib3NzYXJ0QGdtYWlsLmNvbT4KUmV2
aWV3ZWQtYnk6IE5laWwgQ29ud2F5IDxuZWlsLmNvbndheUBnbWFpbC5jb20+ClJldmlld2VkLWJ5
OiBBbmRyZXcgRHVuc3RhbiA8YW5kcmV3QGR1bnNsYW5lLm5ldD4KUmV2aWV3ZWQtYnk6IE1hbm5p
IFdvb2QgPG1hbm5pLndvb2RAZW50ZXJwcmlzZWRiLmNvbT4KUmV2aWV3ZWQtYnk6IE1hcmsgV29u
ZyA8bWFya3drbUBnbWFpbC5jb20+CkRpc2N1c3Npb246IGh0dHBzOi8vcG9zdGdyLmVzL20vQ0FP
ekV1clNXOGNOcjZUUEtzanJzdG5QZmhmNFF5UXFCNHRuUFhHR2U4TjRlX3Y3SmlnJTQwbWFpbC5n
bWFpbC5jb20KLS0tCiBzcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5ZnJvbS5jICAgICAgICAgIHwg
ICA0ICsKIHNyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHlmcm9tcGFyc2UuYyAgICAgfCAyMjIgKysr
KysrKysrKysrKysrKysrKysrKy0KIHNyYy9pbmNsdWRlL2NvbW1hbmRzL2NvcHlmcm9tX2ludGVy
bmFsLmggfCAgIDQgKwogMyBmaWxlcyBjaGFuZ2VkLCAyMjMgaW5zZXJ0aW9ucygrKSwgNyBkZWxl
dGlvbnMoLSkKCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5ZnJvbS5jIGIv
c3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb20uYwppbmRleCAyZjQyZjU1ZTIyOS4uMmFhNTI4
MTBmZjEgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHlmcm9tLmMKKysrIGIv
c3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb20uYwpAQCAtMTc0Nyw2ICsxNzQ3LDEwIEBAIEJl
Z2luQ29weUZyb20oUGFyc2VTdGF0ZSAqcHN0YXRlLAogCWNzdGF0ZS0+Y3VyX2F0dHZhbCA9IE5V
TEw7CiAJY3N0YXRlLT5yZWxuYW1lX29ubHkgPSBmYWxzZTsKIAorCS8qIEluaXRpYWxpemUgU0lN
RCAqLworCWNzdGF0ZS0+c2ltZF9lbmFibGVkID0gdHJ1ZTsKKwljc3RhdGUtPnNpbWRfZmFpbGVk
X2ZpcnN0X3ZlY3RvciA9IGZhbHNlOworCiAJLyoKIAkgKiBBbGxvY2F0ZSBidWZmZXJzIGZvciB0
aGUgaW5wdXQgcGlwZWxpbmUuCiAJICoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2NvbW1hbmRz
L2NvcHlmcm9tcGFyc2UuYyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHlmcm9tcGFyc2UuYwpp
bmRleCBmYmQxMzM1M2VmYy4uNzBlMWE1YTA0MTAgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2Nv
bW1hbmRzL2NvcHlmcm9tcGFyc2UuYworKysgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5ZnJv
bXBhcnNlLmMKQEAgLTcyLDYgKzcyLDcgQEAKICNpbmNsdWRlICJtaXNjYWRtaW4uaCIKICNpbmNs
dWRlICJwZ3N0YXQuaCIKICNpbmNsdWRlICJwb3J0L3BnX2Jzd2FwLmgiCisjaW5jbHVkZSAicG9y
dC9zaW1kLmgiCiAjaW5jbHVkZSAidXRpbHMvYnVpbHRpbnMuaCIKICNpbmNsdWRlICJ1dGlscy9y
ZWwuaCIKIApAQCAtMTU4LDYgKzE1OSwxMiBAQCBzdGF0aWMgcGdfYXR0cmlidXRlX2Fsd2F5c19p
bmxpbmUgYm9vbCBOZXh0Q29weUZyb21SYXdGaWVsZHNJbnRlcm5hbChDb3B5RnJvbVN0YQogCQkJ
CQkJCQkJCQkJCQkJCQkgaW50ICpuZmllbGRzLAogCQkJCQkJCQkJCQkJCQkJCQkgYm9vbCBpc19j
c3YpOwogCisvKiBTSU1EIGZ1bmN0aW9ucyAqLworI2lmbmRlZiBVU0VfTk9fU0lNRAorc3RhdGlj
IGJvb2wgQ29weVJlYWRMaW5lVGV4dFNJTURIZWxwZXIoQ29weUZyb21TdGF0ZSBjc3RhdGUsIGJv
b2wgaXNfY3N2LAorCQkJCQkJCQkJICAgYm9vbCAqdGVtcF9oaXRfZW9mLCBpbnQgKnRlbXBfaW5w
dXRfYnVmX3B0cik7CisjZW5kaWYKKwogCiAvKiBMb3ctbGV2ZWwgY29tbXVuaWNhdGlvbnMgZnVu
Y3Rpb25zICovCiBzdGF0aWMgaW50CUNvcHlHZXREYXRhKENvcHlGcm9tU3RhdGUgY3N0YXRlLCB2
b2lkICpkYXRhYnVmLApAQCAtMTMxMCw2ICsxMzE3LDE4MiBAQCBDb3B5UmVhZExpbmUoQ29weUZy
b21TdGF0ZSBjc3RhdGUsIGJvb2wgaXNfY3N2KQogCXJldHVybiByZXN1bHQ7CiB9CiAKKyNpZm5k
ZWYgVVNFX05PX1NJTUQKKy8qCisgKiBVc2UgU0lNRCBpbnN0cnVjdGlvbnMgdG8gZWZmaWNpZW50
bHkgc2NhbiB0aGUgaW5wdXQgYnVmZmVyIGZvciBzcGVjaWFsCisgKiBjaGFyYWN0ZXJzIChlLmcu
LCBuZXdsaW5lLCBjYXJyaWFnZSByZXR1cm4sIHF1b3RlLCBhbmQgZXNjYXBlKS4gVGhpcyBpcwor
ICogZmFzdGVyIHRoYW4gYnl0ZS1ieS1ieXRlIGl0ZXJhdGlvbiwgZXNwZWNpYWxseSBvbiBsYXJn
ZSBidWZmZXJzLgorICoKKyAqIE5vdGUgdGhhdCwgU0lNRCBtYXkgYmVjb21lIHNsb3dlciB3aGVu
IHRoZSBpbnB1dCBjb250YWlucyBtYW55IHNwZWNpYWwKKyAqIGNoYXJhY3RlcnMuIFRvIGF2b2lk
IHRoaXMgcmVncmVzc2lvbiwgd2UgZGlzYWJsZSBTSU1EIGZvciB0aGUgcmVzdCBvZiB0aGUKKyAq
IGlucHV0IG9uY2Ugd2UgZW5jb3VudGVyIGEgc3BlY2lhbCBjaGFyYWN0ZXIgd2hpY2ggaXMgbmVp
dGhlciBFT0Ygbm9yIEVPTC4KKyAqIEFsc28sIFNJTUQgaXMgZGlzYWJsZWQgd2hlbiBpdCBlbmNv
dW50ZXJzIHR3byBjb25zZWN1dGl2ZSBzaG9ydCBsaW5lcyB0aGF0CisgKiBTSU1EIGNhbid0IGNy
ZWF0ZSBhIGZ1bGwgc2l6ZWQgVmVjdG9yLCB0b28uCisgKi8KK3N0YXRpYyBib29sCitDb3B5UmVh
ZExpbmVUZXh0U0lNREhlbHBlcihDb3B5RnJvbVN0YXRlIGNzdGF0ZSwgYm9vbCBpc19jc3YsIGJv
b2wgKnRlbXBfaGl0X2VvZiwgaW50ICp0ZW1wX2lucHV0X2J1Zl9wdHIpCit7CisJY2hhcgkJcXVv
dGVjID0gJ1wwJzsKKwljaGFyCQllc2NhcGVjID0gJ1wwJzsKKwljaGFyCSAgICpjb3B5X2lucHV0
X2J1ZjsKKwlpbnQJCQlpbnB1dF9idWZfcHRyOworCWludAkJCWNvcHlfYnVmX2xlbjsKKwlib29s
CQlyZXN1bHQgPSBmYWxzZTsKKwlib29sCQl1bmlxdWVfZXNjYXBlYyA9IGZhbHNlOworCWJvb2wJ
CWZpcnN0X3ZlY3RvciA9IHRydWU7CisJVmVjdG9yOAkJbmwgPSB2ZWN0b3I4X2Jyb2FkY2FzdCgn
XG4nKTsKKwlWZWN0b3I4CQljciA9IHZlY3RvcjhfYnJvYWRjYXN0KCdccicpOworCVZlY3RvcjgJ
CWJzID0gdmVjdG9yOF9icm9hZGNhc3QoJ1xcJyk7CisJVmVjdG9yOAkJcXVvdGUgPSB2ZWN0b3I4
X2Jyb2FkY2FzdCgwKTsKKwlWZWN0b3I4CQllc2NhcGUgPSB2ZWN0b3I4X2Jyb2FkY2FzdCgwKTsK
KworCWlmIChpc19jc3YpCisJeworCQlxdW90ZWMgPSBjc3RhdGUtPm9wdHMucXVvdGVbMF07CisJ
CWVzY2FwZWMgPSBjc3RhdGUtPm9wdHMuZXNjYXBlWzBdOworCisJCXF1b3RlID0gdmVjdG9yOF9i
cm9hZGNhc3QocXVvdGVjKTsKKwkJaWYgKHF1b3RlYyAhPSBlc2NhcGVjKQorCQl7CisJCQl1bmlx
dWVfZXNjYXBlYyA9IHRydWU7CisJCQllc2NhcGUgPSB2ZWN0b3I4X2Jyb2FkY2FzdChlc2NhcGVj
KTsKKwkJfQorCX0KKworCS8qIEZvciBhIGxpdHRsZSBleHRyYSBzcGVlZCB3ZSBjb3B5IHRoZXNl
IGludG8gbG9jYWwgdmFyaWFibGVzICovCisJY29weV9pbnB1dF9idWYgPSBjc3RhdGUtPmlucHV0
X2J1ZjsKKwlpbnB1dF9idWZfcHRyID0gY3N0YXRlLT5pbnB1dF9idWZfaW5kZXg7CisJY29weV9i
dWZfbGVuID0gY3N0YXRlLT5pbnB1dF9idWZfbGVuOworCisJd2hpbGUgKHRydWUpCisJeworCQkv
KiBMb2FkIG1vcmUgZGF0YSBpZiBuZWVkZWQgKi8KKwkJaWYgKHNpemVvZihWZWN0b3I4KSA+PSBj
b3B5X2J1Zl9sZW4gLSBpbnB1dF9idWZfcHRyKQorCQl7CisJCQlSRUZJTExfTElORUJVRjsKKwor
CQkJQ29weUxvYWRJbnB1dEJ1Zihjc3RhdGUpOworCQkJLyogdXBkYXRlIG91ciBsb2NhbCB2YXJp
YWJsZXMgKi8KKwkJCSp0ZW1wX2hpdF9lb2YgPSBjc3RhdGUtPmlucHV0X3JlYWNoZWRfZW9mOwor
CQkJaW5wdXRfYnVmX3B0ciA9IGNzdGF0ZS0+aW5wdXRfYnVmX2luZGV4OworCQkJY29weV9idWZf
bGVuID0gY3N0YXRlLT5pbnB1dF9idWZfbGVuOworCisJCQkvKgorCQkJICogSWYgd2UgYXJlIGNv
bXBsZXRlbHkgb3V0IG9mIGRhdGEsIGJyZWFrIG91dCBvZiB0aGUgbG9vcCwKKwkJCSAqIHJlcG9y
dGluZyBFT0YuCisJCQkgKi8KKwkJCWlmIChJTlBVVF9CVUZfQllURVMoY3N0YXRlKSA8PSAwKQor
CQkJeworCQkJCXJlc3VsdCA9IHRydWU7CisJCQkJYnJlYWs7CisJCQl9CisJCX0KKworCQlpZiAo
Y29weV9idWZfbGVuIC0gaW5wdXRfYnVmX3B0ciA+IHNpemVvZihWZWN0b3I4KSkKKwkJeworCQkJ
VmVjdG9yOAkJY2h1bms7CisJCQlWZWN0b3I4CQltYXRjaCA9IHZlY3RvcjhfYnJvYWRjYXN0KDAp
OworCisJCQkvKiBMb2FkIGEgY2h1bmsgb2YgZGF0YSBpbnRvIGEgdmVjdG9yIHJlZ2lzdGVyICov
CisJCQl2ZWN0b3I4X2xvYWQoJmNodW5rLCAoY29uc3QgdWludDggKikgJmNvcHlfaW5wdXRfYnVm
W2lucHV0X2J1Zl9wdHJdKTsKKworCQkJaWYgKGlzX2NzdikKKwkJCXsKKwkJCQltYXRjaCA9IHZl
Y3Rvcjhfb3IodmVjdG9yOF9lcShjaHVuaywgbmwpLCB2ZWN0b3I4X2VxKGNodW5rLCBjcikpOwor
CQkJCW1hdGNoID0gdmVjdG9yOF9vcihtYXRjaCwgdmVjdG9yOF9lcShjaHVuaywgcXVvdGUpKTsK
KwkJCQlpZiAodW5pcXVlX2VzY2FwZWMpCisJCQkJCW1hdGNoID0gdmVjdG9yOF9vcihtYXRjaCwg
dmVjdG9yOF9lcShjaHVuaywgZXNjYXBlKSk7CisJCQl9CisJCQllbHNlCisJCQl7CisJCQkJbWF0
Y2ggPSB2ZWN0b3I4X29yKHZlY3RvcjhfZXEoY2h1bmssIG5sKSwgdmVjdG9yOF9lcShjaHVuaywg
Y3IpKTsKKwkJCQltYXRjaCA9IHZlY3Rvcjhfb3IobWF0Y2gsIHZlY3RvcjhfZXEoY2h1bmssIGJz
KSk7CisJCQl9CisKKwkJCS8qIENoZWNrIGlmIHdlIGZvdW5kIGFueSBzcGVjaWFsIGNoYXJhY3Rl
cnMgKi8KKwkJCWlmICh2ZWN0b3I4X2lzX2hpZ2hiaXRfc2V0KG1hdGNoKSkKKwkJCXsKKwkJCQkv
KgorCQkJCSAqIEZvdW5kIGEgc3BlY2lhbCBjaGFyYWN0ZXIuIEFkdmFuY2UgdXAgdG8gdGhhdCBw
b2ludCBhbmQgbGV0CisJCQkJICogdGhlIHNjYWxhciBjb2RlIGhhbmRsZSBpdC4KKwkJCQkgKi8K
KwkJCQl1aW50MzIJCW1hc2s7CisJCQkJaW50CQkJYWR2YW5jZTsKKwkJCQljaGFyCQljMSwKKwkJ
CQkJCQljMjsKKwkJCQlib29sCQlzaW1kX2hpdF9lb2wsCisJCQkJCQkJc2ltZF9oaXRfZW9mOwor
CisJCQkJbWFzayA9IHZlY3RvcjhfaGlnaGJpdF9tYXNrKG1hdGNoKTsKKwkJCQlhZHZhbmNlID0g
cGdfcmlnaHRtb3N0X29uZV9wb3MzMihtYXNrKTsKKworCQkJCWlucHV0X2J1Zl9wdHIgKz0gYWR2
YW5jZTsKKwkJCQljMSA9IGNvcHlfaW5wdXRfYnVmW2lucHV0X2J1Zl9wdHJdOworCisJCQkJLyoK
KwkJCQkgKiBTaW5jZSB3ZSBzdG9wcGVkIHdpdGhpbiB0aGUgY2h1bmsgYW5kICgoY29weV9idWZf
bGVuIC0KKwkJCQkgKiBpbnB1dF9idWZfcHRyKSA+IHNpemVvZihWZWN0b3I4KSkgaXMgdHJ1ZSwK
KwkJCQkgKiBjb3B5X2lucHV0X2J1ZltpbnB1dF9idWZfcHRyICsgMV0gaXMgZ3VhcmFudGVlZCB0
byBiZQorCQkJCSAqIHJlYWRhYmxlLgorCQkJCSAqLworCQkJCWMyID0gY29weV9pbnB1dF9idWZb
aW5wdXRfYnVmX3B0ciArIDFdOworCisJCQkJc2ltZF9oaXRfZW9mID0gKGMxID09ICdcXCcgJiYg
YzIgPT0gJy4nICYmICFpc19jc3YpOworCQkJCXNpbWRfaGl0X2VvbCA9IChjMSA9PSAnXHInIHx8
IGMxID09ICdcbicpOworCisJCQkJLyoKKwkJCQkgKiBEbyBub3QgZGlzYWJsZSBTSU1EIHdoZW4g
d2UgaGl0IEVPTCBvciBFT0YgY2hhcmFjdGVycy4gSW4KKwkJCQkgKiBwcmFjdGljZSwgaXQgZG9l
cyBub3QgbWF0dGVyIGZvciBFT0YgYmVjYXVzZSBwYXJzaW5nIGVuZHMKKwkJCQkgKiB0aGVyZSwg
YnV0IHdlIGtlZXAgdGhlIGJlaGF2aW9yIGNvbnNpc3RlbnQuCisJCQkJICovCisJCQkJaWYgKCEo
c2ltZF9oaXRfZW9mIHx8IHNpbWRfaGl0X2VvbCkpCisJCQkJCWNzdGF0ZS0+c2ltZF9lbmFibGVk
ID0gZmFsc2U7CisKKwkJCQkvKgorCQkJCSAqIFdlIGVuY291bnRlcmVkIGEgRU9MIG9yIEVPRiBv
biB0aGUgZmlyc3QgdmVjdG9yLiBUaGlzIG1lYW5zCisJCQkJICogbGluZXMgYXJlIG5vdCBsb25n
IGVub3VnaCB0byBza2lwIGZ1bGx5IHNpemVkIHZlY3Rvci4gSWYKKwkJCQkgKiB0aGlzIGhhcHBl
bnMgdHdvIHRpbWVzIGNvbnNlY3V0aXZlbHksIHRoZW4gZGlzYWJsZSB0aGUKKwkJCQkgKiBTSU1E
LgorCQkJCSAqLworCQkJCWlmIChmaXJzdF92ZWN0b3IpCisJCQkJeworCQkJCQlpZiAoY3N0YXRl
LT5zaW1kX2ZhaWxlZF9maXJzdF92ZWN0b3IpCisJCQkJCQljc3RhdGUtPnNpbWRfZW5hYmxlZCA9
IGZhbHNlOworCisJCQkJCWNzdGF0ZS0+c2ltZF9mYWlsZWRfZmlyc3RfdmVjdG9yID0gdHJ1ZTsK
KwkJCQl9CisKKwkJCQlicmVhazsKKwkJCX0KKwkJCWVsc2UKKwkJCXsKKwkJCQkvKiBObyBzcGVj
aWFsIGNoYXJhY3RlcnMgZm91bmQsIHNvIHNraXAgdGhlIGVudGlyZSBjaHVuayAqLworCQkJCWlu
cHV0X2J1Zl9wdHIgKz0gc2l6ZW9mKFZlY3RvcjgpOworCQkJCWZpcnN0X3ZlY3RvciA9IGZhbHNl
OworCQkJfQorCQl9CisKKwkJLyoKKwkJICogQWx0aG91Z2ggd2UgcmVmaWxsIGxpbmVidWYsIHRo
ZXJlIGlzIG5vdCBlbm91Z2ggY2hhcmFjdGVyIHRvIGZpbGwKKwkJICogZnVsbCBzaXplZCB2ZWN0
b3IuIFRoaXMgZG9lc24ndCBtZWFuIHRoYXQgd2UgZW5jb3VudGVyZWQgYSBsaW5lCisJCSAqIHRo
YXQgaXMgbm90IGVub3VnaCB0byBmaWxsIGEgZnVsbCBzaXplZCB2ZWN0b3IuCisJCSAqCisJCSAq
IFNjYWxhciBjb2RlIHdpbGwgaGFuZGxlIHRoZSByZXN0IGZvciB0aGlzIGxpbmUuIFRoZW4sIFNJ
TUQgd2lsbAorCQkgKiBjb250aW51ZSBmcm9tIHRoZSBuZXh0IGxpbmUuCisJCSAqLworCQllbHNl
CisJCXsKKwkJCWZpcnN0X3ZlY3RvciA9IGZhbHNlOworCQkJYnJlYWs7CisJCX0KKwl9CisKKwlj
c3RhdGUtPnNpbWRfZmFpbGVkX2ZpcnN0X3ZlY3RvciA9IGZpcnN0X3ZlY3RvcjsKKwkqdGVtcF9p
bnB1dF9idWZfcHRyID0gaW5wdXRfYnVmX3B0cjsKKwlyZXR1cm4gcmVzdWx0OworfQorI2VuZGlm
CisKIC8qCiAgKiBDb3B5UmVhZExpbmVUZXh0IC0gaW5uZXIgbG9vcCBvZiBDb3B5UmVhZExpbmUg
Zm9yIHRleHQgbW9kZQogICovCkBAIC0xMzM4LDYgKzE1MjEsMzggQEAgQ29weVJlYWRMaW5lVGV4
dChDb3B5RnJvbVN0YXRlIGNzdGF0ZSwgYm9vbCBpc19jc3YpCiAJCQllc2NhcGVjID0gJ1wwJzsK
IAl9CiAKKwkvKiBpbnB1dF9idWZfcHRyIHdpbGwgYmUgdXNlZCBpbiB0aGUgU0lNRCBIZWxwZXIg
ZnVuY3Rpb24gKi8KKwlpbnB1dF9idWZfcHRyID0gY3N0YXRlLT5pbnB1dF9idWZfaW5kZXg7CisK
KyNpZm5kZWYgVVNFX05PX1NJTUQKKwkvKiBGaXJzdCB0cnkgdG8gcnVuIFNJTUQsIHRoZW4gY29u
dGludWUgd2l0aCB0aGUgc2NhbGFyIHBhdGggKi8KKwlpZiAoY3N0YXRlLT5zaW1kX2VuYWJsZWQp
CisJeworCQlpbnQJCQl0ZW1wX2lucHV0X2J1Zl9wdHIgPSBpbnB1dF9idWZfcHRyOworCQlib29s
CQl0ZW1wX2hpdF9lb2YgPSBmYWxzZTsKKworCQlyZXN1bHQgPSBDb3B5UmVhZExpbmVUZXh0U0lN
REhlbHBlcihjc3RhdGUsIGlzX2NzdiwgJnRlbXBfaGl0X2VvZiwKKwkJCQkJCQkJCQkJJnRlbXBf
aW5wdXRfYnVmX3B0cik7CisJCWlucHV0X2J1Zl9wdHIgPSB0ZW1wX2lucHV0X2J1Zl9wdHI7CisJ
CWhpdF9lb2YgPSB0ZW1wX2hpdF9lb2Y7CisKKwkJLyogU2hvcnQgZXhpdCBmcm9tIFNJTUQgKi8K
KwkJaWYgKHJlc3VsdCkKKwkJeworCQkJLyoKKwkJCSAqIFRyYW5zZmVyIGFueSBzdGlsbC11bmNv
cGllZCBkYXRhIHRvIGxpbmVfYnVmLgorCQkJICovCisJCQlSRUZJTExfTElORUJVRjsKKworCQkJ
cmV0dXJuIHJlc3VsdDsKKwkJfQorCX0KKyNlbmRpZgorCisJLyogRm9yIGEgbGl0dGxlIGV4dHJh
IHNwZWVkIHdlIGNvcHkgdGhlc2UgaW50byBsb2NhbCB2YXJpYWJsZXMgKi8KKwljb3B5X2lucHV0
X2J1ZiA9IGNzdGF0ZS0+aW5wdXRfYnVmOworCWNvcHlfYnVmX2xlbiA9IGNzdGF0ZS0+aW5wdXRf
YnVmX2xlbjsKKwogCS8qCiAJICogVGhlIG9iamVjdGl2ZSBvZiB0aGlzIGxvb3AgaXMgdG8gdHJh
bnNmZXIgdGhlIGVudGlyZSBuZXh0IGlucHV0IGxpbmUKIAkgKiBpbnRvIGxpbmVfYnVmLiAgSGVu
Y2UsIHdlIG9ubHkgY2FyZSBmb3IgZGV0ZWN0aW5nIG5ld2xpbmVzIChcciBhbmQvb3IKQEAgLTEz
NTksMTQgKzE1NzQsNyBAQCBDb3B5UmVhZExpbmVUZXh0KENvcHlGcm9tU3RhdGUgY3N0YXRlLCBi
b29sIGlzX2NzdikKIAkgKiBjaGFyYWN0ZXIgdG8gZXhhbWluZTsgYW55IGNoYXJhY3RlcnMgZnJv
bSBpbnB1dF9idWZfaW5kZXggdG8KIAkgKiBpbnB1dF9idWZfcHRyIGhhdmUgYmVlbiBkZXRlcm1p
bmVkIHRvIGJlIHBhcnQgb2YgdGhlIGxpbmUsIGJ1dCBub3QgeWV0CiAJICogdHJhbnNmZXJyZWQg
dG8gbGluZV9idWYuCi0JICoKLQkgKiBGb3IgYSBsaXR0bGUgZXh0cmEgc3BlZWQgd2l0aGluIHRo
ZSBsb29wLCB3ZSBjb3B5IGlucHV0X2J1ZiBhbmQKLQkgKiBpbnB1dF9idWZfbGVuIGludG8gbG9j
YWwgdmFyaWFibGVzLgogCSAqLwotCWNvcHlfaW5wdXRfYnVmID0gY3N0YXRlLT5pbnB1dF9idWY7
Ci0JaW5wdXRfYnVmX3B0ciA9IGNzdGF0ZS0+aW5wdXRfYnVmX2luZGV4OwotCWNvcHlfYnVmX2xl
biA9IGNzdGF0ZS0+aW5wdXRfYnVmX2xlbjsKLQogCWZvciAoOzspCiAJewogCQlpbnQJCQlwcmV2
X3Jhd19wdHI7CmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS9jb21tYW5kcy9jb3B5ZnJvbV9pbnRl
cm5hbC5oIGIvc3JjL2luY2x1ZGUvY29tbWFuZHMvY29weWZyb21faW50ZXJuYWwuaAppbmRleCBm
ODkyYzM0MzE1Ny4uNGE3NDhkZjhhYzggMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL2NvbW1hbmRz
L2NvcHlmcm9tX2ludGVybmFsLmgKKysrIGIvc3JjL2luY2x1ZGUvY29tbWFuZHMvY29weWZyb21f
aW50ZXJuYWwuaApAQCAtODksNiArODksMTAgQEAgdHlwZWRlZiBzdHJ1Y3QgQ29weUZyb21TdGF0
ZURhdGEKIAljb25zdCBjaGFyICpjdXJfYXR0dmFsOwkJLyogY3VycmVudCBhdHQgdmFsdWUgZm9y
IGVycm9yIG1lc3NhZ2VzICovCiAJYm9vbAkJcmVsbmFtZV9vbmx5OwkvKiBkb24ndCBvdXRwdXQg
bGluZSBudW1iZXIsIGF0dCwgZXRjLiAqLwogCisJLyogU0lNRCB2YXJpYWJsZXMgKi8KKwlib29s
CQlzaW1kX2VuYWJsZWQ7CisJYm9vbAkJc2ltZF9mYWlsZWRfZmlyc3RfdmVjdG9yOworCiAJLyoK
IAkgKiBXb3JraW5nIHN0YXRlCiAJICovCi0tIAoyLjQ3LjMKCg==
--0000000000008700c4064c344ca1--