Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pgnGB-0002vl-SJ for pgsql-hackers@arkaria.postgresql.org; Mon, 27 Mar 2023 13:51:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pgnGA-0006KI-OR for pgsql-hackers@arkaria.postgresql.org; Mon, 27 Mar 2023 13:51:42 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pgnGA-0006K7-CH for pgsql-hackers@lists.postgresql.org; Mon, 27 Mar 2023 13:51:42 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pgnG6-00039u-DX for pgsql-hackers@postgresql.org; Mon, 27 Mar 2023 13:51:42 +0000 Received: from oss.nttdata.com (localhost [127.0.0.1]) by oss.nttdata.com (Postfix) with ESMTPA id 7BB0E600FF; Mon, 27 Mar 2023 22:51:32 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.8 at oss.nttdata.com MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_bac8f6fe4607d4ed50972ec196ed62d6" Content-Transfer-Encoding: 7bit Date: Mon, 27 Mar 2023 22:51:32 +0900 From: torikoshia To: Andres Freund Cc: tgl@sss.pgh.pa.us, Daniel Gustafsson , Damir Belyalov , PostgreSQL Hackers , Danil Anisimow , Nikita Malakhov , a.lepikhov@postgrespro.ru Subject: Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) In-Reply-To: <16e09747fcfbb21c30e4c1009c416aa4@oss.nttdata.com> References: <483E1B59-4A80-450B-8685-7CC12762FD5A@anarazel.de> <2592557.1675663234@sss.pgh.pa.us> <39143e83571d59dfc04c75707fa0ea5a@oss.nttdata.com> <0DE0602F-CC12-40ED-B259-3AB91FB02C3B@yesql.se> <8ad8492ff9fae3481d87c7aab4e0aed0@oss.nttdata.com> <9a94e684d6528dff74de5959e97032f3@oss.nttdata.com> <8e5c596e47435e3b37b7a751ebcd9569@oss.nttdata.com> <20230322175000.qbdctk7bnmifh5an@awork3.anarazel.de> <16e09747fcfbb21c30e4c1009c416aa4@oss.nttdata.com> User-Agent: Roundcube Webmail/1.4.11 Message-ID: <19551e8c2717c24689913083f841ddb5@oss.nttdata.com> X-Sender: torikoshia@oss.nttdata.com List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --=_bac8f6fe4607d4ed50972ec196ed62d6 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed On 2023-03-23 02:50, Andres Freund wrote: Thanks again for your review. Attached v5 patch. > Have you measured whether this has negative performance effects when > *NOT* > using the new option? I loaded 10000000 rows of pgbench_accounts on my laptop and compared the elapsed time. GUCs changed from the default are logging_collector = on, log_error_verbosity = verbose. Three tests were run under each condition and the middle of them is listed below: - patch NOT applied(36f40ce2dc66): 35299ms - patch applied, without IGNORE_DATATYPE_ERRORS: 34409ms - patch applied, with IGNORE_DATATYPE_ERRORS: 35510ms It seems there are no significant degradation. Also tested the elapsed time when loading data which has some datatype error with IGNORE_DATATYPE_ERRORS: - data has 100 rows of error: 35269ms - data has 1000 rows of error: 34577ms - data has 5000000 rows of error: 48925ms 5000000 rows of error consumes much time, but it seems to be influenced by logging time. Here are test results under log_min_messages and client_min_messages are 'error': - data has 5000000 data type error: 23972ms - data has 0 data type error: 34320ms Now conversely, when there were many data type errors, it consumes less time. This seems like a reasonable result since the amount of skipped data is increasing. > As-is this does not work with FORMAT BINARY - and converting the binary > input > functions to support soft errors won't happen for 16. So I think you > need to > raise an error if BINARY and IGNORE_DATATYPE_ERRORS are specified. Added the option check. > On 2023-03-22 22:34:20 +0900, torikoshia wrote: > > @@ -985,9 +986,28 @@ CopyFrom(CopyFromState cstate) > > > > ExecClearTuple(myslot); > > > > + if (cstate->opts.ignore_datatype_errors) > > + { > > + escontext.details_wanted = true; > > + cstate->escontext = escontext; > > + } > > I think it might be worth pulling this out of the loop. That does mean > you'd > have to reset escontext.error_occurred after an error, but that doesn't > seem > too bad, you need to do other cleanup anyway. Pull this out of the loop and added process for resetting escontext. > > @@ -956,10 +957,20 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext, > > values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]); > > } > > else > > - values[m] = InputFunctionCall(&in_functions[m], > > - string, > > - typioparams[m], > > - att->atttypmod); > > + /* If IGNORE_DATATYPE_ERRORS is enabled skip rows with datatype errors */ > > + if (!InputFunctionCallSafe(&in_functions[m], > > + string, > > + typioparams[m], > > + att->atttypmod, > > + (Node *) &cstate->escontext, > > + &values[m])) > > + { > > + cstate->ignored_errors++; > > + > > + ereport(WARNING, > > + errmsg("%s", cstate->escontext.error_data->message)); > > That isn't right - you loose all the details of the message. As is > you'd also > leak the error context. > > I think the best bet for now is to do something like > /* adjust elevel so we don't jump out */ > cstate->escontext.error_data->elevel = WARNING; > /* despite the name, this won't raise an error if elevel < ERROR */ > ThrowErrorData(cstate->escontext.error_data); As I mentioned in one previous email, added above codes for now. > I wonder if we ought to provide a wrapper for this? It could e.g. know > to > mention the original elevel and such? > > > I don't think NextCopyFrom() is the right place to emit this warning - > it > e.g. is also called from file_fdw.c, which might want to do something > else > with the error. From a layering POV it seems cleaner to do this in > CopyFrom(). You already have a check for escontext.error_occurred there > anyway. Agreed. > > @@ -3378,6 +3378,10 @@ copy_opt_item: > > { > > $$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1); > > } > > + | IGNORE_DATATYPE_ERRORS > > + { > > + $$ = makeDefElem("ignore_datatype_errors", (Node *)makeBoolean(true), @1); > > + } > > | DELIMITER opt_as Sconst > > { > > $$ = makeDefElem("delimiter", (Node *) makeString($3), @1); > > > I think we shouldn't add a new keyword for this, but only support this > via > /* new COPY option syntax */ > copy_generic_opt_list: > copy_generic_opt_elem > > Further increasing the size of the grammar with random keywords when we > have > more generic ways to represent them seems unnecessary. Agreed. > > +-- tests for IGNORE_DATATYPE_ERRORS option > > +CREATE TABLE check_ign_err (n int, m int[], k int); > > +COPY check_ign_err FROM STDIN WITH IGNORE_DATATYPE_ERRORS; > > +1 {1} 1 > > +a {2} 2 > > +3 {3} 3333333333 > > +4 {a, 4} 4 > > + > > +5 {5} 5 > > +\. > > +SELECT * FROM check_ign_err; > > + > > I suggest adding a few more tests: > > - COPY with a datatype error that can't be handled as a soft error Added a test for cases with missing columns. However it's not datatype error and not what you expected, is it? > - test documenting that COPY FORMAT BINARY is incompatible with > IGNORE_DATATYPE_ERRORS Added it. > - a soft error showing the error context - although that will require > some > care to avoid the function name + line in the output I assume you mean a test to check the server log, but I haven't come up with a way to do it. Adding a TAP test might do it, but I think it would be overkill to add one just for this. -- Regards, -- Atsushi Torikoshi NTT DATA CORPORATION --=_bac8f6fe4607d4ed50972ec196ed62d6 Content-Transfer-Encoding: base64 Content-Type: text/x-diff; name=v5-0001-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch Content-Disposition: attachment; filename=v5-0001-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch; size=12749 RnJvbSA2YjY0NmQ5NmE5YzJhMzEwODM2NjkzNDUyZGViMjEyODYzNmQxYmViIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBBdHN1c2hpIFRvcmlrb3NoaSA8dG9yaWtvc2hpYUBvc3MubnR0 ZGF0YS5jb20+CkRhdGU6IE1vbiwgMjcgTWFyIDIwMjMgMjI6MTU6NDkgKzA5MDAKU3ViamVjdDog W1BBVENIIHY1XSBBZGQgbmV3IENPUFkgb3B0aW9uIElHTk9SRV9EQVRBVFlQRV9FUlJPUlMKCkN1 cnJlbnRseSBlbnRpcmUgQ09QWSBmYWlscyB3aGVuIHRoZXJlIGV4aXN0cyB1bmV4cGVjdGVkCmRh dGEgcmVnYXJkaW5nIGRhdGEgdHlwZSBvciByYW5nZS4KSW4gc29tZSBjYXNlcywgaXQgd291bGQg YmUgdXNlZnVsIHRvIHNraXAgY29weWluZyBzdWNoIGRhdGEKYW5kIGNvbnRpbnVlIGNvcHlpbmcg YW5kIElHTk9SRV9EQVRBVFlQRV9FUlJPUlMgZG9lcyB0aGlzLgoKVGhpcyBwYXRjaCB1c2VzIHRo ZSBzb2Z0IGVycm9yIGhhbmRsaW5nIGluZnJhc3RydWN0dXJlLCB3aGljaAppcyBpbnRyb2R1Y2Vk IGJ5IGQ5ZjdmNWQzMmYyMC4KCkF1dGhvcjogRGFtaXIgQmVseWFsb3YsIEF0c3VzaGkgVG9yaWtv c2hpCi0tLQogZG9jL3NyYy9zZ21sL3JlZi9jb3B5LnNnbWwgICAgICAgICAgICAgICB8IDEzICsr KysrKysrKysKIHNyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHkuYyAgICAgICAgICAgICAgfCAxNSAr KysrKysrKysrLQogc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb20uYyAgICAgICAgICB8IDMy ICsrKysrKysrKysrKysrKysrKysrKysrKwogc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb21w YXJzZS5jICAgICB8IDE3ICsrKysrKysrKystLS0KIHNyYy9iaW4vcHNxbC90YWItY29tcGxldGUu YyAgICAgICAgICAgICAgfCAgMyArKy0KIHNyYy9pbmNsdWRlL2NvbW1hbmRzL2NvcHkuaCAgICAg ICAgICAgICAgfCAgMSArCiBzcmMvaW5jbHVkZS9jb21tYW5kcy9jb3B5ZnJvbV9pbnRlcm5hbC5o IHwgIDMgKysrCiBzcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL2NvcHkyLm91dCAgICAgIHwgMjIg KysrKysrKysrKysrKysrKwogc3JjL3Rlc3QvcmVncmVzcy9zcWwvY29weTIuc3FsICAgICAgICAg ICB8IDE5ICsrKysrKysrKysrKysrCiA5IGZpbGVzIGNoYW5nZWQsIDExOSBpbnNlcnRpb25zKCsp LCA2IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9yZWYvY29weS5zZ21s IGIvZG9jL3NyYy9zZ21sL3JlZi9jb3B5LnNnbWwKaW5kZXggNWU1OTFlZDJlNi4uY2VhNTZkNjVl YiAxMDA2NDQKLS0tIGEvZG9jL3NyYy9zZ21sL3JlZi9jb3B5LnNnbWwKKysrIGIvZG9jL3NyYy9z Z21sL3JlZi9jb3B5LnNnbWwKQEAgLTM0LDYgKzM0LDcgQEAgQ09QWSB7IDxyZXBsYWNlYWJsZSBj bGFzcz0icGFyYW1ldGVyIj50YWJsZV9uYW1lPC9yZXBsYWNlYWJsZT4gWyAoIDxyZXBsYWNlYWJs ZQogCiAgICAgRk9STUFUIDxyZXBsYWNlYWJsZSBjbGFzcz0icGFyYW1ldGVyIj5mb3JtYXRfbmFt ZTwvcmVwbGFjZWFibGU+CiAgICAgRlJFRVpFIFsgPHJlcGxhY2VhYmxlIGNsYXNzPSJwYXJhbWV0 ZXIiPmJvb2xlYW48L3JlcGxhY2VhYmxlPiBdCisgICAgSUdOT1JFX0RBVEFUWVBFX0VSUk9SUyBb IDxyZXBsYWNlYWJsZSBjbGFzcz0icGFyYW1ldGVyIj5ib29sZWFuPC9yZXBsYWNlYWJsZT4gXQog ICAgIERFTElNSVRFUiAnPHJlcGxhY2VhYmxlIGNsYXNzPSJwYXJhbWV0ZXIiPmRlbGltaXRlcl9j aGFyYWN0ZXI8L3JlcGxhY2VhYmxlPicKICAgICBOVUxMICc8cmVwbGFjZWFibGUgY2xhc3M9InBh cmFtZXRlciI+bnVsbF9zdHJpbmc8L3JlcGxhY2VhYmxlPicKICAgICBIRUFERVIgWyA8cmVwbGFj ZWFibGUgY2xhc3M9InBhcmFtZXRlciI+Ym9vbGVhbjwvcmVwbGFjZWFibGU+IHwgTUFUQ0ggXQpA QCAtMjM0LDYgKzIzNSwxOCBAQCBDT1BZIHsgPHJlcGxhY2VhYmxlIGNsYXNzPSJwYXJhbWV0ZXIi PnRhYmxlX25hbWU8L3JlcGxhY2VhYmxlPiBbICggPHJlcGxhY2VhYmxlCiAgICAgPC9saXN0aXRl bT4KICAgIDwvdmFybGlzdGVudHJ5PgogCisgICA8dmFybGlzdGVudHJ5PgorICAgIDx0ZXJtPjxs aXRlcmFsPklHTk9SRV9EQVRBVFlQRV9FUlJPUlM8L2xpdGVyYWw+PC90ZXJtPgorICAgIDxsaXN0 aXRlbT4KKyAgICAgPHBhcmE+CisgICAgICBEcm9wcyByb3dzIHRoYXQgY29udGFpbiBtYWxmb3Jt ZWQgZGF0YSB3aGlsZSBjb3B5aW5nLiBUaGVzZSBhcmUgcm93cworICAgICAgd2l0aCBjb2x1bW5z IHdoZXJlIHRoZSBkYXRhIHR5cGUncyBpbnB1dC1mdW5jdGlvbiByYWlzZXMgYW4gZXJyb3IuCisg ICAgICBUaGlzIG9wdGlvbiBpcyBub3QgYWxsb3dlZCB3aGVuIHVzaW5nIGJpbmFyeSBmb3JtYXQu ICBOb3RlIHRoYXQgdGhpcworICAgICAgaXMgb25seSBzdXBwb3J0ZWQgaW4gY3VycmVudCA8Y29t bWFuZD5DT1BZPC9jb21tYW5kPiBzeW50YXguCisgICAgIDwvcGFyYT4KKyAgICA8L2xpc3RpdGVt PgorICAgPC92YXJsaXN0ZW50cnk+CisKICAgIDx2YXJsaXN0ZW50cnk+CiAgICAgPHRlcm0+PGxp dGVyYWw+REVMSU1JVEVSPC9saXRlcmFsPjwvdGVybT4KICAgICA8bGlzdGl0ZW0+CmRpZmYgLS1n aXQgYS9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5LmMgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9j b3B5LmMKaW5kZXggZjE0ZmFlMzMwOC4uYWE1MGNjMWVlMSAxMDA2NDQKLS0tIGEvc3JjL2JhY2tl bmQvY29tbWFuZHMvY29weS5jCisrKyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHkuYwpAQCAt NDE5LDYgKzQxOSw3IEBAIFByb2Nlc3NDb3B5T3B0aW9ucyhQYXJzZVN0YXRlICpwc3RhdGUsCiAJ Ym9vbAkJZm9ybWF0X3NwZWNpZmllZCA9IGZhbHNlOwogCWJvb2wJCWZyZWV6ZV9zcGVjaWZpZWQg PSBmYWxzZTsKIAlib29sCQloZWFkZXJfc3BlY2lmaWVkID0gZmFsc2U7CisJYm9vbAkJaWdub3Jl X2RhdGF0eXBlX2Vycm9yc19zcGVjaWZpZWQgPSBmYWxzZTsKIAlMaXN0Q2VsbCAgICpvcHRpb247 CiAKIAkvKiBTdXBwb3J0IGV4dGVybmFsIHVzZSBmb3Igb3B0aW9uIHNhbml0eSBjaGVja2luZyAq LwpAQCAtNDU4LDYgKzQ1OSwxMyBAQCBQcm9jZXNzQ29weU9wdGlvbnMoUGFyc2VTdGF0ZSAqcHN0 YXRlLAogCQkJZnJlZXplX3NwZWNpZmllZCA9IHRydWU7CiAJCQlvcHRzX291dC0+ZnJlZXplID0g ZGVmR2V0Qm9vbGVhbihkZWZlbCk7CiAJCX0KKwkJZWxzZSBpZiAoc3RyY21wKGRlZmVsLT5kZWZu YW1lLCAiaWdub3JlX2RhdGF0eXBlX2Vycm9ycyIpID09IDApCisJCXsKKwkJCWlmIChpZ25vcmVf ZGF0YXR5cGVfZXJyb3JzX3NwZWNpZmllZCkKKwkJCQllcnJvckNvbmZsaWN0aW5nRGVmRWxlbShk ZWZlbCwgcHN0YXRlKTsKKwkJCWlnbm9yZV9kYXRhdHlwZV9lcnJvcnNfc3BlY2lmaWVkID0gdHJ1 ZTsKKwkJCW9wdHNfb3V0LT5pZ25vcmVfZGF0YXR5cGVfZXJyb3JzID0gZGVmR2V0Qm9vbGVhbihk ZWZlbCk7CisJCX0KIAkJZWxzZSBpZiAoc3RyY21wKGRlZmVsLT5kZWZuYW1lLCAiZGVsaW1pdGVy IikgPT0gMCkKIAkJewogCQkJaWYgKG9wdHNfb3V0LT5kZWxpbSkKQEAgLTU3Niw3ICs1ODQsNyBA QCBQcm9jZXNzQ29weU9wdGlvbnMoUGFyc2VTdGF0ZSAqcHN0YXRlLAogCX0KIAogCS8qCi0JICog Q2hlY2sgZm9yIGluY29tcGF0aWJsZSBvcHRpb25zIChtdXN0IGRvIHRoZXNlIHR3byBiZWZvcmUg aW5zZXJ0aW5nCisJICogQ2hlY2sgZm9yIGluY29tcGF0aWJsZSBvcHRpb25zIChtdXN0IGRvIHRo ZXNlIGJlZm9yZSBpbnNlcnRpbmcKIAkgKiBkZWZhdWx0cykKIAkgKi8KIAlpZiAob3B0c19vdXQt PmJpbmFyeSAmJiBvcHRzX291dC0+ZGVsaW0pCkBAIC01OTQsNiArNjAyLDExIEBAIFByb2Nlc3ND b3B5T3B0aW9ucyhQYXJzZVN0YXRlICpwc3RhdGUsCiAJCQkJKGVycmNvZGUoRVJSQ09ERV9TWU5U QVhfRVJST1IpLAogCQkJCSBlcnJtc2coImNhbm5vdCBzcGVjaWZ5IERFRkFVTFQgaW4gQklOQVJZ IG1vZGUiKSkpOwogCisJaWYgKG9wdHNfb3V0LT5iaW5hcnkgJiYgb3B0c19vdXQtPmlnbm9yZV9k YXRhdHlwZV9lcnJvcnMpCisJCWVyZXBvcnQoRVJST1IsCisJCQkJKGVycmNvZGUoRVJSQ09ERV9T WU5UQVhfRVJST1IpLAorCQkJCSBlcnJtc2coImNhbm5vdCBzcGVjaWZ5IElHTk9SRV9EQVRBVFlQ RV9FUlJPUlMgaW4gQklOQVJZIG1vZGUiKSkpOworCiAJLyogU2V0IGRlZmF1bHRzIGZvciBvbWl0 dGVkIG9wdGlvbnMgKi8KIAlpZiAoIW9wdHNfb3V0LT5kZWxpbSkKIAkJb3B0c19vdXQtPmRlbGlt ID0gb3B0c19vdXQtPmNzdl9tb2RlID8gIiwiIDogIlx0IjsKZGlmZiAtLWdpdCBhL3NyYy9iYWNr ZW5kL2NvbW1hbmRzL2NvcHlmcm9tLmMgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5ZnJvbS5j CmluZGV4IDgwYmNhNzljZDAuLjllMjNjZDQ1ZDUgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2Nv bW1hbmRzL2NvcHlmcm9tLmMKKysrIGIvc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb20uYwpA QCAtNjUyLDYgKzY1Miw3IEBAIENvcHlGcm9tKENvcHlGcm9tU3RhdGUgY3N0YXRlKQogCWJvb2wJ CWhhc19iZWZvcmVfaW5zZXJ0X3Jvd190cmlnOwogCWJvb2wJCWhhc19pbnN0ZWFkX2luc2VydF9y b3dfdHJpZzsKIAlib29sCQlsZWFmcGFydF91c2VfbXVsdGlfaW5zZXJ0ID0gZmFsc2U7CisJRXJy b3JTYXZlQ29udGV4dCBlc2NvbnRleHQgPSB7VF9FcnJvclNhdmVDb250ZXh0fTsKIAogCUFzc2Vy dChjc3RhdGUtPnJlbCk7CiAJQXNzZXJ0KGxpc3RfbGVuZ3RoKGNzdGF0ZS0+cmFuZ2VfdGFibGUp ID09IDEpOwpAQCAtNzUyLDYgKzc1MywxMyBAQCBDb3B5RnJvbShDb3B5RnJvbVN0YXRlIGNzdGF0 ZSkKIAkJdGlfb3B0aW9ucyB8PSBUQUJMRV9JTlNFUlRfRlJPWkVOOwogCX0KIAorCS8qIFNldCB1 cCBzb2Z0IGVycm9yIGhhbmRsZXIgZm9yIElHTk9SRV9EQVRBVFlQRV9FUlJPUlMgKi8KKwlpZiAo Y3N0YXRlLT5vcHRzLmlnbm9yZV9kYXRhdHlwZV9lcnJvcnMpCisJeworCQllc2NvbnRleHQuZGV0 YWlsc193YW50ZWQgPSB0cnVlOworCQljc3RhdGUtPmVzY29udGV4dCA9IGVzY29udGV4dDsKKwl9 CisKIAkvKgogCSAqIFdlIG5lZWQgYSBSZXN1bHRSZWxJbmZvIHNvIHdlIGNhbiB1c2UgdGhlIHJl Z3VsYXIgZXhlY3V0b3IncwogCSAqIGluZGV4LWVudHJ5LW1ha2luZyBtYWNoaW5lcnkuICAoVGhl cmUgdXNlZCB0byBiZSBhIGh1Z2UgYW1vdW50IG9mIGNvZGUKQEAgLTk4Nyw3ICs5OTUsMzEgQEAg Q29weUZyb20oQ29weUZyb21TdGF0ZSBjc3RhdGUpCiAKIAkJLyogRGlyZWN0bHkgc3RvcmUgdGhl IHZhbHVlcy9udWxscyBhcnJheSBpbiB0aGUgc2xvdCAqLwogCQlpZiAoIU5leHRDb3B5RnJvbShj c3RhdGUsIGVjb250ZXh0LCBteXNsb3QtPnR0c192YWx1ZXMsIG15c2xvdC0+dHRzX2lzbnVsbCkp CisJCXsKKwkJCWlmIChjc3RhdGUtPm9wdHMuaWdub3JlX2RhdGF0eXBlX2Vycm9ycyAmJiBjc3Rh dGUtPmlnbm9yZWRfZXJyb3JzID4gMCkKKwkJCQllcmVwb3J0KFdBUk5JTkcsCisJCQkJCQllcnJt c2coIiV6ZCByb3dzIHdlcmUgc2tpcHBlZCBkdWUgdG8gZGF0YSB0eXBlIGluY29tcGF0aWJpbGl0 eSIsCisJCQkJCQkJY3N0YXRlLT5pZ25vcmVkX2Vycm9ycykpOwogCQkJYnJlYWs7CisJCX0KKwor CQkvKiBTb2Z0IGVycm9yIG9jY3VyZWQsIHNraXAgdGhpcyB0dXBsZSBhbmQgY2xlYW51cCB0aGUg ZXNjb250ZXh0ICovCisJCWlmIChjc3RhdGUtPmVzY29udGV4dC5lcnJvcl9vY2N1cnJlZCkKKwkJ eworCQkJRXJyb3JTYXZlQ29udGV4dCBuZXdfZXNjb250ZXh0ID0ge1RfRXJyb3JTYXZlQ29udGV4 dH07CisKKwkJCS8qIGFkanVzdCBlbGV2ZWwgc28gd2UgZG9uJ3QganVtcCBvdXQgKi8KKwkJCWNz dGF0ZS0+ZXNjb250ZXh0LmVycm9yX2RhdGEtPmVsZXZlbCA9IFdBUk5JTkc7CisJCQkvKiBkZXNw aXRlIHRoZSBuYW1lLCB0aGlzIHdvbid0IHJhaXNlIGFuIGVycm9yIHNpbmNlIGVsZXZlbCBpcyBX QVJOSU5HIG5vdyAqLworCQkJVGhyb3dFcnJvckRhdGEoY3N0YXRlLT5lc2NvbnRleHQuZXJyb3Jf ZGF0YSk7CisKKwkJCUV4ZWNDbGVhclR1cGxlKG15c2xvdCk7CisKKwkJCW5ld19lc2NvbnRleHQu ZGV0YWlsc193YW50ZWQgPSB0cnVlOworCQkJY3N0YXRlLT5lc2NvbnRleHQgPSBuZXdfZXNjb250 ZXh0OworCisJCQljb250aW51ZTsKKwkJfQogCiAJCUV4ZWNTdG9yZVZpcnR1YWxUdXBsZShteXNs b3QpOwogCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5ZnJvbXBhcnNlLmMg Yi9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5ZnJvbXBhcnNlLmMKaW5kZXggMzg1MzkwMmExNi4u NmIwNDQ3NzgyZCAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb21wYXJz ZS5jCisrKyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHlmcm9tcGFyc2UuYwpAQCAtNzAsNiAr NzAsNyBAQAogI2luY2x1ZGUgImxpYnBxL3BxZm9ybWF0LmgiCiAjaW5jbHVkZSAibWIvcGdfd2No YXIuaCIKICNpbmNsdWRlICJtaXNjYWRtaW4uaCIKKyNpbmNsdWRlICJub2Rlcy9taXNjbm9kZXMu aCIKICNpbmNsdWRlICJwZ3N0YXQuaCIKICNpbmNsdWRlICJwb3J0L3BnX2Jzd2FwLmgiCiAjaW5j bHVkZSAidXRpbHMvYnVpbHRpbnMuaCIKQEAgLTk1NiwxMCArOTU3LDE4IEBAIE5leHRDb3B5RnJv bShDb3B5RnJvbVN0YXRlIGNzdGF0ZSwgRXhwckNvbnRleHQgKmVjb250ZXh0LAogCQkJCXZhbHVl c1ttXSA9IEV4ZWNFdmFsRXhwcihkZWZleHByc1ttXSwgZWNvbnRleHQsICZudWxsc1ttXSk7CiAJ CQl9CiAJCQllbHNlCi0JCQkJdmFsdWVzW21dID0gSW5wdXRGdW5jdGlvbkNhbGwoJmluX2Z1bmN0 aW9uc1ttXSwKLQkJCQkJCQkJCQkJICBzdHJpbmcsCi0JCQkJCQkJCQkJCSAgdHlwaW9wYXJhbXNb bV0sCi0JCQkJCQkJCQkJCSAgYXR0LT5hdHR0eXBtb2QpOworCQkJCS8qIElmIElHTk9SRV9EQVRB VFlQRV9FUlJPUlMgaXMgZW5hYmxlZCBza2lwIHJvd3Mgd2l0aCBkYXRhdHlwZSBlcnJvcnMgKi8K KwkJCQlpZiAoIUlucHV0RnVuY3Rpb25DYWxsU2FmZSgmaW5fZnVuY3Rpb25zW21dLAorCQkJCQkJ CQkJCSAgIHN0cmluZywKKwkJCQkJCQkJCQkgICB0eXBpb3BhcmFtc1ttXSwKKwkJCQkJCQkJCQkg ICBhdHQtPmF0dHR5cG1vZCwKKwkJCQkJCQkJCQkgICAoTm9kZSAqKSAmY3N0YXRlLT5lc2NvbnRl eHQsCisJCQkJCQkJCQkJICAgJnZhbHVlc1ttXSkpCisJCQkJeworCQkJCQljc3RhdGUtPmlnbm9y ZWRfZXJyb3JzKys7CisKKwkJCQkJcmV0dXJuIHRydWU7CisJCQkJfQogCiAJCQljc3RhdGUtPmN1 cl9hdHRuYW1lID0gTlVMTDsKIAkJCWNzdGF0ZS0+Y3VyX2F0dHZhbCA9IE5VTEw7CmRpZmYgLS1n aXQgYS9zcmMvYmluL3BzcWwvdGFiLWNvbXBsZXRlLmMgYi9zcmMvYmluL3BzcWwvdGFiLWNvbXBs ZXRlLmMKaW5kZXggNDJlODdiOWU0OS4uNGI0NDNkNGVhNyAxMDA2NDQKLS0tIGEvc3JjL2Jpbi9w c3FsL3RhYi1jb21wbGV0ZS5jCisrKyBiL3NyYy9iaW4vcHNxbC90YWItY29tcGxldGUuYwpAQCAt Mjg1Nyw3ICsyODU3LDggQEAgcHNxbF9jb21wbGV0aW9uKGNvbnN0IGNoYXIgKnRleHQsIGludCBz dGFydCwgaW50IGVuZCkKIAllbHNlIGlmIChNYXRjaGVzKCJDT1BZfFxcY29weSIsIE1hdGNoQW55 LCAiRlJPTXxUTyIsIE1hdGNoQW55LCAiV0lUSCIsICIoIikpCiAJCUNPTVBMRVRFX1dJVEgoIkZP Uk1BVCIsICJGUkVFWkUiLCAiREVMSU1JVEVSIiwgIk5VTEwiLAogCQkJCQkgICJIRUFERVIiLCAi UVVPVEUiLCAiRVNDQVBFIiwgIkZPUkNFX1FVT1RFIiwKLQkJCQkJICAiRk9SQ0VfTk9UX05VTEwi LCAiRk9SQ0VfTlVMTCIsICJFTkNPRElORyIsICJERUZBVUxUIik7CisJCQkJCSAgIkZPUkNFX05P VF9OVUxMIiwgIkZPUkNFX05VTEwiLCAiRU5DT0RJTkciLCAiREVGQVVMVCIsCisJCQkJCSAgIklH Tk9SRV9EQVRBVFlQRV9FUlJPUlMiKTsKIAogCS8qIENvbXBsZXRlIENPUFkgPHN0aD4gRlJPTXxU TyBmaWxlbmFtZSBXSVRIIChGT1JNQVQgKi8KIAllbHNlIGlmIChNYXRjaGVzKCJDT1BZfFxcY29w eSIsIE1hdGNoQW55LCAiRlJPTXxUTyIsIE1hdGNoQW55LCAiV0lUSCIsICIoIiwgIkZPUk1BVCIp KQpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvY29tbWFuZHMvY29weS5oIGIvc3JjL2luY2x1ZGUv Y29tbWFuZHMvY29weS5oCmluZGV4IDMzMTc1ODY4ZjYuLmMyZTU1YWMyMWYgMTAwNjQ0Ci0tLSBh L3NyYy9pbmNsdWRlL2NvbW1hbmRzL2NvcHkuaAorKysgYi9zcmMvaW5jbHVkZS9jb21tYW5kcy9j b3B5LmgKQEAgLTQyLDYgKzQyLDcgQEAgdHlwZWRlZiBzdHJ1Y3QgQ29weUZvcm1hdE9wdGlvbnMK IAkJCQkJCQkJICogLTEgaWYgbm90IHNwZWNpZmllZCAqLwogCWJvb2wJCWJpbmFyeTsJCQkvKiBi aW5hcnkgZm9ybWF0PyAqLwogCWJvb2wJCWZyZWV6ZTsJCQkvKiBmcmVlemUgcm93cyBvbiBsb2Fk aW5nPyAqLworCWJvb2wJCWlnbm9yZV9kYXRhdHlwZV9lcnJvcnM7ICAvKiBpZ25vcmUgcm93cyB3 aXRoIGRhdGF0eXBlIGVycm9ycyAqLwogCWJvb2wJCWNzdl9tb2RlOwkJLyogQ29tbWEgU2VwYXJh dGVkIFZhbHVlIGZvcm1hdD8gKi8KIAlDb3B5SGVhZGVyQ2hvaWNlIGhlYWRlcl9saW5lOwkvKiBo ZWFkZXIgbGluZT8gKi8KIAljaGFyCSAgICpudWxsX3ByaW50OwkJLyogTlVMTCBtYXJrZXIgc3Ry aW5nIChzZXJ2ZXIgZW5jb2RpbmchKSAqLwpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvY29tbWFu ZHMvY29weWZyb21faW50ZXJuYWwuaCBiL3NyYy9pbmNsdWRlL2NvbW1hbmRzL2NvcHlmcm9tX2lu dGVybmFsLmgKaW5kZXggYWMyYzE2ZjhiOC4uMTE2NGM3MTYzMSAxMDA2NDQKLS0tIGEvc3JjL2lu Y2x1ZGUvY29tbWFuZHMvY29weWZyb21faW50ZXJuYWwuaAorKysgYi9zcmMvaW5jbHVkZS9jb21t YW5kcy9jb3B5ZnJvbV9pbnRlcm5hbC5oCkBAIC0xNiw2ICsxNiw3IEBACiAKICNpbmNsdWRlICJj b21tYW5kcy9jb3B5LmgiCiAjaW5jbHVkZSAiY29tbWFuZHMvdHJpZ2dlci5oIgorI2luY2x1ZGUg Im5vZGVzL21pc2Nub2Rlcy5oIgogCiAvKgogICogUmVwcmVzZW50cyB0aGUgZGlmZmVyZW50IHNv dXJjZSBjYXNlcyB3ZSBuZWVkIHRvIHdvcnJ5IGFib3V0IGF0CkBAIC05NCw2ICs5NSw4IEBAIHR5 cGVkZWYgc3RydWN0IENvcHlGcm9tU3RhdGVEYXRhCiAJCQkJCQkJCSAqIGRlZmF1bHQgdmFsdWUg Ki8KIAlGbWdySW5mbyAgICppbl9mdW5jdGlvbnM7CS8qIGFycmF5IG9mIGlucHV0IGZ1bmN0aW9u cyBmb3IgZWFjaCBhdHRycyAqLwogCU9pZAkJICAgKnR5cGlvcGFyYW1zOwkvKiBhcnJheSBvZiBl bGVtZW50IHR5cGVzIGZvciBpbl9mdW5jdGlvbnMgKi8KKwlFcnJvclNhdmVDb250ZXh0IGVzY29u dGV4dDsgLyogc29mdCBlcnJvciB0cmFwcGVyIGR1cmluZyBpbl9mdW5jdGlvbnMgZXhlY3V0aW9u ICovCisJaW50NjQJCWlnbm9yZWRfZXJyb3JzOwkvKiB0b3RhbCBudW1iZXIgb2YgaWdub3JlZCBl cnJvcnMgKi8KIAlpbnQJCSAgICpkZWZtYXA7CQkJLyogYXJyYXkgb2YgZGVmYXVsdCBhdHQgbnVt YmVycyByZWxhdGVkIHRvCiAJCQkJCQkJCSAqIG1pc3NpbmcgYXR0ICovCiAJRXhwclN0YXRlICoq ZGVmZXhwcnM7CQkvKiBhcnJheSBvZiBkZWZhdWx0IGF0dCBleHByZXNzaW9ucyBmb3IgYWxsCmRp ZmYgLS1naXQgYS9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL2NvcHkyLm91dCBiL3NyYy90ZXN0 L3JlZ3Jlc3MvZXhwZWN0ZWQvY29weTIub3V0CmluZGV4IDhlMzNlZWU3MTkuLjk5ZjcwZjQ0ZWQg MTAwNjQ0Ci0tLSBhL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvY29weTIub3V0CisrKyBiL3Ny Yy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvY29weTIub3V0CkBAIC04Miw2ICs4Miw4IEBAIENPUFkg eCB0byBzdGRpbiAoZm9ybWF0IEJJTkFSWSwgZGVsaW1pdGVyICcsJyk7CiBFUlJPUjogIGNhbm5v dCBzcGVjaWZ5IERFTElNSVRFUiBpbiBCSU5BUlkgbW9kZQogQ09QWSB4IHRvIHN0ZGluIChmb3Jt YXQgQklOQVJZLCBudWxsICd4Jyk7CiBFUlJPUjogIGNhbm5vdCBzcGVjaWZ5IE5VTEwgaW4gQklO QVJZIG1vZGUKK0NPUFkgeCB0byBzdGRpbiAoZm9ybWF0IEJJTkFSWSwgaWdub3JlX2RhdGF0eXBl X2Vycm9ycyk7CitFUlJPUjogIGNhbm5vdCBzcGVjaWZ5IElHTk9SRV9EQVRBVFlQRV9FUlJPUlMg aW4gQklOQVJZIG1vZGUKIENPUFkgeCB0byBzdGRpbiAoZm9ybWF0IFRFWFQsIGZvcmNlX3F1b3Rl KGEpKTsKIEVSUk9SOiAgQ09QWSBmb3JjZSBxdW90ZSBhdmFpbGFibGUgb25seSBpbiBDU1YgbW9k ZQogQ09QWSB4IGZyb20gc3RkaW4gKGZvcm1hdCBDU1YsIGZvcmNlX3F1b3RlKGEpKTsKQEAgLTY2 Niw2ICs2NjgsMjUgQEAgU0VMRUNUICogRlJPTSBpbnN0ZWFkX29mX2luc2VydF90Ymw7CiAoMiBy b3dzKQogCiBDT01NSVQ7CistLSB0ZXN0cyBmb3IgSUdOT1JFX0RBVEFUWVBFX0VSUk9SUyBvcHRp b24KK0NSRUFURSBUQUJMRSBjaGVja19pZ25fZXJyIChuIGludCwgbSBpbnRbXSwgayBpbnQpOwor Q09QWSBjaGVja19pZ25fZXJyIEZST00gU1RESU4gV0lUSCAoSUdOT1JFX0RBVEFUWVBFX0VSUk9S Uyk7CitXQVJOSU5HOiAgaW52YWxpZCBpbnB1dCBzeW50YXggZm9yIHR5cGUgaW50ZWdlcjogImEi CitXQVJOSU5HOiAgdmFsdWUgIjMzMzMzMzMzMzMiIGlzIG91dCBvZiByYW5nZSBmb3IgdHlwZSBp bnRlZ2VyCitXQVJOSU5HOiAgaW52YWxpZCBpbnB1dCBzeW50YXggZm9yIHR5cGUgaW50ZWdlcjog ImEiCitXQVJOSU5HOiAgaW52YWxpZCBpbnB1dCBzeW50YXggZm9yIHR5cGUgaW50ZWdlcjogIiIK K1dBUk5JTkc6ICA0IHJvd3Mgd2VyZSBza2lwcGVkIGR1ZSB0byBkYXRhIHR5cGUgaW5jb21wYXRp YmlsaXR5CitTRUxFQ1QgKiBGUk9NIGNoZWNrX2lnbl9lcnI7CisgbiB8ICBtICB8IGsgCistLS0r LS0tLS0rLS0tCisgMSB8IHsxfSB8IDEKKyA1IHwgezV9IHwgNQorKDIgcm93cykKKworLS0gdGVz dCBtaXNzaW5nIGRhdGE6IHNob3VsZCBmYWlsCitDT1BZIGNoZWNrX2lnbl9lcnIgRlJPTSBTVERJ TiBXSVRIIChJR05PUkVfREFUQVRZUEVfRVJST1JTKTsKK0VSUk9SOiAgbWlzc2luZyBkYXRhIGZv ciBjb2x1bW4gImsiCitDT05URVhUOiAgQ09QWSBjaGVja19pZ25fZXJyLCBsaW5lIDE6ICIxCXsx fSIKIC0tIGNsZWFuIHVwCiBEUk9QIFRBQkxFIGZvcmNldGVzdDsKIERST1AgVEFCTEUgdmlzdGVz dDsKQEAgLTY4MCw2ICs3MDEsNyBAQCBEUk9QIFRBQkxFIGluc3RlYWRfb2ZfaW5zZXJ0X3RibDsK IERST1AgVklFVyBpbnN0ZWFkX29mX2luc2VydF90YmxfdmlldzsKIERST1AgVklFVyBpbnN0ZWFk X29mX2luc2VydF90Ymxfdmlld18yOwogRFJPUCBGVU5DVElPTiBmdW5faW5zdGVhZF9vZl9pbnNl cnRfdGJsKCk7CitEUk9QIFRBQkxFIGNoZWNrX2lnbl9lcnI7CiAtLQogLS0gQ09QWSBGUk9NIC4u LiBERUZBVUxUCiAtLQpkaWZmIC0tZ2l0IGEvc3JjL3Rlc3QvcmVncmVzcy9zcWwvY29weTIuc3Fs IGIvc3JjL3Rlc3QvcmVncmVzcy9zcWwvY29weTIuc3FsCmluZGV4IGQ3NTk2MzUwNjguLmU4OGYy N2JiZmQgMTAwNjQ0Ci0tLSBhL3NyYy90ZXN0L3JlZ3Jlc3Mvc3FsL2NvcHkyLnNxbAorKysgYi9z cmMvdGVzdC9yZWdyZXNzL3NxbC9jb3B5Mi5zcWwKQEAgLTcwLDYgKzcwLDcgQEAgQ09QWSB4IGZy b20gc3RkaW4gKGVuY29kaW5nICdzcWxfYXNjaWknLCBlbmNvZGluZyAnc3FsX2FzY2lpJyk7CiAt LSBpbmNvcnJlY3Qgb3B0aW9ucwogQ09QWSB4IHRvIHN0ZGluIChmb3JtYXQgQklOQVJZLCBkZWxp bWl0ZXIgJywnKTsKIENPUFkgeCB0byBzdGRpbiAoZm9ybWF0IEJJTkFSWSwgbnVsbCAneCcpOwor Q09QWSB4IHRvIHN0ZGluIChmb3JtYXQgQklOQVJZLCBpZ25vcmVfZGF0YXR5cGVfZXJyb3JzKTsK IENPUFkgeCB0byBzdGRpbiAoZm9ybWF0IFRFWFQsIGZvcmNlX3F1b3RlKGEpKTsKIENPUFkgeCBm cm9tIHN0ZGluIChmb3JtYXQgQ1NWLCBmb3JjZV9xdW90ZShhKSk7CiBDT1BZIHggdG8gc3Rkb3V0 IChmb3JtYXQgVEVYVCwgZm9yY2Vfbm90X251bGwoYSkpOwpAQCAtNDY0LDYgKzQ2NSwyMyBAQCB0 ZXN0MQogU0VMRUNUICogRlJPTSBpbnN0ZWFkX29mX2luc2VydF90Ymw7CiBDT01NSVQ7CiAKKy0t IHRlc3RzIGZvciBJR05PUkVfREFUQVRZUEVfRVJST1JTIG9wdGlvbgorQ1JFQVRFIFRBQkxFIGNo ZWNrX2lnbl9lcnIgKG4gaW50LCBtIGludFtdLCBrIGludCk7CitDT1BZIGNoZWNrX2lnbl9lcnIg RlJPTSBTVERJTiBXSVRIIChJR05PUkVfREFUQVRZUEVfRVJST1JTKTsKKzEJezF9CTEKK2EJezJ9 CTIKKzMJezN9CTMzMzMzMzMzMzMKKzQJe2EsIDR9CTQKKworNQl7NX0JNQorXC4KK1NFTEVDVCAq IEZST00gY2hlY2tfaWduX2VycjsKKworLS0gdGVzdCBtaXNzaW5nIGRhdGE6IHNob3VsZCBmYWls CitDT1BZIGNoZWNrX2lnbl9lcnIgRlJPTSBTVERJTiBXSVRIIChJR05PUkVfREFUQVRZUEVfRVJS T1JTKTsKKzEJezF9CitcLgorCiAtLSBjbGVhbiB1cAogRFJPUCBUQUJMRSBmb3JjZXRlc3Q7CiBE Uk9QIFRBQkxFIHZpc3Rlc3Q7CkBAIC00NzgsNiArNDk2LDcgQEAgRFJPUCBUQUJMRSBpbnN0ZWFk X29mX2luc2VydF90Ymw7CiBEUk9QIFZJRVcgaW5zdGVhZF9vZl9pbnNlcnRfdGJsX3ZpZXc7CiBE Uk9QIFZJRVcgaW5zdGVhZF9vZl9pbnNlcnRfdGJsX3ZpZXdfMjsKIERST1AgRlVOQ1RJT04gZnVu X2luc3RlYWRfb2ZfaW5zZXJ0X3RibCgpOworRFJPUCBUQUJMRSBjaGVja19pZ25fZXJyOwogCiAt LQogLS0gQ09QWSBGUk9NIC4uLiBERUZBVUxUCgpiYXNlLWNvbW1pdDogMzZmNDBjZTJkYzY2ZjFh MzZkNmExMmY3YTAzNTJlMWM1YmYxMDYzZQotLSAKMi4yNS4xCgo= --=_bac8f6fe4607d4ed50972ec196ed62d6--