Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rTIpy-001n6l-Fi for pgsql-hackers@arkaria.postgresql.org; Fri, 26 Jan 2024 09:49:27 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rTIpx-00Ct7s-73 for pgsql-hackers@arkaria.postgresql.org; Fri, 26 Jan 2024 09:49:25 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rTIpw-00Ct7k-S1 for pgsql-hackers@lists.postgresql.org; Fri, 26 Jan 2024 09:49:24 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rTIpo-003ncJ-9w for pgsql-hackers@postgresql.org; Fri, 26 Jan 2024 09:49:23 +0000 Received: from oss.nttdata.com (localhost [127.0.0.1]) by oss.nttdata.com (Postfix) with ESMTPA id 785F86023F for ; Fri, 26 Jan 2024 18:49:10 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at oss.nttdata.com MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_a37dbf443803b26fb65b020e5507fdc4" Content-Transfer-Encoding: 7bit Date: Fri, 26 Jan 2024 18:49:10 +0900 From: torikoshia To: Pgsql Hackers Subject: Add new COPY option REJECT_LIMIT User-Agent: Roundcube Webmail/1.4.11 Message-ID: <63f99327aa6b404cc951217fa3e61fe4@oss.nttdata.com> X-Sender: torikoshia@oss.nttdata.com List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --=_a37dbf443803b26fb65b020e5507fdc4 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Hi, 9e2d870 enabled the COPY command to skip soft error, and I think we can add another option which specifies the maximum tolerable number of soft errors. I remember this was discussed in [1], and feel it would be useful when loading 'dirty' data but there is a limit to how dirty it can be. Attached a patch for this. What do you think? [1] https://www.postgresql.org/message-id/752672.1699474336%40sss.pgh.pa.us -- Regards, -- Atsushi Torikoshi NTT DATA Group Corporation --=_a37dbf443803b26fb65b020e5507fdc4 Content-Transfer-Encoding: base64 Content-Type: text/x-diff; name=v1-0001-Add-new-COPY-option-REJECT_LIMIT.patch Content-Disposition: attachment; filename=v1-0001-Add-new-COPY-option-REJECT_LIMIT.patch; size=6364 RnJvbSA3ZjExMWU5OGUyMTY1NGM0Y2EzMzhjOTNkN2NiYjRlYzlhY2FhYmNiIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBBdHN1c2hpIFRvcmlrb3NoaSA8dG9yaWtvc2hpYUBvc3MubnR0 ZGF0YS5jb20+CkRhdGU6IEZyaSwgMjYgSmFuIDIwMjQgMTg6MzI6NDAgKzA5MDAKU3ViamVjdDog W1BBVENIIHYxXSBBZGQgbmV3IENPUFkgb3B0aW9uIFJFSkVDVF9MSU1JVAoKUkVKRUNUX0xJTUlU IHNwZWNpZmllcyB0aGUgbWF4aW11bSB0b2xlcmFibGUgbnVtYmVyIG9mIG1hbGZvcm1lZCByb3dz LgpJZiBpbnB1dCBkYXRhIGhhcyBtb3JlIG1hbGZvcm1lZCBlcnJvcnMgdGhhbiB0aGlzIHZhbHVl LCBlbnRpcmUgQ09QWSBmYWlscy4KVGhpcyBvcHRpb24gbXVzdCBiZSB1c2VkIHdpdGggT05fRVJS T1IgdG8gYmUgc2V0IHRvIG90aGVyIHRoYW4gc3RvcC4KLS0tCiBkb2Mvc3JjL3NnbWwvcmVmL2Nv cHkuc2dtbCAgICAgICAgICB8IDEzICsrKysrKysrKysrKysKIHNyYy9iYWNrZW5kL2NvbW1hbmRz L2NvcHkuYyAgICAgICAgIHwgMTYgKysrKysrKysrKysrKysrKwogc3JjL2JhY2tlbmQvY29tbWFu ZHMvY29weWZyb20uYyAgICAgfCAgNiArKysrKysKIHNyYy9pbmNsdWRlL2NvbW1hbmRzL2NvcHku aCAgICAgICAgIHwgIDEgKwogc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9jb3B5Mi5vdXQgfCAx MCArKysrKysrKysrCiBzcmMvdGVzdC9yZWdyZXNzL3NxbC9jb3B5Mi5zcWwgICAgICB8IDIxICsr KysrKysrKysrKysrKysrKysrKwogNiBmaWxlcyBjaGFuZ2VkLCA2NyBpbnNlcnRpb25zKCspCgpk aWZmIC0tZ2l0IGEvZG9jL3NyYy9zZ21sL3JlZi9jb3B5LnNnbWwgYi9kb2Mvc3JjL3NnbWwvcmVm L2NvcHkuc2dtbAppbmRleCAyMWE1YzRhMDUyLi44OTgyZTg0NjRhIDEwMDY0NAotLS0gYS9kb2Mv c3JjL3NnbWwvcmVmL2NvcHkuc2dtbAorKysgYi9kb2Mvc3JjL3NnbWwvcmVmL2NvcHkuc2dtbApA QCAtMzkzLDYgKzM5MywxOSBAQCBDT1BZIHsgPHJlcGxhY2VhYmxlIGNsYXNzPSJwYXJhbWV0ZXIi PnRhYmxlX25hbWU8L3JlcGxhY2VhYmxlPiBbICggPHJlcGxhY2VhYmxlCiAgICAgPC9saXN0aXRl bT4KICAgIDwvdmFybGlzdGVudHJ5PgogCisgICA8dmFybGlzdGVudHJ5PgorICAgIDx0ZXJtPjxs aXRlcmFsPlJFSkVDVF9MSU1JVDwvbGl0ZXJhbD48L3Rlcm0+CisgICAgPGxpc3RpdGVtPgorICAg ICA8cGFyYT4KKyAgICAgIFNwZWNpZmllcyB0aGUgbWF4aW11bSB0b2xlcmFibGUgbnVtYmVyIG9m IG1hbGZvcm1lZCByb3dzLgorICAgICAgSWYgaW5wdXQgZGF0YSBoYXMgY2F1c2VkIG1vcmUgbWFs Zm9ybWVkIGVycm9ycyB0aGFuIHRoaXMgdmFsdWUsIGVudGlyZQorICAgICAgPGNvbW1hbmQ+Q09Q WTwvY29tbWFuZD4gZmFpbHMuCisgICAgICBUaGlzIG9wdGlvbiBtdXN0IGJlIHVzZWQgd2l0aCA8 bGl0ZXJhbD5PTl9FUlJPUjwvbGl0ZXJhbD4gdG8gYmUgc2V0IHRvCisgICAgICBvdGhlciB0aGFu IDxsaXRlcmFsPnN0b3A8L2xpdGVyYWw+LgorICAgICA8L3BhcmE+CisgICAgPC9saXN0aXRlbT4K KyAgIDwvdmFybGlzdGVudHJ5PgorCiAgICA8dmFybGlzdGVudHJ5PgogICAgIDx0ZXJtPjxsaXRl cmFsPkVOQ09ESU5HPC9saXRlcmFsPjwvdGVybT4KICAgICA8bGlzdGl0ZW0+CmRpZmYgLS1naXQg YS9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5LmMgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5 LmMKaW5kZXggY2MwNzg2YzZmNC4uY2E1MjYzZDU4OCAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQv Y29tbWFuZHMvY29weS5jCisrKyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2NvcHkuYwpAQCAtNjE1 LDYgKzYxNSwyMiBAQCBQcm9jZXNzQ29weU9wdGlvbnMoUGFyc2VTdGF0ZSAqcHN0YXRlLAogCQkJ b25fZXJyb3Jfc3BlY2lmaWVkID0gdHJ1ZTsKIAkJCW9wdHNfb3V0LT5vbl9lcnJvciA9IGRlZkdl dENvcHlPbkVycm9yQ2hvaWNlKGRlZmVsLCBwc3RhdGUsIGlzX2Zyb20pOwogCQl9CisJCWVsc2Ug aWYgKHN0cmNtcChkZWZlbC0+ZGVmbmFtZSwgInJlamVjdF9saW1pdCIpID09IDApCisJCXsKKwkJ CWludDY0CXJlamVjdF9saW1pdCA9IGRlZkdldEludDY0KGRlZmVsKTsKKworCQkJaWYgKCFvcHRz X291dC0+b25fZXJyb3IpCisJCQkJZXJlcG9ydChFUlJPUiwKKwkJCQkJCShlcnJjb2RlKEVSUkNP REVfSU5WQUxJRF9QQVJBTUVURVJfVkFMVUUpLAorCQkJCQkJIGVycm1zZygiUkVKRUNUX0xJTUlU IHJlcXVpcmVzIE9OX0VSUk9SIHRvIGJlIHNldCB0byBvdGhlciB0aGFuIHN0b3AiKSkpOworCQkJ aWYgKG9wdHNfb3V0LT5yZWplY3RfbGltaXQgPiAwKQorCQkJCWVycm9yQ29uZmxpY3RpbmdEZWZF bGVtKGRlZmVsLCBwc3RhdGUpOworCQkJaWYgKHJlamVjdF9saW1pdCA8PSAwKQorCQkJCWVyZXBv cnQoRVJST1IsCisJCQkJCQkoZXJyY29kZShFUlJDT0RFX0lOVkFMSURfUEFSQU1FVEVSX1ZBTFVF KSwKKwkJCQkJCSBlcnJtc2coIlJFSkVDVF9MSU1JVCBtdXN0IGJlIGdyZWF0ZXIgdGhhbiB6ZXJv IikpKTsKKwkJCW9wdHNfb3V0LT5yZWplY3RfbGltaXQgPSByZWplY3RfbGltaXQ7CisJCX0KIAkJ ZWxzZQogCQkJZXJlcG9ydChFUlJPUiwKIAkJCQkJKGVycmNvZGUoRVJSQ09ERV9TWU5UQVhfRVJS T1IpLApkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb20uYyBiL3NyYy9i YWNrZW5kL2NvbW1hbmRzL2NvcHlmcm9tLmMKaW5kZXggMWZlNzBiOTEzMy4uMTUwNjY4ODdlYSAx MDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb20uYworKysgYi9zcmMvYmFj a2VuZC9jb21tYW5kcy9jb3B5ZnJvbS5jCkBAIC0xMDE3LDYgKzEwMTcsMTIgQEAgQ29weUZyb20o Q29weUZyb21TdGF0ZSBjc3RhdGUpCiAJCQlwZ3N0YXRfcHJvZ3Jlc3NfdXBkYXRlX3BhcmFtKFBS T0dSRVNTX0NPUFlfVFVQTEVTX1NLSVBQRUQsCiAJCQkJCQkJCQkJICsrc2tpcHBlZCk7CiAKKwkJ CWlmIChjc3RhdGUtPm9wdHMucmVqZWN0X2xpbWl0ID4gMCAmJiBza2lwcGVkID4gY3N0YXRlLT5v cHRzLnJlamVjdF9saW1pdCkKKwkJCQllcmVwb3J0KEVSUk9SLAorCQkJCQkJKGVycmNvZGUoRVJS Q09ERV9CQURfQ09QWV9GSUxFX0ZPUk1BVCksCisJCQkJCQkgZXJybXNnKCJleGNlZWRlZCB0aGUg bnVtYmVyIHNwZWNpZmllZCBieSBSRUpFQ1QgTElNSVQgXCIlZFwiIiwKKwkJCQkJCQkJY3N0YXRl LT5vcHRzLnJlamVjdF9saW1pdCkpKTsKKwogCQkJY29udGludWU7CiAJCX0KIApkaWZmIC0tZ2l0 IGEvc3JjL2luY2x1ZGUvY29tbWFuZHMvY29weS5oIGIvc3JjL2luY2x1ZGUvY29tbWFuZHMvY29w eS5oCmluZGV4IGIzZGEzY2IwYmUuLjhmOGRhYjk1MjQgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRl L2NvbW1hbmRzL2NvcHkuaAorKysgYi9zcmMvaW5jbHVkZS9jb21tYW5kcy9jb3B5LmgKQEAgLTcz LDYgKzczLDcgQEAgdHlwZWRlZiBzdHJ1Y3QgQ29weUZvcm1hdE9wdGlvbnMKIAlib29sCSAgICpm b3JjZV9udWxsX2ZsYWdzOwkvKiBwZXItY29sdW1uIENTViBGTiBmbGFncyAqLwogCWJvb2wJCWNv bnZlcnRfc2VsZWN0aXZlbHk7CS8qIGRvIHNlbGVjdGl2ZSBiaW5hcnkgY29udmVyc2lvbj8gKi8K IAlDb3B5T25FcnJvckNob2ljZSBvbl9lcnJvcjsgLyogd2hhdCB0byBkbyB3aGVuIGVycm9yIGhh cHBlbmVkICovCisJaW50CQkJcmVqZWN0X2xpbWl0OwkvKiB0b2xlcmFibGUgbnVtYmVyIG9mIG1h bGZvcm1lZCByb3dzICovCiAJTGlzdAkgICAqY29udmVydF9zZWxlY3Q7IC8qIGxpc3Qgb2YgY29s dW1uIG5hbWVzIChjYW4gYmUgTklMKSAqLwogfSBDb3B5Rm9ybWF0T3B0aW9uczsKIApkaWZmIC0t Z2l0IGEvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9jb3B5Mi5vdXQgYi9zcmMvdGVzdC9yZWdy ZXNzL2V4cGVjdGVkL2NvcHkyLm91dAppbmRleCAyNWM0MDFjZTM0Li4yOGRlN2EyNjg1IDEwMDY0 NAotLS0gYS9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL2NvcHkyLm91dAorKysgYi9zcmMvdGVz dC9yZWdyZXNzL2V4cGVjdGVkL2NvcHkyLm91dApAQCAtMTA4LDYgKzEwOCwxMCBAQCBDT1BZIHgg dG8gc3RkaW4gKGZvcm1hdCBCSU5BUlksIG9uX2Vycm9yIHVuc3VwcG9ydGVkKTsKIEVSUk9SOiAg Q09QWSBPTl9FUlJPUiBjYW5ub3QgYmUgdXNlZCB3aXRoIENPUFkgVE8KIExJTkUgMTogQ09QWSB4 IHRvIHN0ZGluIChmb3JtYXQgQklOQVJZLCBvbl9lcnJvciB1bnN1cHBvcnRlZCk7CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgXgorQ09QWSB4IGZyb20gc3RkaW4gd2l0 aCAocmVqZWN0X2xpbWl0IDMpOworRVJST1I6ICBSRUpFQ1RfTElNSVQgcmVxdWlyZXMgT05fRVJS T1IgdG8gYmUgc2V0IHRvIG90aGVyIHRoYW4gc3RvcAorQ09QWSB4IGZyb20gc3RkaW4gd2l0aCAo b25fZXJyb3IgaWdub3JlLCByZWplY3RfbGltaXQgMCk7CitFUlJPUjogIFJFSkVDVF9MSU1JVCBt dXN0IGJlIGdyZWF0ZXIgdGhhbiB6ZXJvCiAtLSB0b28gbWFueSBjb2x1bW5zIGluIGNvbHVtbiBs aXN0OiBzaG91bGQgZmFpbAogQ09QWSB4IChhLCBiLCBjLCBkLCBlLCBkLCBjKSBmcm9tIHN0ZGlu OwogRVJST1I6ICBjb2x1bW4gImQiIHNwZWNpZmllZCBtb3JlIHRoYW4gb25jZQpAQCAtNzUxLDYg Kzc1NSwxMiBAQCBDT05URVhUOiAgQ09QWSBjaGVja19pZ25fZXJyLCBsaW5lIDE6ICIxCXsxfSIK IENPUFkgY2hlY2tfaWduX2VyciBGUk9NIFNURElOIFdJVEggKG9uX2Vycm9yIGlnbm9yZSk7CiBF UlJPUjogIGV4dHJhIGRhdGEgYWZ0ZXIgbGFzdCBleHBlY3RlZCBjb2x1bW4KIENPTlRFWFQ6ICBD T1BZIGNoZWNrX2lnbl9lcnIsIGxpbmUgMTogIjEJezF9CTMJYWJjIgorLS0gdGVzdHMgZm9yIHJl amVjdF9saW1pdCBvcHRpb24KK0NPUFkgY2hlY2tfaWduX2VyciBGUk9NIFNURElOIFdJVEggKG9u X2Vycm9yIGlnbm9yZSwgcmVqZWN0X2xpbWl0IDMpOworRVJST1I6ICBleGNlZWRlZCB0aGUgbnVt YmVyIHNwZWNpZmllZCBieSBSRUpFQ1QgTElNSVQgIjMiCitDT05URVhUOiAgQ09QWSBjaGVja19p Z25fZXJyLCBsaW5lIDUsIGNvbHVtbiBuOiAiIgorQ09QWSBjaGVja19pZ25fZXJyIEZST00gU1RE SU4gV0lUSCAob25fZXJyb3IgaWdub3JlLCByZWplY3RfbGltaXQgNCk7CitOT1RJQ0U6ICA0IHJv d3Mgd2VyZSBza2lwcGVkIGR1ZSB0byBkYXRhIHR5cGUgaW5jb21wYXRpYmlsaXR5CiAtLSBjbGVh biB1cAogRFJPUCBUQUJMRSBmb3JjZXRlc3Q7CiBEUk9QIFRBQkxFIHZpc3Rlc3Q7CmRpZmYgLS1n aXQgYS9zcmMvdGVzdC9yZWdyZXNzL3NxbC9jb3B5Mi5zcWwgYi9zcmMvdGVzdC9yZWdyZXNzL3Nx bC9jb3B5Mi5zcWwKaW5kZXggYjVlNTQ5ZTg1Ni4uZTUyOTliMTMwOCAxMDA2NDQKLS0tIGEvc3Jj L3Rlc3QvcmVncmVzcy9zcWwvY29weTIuc3FsCisrKyBiL3NyYy90ZXN0L3JlZ3Jlc3Mvc3FsL2Nv cHkyLnNxbApAQCAtODAsNiArODAsOCBAQCBDT1BZIHggdG8gc3RkaW4gKGZvcm1hdCBDU1YsIGZv cmNlX25vdF9udWxsKGEpKTsKIENPUFkgeCB0byBzdGRvdXQgKGZvcm1hdCBURVhULCBmb3JjZV9u dWxsKGEpKTsKIENPUFkgeCB0byBzdGRpbiAoZm9ybWF0IENTViwgZm9yY2VfbnVsbChhKSk7CiBD T1BZIHggdG8gc3RkaW4gKGZvcm1hdCBCSU5BUlksIG9uX2Vycm9yIHVuc3VwcG9ydGVkKTsKK0NP UFkgeCBmcm9tIHN0ZGluIHdpdGggKHJlamVjdF9saW1pdCAzKTsKK0NPUFkgeCBmcm9tIHN0ZGlu IHdpdGggKG9uX2Vycm9yIGlnbm9yZSwgcmVqZWN0X2xpbWl0IDApOwogCiAtLSB0b28gbWFueSBj b2x1bW5zIGluIGNvbHVtbiBsaXN0OiBzaG91bGQgZmFpbAogQ09QWSB4IChhLCBiLCBjLCBkLCBl LCBkLCBjKSBmcm9tIHN0ZGluOwpAQCAtNTM0LDYgKzUzNiwyNSBAQCBDT1BZIGNoZWNrX2lnbl9l cnIgRlJPTSBTVERJTiBXSVRIIChvbl9lcnJvciBpZ25vcmUpOwogMQl7MX0JMwlhYmMKIFwuCiAK Ky0tIHRlc3RzIGZvciByZWplY3RfbGltaXQgb3B0aW9uCitDT1BZIGNoZWNrX2lnbl9lcnIgRlJP TSBTVERJTiBXSVRIIChvbl9lcnJvciBpZ25vcmUsIHJlamVjdF9saW1pdCAzKTsKKzYJezZ9CTYK K2EJezd9CTcKKzgJezh9CTg4ODg4ODg4ODgKKzkJe2EsIDl9CTkKKworMTAJezEwfQkxMAorXC4K KworQ09QWSBjaGVja19pZ25fZXJyIEZST00gU1RESU4gV0lUSCAob25fZXJyb3IgaWdub3JlLCBy ZWplY3RfbGltaXQgNCk7Cis2CXs2fQk2CithCXs3fQk3Cis4CXs4fQk4ODg4ODg4ODg4Cis5CXth LCA5fQk5CisKKzEwCXsxMH0JMTAKK1wuCisKIC0tIGNsZWFuIHVwCiBEUk9QIFRBQkxFIGZvcmNl dGVzdDsKIERST1AgVEFCTEUgdmlzdGVzdDsKCmJhc2UtY29tbWl0OiBmMjc0M2E3ZDcwZTdiMjg5 MTI3NzYzMjEyMWJiNTFlNzM5NzQzYTQ3Ci0tIAoyLjM5LjIKCg== --=_a37dbf443803b26fb65b020e5507fdc4--