Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r0oLF-007x36-Pw for pgsql-hackers@arkaria.postgresql.org; Wed, 08 Nov 2023 19:35:57 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r0oKF-007EsN-Rf for pgsql-hackers@arkaria.postgresql.org; Wed, 08 Nov 2023 19:34:55 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r0oKF-007EsF-HK for pgsql-hackers@lists.postgresql.org; Wed, 08 Nov 2023 19:34:55 +0000 Received: from smtp.outgoing.loopia.se ([93.188.3.37]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r0oKD-005Oti-Fr for pgsql-hackers@postgresql.org; Wed, 08 Nov 2023 19:34:54 +0000 Received: from s807.loopia.se (localhost [127.0.0.1]) by s807.loopia.se (Postfix) with ESMTP id 6D3182FC6A72 for ; Wed, 8 Nov 2023 20:34:52 +0100 (CET) Received: from s981.loopia.se (unknown [172.22.191.6]) by s807.loopia.se (Postfix) with ESMTP id 5C1702E27DA6; Wed, 8 Nov 2023 20:34:52 +0100 (CET) Received: from s471.loopia.se (unknown [172.22.191.5]) by s981.loopia.se (Postfix) with ESMTP id 583E722B173C; Wed, 8 Nov 2023 20:34:52 +0100 (CET) X-Virus-Scanned: amavisd-new at amavis.loopia.se X-Spam-Flag: NO X-Spam-Score: -1 X-Spam-Level: X-Spam-Status: No, score=-1 tagged_above=-999 required=6.2 tests=[ALL_TRUSTED=-1] autolearn=disabled Received: from s899.loopia.se ([172.22.191.6]) by s471.loopia.se (s471.loopia.se [172.22.190.35]) (amavisd-new, port 10024) with UTF8LMTP id qPgpZ6IisIoe; Wed, 8 Nov 2023 20:34:51 +0100 (CET) X-Loopia-Auth: user X-Loopia-User: daniel@yesql.se X-Loopia-Originating-IP: 89.255.232.193 Received: from smtpclient.apple (customer-89-255-232-193.stosn.net [89.255.232.193]) (Authenticated sender: daniel@yesql.se) by s899.loopia.se (Postfix) with ESMTPSA id B539C2C8BA29; Wed, 8 Nov 2023 20:34:51 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.3\)) Subject: Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) From: Daniel Gustafsson In-Reply-To: <739953.1699467519@sss.pgh.pa.us> Date: Wed, 8 Nov 2023 20:34:51 +0100 Cc: Damir , torikoshia , PostgreSQL Hackers , Andres Freund , anisimow.d@gmail.com, HukuToc@gmail.com, Andrey Lepikhov , Alena Rybakina Content-Transfer-Encoding: quoted-printable Message-Id: References: <483E1B59-4A80-450B-8685-7CC12762FD5A@anarazel.de> <0DE0602F-CC12-40ED-B259-3AB91FB02C3B@yesql.se> <8ad8492ff9fae3481d87c7aab4e0aed0@oss.nttdata.com> <9a94e684d6528dff74de5959e97032f3@oss.nttdata.com> <8e5c596e47435e3b37b7a751ebcd9569@oss.nttdata.com> <20230322175000.qbdctk7bnmifh5an@awork3.anarazel.de> <16e09747fcfbb21c30e4c1009c416aa4@oss.nttdata.com> <19551e8c2717c24689913083f841ddb5@oss.nttdata.com> <43399b33fe573f0548410893f41f84b6@oss.nttdata.com> <8de9d5e1-2f81-ee2e-ece4-b1a6ede1f285@yandex.ru> <58fa3b63e88ffd0a09273e38aec0661e@oss.nttdat! a.com> <739953.1699467519@sss.pgh.pa.us> To: Tom Lane X-Mailer: Apple Mail (2.3696.120.41.1.3) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk > On 8 Nov 2023, at 19:18, Tom Lane wrote: > I think an actually usable feature of this sort would involve > copying all the failed lines to some alternate output medium, > perhaps a second table with a TEXT column to receive the original > data line. (Or maybe an array of text that could receive the > broken-down field values?) Maybe we could dump the message info, > line number, field name etc into additional columns. I agree that the errors should be easily visible to the user in some = way. The feature is for sure interesting, especially in data warehouse type jobs = where dirty data is often ingested. As a data point, Greenplum has this feature with additional SQL syntax = to control it: COPY .. LOG ERRORS SEGMENT REJECT LIMIT xyz ROWS; LOG ERRORS instructs the database to log the faulty rows and SEGMENT = REJECT LIMIT xyz ROWS sets the limit of how many rows can be faulty before the operation errors out. I'm not at all advocating that we should mimic = this, just wanted to add a reference to postgres derivative where this has = been implemented. -- Daniel Gustafsson