Feedback-ID: id4a34324:Fastmail
Date: Wed, 8 Nov 2023 15:53:58 -0800
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Damir <dam.bel07@gmail.com>, torikoshia <torikoshia@oss.nttdata.com>,
	Pgsql Hackers <pgsql-hackers@postgresql.org>, daniel@yesql.se,
	anisimow.d@gmail.com, HukuToc@gmail.com,
	Andrey Lepikhov <a.lepikhov@postgrespro.ru>,
	Alena Rybakina <lena.ribackina@yandex.ru>
Subject: Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
Message-ID: <20231108235358.jmay4ps767mecrnu@awork3.anarazel.de>
References: <CALH1LguOY-VvJBWvjpLR2W3t0Z2p=LGSK5tzLrKwtzZ2UsqwGg@mail.gmail.com>
 <43399b33fe573f0548410893f41f84b6@oss.nttdata.com>
 <CALH1LgseNT7mxpGzQsQwOJz_UwYBo6L-4vR9tmJqb-WNPkt6Bg@mail.gmail.com>
 <8de9d5e1-2f81-ee2e-ece4-b1a6ede1f285@yandex.ru>
 <b74c1c3e3cc1c5d7ad5fb527b27d2b60@oss.nttdata.com>
 <c0fb57b82b150953f26a5c7e340412e8@oss.nttdata.com>
 <CALH1Lgtg39crESw644KDH+ejBWuW3Vospe67F-ZWXNeBMEE=PA@mail.gmail.com>
 <58fa3b63e88ffd0a09273e38aec0661e@oss.nttdat! a.com>
 <a303c70a-ca58-fd8d-e5f5-00b941f24667@gmail.com>
 <739953.1699467519@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <739953.1699467519@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/20231108235358.jmay4ps767mecrnu%40awork3.anarazel.de>
Precedence: bulk

Hi,

On 2023-11-08 13:18:39 -0500, Tom Lane wrote:
> Damir <dam.bel07@gmail.com> writes:
> > [ v7-0002-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch ]
> 
> Sorry for being so late to the party, but ... I don't think this
> is a well-designed feature as it stands.  Simply dropping failed rows
> seems like an unusable definition for any application that has
> pretensions of robustness.

Not everything needs to be a robust application though. I've definitely cursed
at postgres for lacking this.


> I think an actually usable feature of this sort would involve
> copying all the failed lines to some alternate output medium,
> perhaps a second table with a TEXT column to receive the original
> data line.  (Or maybe an array of text that could receive the
> broken-down field values?)  Maybe we could dump the message info,
> line number, field name etc into additional columns.

If we go in that direction, we should make it possible to *not* use such a
table as well, for some uses it'd be pointless.


Another way of reporting errors could be for copy to return invalid input back
to the client, via the copy protocol. That would allow the client to handle
failing rows and also to abort if the number of errors or the type of errors
gets to be too big.

Greetings,

Andres Freund