Message-ID: <fab7018f-6e00-4be4-bd1e-4a013096f6c9@oss.nttdata.com>
Date: Wed, 3 Jul 2024 02:07:16 +0900
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Add new COPY option REJECT_LIMIT
To: pgsql-hackers@lists.postgresql.org
References: <63f99327aa6b404cc951217fa3e61fe4@oss.nttdata.com>
Content-Language: en-US
From: Fujii Masao <masao.fujii@oss.nttdata.com>
In-Reply-To: <63f99327aa6b404cc951217fa3e61fe4@oss.nttdata.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/fab7018f-6e00-4be4-bd1e-4a013096f6c9%40oss.nttdata.com>
Precedence: bulk


On 2024/01/26 18:49, torikoshia wrote:
> Hi,
> 
> 9e2d870 enabled the COPY command to skip soft error, and I think we can add another option which specifies the maximum tolerable number of soft errors.
> 
> I remember this was discussed in [1], and feel it would be useful when loading 'dirty' data but there is a limit to how dirty it can be.
> 
> Attached a patch for this.
> 
> What do you think?

The patch no longer applies cleanly to HEAD. Could you update it?

I think the REJECT_LIMIT feature is useful. Allowing it to be set as either the absolute number of skipped rows or a percentage of the total input rows is a good idea.

However, if we support REJECT_LIMIT, I'm not sure if the ON_ERROR option is still necessary. REJECT_LIMIT seems to cover the same cases. For instance, REJECT_LIMIT=infinity can act like ON_ERROR=ignore, and REJECT_LIMIT=0 can act like ON_ERROR=stop.

Therefore, having both ON_ERROR and REJECT_LIMIT might be confusing.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION