Message-ID: <54bea7dd-94b1-41f1-a4e9-c5c53cc82ce7@oss.nttdata.com>
Date: Sat, 20 Jul 2024 02:08:39 +0900
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Add new COPY option REJECT_LIMIT
From: Fujii Masao <masao.fujii@oss.nttdata.com>
To: torikoshia <torikoshia@oss.nttdata.com>
Cc: pgsql-hackers@lists.postgresql.org
References: <63f99327aa6b404cc951217fa3e61fe4@oss.nttdata.com>
 <fab7018f-6e00-4be4-bd1e-4a013096f6c9@oss.nttdata.com>
 <de0551d0f9d9e9072e324e51ed5c426d@oss.nttdata.com>
 <6e3dcdc6-8e18-4716-a23b-3bee316aeaa7@oss.nttdata.com>
 <979c6d306343f3d665d14eab2cd1b7bc@oss.nttdata.com>
 <920a29a36032befbeb355cbdfbbe0f63@oss.nttdata.com>
 <a873643c-e7d1-456c-b16b-6162ab698036@oss.nttdata.com>
Content-Language: en-US
In-Reply-To: <a873643c-e7d1-456c-b16b-6162ab698036@oss.nttdata.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/54bea7dd-94b1-41f1-a4e9-c5c53cc82ce7%40oss.nttdata.com>
Precedence: bulk


On 2024/07/19 22:03, Fujii Masao wrote:
> 
> 
> On 2024/07/17 22:21, torikoshia wrote:
>> On 2024-07-03 02:07, Fujii Masao wrote:
>>> However, if we support REJECT_LIMIT, I'm not sure if the ON_ERROR option is still necessary.
>>
>> I remembered another reason for the necessity of ON_ERROR.
>>
>> ON_ERROR defines how to behave when encountering an error and it just accepts 'ignore' and 'stop' currently, but is expected to support other options such as saving details of errors to a table[1].
> 
> Wouldn't it be better to separate the option specifying where
> error details are output from the ON_ERROR option
> (which determines behavior when encountering errors)?
> "table" seems valid for both ON_ERROR=ignore and ON_ERROR=stop.

I still find it odd to accept "table" as a value for ON_ERROR. However,
"set_to_null" or "replace-column" proposed in [1] seem valid for
ON_ERROR. So, I'm okay with keeping the ON_ERROR option.


> On my second thought, whatever value ON_ERROR is specified(e.g. ignore, stop, table), it seems fine to use REJECT_LIMIT.
> I feel REJECT_LIMIT has both "ignore" and "stop" characteristics, meaning it ignores errors until it reaches REJECT_LIMIT and stops when it exceeds the REJECT_LIMIT.

ON_ERROR specifies how to handle errors, and "stop" means to fail
the command. So, if ON_ERROR=stop, REJECT_LIMIT should have no effect,
and the command should fail immediately upon encountering an error.

As in your original proposal, I now think REJECT_LIMIT should only
apply when ON_ERROR=ignore. The command would ignore errors and
continue processing, but if the number of errors exceeds REJECT_LIMIT,
the command should fail. Thought?

BTW if "set_to_null" is supported someday, REJECT_LIMIT can also
apply. The command would cinsert NULL into the target table upon
encountering errors and continue, but fail if the number of errors
exceed REJECT_LIMIT.

Regards,

[1] https://www.postgresql.org/message-id/flat/CAKFQuwawy1e6YR4S=j+y7pXqg_Dw1WBVrgvf=BP3d1_aSfe_+Q@mail.gmail.com

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION