Date: Tue, 12 Nov 2024 14:03:50 +0900
From: Yugo Nagata <nagata@sraoss.co.jp>
To: Kirill Reshke <reshkekirill@gmail.com>
Cc: torikoshia <torikoshia@oss.nttdata.com>, Fujii Masao
 <masao.fujii@oss.nttdata.com>, jian he <jian.universality@gmail.com>, Jim
 Jones <jim.jones@uni-muenster.de>, "David G. Johnston"
 <david.g.johnston@gmail.com>, PostgreSQL Hackers
 <pgsql-hackers@lists.postgresql.org>
Subject: Re: Change COPY ... ON_ERROR ignore to ON_ERROR ignore_row
Message-Id: <20241112140350.e6599cece83d59387d28b5c0@sraoss.co.jp>
In-Reply-To: <CALdSSPhKs69u20a5CfYaYRegmfoz13+cjBngkC=RkTMvqtbH2g@mail.gmail.com>
References: <CAKFQuwawy1e6YR4S=j+y7pXqg_Dw1WBVrgvf=BP3d1_aSfe_+Q@mail.gmail.com>
	<CACJufxFT9j8o5kEC8dPCQqLomWjeJm9V9m8eZjj2Gvc_F5ha=g@mail.gmail.com>
	<76da9fcc-93c5-4053-872e-12932a95356d@uni-muenster.de>
	<CAKFQuwaYNw8U-9JkFdyOX4i4Y3J1sp6+dk-sh8YmZGCq8gMeVQ@mail.gmail.com>
	<6eac5b45-7f45-4c7a-aae1-e90db8be2e08@uni-muenster.de>
	<CACJufxFFdtPKk4B5rSVNEk6yCH2Amvi_8w3Gaz5wg9M_t9c5Rw@mail.gmail.com>
	<3d6b5885-16a1-475d-b56f-41701c48d9d4@uni-muenster.de>
	<CACJufxEgiysa2SMJPGp0aN476Ojm636MfJK88DZC7TVYsXYBBQ@mail.gmail.com>
	<CALdSSPhgjCbyb=ZRgr4LaCFJV2-F9_CxMeX6poHuGCt_f9GYAw@mail.gmail.com>
	<63595e8f-a245-4335-aa22-7e449a70e210@oss.nttdata.com>
	<CALdSSPi1JE9xc31W6DPAdk-bQHeo3HNAYB-10Biruu-w4GJN0Q@mail.gmail.com>
	<07587c36-18b3-4ccb-b5fb-579bcb04ed37@oss.nttdata.com>
	<CALdSSPjYw5g7_sc++bRcxOnC7jW6O2qiSkgdKRUYFXZZv3-Ktw@mail.gmail.com>
	<501dd655ddb04693c15baeb6485bc601@oss.nttdata.com>
	<CALdSSPhKs69u20a5CfYaYRegmfoz13+cjBngkC=RkTMvqtbH2g@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?sb5NQyoLqU+FVQAfMIcEk92q1jsztUtChlhGcuelpCFoDdNo7WUL/1h0k74K?=
 =?us-ascii?Q?rEQcPOQ0u1Swpd/C9pZivRBwrJNTYL1o53YZwLIdNjTu7davLS69DWyHOt9v?=
 =?us-ascii?Q?zTP0SrgjDHvLA/QT/Epbw8ezDZi7JMHJDsi6Fp93uM8Y3+m3MhFr0/kbRvYq?=
 =?us-ascii?Q?g1mrCpmlHvazmWa1xxwIHdDKrIwzfUYaznz3Z3g3ZDT0r2dab0QLHaIWak1Q?=
 =?us-ascii?Q?JOq3oUvBx3lY7WjFD0sYN6QiIaepZhMbla9ucW+NBjs62FqxjactJF1wY2Vx?=
 =?us-ascii?Q?tAjC6CMwRsu+ra/RubOc+gzMJv4xoY6qxzhx5u9aFB1yBMWNLXt7wILZ7wp0?=
 =?us-ascii?Q?9SSLp1uW9Ub80aIAId7lTijbEu7OL9Sy2/Y23VCo2oiM4/32+KLWWhvDLjUA?=
 =?us-ascii?Q?eTy0ogRrL/ohTwkwHel3ev9gP9Ro9MuVWNUiWCoYPi0BIklwf9ajGMbX9PHC?=
 =?us-ascii?Q?PZDzwLTTreE6t89H3A9RRa6bTttx/A+XlYLuvONYj8ApFKebIjjXijnu8K1O?=
 =?us-ascii?Q?RROR3BJyGgqtNFewdOhL7WfOLXMRQcTd3T0VblsT5mgulCuhPVkbl7djzVt7?=
 =?us-ascii?Q?xRY1ScowOZweLv4MdxPA2aHPAQCsre+cfV0p/74VDw/ob7ircMZKC5j4HvHP?=
 =?us-ascii?Q?0c+HMtUK+SEozQAHfDZFZQpRzcKn5rAKiUpWthi5DWy3ickzYNGzZP+FPqs8?=
 =?us-ascii?Q?vYF46UjH1Cd9MdoNkyyPcZAPVK495UvgNpXJ1ZuNzaty5hFVkkKsGEhG9aN3?=
 =?us-ascii?Q?diRmpIDZ6qyRDuuJzj9GUdUeN2i+FW7KJUcHcyCDnZr4cp+JaeSDmlNzTx+F?=
 =?us-ascii?Q?I+BSuGMN6CgL3rwolAjYjZov9rlsDPK7M5UQ7mZoB75rxJfKDIWayn94n7lZ?=
 =?us-ascii?Q?BkdtSIp3DUAyHAU4WFDSLW3vgnG12diaC9QAJvAsZAPAsbJc+3GxSdOaHtcD?=
 =?us-ascii?Q?hzQJwfnbx8NyT/+5HRZXub8ZqliR4WXzm82hUXQwV86uQ2xaX/E+K8fUnfhM?=
 =?us-ascii?Q?I+a0TNiLVb2rcWVj4Srla9owcZcOsPjJF3vtyVz+mQ48bZ0rLfpxQ54eiith?=
 =?us-ascii?Q?y2Eo16WZOI7tJk0+DVRAPwv415/hYae7SVFD8lHG/cmvXE5oiDrSz4bVP0sX?=
 =?us-ascii?Q?qhJifYnuPylCSe4Ae6RLNNBs+FFebXDTgWnbNpM0bF7dk7VT7NWy5OEyIiBp?=
 =?us-ascii?Q?XeXVgl1hXULp2Z2IsFN86Iovqz4zKuNSRaCgST8p+0E20P603CYCo9tgAUm8?=
 =?us-ascii?Q?MPUfNaB8UVvuujHemsv0AMx0KbJ6AhuKmrMRSiZxumXsD/k9dDvirbXKhRy7?=
 =?us-ascii?Q?eAflJdGkVONV+ILwSXd2wxyNycz9Afy9A8GzoHSxrIQdAoIL70NORs0kPzyU?=
 =?us-ascii?Q?dmTitqq6YIAosgugfTXq/Ta+kL06hIChD0fJPmQQO+D71zLspV22jegrTiY8?=
 =?us-ascii?Q?w3BfP2F7CYlb2Y4ITADcEt7YN7uIKKAFjE2f59Tu6EAMb0uvb1RhY6BhGoMZ?=
 =?us-ascii?Q?Rwu0Xf3gnMvKU1VR8/VgQqZYtDA++hr204fPKBbgTbzJLWN3qgntYMlRXcqc?=
 =?us-ascii?Q?q/xgtO8/c85na34b2EZT7w82MjCyZXBDBrzMaiPVRKkSRJRntejtWN4oAihB?=
 =?us-ascii?Q?ijM2T4DH3x6+PEe7aDHLVagrXW73eXALJd237mK09IIi?=
X-OriginatorOrg: sraoss.co.jp
X-MS-Exchange-CrossTenant-Network-Message-Id: 9031e5d3-9d90-4407-fc8c-08dd02d7663c
X-MS-Exchange-CrossTenant-AuthSource: TYBP286MB0111.JPNP286.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Nov 2024 05:03:51.4199
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 75db6c32-352e-4063-ae79-cafb8623f6db
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4LodNjJHKQcCnchcdONVMDrb0SngTx8nYizKSKy/NoG9GDfeXDThNSY5Fo298BN7ELYUKjE7tuKWtcAQONVC1A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYWP286MB2682
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: <https://www.postgresql.org/message-id/20241112140350.e6599cece83d59387d28b5c0%40sraoss.co.jp>
Precedence: bulk

On Tue, 12 Nov 2024 01:27:53 +0500
Kirill Reshke <reshkekirill@gmail.com> wrote:

> On Mon, 11 Nov 2024 at 16:11, torikoshia <torikoshia@oss.nttdata.com> wrote:
> >
> > On 2024-11-09 21:55, Kirill Reshke wrote:
> >
> > Thanks for working on this!
> 
> Thanks for reviewing the v7 patch series!
> 
> > > On Thu, 7 Nov 2024 at 23:00, Fujii Masao <masao.fujii@oss.nttdata.com>
> > > wrote:
> > >>
> > >>
> > >>
> > >> On 2024/10/26 6:03, Kirill Reshke wrote:
> > >> > when the REJECT LIMIT is set to some non-zero number and the number of
> > >> > row NULL replacements exceeds the limit, is it OK to fail. Because
> > >> > there WAS errors, and we should not tolerate more than $limit errors .
> > >> > I do find this behavior to be consistent.
> > >>
> > >> +1
> > >>
> > >>
> > >> > But what if we don't set a REJECT LIMIT, it is sane to do all
> > >> > replacements, as if REJECT LIMIT is inf.
> > >>
> > >> +1
> > >
> > > After thinking for a while, I'm now more opposed to this approach. I
> > > think we should count rows with erroneous data as errors only if
> > > null substitution for these rows failed, not the total number of rows
> > > which were modified.
> > > Then, to respect the REJECT LIMIT option, we compare this number with
> > > the limit. This is actually simpler approach IMHO. What do You think?
> >
> > IMHO I prefer the previous interpretation.
> > I'm not sure this is what people expect, but I assume that REJECT_LIMIT
> > is used to specify how many malformed rows are acceptable in the
> > "original" data source.

I also prefer the previous version.
 
> I do like the first version of interpretation, but I have a struggle
> with it. According to this interpretation, we will fail COPY command
> if the number
> of malformed data rows exceeds the limit, not the number of rejected
> rows (some percentage of malformed rows are accepted with null
> substitution)
> So, a proper name for the limit will be MALFORMED_LIMIT, or something.
> However, we are unable to change the name since the REJECT_LIMIT
> option has already been committed.
> I guess I'll just have to put up with this contradiction. I will send
> an updated patch shortly...

I think we can rename the REJECT_LIMIT option because it is not yet released.

The documentation says that REJECT_LIMIT "Specifies the maximum number of errors",
and there are no wording "reject" in the description, so I wonder it is unclear
what means in "REJECT" in REJECT_LIMIT. It may be proper to use ERROR_LIMIT
since it is supposed to be used with ON_ERROR. 

Alternatively, if we emphasize that errors are handled other than terminating
the command,perhaps MALFORMED_LIMIT as proposed above or TOLERANCE_LIMIT may be
good, for example.

Regards,
Yugo Nagata

-- 
Yugo Nagata <nagata@sraoss.co.jp>