MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Mon, 18 Dec 2023 14:09:03 +0900
From: torikoshia <torikoshia@oss.nttdata.com>
To: jian he <jian.universality@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, Alena Rybakina
 <lena.ribackina@yandex.ru>, Damir Belyalov <dam.bel07@gmail.com>,
 zhihuifan1213@163.com, Tom Lane <tgl@sss.pgh.pa.us>, Daniel Gustafsson
 <daniel@yesql.se>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Andres
 Freund <andres@anarazel.de>, anisimow.d@gmail.com, HukuToc@gmail.com,
 pgsql-hackers@lists.postgresql.org, Andrei Lepikhov
 <a.lepikhov@postgrespro.ru>
Subject: Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
In-Reply-To: <CACJufxHi53OpGYPAe6SdCb4m=-+H8L+7LDbUWvTiJp=V4YYEqA@mail.gmail.com>
References: <483E1B59-4A80-450B-8685-7CC12762FD5A@anarazel.de>
 <43399b33fe573f0548410893f41f84b6@oss.nttdata.com>
 <CALH1LgseNT7mxpGzQsQwOJz_UwYBo6L-4vR9tmJqb-WNPkt6Bg@mail.gmail.com>
 <8de9d5e1-2f81-ee2e-ece4-b1a6ede1f285@yandex.ru>
 <b74c1c3e3cc1c5d7ad5fb527b27d2b60@oss.nttdata.com>
 <c0fb57b82b150953f26a5c7e340412e8@oss.nttdata.com>
 <CALH1Lgtg39crESw644KDH+ejBWuW3Vospe67F-ZWXNeBMEE=PA@mail.gmail.com>
 <58fa3b63e88ffd0a09273e38aec0661e@oss.nttdat!>
 <B082C6BE-D2F2-4DD4-8649-E81E24A6840E@yesql.se>
 <752672.1699474336@sss.pgh.pa.us> <878r77mtlt.fsf@163.com>
 <CALH1LgueaSXWJ=eZ-0-LYCm8tF45SoOM0QFc55C-xpbGfNkFyg@mail.gmail.com>
 <760ee97a-e336-4d7b-9477-9d9e99370097@postgrespro.ru>
 <CACJufxGBb-WE7R1fM4kJxvmsgsyKL_4vce3s_KxLbo_ZEnFnWw@mail.gmail.com>
 <c7c80c6d-4c02-4629-8a27-5c635d27ad10@yandex.ru>
 <CACJufxH3KGOGhnS-J2bYnA-mpABg4HG9Xddfj9zBBgoK+iGSsQ@mail.gmail.com>
 <23a89ba0-b5d5-419c-b798-cd2c59490148@yandex.ru>
 <CACJufxHjfcetqip_3RLyvMQSB-SkvxwXkCP3VUN8QsKMKfjoUA@mail.gmail.com>
 <f03bc3de-9769-4106-80cf-158c5f184a1e@yandex.ru>
 <CACJufxEFXzxjD9oOq3LoVQAy0KH0TJsDS3UnTtecxx-4J0+2NA@mail.gmail.com>
 <CAD21AoCeEOBN49fu43e6tBTynnswugA3oZ5AZvLeyDCpxpCXPg@mail.gmail.com>
 <CACJufxHi53OpGYPAe6SdCb4m=-+H8L+7LDbUWvTiJp=V4YYEqA@mail.gmail.com>
User-Agent: Roundcube Webmail/1.4.11
Message-ID: <1264ee0c8b68d57cb4035a68fdb0eba2@oss.nttdata.com>
Archived-At: <https://www.postgresql.org/message-id/1264ee0c8b68d57cb4035a68fdb0eba2%40oss.nttdata.com>
Precedence: bulk

Hi,

> save the error metadata to  system catalogs would be more expensive,
> please see below explanation.
> I have no knowledge of publications.
> but i feel there is a feature request: publication FOR ALL TABLES
> exclude regex_pattern.
> Anyway, that would be another topic.

I think saving error metadata to system catalog is not a good idea, too.
And I believe Sawada-san just pointed out missing features and did not 
suggested that we use system catalog.

> I don't think "specify the maximum number  of errors to tolerate
> before raising an ERROR." is very useful....

That may be so.
I imagine it's useful in some use case since some loading tools have 
such options.
Anyway I agree it's not necessary for initial patch as mentioned in [1].

> I suppose we can specify an ERRORFILE directory. similar
> implementation [2], demo in [3]
> it will generate 2 files, one file shows the malform line content as
> is, another file shows the error info.

That may be a good option when considering "(2) logging errors to 
somewhere".

What do you think about the proposal to develop these features in 
incrementally?

On 2023-12-15 05:48, Masahiko Sawada wrote:
> So I'm thinking we may be able to implement this
> feature incrementally. The first step would be something like an
> option to ignore all errors or an option to specify the maximum number
> of errors to tolerate before raising an ERROR. The second step would
> be to support logging destinations such as server logs and tables.

[1] 
https://www.postgresql.org/message-id/752672.1699474336%40sss.pgh.pa.us

-- 
Regards,

--
Atsushi Torikoshi
NTT DATA Group Corporation