Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\))
Subject: Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
From: Daniel Gustafsson <daniel@yesql.se>
In-Reply-To: <CALH1LgvQ5eRMi3zO7RSmod4ucru=9ekASpReahR7zDcCabANZw@mail.gmail.com>
Date: Mon, 6 Mar 2023 15:03:38 +0100
Cc: torikoshia <torikoshia@oss.nttdata.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
 Andres Freund <andres@anarazel.de>,
 Danil Anisimow <anisimow.d@gmail.com>,
 HukuToc@gmail.com,
 a.lepikhov@postgrespro.ru,
 tgl@sss.pgh.pa.us
Content-Transfer-Encoding: quoted-printable
Message-Id: <0DE0602F-CC12-40ED-B259-3AB91FB02C3B@yesql.se>
References: <CALH1Lgv5QeYSWYDgMBd=wvKum8egiaMR-_77VGsiHPt7E8dK9Q@mail.gmail.com>
 <CALH1LgvzpORM+EHVXRz4-Dv+_aHvm9m9Hu6eYM4PfppwKgXvFQ@mail.gmail.com>
 <686e46d6718f9535bc39fac7cad7408b@oss.nttdata.com>
 <CALH1Lgsr7Asz6r6jnRta7DVEjKwyz=b3XJhyCMoYv8Yy9y4cXg@mail.gmail.com>
 <769423519e16d18a1e7611c6931da58b@oss.nttdata.com>
 <CALH1LguQ5P0uZSYQFy4gJihvV11EkLx+gwxZ-+1DFq1Z6dJqbg@mail.gmail.com>
 <CALH1Lgug+mR9A1bBzR8hrp2oW4=jvR_N0An-i2X_j_y9C1fkiA@mail.gmail.com>
 <CALH1LguPAN=p5wHKwGcHrjE8Vj1x4o1z3Pk228YDif6LFzbeYQ@mail.gmail.com>
 <CABm2Ma6_=qf7kZPyOmiK1CUXeio7q08HNPYXrqkCwF3P_KQu7A@mail.gmail.com>
 <CALH1Lgv+L_HNN2NKo=WD8BbgLy7tEH7=OhNTJ2QWCU_CPMum0g@mail.gmail.com>
 <20230205233723.xw5z5zmeyc57ezq3@alap3.anarazel.de>
 <CALH1LgsbWZWS3qMxdGjg4n57DUHxqyBgTzvYSzfi-sP1N-ZORQ@mail.gmail.com>
 <2587470.1675660337@sss.pgh.pa.us>
 <483E1B59-4A80-450B-8685-7CC12762FD5A@anarazel.de>
 <2592557.1675663234@sss.pgh.pa.us>
 <39143e83571d59dfc04c75707fa0ea5a@oss.nttdata.com>
 <CALH1LgvQ5eRMi3zO7RSmod4ucru=9ekASpReahR7zDcCabANZw@mail.gmail.com>
To: Damir Belyalov <dam.bel07@gmail.com>
Archived-At: <https://www.postgresql.org/message-id/0DE0602F-CC12-40ED-B259-3AB91FB02C3B%40yesql.se>
Precedence: bulk

> On 28 Feb 2023, at 15:28, Damir Belyalov <dam.bel07@gmail.com> wrote:

> Tested patch on all cases: CIM_SINGLE, CIM_MULTI, CIM_MULTI_CONDITION. =
As expected it works.
> Also added a description to copy.sgml and made a review on patch.
>=20
> I added 'ignored_errors' integer parameter that should be output after =
the option is finished.
> All errors were added to the system logfile with full detailed =
context. Maybe it's better to log only error message.

FWIW, Greenplum has a similar construct (but which also logs the errors =
in the
db) where data type errors are skipped as long as the number of errors =
don't
exceed a reject limit.  If the reject limit is reached then the COPY =
fails:

	LOG ERRORS [ SEGMENT REJECT LIMIT <count> [ ROWS | PERCENT ]]

IIRC the gist of this was to catch then the user copies the wrong input =
data or
plain has a broken file.  Rather than finding out after copying n rows =
which
are likely to be garbage the process can be restarted.

This version of the patch has a compiler error in the error message:

copyfrom.c: In function =E2=80=98CopyFrom=E2=80=99:
copyfrom.c:1008:29: error: format =E2=80=98%ld=E2=80=99 expects argument =
of type =E2=80=98long int=E2=80=99, but argument 2 has type =E2=80=98uint6=
4=E2=80=99 {aka =E2=80=98long long unsigned int=E2=80=99} =
[-Werror=3Dformat=3D]
1008 | ereport(WARNING, errmsg("Errors: %ld", cstate->ignored_errors));
     |                          ^~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
     |                                              |
     |                                              uint64 {aka long =
long unsigned int}


On that note though, it seems to me that this error message leaves a bit =
to be
desired with regards to the level of detail.

--
Daniel Gustafsson