MIME-Version: 1.0
References: <80c44b4d-1185-478b-939c-62e0d4377106@gmail.com>
In-Reply-To: <80c44b4d-1185-478b-939c-62e0d4377106@gmail.com>
From: Ron Johnson <ronljohnsonjr@gmail.com>
Date: Thu, 25 Sep 2025 16:08:34 -0400
Message-ID: <CANzqJaABFc15uQp=ORSw066izoe8+v9PpE4HuCUu7wtKryjiuw@mail.gmail.com>
Subject: Re: Additional options for COPY from
To: "pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="0000000000007ed6cf063fa5bcba"
Archived-At: <https://www.postgresql.org/message-id/CANzqJaABFc15uQp%3DORSw066izoe8%2Bv9PpE4HuCUu7wtKryjiuw%40mail.gmail.com>
Precedence: bulk

--0000000000007ed6cf063fa5bcba
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Sep 25, 2025 at 3:49=E2=80=AFPM Bryan Sayer <brysayer@gmail.com> wr=
ote:

> Hi,
>
> I'm not sure if this is the best list to ask this (and I am very new to
> PostgreSQL) but I think more options are needed in the COPY from command,
> in order to better deal with exceptions. By exceptions I mean data not
> consistent with the format, empty rows, extra delimiters at the end of
> rows, etc.
>

PG 17 COPY has ON_ERROR.


> Is there someone or a party that deals with the details of the COPY
> command that I could discuss this with?
>
> Just for context, I am dealing with delimited data of millions of rows an=
d
> perhaps 60 columns or so that I wish to read into tables in a Postgres 17
> database under Windows 11, with everything on local drives.
>
COPY is pretty simplistic.  3rd party CSV loaders like pgloader and
pg_bulkload exist for that reason. And there's always Python & Perl for
when your data is really dirty.

--=20
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

--0000000000007ed6cf063fa5bcba
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Thu, Sep 25, 2025 at 3:49=E2=80=AFPM B=
ryan Sayer &lt;<a href=3D"mailto:brysayer@gmail.com">brysayer@gmail.com</a>=
&gt; wrote:</div><div class=3D"gmail_quote gmail_quote_container"><blockquo=
te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px =
solid rgb(204,204,204);padding-left:1ex"><u></u>

 =20

   =20
 =20
  <div>
    <p>Hi,</p>
    <p>I&#39;m not sure if this is the best list to ask this (and I am very
      new to PostgreSQL) but I think more options are needed in the COPY
      from command, in order to better deal with exceptions. By
      exceptions I mean data not consistent with the format, empty rows,
      extra delimiters at the end of rows, etc.</p></div></blockquote><div>=
<br></div><div>PG 17 COPY has ON_ERROR.</div><div>=C2=A0</div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex"><div>
    <p>Is there someone or a party that deals with the details of the
      COPY command that I could discuss this with?</p>
    <p>Just for context, I am dealing with delimited data of millions of
      rows and perhaps 60 columns or so that I wish to read into tables
      in a Postgres 17 database under Windows 11, with everything on
      local drives.</p></div></blockquote><div>COPY is pretty simplistic.=
=C2=A0 3rd party CSV loaders like pgloader and pg_bulkload exist for that r=
eason. And there&#39;s always Python &amp; Perl for when your data is reall=
y dirty.</div><div><br></div></div><span class=3D"gmail_signature_prefix">-=
- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"ltr">De=
ath to &lt;Redacted&gt;, and butter sauce.<div>Don&#39;t boil me, I&#39;m s=
till alive.<br><div><div>&lt;Redacted&gt; lobster!</div></div></div></div><=
/div></div>

--0000000000007ed6cf063fa5bcba--