Feedback-ID: i0fe9450f:Fastmail
Date: Tue, 23 Jul 2024 08:57:37 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: torikoshia <torikoshia@oss.nttdata.com>,
	Pgsql Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Add on_error and log_verbosity options to file_fdw
Message-ID: <Zp7x8Y-SpbkZtpE-@paquier.xyz>
References: <ab59dad10490ea3734cf022b16c24cfd@oss.nttdata.com>
 <841721f3ca9a69574a72eb141c83933c@oss.nttdata.com>
 <CAD21AoBij_2cy5T7ihk9ArRQzDcXVHoK13vzy34zAUAMLgrKkQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="5156sOm2JBj5jbFq"
Content-Disposition: inline
In-Reply-To: <CAD21AoBij_2cy5T7ihk9ArRQzDcXVHoK13vzy34zAUAMLgrKkQ@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/Zp7x8Y-SpbkZtpE-%40paquier.xyz>
Precedence: bulk


--5156sOm2JBj5jbFq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Mon, Jul 22, 2024 at 03:07:46PM -0700, Masahiko Sawada wrote:
> I'm slightly concerned that users might not want to see the NOTICE
> message for every scan. Unlike COPY FROM, scanning a file via file_fdw
> could be frequent. An alternative idea of place to write the
> information of the number of malformed rows would be the EXPLAIN
> command as follow:

Yeah, I also have some concerns regarding the noise that this could
produce if called on a foreign table on a regular basis.  The verbose
mode is disabled by default so I don't see why we should not allow it
if the relation owner wants to show it.

Perhaps we should first do a silence mode for log_verbosity to skip
the NOTICE produced at the end of the COPY FROM summarizing the whole?
It would be confusing to have different defaults between COPY and
file_fdw, but having the option to silence that completely is also
appealing from the user point of view.

>                            QUERY PLAN
> ----------------------------------------------------------------
>  Foreign Scan on public.test  (cost=0.00..1.10 rows=1 width=12)
>    Output: a, b, c
>    Foreign File: test.csv
>    Foreign File Size: 12 b
>    Skipped Rows: 10

Interesting idea linked to the idea of pushing the error state to
something else than the logs.  Sounds like a separate feature.

> IIUC we have to execute ALTER FOREIGN TABLE to change the
> log_verbosity value and which requires to be the owner. Which seems
> not to be user-friendly.

I am not sure about allowing scans to force an option to be a
different thing at runtime vs what's been defined in the relation
itself with CREATE/ALTER.
--
Michael

--5156sOm2JBj5jbFq
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmae8fEACgkQnvQgOdby
QH1/7w//blne8JEspQjX0oOqhd/Ovm/NqKOE7538LyFW52klwMqVfpXHvuHOn1Fk
qDB/5n66gRfuu5e42VG28GOktowedYbn30GEg7f3p71Pak+8aWYpVOkhECTf/qOD
ZowIv3y8cAFtMoO+6nHh6WQ1GxeczEMzWGpD/EpT+b39WGl707Xgyc5guXDxqrdw
IbSYA7/0kTbiGhWl27xBepBDcFBczJjQTN1R5VyWo5ivbEVyMh9ktzJ0CK8wh4HP
NjYBp1d6D/YerL5l2UPweYJ6/gPsAzxValgyyF0/qCHxoVQa06K4cmrUyPCp5uGE
+pblrwt/Co6EcQuyrsMKskkSH9RC6xgmQeSL9B1zkbc48vvkKGsUY3XxcQLyo/um
CTkCmTzTE9oA7VNyIqM6DaT4b1l43MQMiRjc7Kdh7Qdob9PxILWOkvPUplZTstCN
ACg2dNrpO1+lVJVqd6RexbXwUmyFOtposcS7htuRzOnmts7Xm7inXxxV5oB6RYTi
INKIUM6U1IlsQMyrukuQSFsqw+0OrtL3EFqwhnsx4a/JSARKLwgrVAiE+M0vtL5e
9vvn+gslQU8hjR9DzG5MMw5NNT7evSHjzSbyyeONQL+TbTCe3ymY1qKBkyhKelTc
OR2Ibsa3hHq87H+M8imYgrxbE4z8V2jnRSALgc+0FkptqeNp9lI=
=KirA
-----END PGP SIGNATURE-----

--5156sOm2JBj5jbFq--