MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Wed, 24 Jul 2024 19:43:37 +0900
From: torikoshia <torikoshia@oss.nttdata.com>
To: Michael Paquier <michael@paquier.xyz>, sawada.mshk@gmail.com
Cc: Pgsql Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Add on_error and log_verbosity options to file_fdw
In-Reply-To: <Zp7x8Y-SpbkZtpE-@paquier.xyz>
References: <ab59dad10490ea3734cf022b16c24cfd@oss.nttdata.com>
 <841721f3ca9a69574a72eb141c83933c@oss.nttdata.com>
 <CAD21AoBij_2cy5T7ihk9ArRQzDcXVHoK13vzy34zAUAMLgrKkQ@mail.gmail.com>
 <Zp7x8Y-SpbkZtpE-@paquier.xyz>
User-Agent: Roundcube Webmail/1.4.11
Message-ID: <c22f047faf8aea56acecb26449b3f454@oss.nttdata.com>
Archived-At: <https://www.postgresql.org/message-id/c22f047faf8aea56acecb26449b3f454%40oss.nttdata.com>
Precedence: bulk

On 2024-07-23 08:57, Michael Paquier wrote:
> On Mon, Jul 22, 2024 at 03:07:46PM -0700, Masahiko Sawada wrote:
>> I'm slightly concerned that users might not want to see the NOTICE
>> message for every scan. Unlike COPY FROM, scanning a file via file_fdw
>> could be frequent.

Agreed.

> Yeah, I also have some concerns regarding the noise that this could
> produce if called on a foreign table on a regular basis.  The verbose
> mode is disabled by default so I don't see why we should not allow it
> if the relation owner wants to show it.
> 
> Perhaps we should first do a silence mode for log_verbosity to skip
> the NOTICE produced at the end of the COPY FROM summarizing the whole?

I like this idea.
If there are no objections, I'm going to make a patch for this.

> It would be confusing to have different defaults between COPY and
> file_fdw, but having the option to silence that completely is also
> appealing from the user point of view.

I'm not sure we should change the defaults.
If the default of file_fdw is silence mode, I am a little concerned that 
there may be cases where people think they have no errors, but in fact 
they have.

>>                            QUERY PLAN
>> ----------------------------------------------------------------
>>  Foreign Scan on public.test  (cost=0.00..1.10 rows=1 width=12)
>>    Output: a, b, c
>>    Foreign File: test.csv
>>    Foreign File Size: 12 b
>>    Skipped Rows: 10
> 
> Interesting idea linked to the idea of pushing the error state to
> something else than the logs.  Sounds like a separate feature.

+1


-- 
Regards,

--
Atsushi Torikoshi
NTT DATA Group Corporation