Date: Mon, 10 Oct 2022 09:38:59 -0700
From: Nathan Bossart <nathandbossart@gmail.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>,
	Soumyadeep Chakraborty <soumyadeep2007@gmail.com>,
	"Sanaba, Bilva" <bilvas@amazon.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: Adding Support for Copy callback functionality on COPY TO api
Message-ID: <20221010163859.GA993859@nathanxps13>
References: <253C21D1-FCEB-41D9-A2AF-E6517015B7D7@amazon.com>
 <CAE-ML+_ebFzUnsgj-4MLshLGFfT05Lo15KK6rpcUt8-ZZKL1XQ@mail.gmail.com>
 <20200930074151.GR1996@paquier.xyz>
 <20220802234919.GA841243@nathanxps13>
 <Yz/L+3aC6CJ/hZ65@paquier.xyz>
 <20221007214824.GA814385@nathanxps13>
 <Y0EGighrcVgNnp30@paquier.xyz>
 <20221008173741.GB882834@nathanxps13>
 <20221008211404.GA893744@nathanxps13>
 <CALj2ACXnEd6t1DQkaW7=QLyrvsYtd=+OeYMfxTwM+ZmV6EUC_A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CALj2ACXnEd6t1DQkaW7=QLyrvsYtd=+OeYMfxTwM+ZmV6EUC_A@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/20221010163859.GA993859%40nathanxps13>
Precedence: bulk

On Mon, Oct 10, 2022 at 12:41:40PM +0530, Bharath Rupireddy wrote:
> IIUC, COPY TO callback helps move a table's data out of postgres
> server. Just wondering, how is it different from existing solutions
> like COPY TO ... PROGRAM/FILE, logical replication, pg_dump etc. that
> can move a table's data out? I understandb that the COPY FROM callback
> was needed for logical replication 7c4f52409. Mentioning a concrete
> use-case helps here.

This new callback allows the use of COPY TO's machinery in extensions.  A
couple of generic use-cases are listed upthread [0], and one concrete
use-case is the aws_s3 extension [1].

> I'm not quite sure if we need a separate module to just tell how to
> use this new callback. I strongly feel that it's not necessary. It
> unnecessarily creates extra code (actual code is 25 LOC with v1 patch
> but 150 LOC with v5 patch) and can cause maintenance burden. These
> callback APIs are simple enough to understand for those who know
> BeginCopyTo() or BeginCopyFrom() and especially for those who know how
> to write extensions. These are not APIs that an end-user uses. The
> best would be to document both COPY FROM and COPY TO callbacks,
> perhaps with a pseudo code specifying just the essence [1], and their
> possible usages somewhere here
> https://www.postgresql.org/docs/devel/sql-copy.html.
> 
> The order of below NOTICE messages isn't guaranteed and it can change
> depending on platforms. Previously, we've had to suppress such
> messages in the test output 6adc5376d.

I really doubt that this small test case is going to cause anything
approaching undue maintenance burden.  I think it's important to ensure
this functionality continues to work as expected long into the future.

[0] https://postgr.es/m/253C21D1-FCEB-41D9-A2AF-E6517015B7D7%40amazon.com
[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postgresql-s3-export.html#aws_s3.export_query_to_s3

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com