From: Nathan Bossart <nathandbossart@gmail.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
Cc: Sanaba, Bilva <bilvas@amazon.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: Adding Support for Copy callback functionality on COPY TO api
Date: Mon, 10 Oct 2022 09:38:59 -0700
Message-ID: <20221010163859.GA993859@nathanxps13> (raw)
In-Reply-To: <CALj2ACXnEd6t1DQkaW7=QLyrvsYtd=+OeYMfxTwM+ZmV6EUC_A@mail.gmail.com>
References: <253C21D1-FCEB-41D9-A2AF-E6517015B7D7@amazon.com>
<CAE-ML+_ebFzUnsgj-4MLshLGFfT05Lo15KK6rpcUt8-ZZKL1XQ@mail.gmail.com>
<20200930074151.GR1996@paquier.xyz>
<20220802234919.GA841243@nathanxps13>
<Yz/L+3aC6CJ/hZ65@paquier.xyz>
<20221007214824.GA814385@nathanxps13>
<Y0EGighrcVgNnp30@paquier.xyz>
<20221008173741.GB882834@nathanxps13>
<20221008211404.GA893744@nathanxps13>
<CALj2ACXnEd6t1DQkaW7=QLyrvsYtd=+OeYMfxTwM+ZmV6EUC_A@mail.gmail.com>
On Mon, Oct 10, 2022 at 12:41:40PM +0530, Bharath Rupireddy wrote:
> IIUC, COPY TO callback helps move a table's data out of postgres
> server. Just wondering, how is it different from existing solutions
> like COPY TO ... PROGRAM/FILE, logical replication, pg_dump etc. that
> can move a table's data out? I understandb that the COPY FROM callback
> was needed for logical replication 7c4f52409. Mentioning a concrete
> use-case helps here.
This new callback allows the use of COPY TO's machinery in extensions. A
couple of generic use-cases are listed upthread [0], and one concrete
use-case is the aws_s3 extension [1].
> I'm not quite sure if we need a separate module to just tell how to
> use this new callback. I strongly feel that it's not necessary. It
> unnecessarily creates extra code (actual code is 25 LOC with v1 patch
> but 150 LOC with v5 patch) and can cause maintenance burden. These
> callback APIs are simple enough to understand for those who know
> BeginCopyTo() or BeginCopyFrom() and especially for those who know how
> to write extensions. These are not APIs that an end-user uses. The
> best would be to document both COPY FROM and COPY TO callbacks,
> perhaps with a pseudo code specifying just the essence [1], and their
> possible usages somewhere here
> https://www.postgresql.org/docs/devel/sql-copy.html.
>
> The order of below NOTICE messages isn't guaranteed and it can change
> depending on platforms. Previously, we've had to suppress such
> messages in the test output 6adc5376d.
I really doubt that this small test case is going to cause anything
approaching undue maintenance burden. I think it's important to ensure
this functionality continues to work as expected long into the future.
[0] https://postgr.es/m/253C21D1-FCEB-41D9-A2AF-E6517015B7D7%40amazon.com
[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postgresql-s3-export.html#aws_s3.export_query...
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nathandbossart@gmail.com, bharath.rupireddyforpostgres@gmail.com, michael@paquier.xyz, soumyadeep2007@gmail.com, bilvas@amazon.com
Subject: Re: Adding Support for Copy callback functionality on COPY TO api
In-Reply-To: <20221010163859.GA993859@nathanxps13>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox