pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedpostgres_fdw: Use COPY to speed up batch inserts
37+ messages / 9 participants
[nested] [flat]
Commit outlook (model estimate bayes-2.0, computed 2026-11-02; about the patch, not a person):
Unhealthy, not receiving community buy-in.
The activity on this thread indicates a low probability (7%) of commit in PG20-3 (closes 2026-11-30), and medium (32%) for PG20 (feature freeze 2027-04-08).
Seek out Tom Lane or Peter Eisentraut or Etsuro Fujita or Bruce Momjian or Michael Paquier (committers): they have history with this code.
What the estimate rests on: prior cfs 6.0 (lowers it), file commits since 2.197 (raises it), final cf 0.0 (lowers it), author thread rate -0.359 (raises it)
* postgres_fdw: Use COPY to speed up batch inserts
@ 2025-10-15 15:02 Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-15 15:02 UTC (permalink / raw)
To: pgsql-hackers
Hi all,
Currently on postgres_fdw we use prepared statements to insert batches
into foreign tables. Although this works fine for the most use cases the
COPY command can also be used in some scenarios to speed up large batch
inserts.
The attached patch implements this idea of using the COPY command for
batch inserts on postgres_fdw foreign tables. I've performed some
benchmarks using pgbench and the results seem good to consider this.
I've performed the benchmark using different batch_size values to see
when this optimization could be useful. The following results are the
best tps of 3 runs.
Command: pgbench -n -c 10 -j 10 -t 100 -f bench.sql postgres
batch_size: 10
master tps: 76.360406
patch tps: 68.917109
batch_size: 100
master tps: 123.427731
patch tps: 243.737055
batch_size: 1000
master tps: 132.500506
patch tps: 239.295132
It seems that using a batch_size greater than 100 we can have a
considerable speed up for batch inserts.
The attached patch uses the COPY command whenever we have a *numSlots >
1 but the tests show that maybe we should have a GUC to enable this?
I also think that we can have a better patch by removing the duplicated
code introduced on this first version, specially on the clean up phase,
but I tried to keep things more simple on this initial phase to keep the
review more easier and also just to test the idea.
Lastly, I don't know if we should change the EXPLAIN(ANALYZE, VERBOSE)
output for batch inserts that use the COPY to mention that we are
sending the COPY command to the remote server. I guess so?
(this proposal is based on a patch idea written by Tomas Vondra in one
of his blogs posts)
--
Matheus Alcantara
Attachments:
v1-0001-postgres_fdw-Use-COPY-to-speed-up-batch-inserts.patch (text/plain, 5.5K) download
From d4041814ba377475a1fa36b6972d5cf5f989a38f Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Fri, 10 Oct 2025 16:07:08 -0300
Subject: [PATCH v1] postgres_fdw: Use COPY to speed up batch inserts
---
contrib/postgres_fdw/deparse.c | 32 +++++++++
contrib/postgres_fdw/postgres_fdw.c | 108 ++++++++++++++++++++++++++++
contrib/postgres_fdw/postgres_fdw.h | 1 +
3 files changed, 141 insertions(+)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index e5b5e1a5f51..cd80bb7306a 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2238,6 +2238,38 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+buildCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ ListCell *lc;
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach(lc, target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN (FORMAT TEXT, DELIMITER ',')");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 456b267f70b..985c9bc5be7 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -4066,6 +4066,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ ListCell *lc;
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+
+ foreach(lc, fmstate->target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ",");
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[attnum - 1],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ }
+
+ appendStringInfoChar(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,6 +4141,70 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /*
+ * Use COPY command for batch insert if the original query don't include a
+ * RETURNING clause
+ */
+ if (operation == CMD_INSERT && *numSlots > 1 && !fmstate->has_returning)
+ {
+ int i;
+ StringInfoData copy_data;
+
+ /* Build COPY command */
+ initStringInfo(&sql);
+ buildCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, sql.data))
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, sql.data);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ /*
+ * XXX(matheus): Should we have a COPYBUFSIZ limit to send large
+ * data in batches instead of grow the buffer too much?
+ */
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+ }
+
+ /* Send COPY data */
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, sql.data);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted/updated/deleted on the remote
+ * end
+ */
+ return (n_rows > 0) ? slots : NULL;
+ }
+
/*
* If the existing query was deparsed and prepared for a different number
* of rows, rebuild it for the proper number.
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..c0198b865f3 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void buildCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
--
2.51.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-10-16 20:39 ` Tomas Vondra <tomas@vondra.me>
2025-10-17 09:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 2 replies; 37+ messages in thread
From: Tomas Vondra @ 2025-10-16 20:39 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; pgsql-hackers
Hi Matheus,
Thanks for the patch. Please add it to the next committfest (PG19-3) at
https://commitfest.postgresql.org/
so that we don't lose track of the patch.
On 10/15/25 17:02, Matheus Alcantara wrote:
> Hi all,
>
> Currently on postgres_fdw we use prepared statements to insert batches
> into foreign tables. Although this works fine for the most use cases the
> COPY command can also be used in some scenarios to speed up large batch
> inserts.
>
Makes sense.
> The attached patch implements this idea of using the COPY command for
> batch inserts on postgres_fdw foreign tables. I've performed some
> benchmarks using pgbench and the results seem good to consider this.
>
Thanks. The code looks sensible in general, I think. I'll have a couple
minor comments. It'd be good to also update the documentation, and add
some tests to postgres_fdw.sql, to exercise this new code.
The sgml docs (doc/src/sgml/postgres-fdw.sgml) mention batch_size, and
explain how it's related to the number of parameters in the INSERT
state. Which does not matter when using COPY under the hood, so this
should be amended/clarified in some way. It doesn't need to be
super-detailed, though.
A couple minor comments about the code:
1) buildCopySql
- Does this really need the "(FORMAT TEXT, DELIMITER ',')" part? AFAIK
no, if we use the default copy format in convert_slot_to_copy_text.
- Shouldn't we cache the COPY SQL, similarly to how we keep insert SQL?
Instead of rebuilding it over and over for every batch.
2) convert_slot_to_copy_text
- It's probably better to not hard-code the delimiters etc.
- I wonder if the formatting needs to do something more like what
copyto.c does (through CopyToTextOneRow and CopyAttributeOutText). Maybe
not, not sure.
3) execute_foreign_modify
- I think the new block of code is a bit confusing. It essentially does
something similar to the original code, but not entirely. I suggest we
move it to a new function, and call that from execute_foreign_modify.
- I agree it's probably better to do COPYBUFSIZ, or something like that
to send the data in smaller (but not tiny) chunks.
> I've performed the benchmark using different batch_size values to see
> when this optimization could be useful. The following results are the
> best tps of 3 runs.
>
> Command: pgbench -n -c 10 -j 10 -t 100 -f bench.sql postgres
>
> batch_size: 10
> master tps: 76.360406
> patch tps: 68.917109
>
> batch_size: 100
> master tps: 123.427731
> patch tps: 243.737055
>
> batch_size: 1000
> master tps: 132.500506
> patch tps: 239.295132
>
> It seems that using a batch_size greater than 100 we can have a
> considerable speed up for batch inserts.
>
I did a bunch of benchmarks too, and I see similar speedups (depending
on the batch and data size). Attached is the script I used to run this.
It runs COPY into a foreign table, pointing to a second instance on the
same machine. And it does that for a range of data sizes, batch sizes,
client counts and logged/unlogged table.
The attached PDF summarizes the results, comparing "copy" build (with
this patch) to "master". There's also two "resourceowner" builds, with
this patch [1] - that helps COPY in general a lot, and it improves the
case with batch_size=1000.
I think the results are good, at least with larger batch sizes. The
regressions with batch_size=10 on UNLOGGED table are not great, though.
I have results from another machine, and there it affects even LOGGED
table. I'm not sure if this inherent, or something the patch can fix.
In a way, it's not surprising - batching with tiny batches adds the
overhead without much benefit. I don't think that kills the patch.
> The attached patch uses the COPY command whenever we have a *numSlots >
> 1 but the tests show that maybe we should have a GUC to enable this?
>
I can imagine having a GUC for testing, but it's not strictly necessary.
> I also think that we can have a better patch by removing the duplicated
> code introduced on this first version, specially on the clean up phase,
> but I tried to keep things more simple on this initial phase to keep the
> review more easier and also just to test the idea.
>
> Lastly, I don't know if we should change the EXPLAIN(ANALYZE, VERBOSE)
> output for batch inserts that use the COPY to mention that we are
> sending the COPY command to the remote server. I guess so?
>
Good point. We definitely should not show SQL for INSERT, when we're
actually running a COPY.
regards
[1]
https://www.postgresql.org/message-id/84f20db7-7d57-4dc0-8144-7e38e0bbe75d%40vondra.me
--
Tomas Vondra
Attachments:
fdw-copy-results.pdf (application/pdf, 66.5K) download
scripts.tgz (application/x-compressed-tar, 992B) download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
@ 2025-10-17 09:28 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-10-21 14:26 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Jakub Wartak @ 2025-10-17 09:28 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; +Cc: Matheus Alcantara <matheusssilv97@gmail.com>; pgsql-hackers
On Thu, Oct 16, 2025 at 10:42 PM Tomas Vondra <tomas@vondra.me> wrote:
> Thanks for the patch. Please add it to the next committfest (PG19-3) at
Hi Matheus! same here - thanks for the patch!
> > The attached patch uses the COPY command whenever we have a *numSlots >
> > 1 but the tests show that maybe we should have a GUC to enable this?
> >
>
> I can imagine having a GUC for testing, but it's not strictly necessary.
Just note, I've played maybe like 20mins with this patch and it works,
however if we would like to have yet another GUCs then we would need
to enable two of those? (enable batch_size and this hypothetical
`batch_use_copy`?)
Some other stuff I've tried to cover:
1. how this works with INSERT RETURNING -> as per patch it fallbacks
from COPY to INSERT as expected
2. how this works with INSERT ON CONFLICT -> well, we cannot have
constraints on postgres_fdw, so it is impossible
3. how this works with MERGE -> well, MERGE doesnt work with postgres_fdw
4. I've found that big rows don't play with COPY feature without
memory limitation, so probably some special handling should be done
here, it's nonsense , but:
postgres@postgres:1236 : 15836 # INSERT INTO local_t1 (id, t)
SELECT s, repeat(md5(s::text), 10000000) from generate_series(100,
103) s;
2025-10-17 11:17:08.742 CEST [15836] LOG: statement: INSERT INTO
local_t1 (id, t) SELECT s, repeat(md5(s::text), 10000000) from
generate_series(100, 103) s;
2025-10-17 11:17:08.743 CEST [15838] LOG: statement: START
TRANSACTION ISOLATION LEVEL REPEATABLE READ
2025-10-17 11:17:38.302 CEST [15838] LOG: statement: COPY
public.t1(id, t, counter) FROM STDIN (FORMAT TEXT, DELIMITER ',')
ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)
DETAIL: Cannot enlarge string buffer containing 960000028 bytes
by 320000000 more bytes.
2025-10-17 11:17:40.213 CEST [15836] ERROR: string buffer exceeds
maximum allowed length (1073741823 bytes)
2025-10-17 11:17:40.213 CEST [15836] DETAIL: Cannot enlarge
string buffer containing 960000028 bytes by 320000000 more bytes.
2025-10-17 11:17:40.213 CEST [15836] STATEMENT: INSERT INTO
local_t1 (id, t) SELECT s, repeat(md5(s::text), 10000000) from
generate_series(100, 103) s;
but then it never wants to finish that backend (constant loop[ in
PQCleanup() or somewhere close to that), server behaves unstable.
Without batch_size set the very same INSERT behaves OK.
Regards,
-J.
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-17 09:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Jakub Wartak <jakub.wartak@enterprisedb.com>
@ 2025-10-21 14:26 ` Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 0 replies; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-21 14:26 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; Tomas Vondra <tomas@vondra.me>; +Cc: Matheus Alcantara <matheusssilv97@gmail.com>; pgsql-hackers
Thanks for testing and for the comments!
On Fri Oct 17, 2025 at 6:28 AM -03, Jakub Wartak wrote:
> On Thu, Oct 16, 2025 at 10:42 PM Tomas Vondra <tomas@vondra.me> wrote:
>> Thanks for the patch. Please add it to the next committfest (PG19-3) at
>
> Hi Matheus! same here - thanks for the patch!
>
>> > The attached patch uses the COPY command whenever we have a *numSlots >
>> > 1 but the tests show that maybe we should have a GUC to enable this?
>> >
>>
>> I can imagine having a GUC for testing, but it's not strictly necessary.
>
> Just note, I've played maybe like 20mins with this patch and it works,
> however if we would like to have yet another GUCs then we would need
> to enable two of those? (enable batch_size and this hypothetical
> `batch_use_copy`?)
>
I was thinking in a GUC to enable to use the COPY command if the number
of rows being ingested during the batch insert is greater than the value
configured on this GUC, something like, batch_size_for_copy. But for now
I'm starting to think that perhaps we may use the COPY if the batch_size
is configured to a number > 1(or make this configurable?). With this it
would make more easier to show on EXPLAIN that we will send a COPY
command to the remote server instead of INSERT. The currently patch
relies on the number of rows being sent to the foreign server to enable
the COPY usage or not, and IUUC we don't have this information during
EXPLAIN. I wrote more about this on my previous reply [1].
> Some other stuff I've tried to cover:
>...
> 4. I've found that big rows don't play with COPY feature without
> memory limitation, so probably some special handling should be done
> here, it's nonsense , but:
>
> postgres@postgres:1236 : 15836 # INSERT INTO local_t1 (id, t)
> SELECT s, repeat(md5(s::text), 10000000) from generate_series(100,
> 103) s;
> 2025-10-17 11:17:08.742 CEST [15836] LOG: statement: INSERT INTO
> local_t1 (id, t) SELECT s, repeat(md5(s::text), 10000000) from
> generate_series(100, 103) s;
> 2025-10-17 11:17:08.743 CEST [15838] LOG: statement: START
> TRANSACTION ISOLATION LEVEL REPEATABLE READ
> 2025-10-17 11:17:38.302 CEST [15838] LOG: statement: COPY
> public.t1(id, t, counter) FROM STDIN (FORMAT TEXT, DELIMITER ',')
> ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)
> DETAIL: Cannot enlarge string buffer containing 960000028 bytes
> by 320000000 more bytes.
> 2025-10-17 11:17:40.213 CEST [15836] ERROR: string buffer exceeds
> maximum allowed length (1073741823 bytes)
> 2025-10-17 11:17:40.213 CEST [15836] DETAIL: Cannot enlarge
> string buffer containing 960000028 bytes by 320000000 more bytes.
> 2025-10-17 11:17:40.213 CEST [15836] STATEMENT: INSERT INTO
> local_t1 (id, t) SELECT s, repeat(md5(s::text), 10000000) from
> generate_series(100, 103) s;
>
> but then it never wants to finish that backend (constant loop[ in
> PQCleanup() or somewhere close to that), server behaves unstable.
> Without batch_size set the very same INSERT behaves OK.
>
On the last version that I sent on [1] I introduce a buffer size limit,
and testing this INSERT statement with the latest version seems to fix
this issue. Could you please check this too?
[1] https://www.postgresql.org/message-id/CAFY6G8ePwjT8GiJX1AK5FDMhfq-sOnny6optgTPg98HQw7oJ0g%40mail.gma...
--
Matheus Alcantara
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
@ 2025-10-21 14:25 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-21 14:25 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; Matheus Alcantara <matheusssilv97@gmail.com>; pgsql-hackers
Thanks for the review!
On Thu Oct 16, 2025 at 5:39 PM -03, Tomas Vondra wrote:
> Thanks for the patch. Please add it to the next committfest (PG19-3) at
>
> https://commitfest.postgresql.org/
>
> so that we don't lose track of the patch.
>
Here it is: https://commitfest.postgresql.org/patch/6137/
> Thanks. The code looks sensible in general, I think. I'll have a couple
> minor comments. It'd be good to also update the documentation, and add
> some tests to postgres_fdw.sql, to exercise this new code.
>
> The sgml docs (doc/src/sgml/postgres-fdw.sgml) mention batch_size, and
> explain how it's related to the number of parameters in the INSERT
> state. Which does not matter when using COPY under the hood, so this
> should be amended/clarified in some way. It doesn't need to be
> super-detailed, though.
>
I'll work on this and intend to have something on the next version.
> A couple minor comments about the code:
>
> 1) buildCopySql
>
> - Does this really need the "(FORMAT TEXT, DELIMITER ',')" part? AFAIK
> no, if we use the default copy format in convert_slot_to_copy_text.
>
You're right. I've removed the (FORMAT TEXT, DELIMITER ',') part.
> - Shouldn't we cache the COPY SQL, similarly to how we keep insert SQL?
> Instead of rebuilding it over and over for every batch.
>
I tried to reuse the fmstate->query field to cache the COPY sql but
running the postgres_fdw.sql regress test shows that this may not
work. When we are running a user supplied COPY command on a foreign
table the CopyMultiInsertBufferFlush() call
ri_FdwRoutine->ExecForeignBatchInsert which may pass different values
for numSlots based on the number of slots already sent to the foreign
server, and eventually it may pass numSlots as 1 which will not use the
COPY under the hood to send to the foreign server and if we cache the
COPY command into the fmstate->query this will not work because the
normal INSERT path on execute_foreign_modify uses the fmstate->query to
build a prepared statement to send to the foreign server. So basically
what I'm trying to say is that when the server is executing a COPY into
a foreign it may use the COPY command or INSERT command to send the data
to the foreign server. That being said, I decided to create a new
copy_query field on PgFdwModifyState to cache only COPY commands. Please
let me know if my understanding is wrong or if we could have a better
approach here.
> 2) convert_slot_to_copy_text
>
> - It's probably better to not hard-code the delimiters etc.
>
Since now we are using the default copy format, it's safe to hard code
the default delimiter which is \t? Or should we still make this
parameterizable or something else?
> - I wonder if the formatting needs to do something more like what
> copyto.c does (through CopyToTextOneRow and CopyAttributeOutText). Maybe
> not, not sure.
>
I'm still checking this, I think that is not needed but I want to do
some more tests to make sure.
> 3) execute_foreign_modify
>
> - I think the new block of code is a bit confusing. It essentially does
> something similar to the original code, but not entirely. I suggest we
> move it to a new function, and call that from execute_foreign_modify.
>
Fixed
> - I agree it's probably better to do COPYBUFSIZ, or something like that
> to send the data in smaller (but not tiny) chunks.
>
Fixed. I've declared the default chunk size as 8192 (which is the same
used on psql/copy.c), let me know if we should use another value or
perhaps make this configurable.
>> Lastly, I don't know if we should change the EXPLAIN(ANALYZE, VERBOSE)
>> output for batch inserts that use the COPY to mention that we are
>> sending the COPY command to the remote server. I guess so?
>>
>
> Good point. We definitely should not show SQL for INSERT, when we're
> actually running a COPY.
>
This seems a bit tricky to implement. The COPY is used based on the
number of slots into the TupleTableSlot array that is used for batch
insert. The numSlots that execute_foreign_modify() receive is coming
from ResultRelInfo->ri_NumSlots during ExecInsert(). We don't have this
information during EXPLAIN that is handled by
postgresExplainForeignModify(), we only have the
ResultRelInfo->ri_BatchSize at this stage. The current idea is to use
the COPY command if the number of slots is > 1 so I'm wondering if we
should use another mechanism to enable the COPY usage, for example, we
could just use if the batch_size is configured to a number greater than
X, but what if the INSERT statement is only inserting a single row,
should we still use the COPY command to ingest a single row into the
foreign table? Any thoughts?
--
Matheus Alcantara
Attachments:
v2-0001-postgres_fdw-Use-COPY-to-speed-up-batch-inserts.patch (application/octet-stream, 7.4K) download
From 5e8545c8641d1afbee22b446bbfcae7970480fbf Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Fri, 10 Oct 2025 16:07:08 -0300
Subject: [PATCH v2] postgres_fdw: Use COPY to speed up batch inserts
---
contrib/postgres_fdw/deparse.c | 32 +++++++
contrib/postgres_fdw/postgres_fdw.c | 140 ++++++++++++++++++++++++++++
contrib/postgres_fdw/postgres_fdw.h | 1 +
3 files changed, 173 insertions(+)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index e5b5e1a5f51..3323a92617a 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2238,6 +2238,38 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+buildCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ ListCell *lc;
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach(lc, target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 456b267f70b..7f345d19102 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -192,6 +195,7 @@ typedef struct PgFdwModifyState
/* extracted fdw_private data */
char *query; /* text of INSERT/UPDATE/DELETE command */
char *orig_query; /* original text of INSERT command */
+ char *copy_query; /* text of COPY command if it's being used */
List *target_attrs; /* list of target attribute numbers */
int values_end; /* length up to the end of VALUES */
int batch_size; /* value of FDW option "batch_size" */
@@ -545,6 +549,9 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -4066,6 +4073,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ ListCell *lc;
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+
+ foreach(lc, fmstate->target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[attnum - 1],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,6 +4148,13 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /*
+ * Use COPY command for batch insert if the original query don't include a
+ * RETURNING clause
+ */
+ if (operation == CMD_INSERT && *numSlots > 1 && !fmstate->has_returning)
+ return execute_foreign_insert_using_copy(fmstate, slots, numSlots);
+
/*
* If the existing query was deparsed and prepared for a different number
* of rows, rebuild it for the proper number.
@@ -7886,3 +7944,85 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/* Execute a batch insert into a foreign table using the COPY command */
+static TupleTableSlot **
+execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData sql;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ if (fmstate->copy_query == NULL)
+ {
+ /* Build COPY command */
+ initStringInfo(&sql);
+ buildCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+
+ /* Cache for reuse. */
+ fmstate->copy_query = sql.data;
+ }
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->copy_query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->copy_query);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->copy_query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..c0198b865f3 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void buildCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
--
2.51.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-10-23 00:00 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-23 00:00 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Tue Oct 21, 2025 at 11:25 AM -03, Matheus Alcantara wrote:
>>> Lastly, I don't know if we should change the EXPLAIN(ANALYZE, VERBOSE)
>>> output for batch inserts that use the COPY to mention that we are
>>> sending the COPY command to the remote server. I guess so?
>>>
>>
>> Good point. We definitely should not show SQL for INSERT, when we're
>> actually running a COPY.
>>
> This seems a bit tricky to implement. The COPY is used based on the
> number of slots into the TupleTableSlot array that is used for batch
> insert. The numSlots that execute_foreign_modify() receive is coming
> from ResultRelInfo->ri_NumSlots during ExecInsert(). We don't have this
> information during EXPLAIN that is handled by
> postgresExplainForeignModify(), we only have the
> ResultRelInfo->ri_BatchSize at this stage. The current idea is to use
> the COPY command if the number of slots is > 1 so I'm wondering if we
> should use another mechanism to enable the COPY usage, for example, we
> could just use if the batch_size is configured to a number greater than
> X, but what if the INSERT statement is only inserting a single row,
> should we still use the COPY command to ingest a single row into the
> foreign table? Any thoughts?
>
Thinking more about this I realize that when we are deparsing the remote
SQL to be sent to the foreign server at the planner phase (via
postgresPlanForeignModify()) we don't have the batch_size and number of
rows information, so currently we can not know at the plan time if the
COPY usage for a batch insert is visible or not because IIUC these
information are only visible at query runtime.
One way to make it possible is that we could simply use the
PgFdwModifyState->copy_data during postgresExplainForeignModify() if
it's not null. Since we will only have this information during query
execution the drawback of this approach is that we would only show the
COPY as a Remote SQL on during EXPLAIN(ANALYZE).
Please see the attached v3 version that implements this idea.
> I tried to reuse the fmstate->query field to cache the COPY sql but
> running the postgres_fdw.sql regress test shows that this may not
> work. When we are running a user supplied COPY command on a foreign
> table the CopyMultiInsertBufferFlush() call
> ri_FdwRoutine->ExecForeignBatchInsert which may pass different values
> for numSlots based on the number of slots already sent to the foreign
> server, and eventually it may pass numSlots as 1 which will not use the
> COPY under the hood to send to the foreign server and if we cache the
> COPY command into the fmstate->query this will not work because the
> normal INSERT path on execute_foreign_modify uses the fmstate->query to
> build a prepared statement to send to the foreign server. So basically
> what I'm trying to say is that when the server is executing a COPY into
> a foreign it may use the COPY command or INSERT command to send the data
> to the foreign server. That being said, I decided to create a new
> copy_query field on PgFdwModifyState to cache only COPY commands. Please
> let me know if my understanding is wrong or if we could have a better
> approach here.
>
Based on the information that I've mention above I think that we need
some way to not mix INSERT with COPY commands when executing a COPY in
a foreign table supplied by the user. Or we should disable the COPY
under the hood and always fallback to INSERT or enable the COPY to use
when the *numSlots is 1, so in case of an EXPLAIN(ANALYZE) output we can
show the Remote SQL correctly. Is that make sense?
I'm still not sure if the trigger to use the COPY command for batch
insert should be *numSlots > 1 or something else. I'm open for better
ideas.
Thoughts?
--
Matheus Alcantara
Attachments:
v3-0001-postgres_fdw-Use-COPY-to-speed-up-batch-inserts.patch (text/plain, 8.1K) download
From 0175f829cc2944eb596f18d701b64c2d90d8e2bb Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Fri, 10 Oct 2025 16:07:08 -0300
Subject: [PATCH v3] postgres_fdw: Use COPY to speed up batch inserts
---
contrib/postgres_fdw/deparse.c | 32 ++++++
contrib/postgres_fdw/postgres_fdw.c | 159 +++++++++++++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
3 files changed, 190 insertions(+), 2 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index f2fb0051843..afd1cc636d7 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,38 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+buildCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ ListCell *lc;
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach(lc, target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 456b267f70b..0ccbff8e390 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -192,6 +195,7 @@ typedef struct PgFdwModifyState
/* extracted fdw_private data */
char *query; /* text of INSERT/UPDATE/DELETE command */
char *orig_query; /* original text of INSERT command */
+ char *copy_query; /* text of COPY command if it's being used */
List *target_attrs; /* list of target attribute numbers */
int values_end; /* length up to the end of VALUES */
int batch_size; /* value of FDW option "batch_size" */
@@ -545,6 +549,9 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -2942,8 +2949,23 @@ postgresExplainForeignModify(ModifyTableState *mtstate,
{
if (es->verbose)
{
- char *sql = strVal(list_nth(fdw_private,
- FdwModifyPrivateUpdateSql));
+ char *sql = NULL;
+
+ /*
+ * We only have ri_FdwState during EXPLAIN(ANALYZE), so check if the
+ * COPY was used during query execution and show it as a Remote SQL.
+ */
+ if (rinfo->ri_FdwState != NULL)
+ {
+ PgFdwModifyState *fmstate = (PgFdwModifyState *) rinfo->ri_FdwState;
+
+ if (fmstate->copy_query != NULL)
+ sql = fmstate->copy_query;
+ }
+
+ if (sql == NULL)
+ sql = strVal(list_nth(fdw_private,
+ FdwModifyPrivateUpdateSql));
ExplainPropertyText("Remote SQL", sql, es);
@@ -4066,6 +4088,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ ListCell *lc;
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+
+ foreach(lc, fmstate->target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[attnum - 1],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,6 +4163,13 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /*
+ * Use COPY command for batch insert if the original query don't include a
+ * RETURNING clause
+ */
+ if (operation == CMD_INSERT && *numSlots > 1 && !fmstate->has_returning)
+ return execute_foreign_insert_using_copy(fmstate, slots, numSlots);
+
/*
* If the existing query was deparsed and prepared for a different number
* of rows, rebuild it for the proper number.
@@ -7886,3 +7959,85 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/* Execute a batch insert into a foreign table using the COPY command */
+static TupleTableSlot **
+execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData sql;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ if (fmstate->copy_query == NULL)
+ {
+ /* Build COPY command */
+ initStringInfo(&sql);
+ buildCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+
+ /* Cache for reuse. */
+ fmstate->copy_query = sql.data;
+ }
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->copy_query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->copy_query);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->copy_query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..c0198b865f3 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void buildCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
--
2.51.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-10-23 09:49 ` jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: jian he @ 2025-10-23 09:49 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Thu, Oct 23, 2025 at 8:01 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> Please see the attached v3 version that implements this idea.
>
hi.
I am not famailith with this module.
some of the foreach can be replaced with foreach_int.
I suspect that somewhere Form_pg_attribute.attisdropped is not handled properly.
the following setup will crash.
---source database
drop table batch_table1;
create table batch_table1(x int);
---foreign table database
drop foreign table if exists ftable1;
CREATE FOREIGN TABLE ftable1 ( x int ) SERVER loopback1 OPTIONS (
table_name 'batch_table1', batch_size '10' );
ALTER FOREIGN TABLE ftable1 DROP COLUMN x;
ALTER FOREIGN TABLE ftable1 add COLUMN x int;
INSERT INTO ftable SELECT * FROM generate_series(1, 10) i; --- this
will cause server crash.
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
@ 2025-10-24 18:27 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-24 18:27 UTC (permalink / raw)
To: jian he <jian.universality@gmail.com>; Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
Hi, thanks for testing this patch!
On Thu Oct 23, 2025 at 6:49 AM -03, jian he wrote:
> On Thu, Oct 23, 2025 at 8:01 AM Matheus Alcantara
> <matheusssilv97@gmail.com> wrote:
>>
>> Please see the attached v3 version that implements this idea.
>>
> hi.
>
> I am not famailith with this module.
> some of the foreach can be replaced with foreach_int.
>
Fixed.
> I suspect that somewhere Form_pg_attribute.attisdropped is not handled properly.
> the following setup will crash.
>
> ---source database
> drop table batch_table1;
> create table batch_table1(x int);
>
> ---foreign table database
> drop foreign table if exists ftable1;
> CREATE FOREIGN TABLE ftable1 ( x int ) SERVER loopback1 OPTIONS (
> table_name 'batch_table1', batch_size '10' );
> ALTER FOREIGN TABLE ftable1 DROP COLUMN x;
> ALTER FOREIGN TABLE ftable1 add COLUMN x int;
>
> INSERT INTO ftable SELECT * FROM generate_series(1, 10) i; --- this
> will cause server crash.
>
I've tested this scenario and the field attisdropped is still being set
to false. After some debugging I realize that the problem was how I was
accessing the fmstate->p_flinfo array - I was using the attum-1 but I
don't think that it's correct.
On create_foreign_modify() we have the following code:
fmstate->p_flinfo = (FmgrInfo *) palloc0(sizeof(FmgrInfo) * n_params);
fmstate->p_nums = 0;
if (operation == CMD_INSERT || operation == CMD_UPDATE)
{
/* Set up for remaining transmittable parameters */
foreach(lc, fmstate->target_attrs)
{
int attnum = lfirst_int(lc);
Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
Assert(!attr->attisdropped);
/* Ignore generated columns; they are set to DEFAULT */
if (attr->attgenerated)
continue;
getTypeOutputInfo(attr->atttypid, &typefnoid, &isvarlena);
fmgr_info(typefnoid, &fmstate->p_flinfo[fmstate->p_nums]);
fmstate->p_nums++;
}
}
So I think that I should access fmstate->p_flinfo array when looping
through the target_attrs using an int value starting at 0 and ++ after
each iteration. Although I'm not sure if my understanding is fully
correct I've implemented this on the attached patch and it seems to fix
the error.
On this new version I also added some regress tests on postgres_fdw.sql
--
Matheus Alcantara
Attachments:
v4-0001-postgres_fdw-Use-COPY-to-speed-up-batch-inserts.patch (text/plain, 16.7K) download
From 1856fba0c49d5aa7b164debb90b376c72cfa3e02 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Fri, 10 Oct 2025 16:07:08 -0300
Subject: [PATCH v4] postgres_fdw: Use COPY to speed up batch inserts
---
contrib/postgres_fdw/deparse.c | 30 ++++
.../postgres_fdw/expected/postgres_fdw.out | 120 +++++++++++--
contrib/postgres_fdw/postgres_fdw.c | 159 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 52 ++++++
5 files changed, 350 insertions(+), 12 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index f2fb0051843..113e6fb7d91 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,36 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+buildCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index cd28126049d..dd507ad6186 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -50,6 +50,18 @@ CREATE TABLE "S 1"."T 4" (
c3 text,
CONSTRAINT t4_pkey PRIMARY KEY (c1)
);
+CREATE TABLE "S 1"."T 5"(
+ x int
+);
+CREATE TABLE "S 1"."T 6"(
+ id int not null,
+ note text,
+ value int NOT NULL
+);
+CREATE TABLE "S 1"."T 7"(
+ id int,
+ t text
+);
-- Disable autovacuum for these tables to avoid unexpected effects of that
ALTER TABLE "S 1"."T 1" SET (autovacuum_enabled = 'false');
ALTER TABLE "S 1"."T 2" SET (autovacuum_enabled = 'false');
@@ -132,6 +144,21 @@ CREATE FOREIGN TABLE ft7 (
c2 int NOT NULL,
c3 text
) SERVER loopback3 OPTIONS (schema_name 'S 1', table_name 'T 4');
+CREATE FOREIGN TABLE ft8 (
+ x int
+)
+SERVER loopback OPTIONS (schema_name 'S 1', table_name 'T 5', batch_size '10');
+CREATE FOREIGN TABLE ft9 (
+ id int not null,
+ note text,
+ value int NOT NULL
+)
+SERVER loopback OPTIONS (schema_name 'S 1', table_name 'T 6', batch_size '10');
+CREATE FOREIGN TABLE ft10 (
+ id int,
+ t text
+)
+SERVER loopback OPTIONS (schema_name 'S 1', table_name 'T 7', batch_size '10');
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -205,16 +232,19 @@ ALTER FOREIGN TABLE ft2 OPTIONS (schema_name 'S 1', table_name 'T 1');
ALTER FOREIGN TABLE ft1 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
ALTER FOREIGN TABLE ft2 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
\det+
- List of foreign tables
- Schema | Table | Server | FDW options | Description
---------+-------+-----------+---------------------------------------+-------------
- public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
- public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
- public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
- public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
-(6 rows)
+ List of foreign tables
+ Schema | Table | Server | FDW options | Description
+--------+-------+-----------+--------------------------------------------------------+-------------
+ public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft10 | loopback | (schema_name 'S 1', table_name 'T 7', batch_size '10') |
+ public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
+ public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
+ public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft8 | loopback | (schema_name 'S 1', table_name 'T 5', batch_size '10') |
+ public | ft9 | loopback | (schema_name 'S 1', table_name 'T 6', batch_size '10') |
+(9 rows)
-- Test that alteration of server options causes reconnection
-- Remote's errors might be non-English, so hide them to ensure stable results
@@ -12664,6 +12694,76 @@ ANALYZE analyze_ftable;
-- cleanup
DROP FOREIGN TABLE analyze_ftable;
DROP TABLE analyze_table;
+-- ===================================================================
+-- test for batch insert using COPY
+-- ===================================================================
+ALTER FOREIGN TABLE ft8 DROP COLUMN x;
+ALTER FOREIGN TABLE ft8 add COLUMN x int;
+INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
+SELECT * FROM ft8;
+ x
+----
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
+SELECT g,
+ g * 2,
+ 'batch insert test data' || g
+FROM generate_series(1, 20) g;
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Insert on public.ft9 (actual rows=0.00 loops=1)
+ Remote SQL: COPY "S 1"."T 6"(id, note, value) FROM STDIN
+ Batch Size: 10
+ -> Function Scan on pg_catalog.generate_series g (actual rows=20.00 loops=1)
+ Output: g.g, ('batch insert test data'::text || (g.g)::text), (g.g * 2)
+ Function Call: generate_series(1, 20)
+(6 rows)
+
+SELECT * FROM ft9;
+ id | note | value
+----+--------------------------+-------
+ 1 | batch insert test data1 | 2
+ 2 | batch insert test data2 | 4
+ 3 | batch insert test data3 | 6
+ 4 | batch insert test data4 | 8
+ 5 | batch insert test data5 | 10
+ 6 | batch insert test data6 | 12
+ 7 | batch insert test data7 | 14
+ 8 | batch insert test data8 | 16
+ 9 | batch insert test data9 | 18
+ 10 | batch insert test data10 | 20
+ 11 | batch insert test data11 | 22
+ 12 | batch insert test data12 | 24
+ 13 | batch insert test data13 | 26
+ 14 | batch insert test data14 | 28
+ 15 | batch insert test data15 | 30
+ 16 | batch insert test data16 | 32
+ 17 | batch insert test data17 | 34
+ 18 | batch insert test data18 | 36
+ 19 | batch insert test data19 | 38
+ 20 | batch insert test data20 | 40
+(20 rows)
+
+-- Test buffer limit of copy data on COPYBUFSIZ
+INSERT INTO ft10 (id, t)
+SELECT s, repeat(md5(s::text), 10000) from generate_series(100, 103) s;
+SELECT COUNT(*) FROM ft10;
+ count
+-------
+ 4
+(1 row)
+
-- ===================================================================
-- test for postgres_fdw_get_connections function with check_conn = true
-- ===================================================================
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 456b267f70b..d8e13e78938 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -192,6 +195,7 @@ typedef struct PgFdwModifyState
/* extracted fdw_private data */
char *query; /* text of INSERT/UPDATE/DELETE command */
char *orig_query; /* original text of INSERT command */
+ char *copy_query; /* text of COPY command if it's being used */
List *target_attrs; /* list of target attribute numbers */
int values_end; /* length up to the end of VALUES */
int batch_size; /* value of FDW option "batch_size" */
@@ -545,6 +549,9 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -2942,8 +2949,23 @@ postgresExplainForeignModify(ModifyTableState *mtstate,
{
if (es->verbose)
{
- char *sql = strVal(list_nth(fdw_private,
- FdwModifyPrivateUpdateSql));
+ char *sql = NULL;
+
+ /*
+ * We only have ri_FdwState during EXPLAIN(ANALYZE), so check if the
+ * COPY was used during query execution and show it as a Remote SQL.
+ */
+ if (rinfo->ri_FdwState != NULL)
+ {
+ PgFdwModifyState *fmstate = (PgFdwModifyState *) rinfo->ri_FdwState;
+
+ if (fmstate->copy_query != NULL)
+ sql = fmstate->copy_query;
+ }
+
+ if (sql == NULL)
+ sql = strVal(list_nth(fdw_private,
+ FdwModifyPrivateUpdateSql));
ExplainPropertyText("Remote SQL", sql, es);
@@ -4066,6 +4088,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,6 +4163,13 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /*
+ * Use COPY command for batch insert if the original query don't include a
+ * RETURNING clause
+ */
+ if (operation == CMD_INSERT && *numSlots > 1 && !fmstate->has_returning)
+ return execute_foreign_insert_using_copy(fmstate, slots, numSlots);
+
/*
* If the existing query was deparsed and prepared for a different number
* of rows, rebuild it for the proper number.
@@ -7886,3 +7959,85 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/* Execute a batch insert into a foreign table using the COPY command */
+static TupleTableSlot **
+execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData sql;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ if (fmstate->copy_query == NULL)
+ {
+ /* Build COPY command */
+ initStringInfo(&sql);
+ buildCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+
+ /* Cache for reuse. */
+ fmstate->copy_query = sql.data;
+ }
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->copy_query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->copy_query);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->copy_query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->copy_query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..c0198b865f3 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void buildCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 9a8f9e28135..79f4f305641 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -54,6 +54,18 @@ CREATE TABLE "S 1"."T 4" (
c3 text,
CONSTRAINT t4_pkey PRIMARY KEY (c1)
);
+CREATE TABLE "S 1"."T 5"(
+ x int
+);
+CREATE TABLE "S 1"."T 6"(
+ id int not null,
+ note text,
+ value int NOT NULL
+);
+CREATE TABLE "S 1"."T 7"(
+ id int,
+ t text
+);
-- Disable autovacuum for these tables to avoid unexpected effects of that
ALTER TABLE "S 1"."T 1" SET (autovacuum_enabled = 'false');
@@ -146,6 +158,24 @@ CREATE FOREIGN TABLE ft7 (
c3 text
) SERVER loopback3 OPTIONS (schema_name 'S 1', table_name 'T 4');
+CREATE FOREIGN TABLE ft8 (
+ x int
+)
+SERVER loopback OPTIONS (schema_name 'S 1', table_name 'T 5', batch_size '10');
+
+CREATE FOREIGN TABLE ft9 (
+ id int not null,
+ note text,
+ value int NOT NULL
+)
+SERVER loopback OPTIONS (schema_name 'S 1', table_name 'T 6', batch_size '10');
+
+CREATE FOREIGN TABLE ft10 (
+ id int,
+ t text
+)
+SERVER loopback OPTIONS (schema_name 'S 1', table_name 'T 7', batch_size '10');
+
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -4379,6 +4409,28 @@ ANALYZE analyze_ftable;
DROP FOREIGN TABLE analyze_ftable;
DROP TABLE analyze_table;
+-- ===================================================================
+-- test for batch insert using COPY
+-- ===================================================================
+ALTER FOREIGN TABLE ft8 DROP COLUMN x;
+ALTER FOREIGN TABLE ft8 add COLUMN x int;
+
+INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
+SELECT * FROM ft8;
+
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
+SELECT g,
+ g * 2,
+ 'batch insert test data' || g
+FROM generate_series(1, 20) g;
+
+SELECT * FROM ft9;
+
+-- Test buffer limit of copy data on COPYBUFSIZ
+INSERT INTO ft10 (id, t)
+SELECT s, repeat(md5(s::text), 10000) from generate_series(100, 103) s;
+SELECT COUNT(*) FROM ft10;
+
-- ===================================================================
-- test for postgres_fdw_get_connections function with check_conn = true
-- ===================================================================
--
2.51.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-10-29 03:10 ` jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: jian he @ 2025-10-29 03:10 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Sat, Oct 25, 2025 at 2:27 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On this new version I also added some regress tests on postgres_fdw.sql
>
In the CopyFrom function, we have the CopyInsertMethod, CIM_SINGLE is slower
than CIM_MULTI, I think.
We should do performance tests for the case where the COPY statement is limited
to use CIM_SINGLE.
You can use triggers to make COPY can only use the CIM_SINGLE copymethod.
for example:
create function dummy() returns trigger as $$ begin return new; end $$
language plpgsql;
create trigger dummy
before insert or update on batch_table_3
for each row execute procedure dummy();
My local tests show that when batch_size is greater than 2, COPY performs faster
than batch inserts into a foreign table, even though COPY can only use
CIM_SINGLE.
However, my tests were done with an enable-assert build, since I
encountered issues compiling the release build.
anyway, I am sharing my test script.
Attachments:
fdw_copy_test.sql (application/sql, 2.0K) download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
@ 2025-10-30 00:28 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-30 00:28 UTC (permalink / raw)
To: jian he <jian.universality@gmail.com>; Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Wed Oct 29, 2025 at 12:10 AM -03, jian he wrote:
> On Sat, Oct 25, 2025 at 2:27 AM Matheus Alcantara
> <matheusssilv97@gmail.com> wrote:
>>
>> On this new version I also added some regress tests on postgres_fdw.sql
>>
>
> In the CopyFrom function, we have the CopyInsertMethod, CIM_SINGLE is slower
> than CIM_MULTI, I think.
> We should do performance tests for the case where the COPY statement is limited
> to use CIM_SINGLE.
>
> You can use triggers to make COPY can only use the CIM_SINGLE copymethod.
> for example:
> create function dummy() returns trigger as $$ begin return new; end $$
> language plpgsql;
> create trigger dummy
> before insert or update on batch_table_3
> for each row execute procedure dummy();
>
> My local tests show that when batch_size is greater than 2, COPY performs faster
> than batch inserts into a foreign table, even though COPY can only use
> CIM_SINGLE.
> However, my tests were done with an enable-assert build, since I
> encountered issues compiling the release build.
>
> anyway, I am sharing my test script.
>
I've benchmarked using buildtype=release with Dcassert=false and
buildtype=debug with Dcassert=true and in both cases I've got a worst
performance when using the COPY for batching insert into a a foreign
table with a trigger. See the results (best of 4 runs).
Batch using INSERT
batch_size: 100
buildtype=debug
Dcassert=true
tps = 13.596754
Batch using COPY
batch_size: 100
buildtype=debug
Dcassert=true
tps = 11.650642
--------------------
Batch using INSERT
batch_size: 100
buildtype=release
Dcassert=false
tps = 28.333161
Batch using COPY
batch_size: 100
buildtype=release
Dcassert=false
tps = 18.499420
It seems to me that we need to disable the COPY usage when the foreign
table has triggers enabled.
--
Matheus Alcantara
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-10-30 16:32 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Andrew Dunstan @ 2025-10-30 16:32 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; jian he <jian.universality@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On 2025-10-29 We 8:28 PM, Matheus Alcantara wrote:
> On Wed Oct 29, 2025 at 12:10 AM -03, jian he wrote:
>> On Sat, Oct 25, 2025 at 2:27 AM Matheus Alcantara
>> <matheusssilv97@gmail.com> wrote:
>>> On this new version I also added some regress tests on postgres_fdw.sql
>>>
>> In the CopyFrom function, we have the CopyInsertMethod, CIM_SINGLE is slower
>> than CIM_MULTI, I think.
>> We should do performance tests for the case where the COPY statement is limited
>> to use CIM_SINGLE.
>>
>> You can use triggers to make COPY can only use the CIM_SINGLE copymethod.
>> for example:
>> create function dummy() returns trigger as $$ begin return new; end $$
>> language plpgsql;
>> create trigger dummy
>> before insert or update on batch_table_3
>> for each row execute procedure dummy();
>>
>> My local tests show that when batch_size is greater than 2, COPY performs faster
>> than batch inserts into a foreign table, even though COPY can only use
>> CIM_SINGLE.
>> However, my tests were done with an enable-assert build, since I
>> encountered issues compiling the release build.
>>
>> anyway, I am sharing my test script.
>>
> I've benchmarked using buildtype=release with Dcassert=false and
> buildtype=debug with Dcassert=true and in both cases I've got a worst
> performance when using the COPY for batching insert into a a foreign
> table with a trigger. See the results (best of 4 runs).
>
> Batch using INSERT
> batch_size: 100
> buildtype=debug
> Dcassert=true
> tps = 13.596754
>
> Batch using COPY
> batch_size: 100
> buildtype=debug
> Dcassert=true
> tps = 11.650642
>
> --------------------
>
> Batch using INSERT
> batch_size: 100
> buildtype=release
> Dcassert=false
> tps = 28.333161
>
>
> Batch using COPY
> batch_size: 100
> buildtype=release
> Dcassert=false
> tps = 18.499420
>
>
> It seems to me that we need to disable the COPY usage when the foreign
> table has triggers enabled.
>
I think it's probably worth finding out why COPY is so much worse in the
presence of triggers. Is there something we can do to improve that, at
least so it's no worse?
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
@ 2025-10-31 19:02 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-10-31 19:02 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; Matheus Alcantara <matheusssilv97@gmail.com>; jian he <jian.universality@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Thu Oct 30, 2025 at 1:32 PM -03, Andrew Dunstan wrote:
>> It seems to me that we need to disable the COPY usage when the foreign
>> table has triggers enabled.
>>
>
> I think it's probably worth finding out why COPY is so much worse in the
> presence of triggers. Is there something we can do to improve that, at
> least so it's no worse?
>
I did a bit of reading on CopyFrom() and found comments that clarifies
why the use of COPY for batch inserts is slower when the target foreign
table has triggers.
/*
* It's generally more efficient to prepare a bunch of tuples for
* insertion, and insert them in one
* table_multi_insert()/ExecForeignBatchInsert() call, than call
* table_tuple_insert()/ExecForeignInsert() separately for every tuple.
* However, there are a number of reasons why we might not be able to do
* this. These are explained below.
*/
if (resultRelInfo->ri_TrigDesc != NULL &&
(resultRelInfo->ri_TrigDesc->trig_insert_before_row ||
resultRelInfo->ri_TrigDesc->trig_insert_instead_row))
{
/*
* Can't support multi-inserts when there are any BEFORE/INSTEAD OF
* triggers on the table. Such triggers might query the table we're
* inserting into and act differently if the tuples that have already
* been processed and prepared for insertion are not there.
*/
insertMethod = CIM_SINGLE;
}
It forces the use of CIM_SINGLE if a BEFORE or INSTEAD OF trigger is
present. This method then relies on table_tuple_insert() or
ExecForeignInsert() to insert one tuple at a time.
IUUC we cannot determine during execute_foreign_modify() whether a
foreign table has triggers enabled on the target remote table. This lack
of information suggests that we would need to query the foreign server
to find out. If it's the only viable path I think that we would have
issues if the the user configured to access the foreign server don't
have access on catalog tables.
It's showing a bit complicated to decide at runtime if we should use the
COPY or INSERT for batch insert into a foreign table. Perhaps we could
add a new option on CREATE FOREIGN TABLE to enable this usage or not? We
could document the performance improvements and the limitations so the
user can decide if it should enable or not.
--
Matheus Alcantara
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-11-06 23:49 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-11-06 23:49 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Fri Oct 31, 2025 at 4:02 PM -03, I wrote:
> It's showing a bit complicated to decide at runtime if we should use the
> COPY or INSERT for batch insert into a foreign table. Perhaps we could
> add a new option on CREATE FOREIGN TABLE to enable this usage or not? We
> could document the performance improvements and the limitations so the
> user can decide if it should enable or not.
>
Here is v5 that implement this idea.
On this version I've introduced a foreign table and foreign server
option "use_copy_for_insert" (I'm open for a better name) that enable
the use of the COPY as remote command to execute an INSERT into a
foreign table. The COPY can be used if the user enable this option on
the foreign table or the foreign server and if the original INSERT
statement don't have a RETURNING clause.
See the benchmark results:
pgbench -n -c 10 -j 10 -t 100 -f bench.sql postgres
Master (batch_size = 1 with a single row to insert):
tps = 16000.768037
Master (batch_size = 1 with 1000 rows to insert):
tps = 133.451518
Master (batch_size = 100 with 1000 rows to insert):
tps = 1274.096347
-----------------
Patch(batch_size = 1, use_copy_for_insert = false with single row to
insert)
tps = 15734.155705
Master (batch_size = 1, use_copy_for_insert = false with 1000 rows to
insert):
tps = 132.644801
Master (batch_size = 100, use_copy_for_insert = false with 1000 rows to
insert):
tps = 1245.514591
-----------------
Patch(batch_size = 1, use_copy_for_insert = true with single row to
insert)
tps = 17604.394057
Master (batch_size = 1, use_copy_for_insert = true with 1000 rows to
insert):
tps = 88.998804
Master (batch_size = 100, use_copy_for_insert = true with 1000 rows to
insert):
tps = 2406.009249
-----------------
We can see that when batching inserting with the batch_size configured
properly we have a very significant performance improvement and when the
"use_copy_for_insert" option is disabled the performance are close
compared with master.
The problem is when the "batch_size" is 1 (default) and
"use_copy_for_insert" is enabled. This is because on this scenario we
are sending multiple COPY commands with a single row to the foreign
server.
One way to fix this would to decide at runtime (at
execute_foreign_modify()) if the COPY can be used based on the number of
rows being insert. I don't think that I like this option because it
would make the EXPLAIN output different when the ANALYZE option is used
since during planning time we don't have the number of rows being
inserted, so if just EXPLAIN(VERBOSE) is executed we would show the
INSERT as remote SQL, and if the ANALYZE is included and we have enough
rows to enable the COPY usage, the remote SQL would show the COPY
command.
Since the new "use_copy_for_insert" option is be disabled by default I
think that we could document this limitation and mention the performance
improvements when used correctly with the batch_size option.
Another option would be to use the COPY command only if the
"use_copy_for_insert" is true and also if the "batch_size" is > 1. We
would still have the performance issue if the user insert a single row
but we would close to less scenarios. The attached 0002 implement this
idea.
Thoughts?
--
Matheus Alcantara
EDB: http://www.enterprisedb.com
Attachments: download all (2 files, .tar.gz)
v5-0001-postgres_fdw-Enable-the-use-of-COPY-to-speed-up-i.patch (text/plain, 22.1K) download
From dffe3f86da95451c85277a17de7fda204b678a21 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Fri, 10 Oct 2025 16:07:08 -0300
Subject: [PATCH v5 1/2] postgres_fdw: Enable the use of COPY to speed up
inserts
---
contrib/postgres_fdw/deparse.c | 30 +++
.../postgres_fdw/expected/postgres_fdw.out | 168 ++++++++++++++-
contrib/postgres_fdw/option.c | 3 +
contrib/postgres_fdw/postgres_fdw.c | 191 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 74 +++++++
6 files changed, 452 insertions(+), 15 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index f2fb0051843..1cdf1d8cc8d 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,36 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index cd28126049d..bc99e278f00 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -50,6 +50,18 @@ CREATE TABLE "S 1"."T 4" (
c3 text,
CONSTRAINT t4_pkey PRIMARY KEY (c1)
);
+CREATE TABLE "S 1"."T 5"(
+ x int
+);
+CREATE TABLE "S 1"."T 6"(
+ id int not null,
+ note text,
+ value int NOT NULL
+);
+CREATE TABLE "S 1"."T 7"(
+ id int,
+ t text
+);
-- Disable autovacuum for these tables to avoid unexpected effects of that
ALTER TABLE "S 1"."T 1" SET (autovacuum_enabled = 'false');
ALTER TABLE "S 1"."T 2" SET (autovacuum_enabled = 'false');
@@ -132,6 +144,24 @@ CREATE FOREIGN TABLE ft7 (
c2 int NOT NULL,
c3 text
) SERVER loopback3 OPTIONS (schema_name 'S 1', table_name 'T 4');
+CREATE FOREIGN TABLE ft8 (
+ x int
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true');
+CREATE FOREIGN TABLE ft9 (
+ id int not null,
+ note text,
+ value int NOT NULL
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true');
+CREATE FOREIGN TABLE ft10 (
+ id int,
+ t text
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true');
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -205,16 +235,19 @@ ALTER FOREIGN TABLE ft2 OPTIONS (schema_name 'S 1', table_name 'T 1');
ALTER FOREIGN TABLE ft1 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
ALTER FOREIGN TABLE ft2 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
\det+
- List of foreign tables
- Schema | Table | Server | FDW options | Description
---------+-------+-----------+---------------------------------------+-------------
- public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
- public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
- public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
- public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
-(6 rows)
+ List of foreign tables
+ Schema | Table | Server | FDW options | Description
+--------+-------+-----------+-------------------------------------------------------------------+-------------
+ public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft10 | loopback | (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true') |
+ public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
+ public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
+ public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft8 | loopback | (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true') |
+ public | ft9 | loopback | (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true') |
+(9 rows)
-- Test that alteration of server options causes reconnection
-- Remote's errors might be non-English, so hide them to ensure stable results
@@ -12665,6 +12698,121 @@ ANALYZE analyze_ftable;
DROP FOREIGN TABLE analyze_ftable;
DROP TABLE analyze_table;
-- ===================================================================
+-- test for COPY usage to perform INSERT's
+-- ===================================================================
+-- Test that target attr is correctly used to build the COPY command
+ALTER FOREIGN TABLE ft8 DROP COLUMN x;
+ALTER FOREIGN TABLE ft8 add COLUMN x int;
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Insert on public.ft8 (actual rows=0.00 loops=1)
+ Remote SQL: COPY "S 1"."T 5"(x) FROM STDIN
+ Batch Size: 1
+ -> Function Scan on pg_catalog.generate_series i (actual rows=10.00 loops=1)
+ Output: NULL::integer, i.i
+ Function Call: generate_series(1, 10)
+(6 rows)
+
+SELECT * FROM ft8;
+ x
+----
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+-- Test outer of order columns and batch_size with COPY
+ALTER FOREIGN TABLE ft9 OPTIONS(ADD batch_size '10');
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
+SELECT g,
+ g * 2,
+ 'batch insert test data' || g
+FROM generate_series(1, 20) g;
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Insert on public.ft9 (actual rows=0.00 loops=1)
+ Remote SQL: COPY "S 1"."T 6"(id, note, value) FROM STDIN
+ Batch Size: 10
+ -> Function Scan on pg_catalog.generate_series g (actual rows=20.00 loops=1)
+ Output: g.g, ('batch insert test data'::text || (g.g)::text), (g.g * 2)
+ Function Call: generate_series(1, 20)
+(6 rows)
+
+SELECT * FROM ft9;
+ id | note | value
+----+--------------------------+-------
+ 1 | batch insert test data1 | 2
+ 2 | batch insert test data2 | 4
+ 3 | batch insert test data3 | 6
+ 4 | batch insert test data4 | 8
+ 5 | batch insert test data5 | 10
+ 6 | batch insert test data6 | 12
+ 7 | batch insert test data7 | 14
+ 8 | batch insert test data8 | 16
+ 9 | batch insert test data9 | 18
+ 10 | batch insert test data10 | 20
+ 11 | batch insert test data11 | 22
+ 12 | batch insert test data12 | 24
+ 13 | batch insert test data13 | 26
+ 14 | batch insert test data14 | 28
+ 15 | batch insert test data15 | 30
+ 16 | batch insert test data16 | 32
+ 17 | batch insert test data17 | 34
+ 18 | batch insert test data18 | 36
+ 19 | batch insert test data19 | 38
+ 20 | batch insert test data20 | 40
+(20 rows)
+
+-- Test buffer limit of copy data on COPYBUFSIZ
+INSERT INTO ft10 (id, t)
+SELECT s, repeat(md5(s::text), 10000) from generate_series(100, 103) s;
+SELECT COUNT(*) FROM ft10;
+ count
+-------
+ 4
+(1 row)
+
+-- Disable the use_copy_for_insert table option and check that the INSERT is
+-- used
+ALTER FOREIGN TABLE ft8 OPTIONS(DROP use_copy_for_insert);
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (10);
+ QUERY PLAN
+------------------------------------------------------
+ Insert on public.ft8
+ Remote SQL: INSERT INTO "S 1"."T 5"(x) VALUES ($1)
+ Batch Size: 1
+ -> Result
+ Output: NULL::integer, 10
+(5 rows)
+
+-- Enable the use_copy_for_insert for the foreign server and check that the
+-- COPY is used
+ALTER SERVER loopback OPTIONS(ADD use_copy_for_insert 'true');
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (20);
+ QUERY PLAN
+----------------------------------------------
+ Insert on public.ft8
+ Remote SQL: COPY "S 1"."T 5"(x) FROM STDIN
+ Batch Size: 1
+ -> Result
+ Output: NULL::integer, 20
+(5 rows)
+
+-- Reset state
+ALTER SERVER loopback OPTIONS(DROP use_copy_for_insert);
+ALTER FOREIGN TABLE ft8 OPTIONS(ADD use_copy_for_insert 'true');
+-- ===================================================================
-- test for postgres_fdw_get_connections function with check_conn = true
-- ===================================================================
-- Disable debug_discard_caches in order to manage remote connections
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 04788b7e8b3..de0f59332c3 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -263,6 +263,9 @@ InitPgFdwOptions(void)
/* batch_size is available on both server and table */
{"batch_size", ForeignServerRelationId, false},
{"batch_size", ForeignTableRelationId, false},
+ /* use_copy_for_insert is available on both server and table */
+ {"use_copy_for_insert", ForeignServerRelationId, false},
+ {"use_copy_for_insert", ForeignTableRelationId, false},
/* async_capable is available on both server and table */
{"async_capable", ForeignServerRelationId, false},
{"async_capable", ForeignTableRelationId, false},
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 06b52c65300..77effdffeb2 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -197,6 +200,7 @@ typedef struct PgFdwModifyState
int batch_size; /* value of FDW option "batch_size" */
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy_for_insert; /* is the COPY enabled for INSERT's? */
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
@@ -545,6 +549,10 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static bool get_use_copy_for_insert(Relation rel);
+static TupleTableSlot **execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -1788,6 +1796,7 @@ postgresPlanForeignModify(PlannerInfo *root,
List *retrieved_attrs = NIL;
bool doNothing = false;
int values_end_len = -1;
+ bool use_copy_for_insert = false;
initStringInfo(&sql);
@@ -1867,17 +1876,25 @@ postgresPlanForeignModify(PlannerInfo *root,
elog(ERROR, "unexpected ON CONFLICT specification: %d",
(int) plan->onConflictAction);
+ if (operation == CMD_INSERT && plan->returningLists == NULL)
+ use_copy_for_insert = get_use_copy_for_insert(rel);
+
/*
* Construct the SQL command string.
*/
switch (operation)
{
case CMD_INSERT:
- deparseInsertSql(&sql, rte, resultRelation, rel,
- targetAttrs, doNothing,
- withCheckOptionList, returningList,
- &retrieved_attrs, &values_end_len);
- break;
+ {
+ if (use_copy_for_insert)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel,
+ targetAttrs, doNothing,
+ withCheckOptionList, returningList,
+ &retrieved_attrs, &values_end_len);
+ break;
+ }
case CMD_UPDATE:
deparseUpdateSql(&sql, rte, resultRelation, rel,
targetAttrs,
@@ -4058,6 +4075,9 @@ create_foreign_modify(EState *estate,
if (operation == CMD_INSERT)
fmstate->batch_size = get_batch_size_option(rel);
+ if (operation == CMD_INSERT && !fmstate->has_returning)
+ fmstate->use_copy_for_insert = get_use_copy_for_insert(rel);
+
fmstate->num_slots = 1;
/* Initialize auxiliary state */
@@ -4066,6 +4086,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,6 +4161,14 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /* Check if the COPY command is enabled to use for INSERT's */
+ if (operation == CMD_INSERT && fmstate->use_copy_for_insert)
+ {
+ /* COPY should only be used with INSERT without RETURNING clause. */
+ Assert(!fmstate->has_returning);
+ return execute_foreign_insert_using_copy(fmstate, slots, numSlots);
+ }
+
/*
* If the existing query was deparsed and prepared for a different number
* of rows, rebuild it for the proper number.
@@ -7886,3 +7958,112 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Determine if the usage of the COPY command to execute a INSERT into a foreign
+ * table is enabled. The option specified for a table has precedence.
+ */
+static bool
+get_use_copy_for_insert(Relation rel)
+{
+ Oid foreigntableid = RelationGetRelid(rel);
+ List *options = NIL;
+ ListCell *lc;
+ ForeignTable *table;
+ ForeignServer *server;
+ bool enable_batch_with_copy = false;
+
+ /*
+ * Load options for table and server. We append server options after table
+ * options, because table options take precedence.
+ */
+ table = GetForeignTable(foreigntableid);
+ server = GetForeignServer(table->serverid);
+
+ options = list_concat(options, table->options);
+ options = list_concat(options, server->options);
+
+ /* See if either table or server specifies enable_batch_with_copy. */
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "use_copy_for_insert") == 0)
+ {
+ (void) parse_bool(defGetString(def), &enable_batch_with_copy);
+ break;
+ }
+ }
+ return enable_batch_with_copy;
+}
+
+/* Execute an insert into a foreign table using the COPY command */
+static TupleTableSlot **
+execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..aa54d6bba53 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 9a8f9e28135..0d29b9d9bae 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -54,6 +54,18 @@ CREATE TABLE "S 1"."T 4" (
c3 text,
CONSTRAINT t4_pkey PRIMARY KEY (c1)
);
+CREATE TABLE "S 1"."T 5"(
+ x int
+);
+CREATE TABLE "S 1"."T 6"(
+ id int not null,
+ note text,
+ value int NOT NULL
+);
+CREATE TABLE "S 1"."T 7"(
+ id int,
+ t text
+);
-- Disable autovacuum for these tables to avoid unexpected effects of that
ALTER TABLE "S 1"."T 1" SET (autovacuum_enabled = 'false');
@@ -146,6 +158,27 @@ CREATE FOREIGN TABLE ft7 (
c3 text
) SERVER loopback3 OPTIONS (schema_name 'S 1', table_name 'T 4');
+CREATE FOREIGN TABLE ft8 (
+ x int
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true');
+
+CREATE FOREIGN TABLE ft9 (
+ id int not null,
+ note text,
+ value int NOT NULL
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true');
+
+CREATE FOREIGN TABLE ft10 (
+ id int,
+ t text
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true');
+
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -4379,6 +4412,47 @@ ANALYZE analyze_ftable;
DROP FOREIGN TABLE analyze_ftable;
DROP TABLE analyze_table;
+-- ===================================================================
+-- test for COPY usage to perform INSERT's
+-- ===================================================================
+
+-- Test that target attr is correctly used to build the COPY command
+ALTER FOREIGN TABLE ft8 DROP COLUMN x;
+ALTER FOREIGN TABLE ft8 add COLUMN x int;
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
+SELECT * FROM ft8;
+
+-- Test outer of order columns and batch_size with COPY
+ALTER FOREIGN TABLE ft9 OPTIONS(ADD batch_size '10');
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
+SELECT g,
+ g * 2,
+ 'batch insert test data' || g
+FROM generate_series(1, 20) g;
+SELECT * FROM ft9;
+
+-- Test buffer limit of copy data on COPYBUFSIZ
+INSERT INTO ft10 (id, t)
+SELECT s, repeat(md5(s::text), 10000) from generate_series(100, 103) s;
+SELECT COUNT(*) FROM ft10;
+
+-- Disable the use_copy_for_insert table option and check that the INSERT is
+-- used
+ALTER FOREIGN TABLE ft8 OPTIONS(DROP use_copy_for_insert);
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (10);
+
+-- Enable the use_copy_for_insert for the foreign server and check that the
+-- COPY is used
+ALTER SERVER loopback OPTIONS(ADD use_copy_for_insert 'true');
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (20);
+
+-- Reset state
+ALTER SERVER loopback OPTIONS(DROP use_copy_for_insert);
+ALTER FOREIGN TABLE ft8 OPTIONS(ADD use_copy_for_insert 'true');
+
-- ===================================================================
-- test for postgres_fdw_get_connections function with check_conn = true
-- ===================================================================
--
2.51.2
v5-0002-postgres_fdw-Only-use-COPY-if-batch_size-is-1.patch (text/plain, 8.3K) download
From 1fb93c8b1548b836c68c3e2e4c124eb57f2ee518 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Thu, 6 Nov 2025 20:17:19 -0300
Subject: [PATCH v5 2/2] postgres_fdw: Only use COPY if batch_size is > 1
---
.../postgres_fdw/expected/postgres_fdw.out | 39 +++++++++----------
contrib/postgres_fdw/postgres_fdw.c | 9 ++++-
contrib/postgres_fdw/sql/postgres_fdw.sql | 9 ++---
3 files changed, 30 insertions(+), 27 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bc99e278f00..956ff12b590 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -148,20 +148,20 @@ CREATE FOREIGN TABLE ft8 (
x int
)
SERVER loopback
-OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true');
+OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true', batch_size '10');
CREATE FOREIGN TABLE ft9 (
id int not null,
note text,
value int NOT NULL
)
SERVER loopback
-OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true');
+OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true', batch_size '10');
CREATE FOREIGN TABLE ft10 (
id int,
t text
)
SERVER loopback
-OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true');
+OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true', batch_size '10');
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -235,18 +235,18 @@ ALTER FOREIGN TABLE ft2 OPTIONS (schema_name 'S 1', table_name 'T 1');
ALTER FOREIGN TABLE ft1 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
ALTER FOREIGN TABLE ft2 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
\det+
- List of foreign tables
- Schema | Table | Server | FDW options | Description
---------+-------+-----------+-------------------------------------------------------------------+-------------
- public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft10 | loopback | (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true') |
- public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
- public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
- public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
- public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
- public | ft8 | loopback | (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true') |
- public | ft9 | loopback | (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true') |
+ List of foreign tables
+ Schema | Table | Server | FDW options | Description
+--------+-------+-----------+------------------------------------------------------------------------------------+-------------
+ public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft10 | loopback | (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true', batch_size '10') |
+ public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
+ public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
+ public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft8 | loopback | (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true', batch_size '10') |
+ public | ft9 | loopback | (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true', batch_size '10') |
(9 rows)
-- Test that alteration of server options causes reconnection
@@ -12709,7 +12709,7 @@ INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
---------------------------------------------------------------------------------
Insert on public.ft8 (actual rows=0.00 loops=1)
Remote SQL: COPY "S 1"."T 5"(x) FROM STDIN
- Batch Size: 1
+ Batch Size: 10
-> Function Scan on pg_catalog.generate_series i (actual rows=10.00 loops=1)
Output: NULL::integer, i.i
Function Call: generate_series(1, 10)
@@ -12730,8 +12730,7 @@ SELECT * FROM ft8;
10
(10 rows)
--- Test outer of order columns and batch_size with COPY
-ALTER FOREIGN TABLE ft9 OPTIONS(ADD batch_size '10');
+-- Test outer of order columns
EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
SELECT g,
g * 2,
@@ -12790,7 +12789,7 @@ INSERT INTO ft8 VALUES (10);
------------------------------------------------------
Insert on public.ft8
Remote SQL: INSERT INTO "S 1"."T 5"(x) VALUES ($1)
- Batch Size: 1
+ Batch Size: 10
-> Result
Output: NULL::integer, 10
(5 rows)
@@ -12804,7 +12803,7 @@ INSERT INTO ft8 VALUES (20);
----------------------------------------------
Insert on public.ft8
Remote SQL: COPY "S 1"."T 5"(x) FROM STDIN
- Batch Size: 1
+ Batch Size: 10
-> Result
Output: NULL::integer, 20
(5 rows)
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 77effdffeb2..4a89522a221 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -1877,7 +1877,12 @@ postgresPlanForeignModify(PlannerInfo *root,
(int) plan->onConflictAction);
if (operation == CMD_INSERT && plan->returningLists == NULL)
- use_copy_for_insert = get_use_copy_for_insert(rel);
+ {
+ int batch_size = get_batch_size_option(rel);
+
+ if (batch_size > 1)
+ use_copy_for_insert = get_use_copy_for_insert(rel);
+ }
/*
* Construct the SQL command string.
@@ -4075,7 +4080,7 @@ create_foreign_modify(EState *estate,
if (operation == CMD_INSERT)
fmstate->batch_size = get_batch_size_option(rel);
- if (operation == CMD_INSERT && !fmstate->has_returning)
+ if (operation == CMD_INSERT && !fmstate->has_returning && fmstate->batch_size > 1)
fmstate->use_copy_for_insert = get_use_copy_for_insert(rel);
fmstate->num_slots = 1;
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 0d29b9d9bae..093f86abeb4 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -162,7 +162,7 @@ CREATE FOREIGN TABLE ft8 (
x int
)
SERVER loopback
-OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true');
+OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true', batch_size '10');
CREATE FOREIGN TABLE ft9 (
id int not null,
@@ -170,14 +170,14 @@ CREATE FOREIGN TABLE ft9 (
value int NOT NULL
)
SERVER loopback
-OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true');
+OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true', batch_size '10');
CREATE FOREIGN TABLE ft10 (
id int,
t text
)
SERVER loopback
-OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true');
+OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true', batch_size '10');
-- ===================================================================
-- tests for validator
@@ -4423,8 +4423,7 @@ EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
SELECT * FROM ft8;
--- Test outer of order columns and batch_size with COPY
-ALTER FOREIGN TABLE ft9 OPTIONS(ADD batch_size '10');
+-- Test outer of order columns
EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
SELECT g,
g * 2,
--
2.51.2
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-11-18 02:03 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2025-11-18 02:03 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Thu, Nov 6, 2025 at 3:49 PM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On Fri Oct 31, 2025 at 4:02 PM -03, I wrote:
> > It's showing a bit complicated to decide at runtime if we should use the
> > COPY or INSERT for batch insert into a foreign table. Perhaps we could
> > add a new option on CREATE FOREIGN TABLE to enable this usage or not? We
> > could document the performance improvements and the limitations so the
> > user can decide if it should enable or not.
> >
> Here is v5 that implement this idea.
>
> On this version I've introduced a foreign table and foreign server
> option "use_copy_for_insert" (I'm open for a better name) that enable
> the use of the COPY as remote command to execute an INSERT into a
> foreign table. The COPY can be used if the user enable this option on
> the foreign table or the foreign server and if the original INSERT
> statement don't have a RETURNING clause.
>
> See the benchmark results:
>
> pgbench -n -c 10 -j 10 -t 100 -f bench.sql postgres
>
> Master (batch_size = 1 with a single row to insert):
> tps = 16000.768037
>
> Master (batch_size = 1 with 1000 rows to insert):
> tps = 133.451518
>
> Master (batch_size = 100 with 1000 rows to insert):
> tps = 1274.096347
>
> -----------------
>
> Patch(batch_size = 1, use_copy_for_insert = false with single row to
> insert)
> tps = 15734.155705
>
> Master (batch_size = 1, use_copy_for_insert = false with 1000 rows to
> insert):
> tps = 132.644801
>
> Master (batch_size = 100, use_copy_for_insert = false with 1000 rows to
> insert):
> tps = 1245.514591
>
> -----------------
>
> Patch(batch_size = 1, use_copy_for_insert = true with single row to
> insert)
> tps = 17604.394057
>
> Master (batch_size = 1, use_copy_for_insert = true with 1000 rows to
> insert):
> tps = 88.998804
>
> Master (batch_size = 100, use_copy_for_insert = true with 1000 rows to
> insert):
> tps = 2406.009249
>
> -----------------
>
> We can see that when batching inserting with the batch_size configured
> properly we have a very significant performance improvement and when the
> "use_copy_for_insert" option is disabled the performance are close
> compared with master.
>
> The problem is when the "batch_size" is 1 (default) and
> "use_copy_for_insert" is enabled. This is because on this scenario we
> are sending multiple COPY commands with a single row to the foreign
> server.
>
> One way to fix this would to decide at runtime (at
> execute_foreign_modify()) if the COPY can be used based on the number of
> rows being insert. I don't think that I like this option because it
> would make the EXPLAIN output different when the ANALYZE option is used
> since during planning time we don't have the number of rows being
> inserted, so if just EXPLAIN(VERBOSE) is executed we would show the
> INSERT as remote SQL, and if the ANALYZE is included and we have enough
> rows to enable the COPY usage, the remote SQL would show the COPY
> command.
>
> Since the new "use_copy_for_insert" option is be disabled by default I
> think that we could document this limitation and mention the performance
> improvements when used correctly with the batch_size option.
>
> Another option would be to use the COPY command only if the
> "use_copy_for_insert" is true and also if the "batch_size" is > 1. We
> would still have the performance issue if the user insert a single row
> but we would close to less scenarios. The attached 0002 implement this
> idea.
>
> Thoughts?
IIUC the performance regression occurs when users insert many rows
into a foreign table with batch_size = 1 and use_copy_for_insert =
true (tps: 133.451518 vs. 132.644801 vs. 88.998804). Since batch_size
defaults to 1, users might experience performance issues if they
enable use_copy_for_insert without adjusting the batch_size. I'm
worried that users could easily find themselves in this situation.
One possible solution would be to introduce a threshold, like
copy_min_row, which would specify the minimum number of rows needed
before switching to the COPY command. However, this would require
coordination with batch_size since having copy_min_row lower than
batch_size wouldn't make sense. Alternatively, when users are using
batch insertion (batch_size > 0), we could use the COPY command only
for full batches and fall back to INSERT for partial ones.
BTW I noticed that use_copy_for_insert option doesn't work with COPY
FROM command. I got the following error with use_copy_for_insert=true
and batch_size=3:
postgres(1:2546195)=# copy t from '/tmp/a.csv'; -- table 't' is a foreign table.
ERROR: there is no parameter $1
CONTEXT: remote SQL command: INSERT INTO public.t(c) VALUES ($1)
COPY t
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2025-11-18 22:13 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-11-18 22:13 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Mon Nov 17, 2025 at 11:03 PM -03, Masahiko Sawada wrote:
> IIUC the performance regression occurs when users insert many rows
> into a foreign table with batch_size = 1 and use_copy_for_insert =
> true (tps: 133.451518 vs. 132.644801 vs. 88.998804). Since batch_size
> defaults to 1, users might experience performance issues if they
> enable use_copy_for_insert without adjusting the batch_size. I'm
> worried that users could easily find themselves in this situation.
>
Yes, you are correct. The 0002 patch aims to reduce this issue by using
the COPY command only if the use_copy_for_insert = true and if
batch_size > 1 which will reduce the cases but the regression can still
happen if the user send a single row to insert into a foreign table.
Inserting a single row into a foreign table using COPY is a bit slower
compared with using INSERT. See the followinw pgbench results:
(Single row using INSERT)
tps = 19814.535944
(Single row using COPY)
tps = 16562.324025
I think that the documentation should mention that just changing
use_copy_for_insert without also changing the batch_size option could
cause performance regression.
> One possible solution would be to introduce a threshold, like
> copy_min_row, which would specify the minimum number of rows needed
> before switching to the COPY command. However, this would require
> coordination with batch_size since having copy_min_row lower than
> batch_size wouldn't make sense.
>
The only problem that I see with this approach is that it would make
EXPLAIN(VERBOSE) and EXPLAIN(ANALYZE, VERBOSE) remote SQL output
different. The user will never know with EXPLAIN (without analyze) if
the COPY will be used or not. Is this a problem or I'm being to much
conservative?
I think that we can do such coordination on postgres_fdw_validator().
Also if we decide to go with this idea it seems to me that we would have
to much table options to configure to enable the COPY opitimization, we
would need "copy_min_row", "batch_size" and "use_copy_for_insert". What
about decide to use the COPY command if use_copy_for_insert = true and
the number of rows being inserted is >= batch_size?
> Alternatively, when users are using batch insertion (batch_size > 0),
> we could use the COPY command only for full batches and fall back to
> INSERT for partial ones.
>
IIUC in this case we would sent COPY and INSERT statements to the
foreign server for the same execution, for example, if batch_size = 100
and the user try insert 105 rows into the foreign table we will send a
COPY statement with 100 rows and then an INSERT with the 5 rows
remaining? If that's the case which SQL we should show on Remote SQL
from EXPLAIN(ANALYZE, VERBOSE) output? I think that this can cause some
confusion.
> BTW I noticed that use_copy_for_insert option doesn't work with COPY
> FROM command. I got the following error with use_copy_for_insert=true
> and batch_size=3:
>
> postgres(1:2546195)=# copy t from '/tmp/a.csv'; -- table 't' is a foreign table.
> ERROR: there is no parameter $1
> CONTEXT: remote SQL command: INSERT INTO public.t(c) VALUES ($1)
> COPY t
>
Thanks for testing this case. The problem was that I as checking if the
COPY can be used inside create_foreign_modify() that is called by
BeginForeignInsert and also BeginForeignModify() and the COPY can be
used only by the foreign modify path. To fix this issue I've moved the
check to postgresBeginForeignModify().
I'm attaching v6 with the following changes:
- I've squashed 0002 into 0001, so now the COPY will only be used if
use_copy_for_insert = true and if batch_size > 1
- Fix for the bug of COPY FROM a foreign table
- New test case for the COPY bug
--
Matheus Alcantara
EDB: http://www.enterprisedb.com
Attachments:
v6-0001-postgres_fdw-Enable-the-use-of-COPY-to-speed-up-i.patch (text/plain, 22.8K) download
From dd29e8b3b0092a4ba54cef9b00892f577d2461da Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Fri, 10 Oct 2025 16:07:08 -0300
Subject: [PATCH v6] postgres_fdw: Enable the use of COPY to speed up inserts
---
contrib/postgres_fdw/deparse.c | 30 +++
.../postgres_fdw/expected/postgres_fdw.out | 170 ++++++++++++++-
contrib/postgres_fdw/option.c | 3 +
contrib/postgres_fdw/postgres_fdw.c | 198 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 79 +++++++
6 files changed, 466 insertions(+), 15 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index f2fb0051843..1cdf1d8cc8d 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,36 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index cd28126049d..0648a964522 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -50,6 +50,18 @@ CREATE TABLE "S 1"."T 4" (
c3 text,
CONSTRAINT t4_pkey PRIMARY KEY (c1)
);
+CREATE TABLE "S 1"."T 5"(
+ x int
+);
+CREATE TABLE "S 1"."T 6"(
+ id int not null,
+ note text,
+ value int NOT NULL
+);
+CREATE TABLE "S 1"."T 7"(
+ id int,
+ t text
+);
-- Disable autovacuum for these tables to avoid unexpected effects of that
ALTER TABLE "S 1"."T 1" SET (autovacuum_enabled = 'false');
ALTER TABLE "S 1"."T 2" SET (autovacuum_enabled = 'false');
@@ -132,6 +144,24 @@ CREATE FOREIGN TABLE ft7 (
c2 int NOT NULL,
c3 text
) SERVER loopback3 OPTIONS (schema_name 'S 1', table_name 'T 4');
+CREATE FOREIGN TABLE ft8 (
+ x int
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true', batch_size '10');
+CREATE FOREIGN TABLE ft9 (
+ id int not null,
+ note text,
+ value int NOT NULL
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true', batch_size '10');
+CREATE FOREIGN TABLE ft10 (
+ id int,
+ t text
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true', batch_size '10');
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -205,16 +235,19 @@ ALTER FOREIGN TABLE ft2 OPTIONS (schema_name 'S 1', table_name 'T 1');
ALTER FOREIGN TABLE ft1 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
ALTER FOREIGN TABLE ft2 ALTER COLUMN c1 OPTIONS (column_name 'C 1');
\det+
- List of foreign tables
- Schema | Table | Server | FDW options | Description
---------+-------+-----------+---------------------------------------+-------------
- public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
- public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
- public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
- public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
- public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
-(6 rows)
+ List of foreign tables
+ Schema | Table | Server | FDW options | Description
+--------+-------+-----------+------------------------------------------------------------------------------------+-------------
+ public | ft1 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft10 | loopback | (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true', batch_size '10') |
+ public | ft2 | loopback | (schema_name 'S 1', table_name 'T 1') |
+ public | ft4 | loopback | (schema_name 'S 1', table_name 'T 3') |
+ public | ft5 | loopback | (schema_name 'S 1', table_name 'T 4') |
+ public | ft6 | loopback2 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft7 | loopback3 | (schema_name 'S 1', table_name 'T 4') |
+ public | ft8 | loopback | (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true', batch_size '10') |
+ public | ft9 | loopback | (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true', batch_size '10') |
+(9 rows)
-- Test that alteration of server options causes reconnection
-- Remote's errors might be non-English, so hide them to ensure stable results
@@ -12665,6 +12698,123 @@ ANALYZE analyze_ftable;
DROP FOREIGN TABLE analyze_ftable;
DROP TABLE analyze_table;
-- ===================================================================
+-- test for COPY usage to perform INSERT's
+-- ===================================================================
+-- Test that target attr is correctly used to build the COPY command
+ALTER FOREIGN TABLE ft8 DROP COLUMN x;
+ALTER FOREIGN TABLE ft8 add COLUMN x int;
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Insert on public.ft8 (actual rows=0.00 loops=1)
+ Remote SQL: COPY "S 1"."T 5"(x) FROM STDIN
+ Batch Size: 10
+ -> Function Scan on pg_catalog.generate_series i (actual rows=10.00 loops=1)
+ Output: NULL::integer, i.i
+ Function Call: generate_series(1, 10)
+(6 rows)
+
+SELECT * FROM ft8;
+ x
+----
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+-- Test outer of order columns
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
+SELECT g,
+ g * 2,
+ 'batch insert test data' || g
+FROM generate_series(1, 20) g;
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Insert on public.ft9 (actual rows=0.00 loops=1)
+ Remote SQL: COPY "S 1"."T 6"(id, note, value) FROM STDIN
+ Batch Size: 10
+ -> Function Scan on pg_catalog.generate_series g (actual rows=20.00 loops=1)
+ Output: g.g, ('batch insert test data'::text || (g.g)::text), (g.g * 2)
+ Function Call: generate_series(1, 20)
+(6 rows)
+
+SELECT * FROM ft9;
+ id | note | value
+----+--------------------------+-------
+ 1 | batch insert test data1 | 2
+ 2 | batch insert test data2 | 4
+ 3 | batch insert test data3 | 6
+ 4 | batch insert test data4 | 8
+ 5 | batch insert test data5 | 10
+ 6 | batch insert test data6 | 12
+ 7 | batch insert test data7 | 14
+ 8 | batch insert test data8 | 16
+ 9 | batch insert test data9 | 18
+ 10 | batch insert test data10 | 20
+ 11 | batch insert test data11 | 22
+ 12 | batch insert test data12 | 24
+ 13 | batch insert test data13 | 26
+ 14 | batch insert test data14 | 28
+ 15 | batch insert test data15 | 30
+ 16 | batch insert test data16 | 32
+ 17 | batch insert test data17 | 34
+ 18 | batch insert test data18 | 36
+ 19 | batch insert test data19 | 38
+ 20 | batch insert test data20 | 40
+(20 rows)
+
+-- Test buffer limit of copy data on COPYBUFSIZ
+INSERT INTO ft10 (id, t)
+SELECT s, repeat(md5(s::text), 10000) from generate_series(100, 103) s;
+SELECT COUNT(*) FROM ft10;
+ count
+-------
+ 4
+(1 row)
+
+-- Disable the use_copy_for_insert table option and check that the INSERT is
+-- used
+ALTER FOREIGN TABLE ft8 OPTIONS(DROP use_copy_for_insert);
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (10);
+ QUERY PLAN
+------------------------------------------------------
+ Insert on public.ft8
+ Remote SQL: INSERT INTO "S 1"."T 5"(x) VALUES ($1)
+ Batch Size: 10
+ -> Result
+ Output: NULL::integer, 10
+(5 rows)
+
+-- Enable the use_copy_for_insert for the foreign server and check that the
+-- COPY is used
+ALTER SERVER loopback OPTIONS(ADD use_copy_for_insert 'true');
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (20);
+ QUERY PLAN
+----------------------------------------------
+ Insert on public.ft8
+ Remote SQL: COPY "S 1"."T 5"(x) FROM STDIN
+ Batch Size: 10
+ -> Result
+ Output: NULL::integer, 20
+(5 rows)
+
+-- Check that COPY work correctly for a foreign table that has
+-- use_copy_for_insert enabled
+COPY ft8(x) FROM stdin;
+-- Reset state
+ALTER SERVER loopback OPTIONS(DROP use_copy_for_insert);
+ALTER FOREIGN TABLE ft8 OPTIONS(ADD use_copy_for_insert 'true');
+-- ===================================================================
-- test for postgres_fdw_get_connections function with check_conn = true
-- ===================================================================
-- Disable debug_discard_caches in order to manage remote connections
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 04788b7e8b3..de0f59332c3 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -263,6 +263,9 @@ InitPgFdwOptions(void)
/* batch_size is available on both server and table */
{"batch_size", ForeignServerRelationId, false},
{"batch_size", ForeignTableRelationId, false},
+ /* use_copy_for_insert is available on both server and table */
+ {"use_copy_for_insert", ForeignServerRelationId, false},
+ {"use_copy_for_insert", ForeignTableRelationId, false},
/* async_capable is available on both server and table */
{"async_capable", ForeignServerRelationId, false},
{"async_capable", ForeignTableRelationId, false},
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 06b52c65300..64174727d09 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -197,6 +200,7 @@ typedef struct PgFdwModifyState
int batch_size; /* value of FDW option "batch_size" */
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy_for_insert; /* is the COPY enabled for INSERT's? */
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
@@ -545,6 +549,10 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static bool get_use_copy_for_insert(Relation rel);
+static TupleTableSlot **execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -1788,6 +1796,7 @@ postgresPlanForeignModify(PlannerInfo *root,
List *retrieved_attrs = NIL;
bool doNothing = false;
int values_end_len = -1;
+ bool use_copy_for_insert = false;
initStringInfo(&sql);
@@ -1867,17 +1876,30 @@ postgresPlanForeignModify(PlannerInfo *root,
elog(ERROR, "unexpected ON CONFLICT specification: %d",
(int) plan->onConflictAction);
+ if (operation == CMD_INSERT && plan->returningLists == NULL)
+ {
+ int batch_size = get_batch_size_option(rel);
+
+ if (batch_size > 1)
+ use_copy_for_insert = get_use_copy_for_insert(rel);
+ }
+
/*
* Construct the SQL command string.
*/
switch (operation)
{
case CMD_INSERT:
- deparseInsertSql(&sql, rte, resultRelation, rel,
- targetAttrs, doNothing,
- withCheckOptionList, returningList,
- &retrieved_attrs, &values_end_len);
- break;
+ {
+ if (use_copy_for_insert)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel,
+ targetAttrs, doNothing,
+ withCheckOptionList, returningList,
+ &retrieved_attrs, &values_end_len);
+ break;
+ }
case CMD_UPDATE:
deparseUpdateSql(&sql, rte, resultRelation, rel,
targetAttrs,
@@ -1925,6 +1947,7 @@ postgresBeginForeignModify(ModifyTableState *mtstate,
int values_end_len;
List *retrieved_attrs;
RangeTblEntry *rte;
+ Relation rel = resultRelInfo->ri_RelationDesc;
/*
* Do nothing in EXPLAIN (no ANALYZE) case. resultRelInfo->ri_FdwState
@@ -1961,6 +1984,10 @@ postgresBeginForeignModify(ModifyTableState *mtstate,
has_returning,
retrieved_attrs);
+ /* Can COPY be used for batch insert? */
+ if (mtstate->operation == CMD_INSERT && !fmstate->has_returning && fmstate->batch_size > 1)
+ fmstate->use_copy_for_insert = get_use_copy_for_insert(rel);
+
resultRelInfo->ri_FdwState = fmstate;
}
@@ -4066,6 +4093,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,6 +4168,14 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /* Check if the COPY command is enabled to use for INSERT's */
+ if (operation == CMD_INSERT && fmstate->use_copy_for_insert)
+ {
+ /* COPY should only be used with INSERT without RETURNING clause. */
+ Assert(!fmstate->has_returning);
+ return execute_foreign_insert_using_copy(fmstate, slots, numSlots);
+ }
+
/*
* If the existing query was deparsed and prepared for a different number
* of rows, rebuild it for the proper number.
@@ -7886,3 +7965,112 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Determine if the usage of the COPY command to execute a INSERT into a foreign
+ * table is enabled. The option specified for a table has precedence.
+ */
+static bool
+get_use_copy_for_insert(Relation rel)
+{
+ Oid foreigntableid = RelationGetRelid(rel);
+ List *options = NIL;
+ ListCell *lc;
+ ForeignTable *table;
+ ForeignServer *server;
+ bool enable_batch_with_copy = false;
+
+ /*
+ * Load options for table and server. We append server options after table
+ * options, because table options take precedence.
+ */
+ table = GetForeignTable(foreigntableid);
+ server = GetForeignServer(table->serverid);
+
+ options = list_concat(options, table->options);
+ options = list_concat(options, server->options);
+
+ /* See if either table or server specifies enable_batch_with_copy. */
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "use_copy_for_insert") == 0)
+ {
+ (void) parse_bool(defGetString(def), &enable_batch_with_copy);
+ break;
+ }
+ }
+ return enable_batch_with_copy;
+}
+
+/* Execute an insert into a foreign table using the COPY command */
+static TupleTableSlot **
+execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..aa54d6bba53 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 9a8f9e28135..0c483cd49a5 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -54,6 +54,18 @@ CREATE TABLE "S 1"."T 4" (
c3 text,
CONSTRAINT t4_pkey PRIMARY KEY (c1)
);
+CREATE TABLE "S 1"."T 5"(
+ x int
+);
+CREATE TABLE "S 1"."T 6"(
+ id int not null,
+ note text,
+ value int NOT NULL
+);
+CREATE TABLE "S 1"."T 7"(
+ id int,
+ t text
+);
-- Disable autovacuum for these tables to avoid unexpected effects of that
ALTER TABLE "S 1"."T 1" SET (autovacuum_enabled = 'false');
@@ -146,6 +158,27 @@ CREATE FOREIGN TABLE ft7 (
c3 text
) SERVER loopback3 OPTIONS (schema_name 'S 1', table_name 'T 4');
+CREATE FOREIGN TABLE ft8 (
+ x int
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 5', use_copy_for_insert 'true', batch_size '10');
+
+CREATE FOREIGN TABLE ft9 (
+ id int not null,
+ note text,
+ value int NOT NULL
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 6', use_copy_for_insert 'true', batch_size '10');
+
+CREATE FOREIGN TABLE ft10 (
+ id int,
+ t text
+)
+SERVER loopback
+OPTIONS (schema_name 'S 1', table_name 'T 7', use_copy_for_insert 'true', batch_size '10');
+
-- ===================================================================
-- tests for validator
-- ===================================================================
@@ -4379,6 +4412,52 @@ ANALYZE analyze_ftable;
DROP FOREIGN TABLE analyze_ftable;
DROP TABLE analyze_table;
+-- ===================================================================
+-- test for COPY usage to perform INSERT's
+-- ===================================================================
+
+-- Test that target attr is correctly used to build the COPY command
+ALTER FOREIGN TABLE ft8 DROP COLUMN x;
+ALTER FOREIGN TABLE ft8 add COLUMN x int;
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 SELECT * FROM generate_series(1, 10) i;
+SELECT * FROM ft8;
+
+-- Test outer of order columns
+EXPLAIN(ANALYZE, VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF) INSERT INTO ft9 (id, value, note)
+SELECT g,
+ g * 2,
+ 'batch insert test data' || g
+FROM generate_series(1, 20) g;
+SELECT * FROM ft9;
+
+-- Test buffer limit of copy data on COPYBUFSIZ
+INSERT INTO ft10 (id, t)
+SELECT s, repeat(md5(s::text), 10000) from generate_series(100, 103) s;
+SELECT COUNT(*) FROM ft10;
+
+-- Disable the use_copy_for_insert table option and check that the INSERT is
+-- used
+ALTER FOREIGN TABLE ft8 OPTIONS(DROP use_copy_for_insert);
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (10);
+
+-- Enable the use_copy_for_insert for the foreign server and check that the
+-- COPY is used
+ALTER SERVER loopback OPTIONS(ADD use_copy_for_insert 'true');
+EXPLAIN(VERBOSE, COSTS OFF, SUMMARY OFF, BUFFERS OFF, TIMING OFF)
+INSERT INTO ft8 VALUES (20);
+
+-- Check that COPY work correctly for a foreign table that has
+-- use_copy_for_insert enabled
+COPY ft8(x) FROM stdin;
+30
+\.
+
+-- Reset state
+ALTER SERVER loopback OPTIONS(DROP use_copy_for_insert);
+ALTER FOREIGN TABLE ft8 OPTIONS(ADD use_copy_for_insert 'true');
+
-- ===================================================================
-- test for postgres_fdw_get_connections function with check_conn = true
-- ===================================================================
--
2.51.2
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-11-19 23:32 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2025-11-19 23:32 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Tue, Nov 18, 2025 at 2:13 PM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On Mon Nov 17, 2025 at 11:03 PM -03, Masahiko Sawada wrote:
> > IIUC the performance regression occurs when users insert many rows
> > into a foreign table with batch_size = 1 and use_copy_for_insert =
> > true (tps: 133.451518 vs. 132.644801 vs. 88.998804). Since batch_size
> > defaults to 1, users might experience performance issues if they
> > enable use_copy_for_insert without adjusting the batch_size. I'm
> > worried that users could easily find themselves in this situation.
> >
> Yes, you are correct. The 0002 patch aims to reduce this issue by using
> the COPY command only if the use_copy_for_insert = true and if
> batch_size > 1 which will reduce the cases but the regression can still
> happen if the user send a single row to insert into a foreign table.
>
> Inserting a single row into a foreign table using COPY is a bit slower
> compared with using INSERT. See the followinw pgbench results:
>
> (Single row using INSERT)
> tps = 19814.535944
>
> (Single row using COPY)
> tps = 16562.324025
>
> I think that the documentation should mention that just changing
> use_copy_for_insert without also changing the batch_size option could
> cause performance regression.
>
> > One possible solution would be to introduce a threshold, like
> > copy_min_row, which would specify the minimum number of rows needed
> > before switching to the COPY command. However, this would require
> > coordination with batch_size since having copy_min_row lower than
> > batch_size wouldn't make sense.
> >
> The only problem that I see with this approach is that it would make
> EXPLAIN(VERBOSE) and EXPLAIN(ANALYZE, VERBOSE) remote SQL output
> different. The user will never know with EXPLAIN (without analyze) if
> the COPY will be used or not. Is this a problem or I'm being to much
> conservative?
I think that's a valid concern. Is it a good idea to show both queries
with some additional information (e.g., threshold to switch using COPY
command)?
> I think that we can do such coordination on postgres_fdw_validator().
>
> Also if we decide to go with this idea it seems to me that we would have
> to much table options to configure to enable the COPY opitimization, we
> would need "copy_min_row", "batch_size" and "use_copy_for_insert". What
> about decide to use the COPY command if use_copy_for_insert = true and
> the number of rows being inserted is >= batch_size?
Sounds like a reasonable idea.
>
> > Alternatively, when users are using batch insertion (batch_size > 0),
> > we could use the COPY command only for full batches and fall back to
> > INSERT for partial ones.
> >
> IIUC in this case we would sent COPY and INSERT statements to the
> foreign server for the same execution, for example, if batch_size = 100
> and the user try insert 105 rows into the foreign table we will send a
> COPY statement with 100 rows and then an INSERT with the 5 rows
> remaining? If that's the case which SQL we should show on Remote SQL
> from EXPLAIN(ANALYZE, VERBOSE) output? I think that this can cause some
> confusion.
>
> > BTW I noticed that use_copy_for_insert option doesn't work with COPY
> > FROM command. I got the following error with use_copy_for_insert=true
> > and batch_size=3:
> >
> > postgres(1:2546195)=# copy t from '/tmp/a.csv'; -- table 't' is a foreign table.
> > ERROR: there is no parameter $1
> > CONTEXT: remote SQL command: INSERT INTO public.t(c) VALUES ($1)
> > COPY t
> >
> Thanks for testing this case. The problem was that I as checking if the
> COPY can be used inside create_foreign_modify() that is called by
> BeginForeignInsert and also BeginForeignModify() and the COPY can be
> used only by the foreign modify path. To fix this issue I've moved the
> check to postgresBeginForeignModify().
>
> I'm attaching v6 with the following changes:
> - I've squashed 0002 into 0001, so now the COPY will only be used if
> use_copy_for_insert = true and if batch_size > 1
> - Fix for the bug of COPY FROM a foreign table
> - New test case for the COPY bug
Thank you for updating the patch!
I think one key point in the patch is whether or not it's okay to
switch using COPY based on the actual number of tuples inserted. While
it should be okay from the performance perspective, it might be an
issue that the remote query shown in EXPLAIN (without ANALYZE) might
be different from the actual query sent. If there is a way to
distinguish the batch insertion between INSERT and COPY in
postgres_fdw, it might be a good idea to use COPY command for the
remote query only when the COPY FROM comes.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2025-11-26 20:51 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-11-26 20:51 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Wed Nov 19, 2025 at 8:32 PM -03, Masahiko Sawada wrote:
> I think one key point in the patch is whether or not it's okay to
> switch using COPY based on the actual number of tuples inserted. While
> it should be okay from the performance perspective, it might be an
> issue that the remote query shown in EXPLAIN (without ANALYZE) might
> be different from the actual query sent. If there is a way to
> distinguish the batch insertion between INSERT and COPY in
> postgres_fdw, it might be a good idea to use COPY command for the
> remote query only when the COPY FROM comes.
>
Yeah, I agree that this EXPLAIN inconsistency is an issue that for now
it doesn't seems easy to fix. That being said I've take a step back and
tried to reduce the scope of the patch to implement this idea of using
COPY as a remote sql when the user is executing a COPY FROM on a foreign
table.
My initial idea was to use the COPY as a remote SQL whenever an user
execute a COPY FROM on a foreign table but this can cause breaking
changes because the table on the foreign server may have triggers for
INSERT's and changing to use only the COPY FROM as remote sql would
break these cases. We have some test cases for this scenario.
So on this new version I introduced two new foreign table and server
options:
- use_copy_for_batch_insert: Enable the usage of COPY when
appropriate
- copy_for_batch_insert_threshold: The number of rows necessary to
switch to use the COPY command instead of an INSERT.
I think that the threshold option is necessary because it can be
configured for a different value than batch_size option based on the
user needs. The default value is 1, so once "use_copy_for_batch_insert"
is set to true, the COPY will start to be used. Note that this option is
set to false by default.
Speeking about the implementation, the CopyFrom() calls
BeginForeignInsert() fdw routine. The postgres_fdw implementation of
this routine create the PgFdwModifyState that is used by
execute_foreign_modify() so I thought that it could be a good idea to
get the table options of COPY usage on this function and save it on
PgFdwModifyState struct and when execute_foreign_modify() is executed we
can just access the table options previously stored and check if the
COPY can be actually be used based on the number of tuples being batch
inserted.
The BeginForeignInsert() routine is also called when inserting tuples
into table partitions, so saving the COPY usage options on this stage
can make it possible to use the COPY command to speed up batch inserts
into partition tables that are postgres_fdw tables. I'm not sure if we
should keep the patch scope only for COPY FROM on foreign table but I
don't see any issue of using the COPY to speed up batch inserts of
postgres_fdw table partitions too since we don't expose the remote sql
being used on this case, and benchmarks shows that we can have a good
performance improvement.
I've implemented this idea on the attached v7 and here it is some
benchmarks that I've run.
Scenario: COPY FROM <fdw_table>
use_copy_for_batch_insert = false
rows being inserted = 100
batch_size = 100
copy_for_batch_insert_threshold = 50
tps = 6500.133253
Scenario: COPY FROM <fdw_table>
use_copy_for_batch_insert = true
rows being inserted = 100
batch_size = 100
copy_for_batch_insert_threshold = 50
tps = 13116.474292
Scenario: COPY FROM <fdw_table>
use_copy_for_batch_insert = false
rows being inserted = 140
batch_size = 100
copy_for_batch_insert_threshold = 50
tps = 4654.865032
Scenario: COPY FROM <fdw_table>
use_copy_for_batch_insert = true
rows being inserted = 140
batch_size = 100
copy_for_batch_insert_threshold = 50
tps = 7441.694325
-------------------------------------
Scenario: INSERT INTO <partitioned_table>
use_copy_for_batch_insert = false
rows being inserted per partition = 100
number of partitions: 3
tps = 3176.872369
Scenario: INSERT INTO <partitioned_table>
use_copy_for_batch_insert = true
rows being inserted per partition = 100
number of partitions: 3
tps = 6993.544958
-------------------------------------
Note that for the "copy_for_batch_insert_threshold = 50" and "rows being
inserted=140" the behaviour is to use the COPY for the first batch
iteration of 100 rows and then fallback to use INSERT for the 40 rows
remaining.
Summary of v7 changes:
- Introduce "use_copy_for_batch_insert" foreign server/table option
to enable the usage of COPY command
- Introduce "copy_for_batch_insert_threshold" option to use the COPY
command if the number of rows being inserted is >= of the
configured value. Default is 1.
- COPY command can only be used if the user is executing a COPY FROM
on a postgres_fdw table or an INSERT into a partitioned table that
has postgres_fdw as table partitions.
- COPY and INSERT can be used for the same execution if there is no
sufficient rows remaining (based on copy_usage_threshold) after
the first batch execution.
--
Matheus Alcantara
EDB: http://www.enterprisedb.com
Attachments:
v7-0001-postgres_fdw-speed-up-batch-inserts-using-COPY.patch (text/plain, 14.5K) download
From 17bbb0b129adc96f595e2eb4641b4b601f9fbf6b Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 26 Nov 2025 16:34:46 -0300
Subject: [PATCH v7] postgres_fdw: speed up batch inserts using COPY
Previously when the user execute a COPY into a foreign table the
statement was translated into a INSERT statement to be executed on the
foreign server. This commit introduce a new foreign table/server option
"use_copy_for_batch_insert" to enable the usage of the COPY command
instead of an INSERT. Another option "copy_for_batch_insert_threshold"
was also added to switch to use the COPY command when the number of rows
being inserted is >= than the configured value.
This logic was implement on postgresBeginForeignInsert() that is the
implementation of BeginForeignInsert() fdw routine. As this function is
also called when inserting tuples into partitions the COPY can also be
used to speed up batch inserts for table partitions that are
postgres_fdw tables.
---
contrib/postgres_fdw/deparse.c | 30 +++
.../postgres_fdw/expected/postgres_fdw.out | 13 +
contrib/postgres_fdw/option.c | 13 +-
contrib/postgres_fdw/postgres_fdw.c | 237 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 13 +
6 files changed, 304 insertions(+), 3 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index f2fb0051843..1cdf1d8cc8d 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,36 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ appendStringInfoString(buf, ") FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index cd28126049d..9d06d9b6eb0 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9524,6 +9524,19 @@ select * from rem2;
2 | bar
(2 rows)
+delete from rem2;
+-- Test COPY with can_use_copy = true
+alter foreign table rem2 options (add use_copy_for_batch_insert 'true', copy_for_batch_insert_threshold '2');
+-- Insert 3 rows so that the third row fallback to normal INSERT statement path
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 | foo
+ 2 | bar
+ 3 | baz
+(3 rows)
+
delete from rem2;
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 04788b7e8b3..d56b6cc142d 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -157,7 +157,8 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
(void) ExtractExtensionList(defGetString(def), true);
}
else if (strcmp(def->defname, "fetch_size") == 0 ||
- strcmp(def->defname, "batch_size") == 0)
+ strcmp(def->defname, "batch_size") == 0 ||
+ strcmp(def->defname, "copy_for_batch_insert_threshold") == 0)
{
char *value;
int int_val;
@@ -263,6 +264,16 @@ InitPgFdwOptions(void)
/* batch_size is available on both server and table */
{"batch_size", ForeignServerRelationId, false},
{"batch_size", ForeignTableRelationId, false},
+ /* use_copy_for_batch_insert is available on both server and table */
+ {"use_copy_for_batch_insert", ForeignServerRelationId, false},
+ {"use_copy_for_batch_insert", ForeignTableRelationId, false},
+
+ /*
+ * copy_for_batch_insert_threshold is available on both server and
+ * table
+ */
+ {"copy_for_batch_insert_threshold", ForeignServerRelationId, false},
+ {"copy_for_batch_insert_threshold", ForeignTableRelationId, false},
/* async_capable is available on both server and table */
{"async_capable", ForeignServerRelationId, false},
{"async_capable", ForeignTableRelationId, false},
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 06b52c65300..02c6312a655 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -198,6 +201,12 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ /* COPY usage stuff */
+ bool use_copy_for_batch_insert; /* COPY command is enabled to use? */
+ int copy_for_batch_insert_threshold; /* # of rows to switch to
+ * use COPY */
+ bool usingcopy; /* is COPY being used ? */
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -545,6 +554,11 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static bool get_use_copy_for_batch_insert(Relation rel);
+static int get_copy_for_batch_insert_threshold(Relation rel);
+static TupleTableSlot **execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -2265,6 +2279,10 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy_for_batch_insert = get_use_copy_for_batch_insert(rel);
+ if (fmstate->use_copy_for_batch_insert)
+ fmstate->copy_for_batch_insert_threshold = get_copy_for_batch_insert_threshold(rel);
+
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
* the foreign table is an UPDATE subplan result rel; in which case, store
@@ -4066,6 +4084,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -4097,11 +4159,34 @@ execute_foreign_modify(EState *estate,
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
+ /*
+ * Check if the COPY command can be used to speed up inserts. The COPY
+ * command can not be used if the original query has a RETURNING clause.
+ */
+ if (operation == CMD_INSERT &&
+ fmstate->use_copy_for_batch_insert &&
+ !fmstate->has_returning &&
+ *numSlots >= fmstate->copy_for_batch_insert_threshold)
+ {
+
+ /* Build the COPY command if it's not already built */
+ if (!fmstate->usingcopy)
+ {
+ pfree(fmstate->query);
+ initStringInfo(&sql);
+ deparseCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+ fmstate->query = sql.data;
+ fmstate->usingcopy = true;
+ }
+ return execute_foreign_insert_using_copy(fmstate, slots, numSlots);
+ }
+
/*
* If the existing query was deparsed and prepared for a different number
- * of rows, rebuild it for the proper number.
+ * of rows or if COPY was being used on the previous execution, rebuild
+ * the INSERT statement and use the proper number.
*/
- if (operation == CMD_INSERT && fmstate->num_slots != *numSlots)
+ if ((operation == CMD_INSERT && fmstate->num_slots != *numSlots) || fmstate->usingcopy)
{
/* Destroy the prepared statement created previously */
if (fmstate->p_name)
@@ -7886,3 +7971,151 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Determine if the usage of the COPY command to execute a INSERT into a foreign
+ * table is enabled. The option specified for a table has precedence.
+ */
+static bool
+get_use_copy_for_batch_insert(Relation rel)
+{
+ Oid foreigntableid = RelationGetRelid(rel);
+ List *options = NIL;
+ ListCell *lc;
+ ForeignTable *table;
+ ForeignServer *server;
+ bool can_use_copy = false;
+
+ /*
+ * Load options for table and server. We append server options after table
+ * options, because table options take precedence.
+ */
+ table = GetForeignTable(foreigntableid);
+ server = GetForeignServer(table->serverid);
+
+ options = list_concat(options, table->options);
+ options = list_concat(options, server->options);
+
+ /* See if either table or server specifies enable_batch_with_copy. */
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "use_copy_for_batch_insert") == 0)
+ {
+ (void) parse_bool(defGetString(def), &can_use_copy);
+ break;
+ }
+ }
+ return can_use_copy;
+}
+
+static int
+get_copy_for_batch_insert_threshold(Relation rel)
+{
+ Oid foreigntableid = RelationGetRelid(rel);
+ List *options = NIL;
+ ListCell *lc;
+ ForeignTable *table;
+ ForeignServer *server;
+
+ /*
+ * We use 1 as default, which means that COPY will be used once
+ * "can_use_copy" is set to true.
+ */
+ int copy_for_batch_insert_threshold = 1;
+
+ /*
+ * Load options for table and server. We append server options after table
+ * options, because table options take precedence.
+ */
+ table = GetForeignTable(foreigntableid);
+ server = GetForeignServer(table->serverid);
+
+ options = list_concat(options, table->options);
+ options = list_concat(options, server->options);
+
+ /* See if either table or server specifies enable_batch_with_copy. */
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "copy_for_batch_insert_threshold") == 0)
+ {
+ (void) parse_int(defGetString(def), ©_for_batch_insert_threshold, 0, NULL);
+ break;
+ }
+ }
+ return copy_for_batch_insert_threshold;
+}
+
+/* Execute an insert into a foreign table using the COPY command */
+static TupleTableSlot **
+execute_foreign_insert_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..aa54d6bba53 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 9a8f9e28135..fac00c55553 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2807,6 +2807,19 @@ select * from rem2;
delete from rem2;
+-- Test COPY with can_use_copy = true
+alter foreign table rem2 options (add use_copy_for_batch_insert 'true', copy_for_batch_insert_threshold '2');
+
+-- Insert 3 rows so that the third row fallback to normal INSERT statement path
+copy rem2 from stdin;
+1 foo
+2 bar
+3 baz
+\.
+select * from rem2;
+
+delete from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
--
2.51.2
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2025-12-11 12:03 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2025-12-11 12:03 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
I've spent some more time on this patch cleaning up some things and
trying to simplify some things.
I've renamed "copy_for_batch_insert_threshold" to
"batch_with_copy_threshold" and removed the boolean option
"use_copy_for_batch_insert", so now to enable the COPY usage for batch
inserts it only need to set batch_with_copy_threshold to a number
greater than 0.
Also the COPY can only be used if batching is also enabled (batch_size >
1) and it will only be used for the COPY FROM on a foreign table and for
inserts into table partitions that are also foreign tables.
--
Matheus Alcantara
EDB: http://www.enterprisedb.com
Attachments:
v8-0001-postgres_fdw-speed-up-batch-inserts-using-COPY.patch (text/plain, 14.7K) download
From dead99e8a2db663df8676f1caca1d834e19ca076 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 26 Nov 2025 16:34:46 -0300
Subject: [PATCH v8] postgres_fdw: speed up batch inserts using COPY
This commit include a new foreign table/server option
"batch_with_copy_threshold" that enable the usage of the COPY command to
speed up batch inserts when a COPY FROM or an insert into a table
partition that is a foreign table is executed. In both cases the
BeginForeignInsert fdw routine is called, so this new option is
retrieved only on this routine. For the other cases that use the
ForeignModify routines still use the INSERT as a remote SQL.
Note that the COPY will only be used for batch inserts and only if the
current number of rows being inserted on the batch operation is >=
batch_with_copy_threshold. If batch_size=100, batch_with_copy_threshold=50
and number of rows being inserted is 120 the first 100 rows will be
inserted using the COPY command and the remaining 20 rows will be
inserted using INSERT statement because it did not reach the copy
threshold.
---
contrib/postgres_fdw/deparse.c | 35 +++
.../postgres_fdw/expected/postgres_fdw.out | 26 +++
contrib/postgres_fdw/option.c | 6 +-
contrib/postgres_fdw/postgres_fdw.c | 210 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 23 ++
6 files changed, 298 insertions(+), 3 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index f2fb0051843..54e821f6bf5 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,41 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 48e3185b227..ffba243dece 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9215,6 +9215,19 @@ with result as (insert into itrtest values (1, 'test1'), (2, 'test2') returning
drop trigger loct1_br_insert_trigger on loct1;
drop trigger loct2_br_insert_trigger on loct2;
+-- Test batch insert using COPY with batch_with_copy_threshold
+delete from itrtest;
+alter server loopback options (add batch_with_copy_threshold '2', batch_size '3');
+insert into itrtest values (1, 'test1'), (2, 'test2'), (2, 'test3');
+select * from itrtest;
+ a | b
+---+-------
+ 1 | test1
+ 2 | test2
+ 2 | test3
+(3 rows)
+
+alter server loopback options (drop batch_with_copy_threshold, drop batch_size);
drop table itrtest;
drop table loct1;
drop table loct2;
@@ -9524,6 +9537,19 @@ select * from rem2;
2 | bar
(2 rows)
+delete from rem2;
+-- Test COPY with batch_with_copy_threshold
+alter foreign table rem2 options (add batch_with_copy_threshold '2');
+-- Insert 3 rows so that the third row fallback to normal INSERT statement path
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 | foo
+ 2 | bar
+ 3 | baz
+(3 rows)
+
delete from rem2;
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 04788b7e8b3..d2696206e75 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -157,7 +157,8 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
(void) ExtractExtensionList(defGetString(def), true);
}
else if (strcmp(def->defname, "fetch_size") == 0 ||
- strcmp(def->defname, "batch_size") == 0)
+ strcmp(def->defname, "batch_size") == 0 ||
+ strcmp(def->defname, "batch_with_copy_threshold") == 0)
{
char *value;
int int_val;
@@ -263,6 +264,9 @@ InitPgFdwOptions(void)
/* batch_size is available on both server and table */
{"batch_size", ForeignServerRelationId, false},
{"batch_size", ForeignTableRelationId, false},
+ /* batch_with_copy_threshold is available on both server and table */
+ {"batch_with_copy_threshold", ForeignServerRelationId, false},
+ {"batch_with_copy_threshold", ForeignTableRelationId, false},
/* async_capable is available on both server and table */
{"async_capable", ForeignServerRelationId, false},
{"async_capable", ForeignTableRelationId, false},
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 06b52c65300..7896760d51a 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -198,6 +201,10 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ /* COPY usage stuff */
+ int batch_with_copy_threshold; /* value of FDW option */
+ char *cmd_copy; /* COPY statement */
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -545,6 +552,10 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static int get_batch_with_copy_threshold(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -2013,8 +2024,30 @@ postgresExecForeignBatchInsert(EState *estate,
*/
if (fmstate->aux_fmstate)
resultRelInfo->ri_FdwState = fmstate->aux_fmstate;
- rslot = execute_foreign_modify(estate, resultRelInfo, CMD_INSERT,
- slots, planSlots, numSlots);
+
+ /*
+ * Check if "batch_with_copy_threshold" is enable (> 0) and if the COPY
+ * can be used based on the number of rows being inserted on this batch.
+ * The original query also should not have a RETURNING clause.
+ */
+ if (fmstate->batch_with_copy_threshold > 0 &&
+ fmstate->batch_with_copy_threshold <= *numSlots &&
+ !fmstate->has_returning)
+ {
+ if (fmstate->cmd_copy == NULL)
+ {
+ StringInfoData sql;
+
+ initStringInfo(&sql);
+ deparseCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+ fmstate->cmd_copy = sql.data;
+ }
+
+ rslot = execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+ }
+ else
+ rslot = execute_foreign_modify(estate, resultRelInfo, CMD_INSERT,
+ slots, planSlots, numSlots);
/* Revert that change */
if (fmstate->aux_fmstate)
resultRelInfo->ri_FdwState = fmstate;
@@ -2265,6 +2298,16 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
retrieved_attrs != NIL,
retrieved_attrs);
+
+ /*
+ * Set batch_with_copy_threshold from foreign server/table options. We do
+ * this outside of create_foreign_modify() because we only want to use
+ * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
+ * an insert is being performed on a table partition. In both cases the
+ * BeginForeignInsert fdw routine is called.
+ */
+ fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
+
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
* the foreign table is an UPDATE subplan result rel; in which case, store
@@ -4066,6 +4109,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -7886,3 +7973,122 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Determine COPY usage threshold for batching inserts for a given foreign
+ * table. The option specified for a table has precedence.
+ */
+static int
+get_batch_with_copy_threshold(Relation rel)
+{
+ Oid foreigntableid = RelationGetRelid(rel);
+ List *options = NIL;
+ ListCell *lc;
+ ForeignTable *table;
+ ForeignServer *server;
+
+ /*
+ * We use 0 as default, which means that COPY will not be used by default
+ * for batching insert.
+ */
+ int copy_for_batch_insert_threshold = 0;
+
+ /*
+ * Load options for table and server. We append server options after table
+ * options, because table options take precedence.
+ */
+ table = GetForeignTable(foreigntableid);
+ server = GetForeignServer(table->serverid);
+
+ options = list_concat(options, table->options);
+ options = list_concat(options, server->options);
+
+ /* See if either table or server specifies enable_batch_with_copy. */
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "batch_with_copy_threshold") == 0)
+ {
+ (void) parse_int(defGetString(def), ©_for_batch_insert_threshold, 0, NULL);
+ break;
+ }
+ }
+ return copy_for_batch_insert_threshold;
+}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ Assert(fmstate->cmd_copy != NULL);
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->cmd_copy))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->cmd_copy);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reach the limit to avoid large
+ * memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+ }
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->cmd_copy);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index e69735298d7..aa54d6bba53 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 9a8f9e28135..f973ef07d80 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2635,6 +2635,16 @@ with result as (insert into itrtest values (1, 'test1'), (2, 'test2') returning
drop trigger loct1_br_insert_trigger on loct1;
drop trigger loct2_br_insert_trigger on loct2;
+-- Test batch insert using COPY with batch_with_copy_threshold
+delete from itrtest;
+alter server loopback options (add batch_with_copy_threshold '2', batch_size '3');
+
+insert into itrtest values (1, 'test1'), (2, 'test2'), (2, 'test3');
+
+select * from itrtest;
+
+alter server loopback options (drop batch_with_copy_threshold, drop batch_size);
+
drop table itrtest;
drop table loct1;
drop table loct2;
@@ -2807,6 +2817,19 @@ select * from rem2;
delete from rem2;
+-- Test COPY with batch_with_copy_threshold
+alter foreign table rem2 options (add batch_with_copy_threshold '2');
+
+-- Insert 3 rows so that the third row fallback to normal INSERT statement path
+copy rem2 from stdin;
+1 foo
+2 bar
+3 baz
+\.
+select * from rem2;
+
+delete from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
--
2.51.2
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-01-02 20:15 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2026-01-02 20:15 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
Sorry for the late reply.
On Thu, Dec 11, 2025 at 4:03 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> I've spent some more time on this patch cleaning up some things and
> trying to simplify some things.
>
> I've renamed "copy_for_batch_insert_threshold" to
> "batch_with_copy_threshold" and removed the boolean option
> "use_copy_for_batch_insert", so now to enable the COPY usage for batch
> inserts it only need to set batch_with_copy_threshold to a number
> greater than 0.
>
> Also the COPY can only be used if batching is also enabled (batch_size >
> 1) and it will only be used for the COPY FROM on a foreign table and for
> inserts into table partitions that are also foreign tables.
Thank you for updating the patch!
+
+ /*
+ * Set batch_with_copy_threshold from foreign server/table options. We do
+ * this outside of create_foreign_modify() because we only want to use
+ * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
+ * an insert is being performed on a table partition. In both cases the
+ * BeginForeignInsert fdw routine is called.
+ */
+ fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
Does it mean that we could end up using the COPY method not only when
executing COPY FROM but also when executing INSERT with tuple
routings? If so, how does the EXPLAIN command show the remote SQL?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2026-01-02 20:33 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-03 12:45 ` Re: Re: postgres_fdw: Use COPY to speed up batch inserts Dewei Dai <daidewei1970@163.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 37+ messages in thread
From: Matheus Alcantara @ 2026-01-02 20:33 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Fri Jan 2, 2026 at 5:15 PM -03, Masahiko Sawada wrote:
> +
> + /*
> + * Set batch_with_copy_threshold from foreign server/table options. We do
> + * this outside of create_foreign_modify() because we only want to use
> + * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
> + * an insert is being performed on a table partition. In both cases the
> + * BeginForeignInsert fdw routine is called.
> + */
> + fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
>
> Does it mean that we could end up using the COPY method not only when
> executing COPY FROM but also when executing INSERT with tuple
> routings? If so, how does the EXPLAIN command show the remote SQL?
>
It meas that we could also use the COPY method to insert rows into a
specific table partition that is a foreign table.
Let's say that an user execute an INSERT INTO on a partitioned table
that has partitions that are postgres_fdw tables, with this patch we
could use the COPY method to insert the rows on these partitions. On
this scenario we would not have issue with EXPLAIN output because
currently we do not show the remote SQL being executed on each partition
that is involved on the INSERT statement.
If an user execute an INSERT directly into a postgres_fdw table we will
use the normal INSERT statement as we use today.
Thanks for taking a look at this.
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-01-03 12:45 ` Dewei Dai <daidewei1970@163.com>
2026-01-05 13:43 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Dewei Dai @ 2026-01-03 12:45 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
Hi Matheus,
I just reviewed the v8 patch and got a few comments:
1 - in function `execute_foreign_modify_using_copy`
The `res` object obtained from the first call to `pgfdw_get_result`
is not freed, maybe you can use `PQclear` to release it
2 - in function `execute_foreign_modify_using_copy`
After using `copy_data`,it appears it can be released by
calling `destroyStringInfo`.
3 - in function `convert_slot_to_copy_text`
The value returned by the OutputFunctionCall function can be
freed by calling pfree
4 - in function `execute_foreign_modify_using_copy`
```Send initial COPY data if the buffer reach the limit to avoid large
```
Typo: reach -> reaches
Best regards，
Dewei Dai
daidewei1970@163.com
From: Matheus Alcantara
Date: 2026-01-03 04:33
To: Masahiko Sawada; Matheus Alcantara
CC: Andrew Dunstan; jian he; Tomas Vondra; pgsql-hackers@postgresql.org
Subject: Re: postgres_fdw: Use COPY to speed up batch inserts
On Fri Jan 2, 2026 at 5:15 PM -03, Masahiko Sawada wrote:
> +
> + /*
> + * Set batch_with_copy_threshold from foreign server/table options. We do
> + * this outside of create_foreign_modify() because we only want to use
> + * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
> + * an insert is being performed on a table partition. In both cases the
> + * BeginForeignInsert fdw routine is called.
> + */
> + fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
>
> Does it mean that we could end up using the COPY method not only when
> executing COPY FROM but also when executing INSERT with tuple
> routings? If so, how does the EXPLAIN command show the remote SQL?
>
It meas that we could also use the COPY method to insert rows into a
specific table partition that is a foreign table.
Let's say that an user execute an INSERT INTO on a partitioned table
that has partitions that are postgres_fdw tables, with this patch we
could use the COPY method to insert the rows on these partitions. On
this scenario we would not have issue with EXPLAIN output because
currently we do not show the remote SQL being executed on each partition
that is involved on the INSERT statement.
If an user execute an INSERT directly into a postgres_fdw table we will
use the normal INSERT statement as we use today.
Thanks for taking a look at this.
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-03 12:45 ` Re: Re: postgres_fdw: Use COPY to speed up batch inserts Dewei Dai <daidewei1970@163.com>
@ 2026-01-05 13:43 ` Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 0 replies; 37+ messages in thread
From: Matheus Alcantara @ 2026-01-05 13:43 UTC (permalink / raw)
To: Dewei Dai <daidewei1970@163.com>; Matheus Alcantara <matheusssilv97@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
Hi, thank you for reviewing this patch!
On Sat Jan 3, 2026 at 9:45 AM -03, Dewei Dai wrote:
> 1 - in function `execute_foreign_modify_using_copy`
> The `res` object obtained from the first call to `pgfdw_get_result`
> is not freed, maybe you can use `PQclear` to release it
>
Fixed.
> 2 - in function `execute_foreign_modify_using_copy`
> After using `copy_data`,it appears it can be released by
> calling `destroyStringInfo`.
>
I'm wondering if we should call destroyStringInfo(©_data) or
pfree(copy_data.data). Other functions use the pfree version, so I
decided to use the same.
(I actually tried to use destroyStringInfo() but the postgres_fdw tests
kept running for longer than usual, so I think that using the pfree is
correct)
> 3 - in function `convert_slot_to_copy_text`
> The value returned by the OutputFunctionCall function can be
> freed by calling pfree
>
We have other calls to OutputFunctionCall() on postgres_fdw.c and I'm
not seeing a subsequent call to pfree. IIUC the returned valued will be
allocated on the current memory context which will be free at the end of
query execution, so I don't think that a pfree here is necessary, or I'm
missing something?
> 4 - in function `execute_foreign_modify_using_copy`
> ```Send initial COPY data if the buffer reach the limit to avoid large
> ```
> Typo: reach -> reaches
>
Fixed
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments:
v9-0001-postgres_fdw-speed-up-batch-inserts-using-COPY.patch (text/plain, 14.8K) download
From f4dcd9d836137589c8345b74cb24ab3e7dc18eeb Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 26 Nov 2025 16:34:46 -0300
Subject: [PATCH v9] postgres_fdw: speed up batch inserts using COPY
This commit include a new foreign table/server option
"batch_with_copy_threshold" that enable the usage of the COPY command to
speed up batch inserts when a COPY FROM or an insert into a table
partition that is a foreign table is executed. In both cases the
BeginForeignInsert fdw routine is called, so this new option is
retrieved only on this routine. For the other cases that use the
ForeignModify routines still use the INSERT as a remote SQL.
Note that the COPY will only be used for batch inserts and only if the
current number of rows being inserted on the batch operation is >=
batch_with_copy_threshold. If batch_size=100, batch_with_copy_threshold=50
and number of rows being inserted is 120 the first 100 rows will be
inserted using the COPY command and the remaining 20 rows will be
inserted using INSERT statement because it did not reach the copy
threshold.
---
contrib/postgres_fdw/deparse.c | 35 +++
.../postgres_fdw/expected/postgres_fdw.out | 26 +++
contrib/postgres_fdw/option.c | 6 +-
contrib/postgres_fdw/postgres_fdw.c | 215 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 23 ++
6 files changed, 303 insertions(+), 3 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index ebe2c3a596a..78335db1889 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2236,6 +2236,41 @@ rebuildInsertSql(StringInfo buf, Relation rel,
appendStringInfoString(buf, orig_query + values_end_len);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 6066510c7c0..8bbc27b7b3b 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9215,6 +9215,19 @@ with result as (insert into itrtest values (1, 'test1'), (2, 'test2') returning
drop trigger loct1_br_insert_trigger on loct1;
drop trigger loct2_br_insert_trigger on loct2;
+-- Test batch insert using COPY with batch_with_copy_threshold
+delete from itrtest;
+alter server loopback options (add batch_with_copy_threshold '2', batch_size '3');
+insert into itrtest values (1, 'test1'), (2, 'test2'), (2, 'test3');
+select * from itrtest;
+ a | b
+---+-------
+ 1 | test1
+ 2 | test2
+ 2 | test3
+(3 rows)
+
+alter server loopback options (drop batch_with_copy_threshold, drop batch_size);
drop table itrtest;
drop table loct1;
drop table loct2;
@@ -9524,6 +9537,19 @@ select * from rem2;
2 | bar
(2 rows)
+delete from rem2;
+-- Test COPY with batch_with_copy_threshold
+alter foreign table rem2 options (add batch_with_copy_threshold '2');
+-- Insert 3 rows so that the third row fallback to normal INSERT statement path
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 | foo
+ 2 | bar
+ 3 | baz
+(3 rows)
+
delete from rem2;
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index b0bd72d1e58..4545c2f9ba1 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -157,7 +157,8 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
(void) ExtractExtensionList(defGetString(def), true);
}
else if (strcmp(def->defname, "fetch_size") == 0 ||
- strcmp(def->defname, "batch_size") == 0)
+ strcmp(def->defname, "batch_size") == 0 ||
+ strcmp(def->defname, "batch_with_copy_threshold") == 0)
{
char *value;
int int_val;
@@ -263,6 +264,9 @@ InitPgFdwOptions(void)
/* batch_size is available on both server and table */
{"batch_size", ForeignServerRelationId, false},
{"batch_size", ForeignTableRelationId, false},
+ /* batch_with_copy_threshold is available on both server and table */
+ {"batch_with_copy_threshold", ForeignServerRelationId, false},
+ {"batch_with_copy_threshold", ForeignTableRelationId, false},
/* async_capable is available on both server and table */
{"async_capable", ForeignServerRelationId, false},
{"async_capable", ForeignTableRelationId, false},
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 3572689e33b..2fb95167a1c 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -198,6 +201,10 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ /* COPY usage stuff */
+ int batch_with_copy_threshold; /* value of FDW option */
+ char *cmd_copy; /* COPY statement */
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -545,6 +552,10 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static int get_batch_with_copy_threshold(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
/*
@@ -2013,8 +2024,30 @@ postgresExecForeignBatchInsert(EState *estate,
*/
if (fmstate->aux_fmstate)
resultRelInfo->ri_FdwState = fmstate->aux_fmstate;
- rslot = execute_foreign_modify(estate, resultRelInfo, CMD_INSERT,
- slots, planSlots, numSlots);
+
+ /*
+ * Check if "batch_with_copy_threshold" is enable (> 0) and if the COPY
+ * can be used based on the number of rows being inserted on this batch.
+ * The original query also should not have a RETURNING clause.
+ */
+ if (fmstate->batch_with_copy_threshold > 0 &&
+ fmstate->batch_with_copy_threshold <= *numSlots &&
+ !fmstate->has_returning)
+ {
+ if (fmstate->cmd_copy == NULL)
+ {
+ StringInfoData sql;
+
+ initStringInfo(&sql);
+ deparseCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+ fmstate->cmd_copy = sql.data;
+ }
+
+ rslot = execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+ }
+ else
+ rslot = execute_foreign_modify(estate, resultRelInfo, CMD_INSERT,
+ slots, planSlots, numSlots);
/* Revert that change */
if (fmstate->aux_fmstate)
resultRelInfo->ri_FdwState = fmstate;
@@ -2265,6 +2298,16 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
retrieved_attrs != NIL,
retrieved_attrs);
+
+ /*
+ * Set batch_with_copy_threshold from foreign server/table options. We do
+ * this outside of create_foreign_modify() because we only want to use
+ * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
+ * an insert is being performed on a table partition. In both cases the
+ * BeginForeignInsert fdw routine is called.
+ */
+ fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
+
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
* the foreign table is an UPDATE subplan result rel; in which case, store
@@ -4066,6 +4109,50 @@ create_foreign_modify(EState *estate,
return fmstate;
}
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
/*
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
@@ -7886,3 +7973,127 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Determine COPY usage threshold for batching inserts for a given foreign
+ * table. The option specified for a table has precedence.
+ */
+static int
+get_batch_with_copy_threshold(Relation rel)
+{
+ Oid foreigntableid = RelationGetRelid(rel);
+ List *options = NIL;
+ ListCell *lc;
+ ForeignTable *table;
+ ForeignServer *server;
+
+ /*
+ * We use 0 as default, which means that COPY will not be used by default
+ * for batching insert.
+ */
+ int copy_for_batch_insert_threshold = 0;
+
+ /*
+ * Load options for table and server. We append server options after table
+ * options, because table options take precedence.
+ */
+ table = GetForeignTable(foreigntableid);
+ server = GetForeignServer(table->serverid);
+
+ options = list_concat(options, table->options);
+ options = list_concat(options, server->options);
+
+ /* See if either table or server specifies enable_batch_with_copy. */
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "batch_with_copy_threshold") == 0)
+ {
+ (void) parse_int(defGetString(def), ©_for_batch_insert_threshold, 0, NULL);
+ break;
+ }
+ }
+ return copy_for_batch_insert_threshold;
+}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ Assert(fmstate->cmd_copy != NULL);
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->cmd_copy))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->cmd_copy);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->cmd_copy);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->cmd_copy);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 4f7ab2ed0ac..840e97fed2f 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2643,6 +2643,16 @@ with result as (insert into itrtest values (1, 'test1'), (2, 'test2') returning
drop trigger loct1_br_insert_trigger on loct1;
drop trigger loct2_br_insert_trigger on loct2;
+-- Test batch insert using COPY with batch_with_copy_threshold
+delete from itrtest;
+alter server loopback options (add batch_with_copy_threshold '2', batch_size '3');
+
+insert into itrtest values (1, 'test1'), (2, 'test2'), (2, 'test3');
+
+select * from itrtest;
+
+alter server loopback options (drop batch_with_copy_threshold, drop batch_size);
+
drop table itrtest;
drop table loct1;
drop table loct2;
@@ -2815,6 +2825,19 @@ select * from rem2;
delete from rem2;
+-- Test COPY with batch_with_copy_threshold
+alter foreign table rem2 options (add batch_with_copy_threshold '2');
+
+-- Insert 3 rows so that the third row fallback to normal INSERT statement path
+copy rem2 from stdin;
+1 foo
+2 bar
+3 baz
+\.
+select * from rem2;
+
+delete from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
--
2.51.2
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-01-27 19:17 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2026-01-27 19:17 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Fri, Jan 2, 2026 at 12:33 PM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On Fri Jan 2, 2026 at 5:15 PM -03, Masahiko Sawada wrote:
> > +
> > + /*
> > + * Set batch_with_copy_threshold from foreign server/table options. We do
> > + * this outside of create_foreign_modify() because we only want to use
> > + * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
> > + * an insert is being performed on a table partition. In both cases the
> > + * BeginForeignInsert fdw routine is called.
> > + */
> > + fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
> >
> > Does it mean that we could end up using the COPY method not only when
> > executing COPY FROM but also when executing INSERT with tuple
> > routings? If so, how does the EXPLAIN command show the remote SQL?
> >
> It meas that we could also use the COPY method to insert rows into a
> specific table partition that is a foreign table.
>
> Let's say that an user execute an INSERT INTO on a partitioned table
> that has partitions that are postgres_fdw tables, with this patch we
> could use the COPY method to insert the rows on these partitions. On
> this scenario we would not have issue with EXPLAIN output because
> currently we do not show the remote SQL being executed on each partition
> that is involved on the INSERT statement.
>
> If an user execute an INSERT directly into a postgres_fdw table we will
> use the normal INSERT statement as we use today.
I'm slightly concerned that it could be confusing for users if we use
the COPY method for the same table based on not only
batch_with_copy_threshold but also how to INSERT. For example, if we
insert tuples directly to a leaf partition, we always use INSERT. On
the other hand, if we insert tuples via its parent table, we would use
either COPY or INSERT based on the number of tuples and
batch_with_copy_threshold value. IIUC this behavior stems from FDW API
design (BeginForeignInsert callback is called only in cases of COPY or
tuple routing), which users would not be aware of in general. Also,
inserting tuples directly to a leaf partition is faster in general
than doing via the parent table, but the COPY method optimization is
available only in the latter case.
How about making use of COPY method only when users execute a COPY
command? Which seems more intuitive and a good start. We can
distinguish BeginForeignInsert called via COPY from called via INSERT
(tuple routing) by adding a flag to ModifyTableState or by checking if
the passed resultRelInfo == resultRelInfo->ri_RootResultRelInfo.
Alternative idea (or an improvement) would be to use the COPY method
whenever the number of buffered tuples exceeds the threshold. It would
cover more cases. Regarding the issue with EXPLAIN output, we could
output both queries (INSERT and COPY) with some contexts (e.g., the
threshold for the COPY method etc).
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2026-01-29 14:02 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2026-01-29 14:02 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Tue Jan 27, 2026 at 4:17 PM -03, Masahiko Sawada wrote:
> On Fri, Jan 2, 2026 at 12:33 PM Matheus Alcantara
> <matheusssilv97@gmail.com> wrote:
>>
>> On Fri Jan 2, 2026 at 5:15 PM -03, Masahiko Sawada wrote:
>> > +
>> > + /*
>> > + * Set batch_with_copy_threshold from foreign server/table options. We do
>> > + * this outside of create_foreign_modify() because we only want to use
>> > + * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
>> > + * an insert is being performed on a table partition. In both cases the
>> > + * BeginForeignInsert fdw routine is called.
>> > + */
>> > + fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
>> >
>> > Does it mean that we could end up using the COPY method not only when
>> > executing COPY FROM but also when executing INSERT with tuple
>> > routings? If so, how does the EXPLAIN command show the remote SQL?
>> >
>> It meas that we could also use the COPY method to insert rows into a
>> specific table partition that is a foreign table.
>>
>> Let's say that an user execute an INSERT INTO on a partitioned table
>> that has partitions that are postgres_fdw tables, with this patch we
>> could use the COPY method to insert the rows on these partitions. On
>> this scenario we would not have issue with EXPLAIN output because
>> currently we do not show the remote SQL being executed on each partition
>> that is involved on the INSERT statement.
>>
>> If an user execute an INSERT directly into a postgres_fdw table we will
>> use the normal INSERT statement as we use today.
>
> I'm slightly concerned that it could be confusing for users if we use
> the COPY method for the same table based on not only
> batch_with_copy_threshold but also how to INSERT. For example, if we
> insert tuples directly to a leaf partition, we always use INSERT. On
> the other hand, if we insert tuples via its parent table, we would use
> either COPY or INSERT based on the number of tuples and
> batch_with_copy_threshold value. IIUC this behavior stems from FDW API
> design (BeginForeignInsert callback is called only in cases of COPY or
> tuple routing), which users would not be aware of in general. Also,
> inserting tuples directly to a leaf partition is faster in general
> than doing via the parent table, but the COPY method optimization is
> available only in the latter case.
Yeah, I agree that this patch ends up in a land that it could introduce
more confusing than improvements for the user.
> How about making use of COPY method only when users execute a COPY
> command? Which seems more intuitive and a good start. We can
> distinguish BeginForeignInsert called via COPY from called via INSERT
> (tuple routing) by adding a flag to ModifyTableState or by checking if
> the passed resultRelInfo == resultRelInfo->ri_RootResultRelInfo.
This sounds a good idea, it simplify the patch scope a lot. During my
tests I've noticed that ri_RootResultRelInfo is null when it's being
called by CopyFrom(), so on postgresBeginForeignInsert I've included a
check that if it's null it means that it's being executed by a COPY
command and then we could use the COPY command as remote SQL.
Note that using COPY as the remote SQL is not always feasible. If the
remote table has a trigger that modifies the row, and the local foreign
table also has an insert trigger, we cannot capture those changes. While
postgres_fdw typically relies on INSERT ... RETURNING * to synchronize
the TupleTableSlot with remote side effects, the COPY command does not
support a RETURNING clause. Without this synchronization, local triggers
would see the original data rather than the actual values inserted. This
limitation is why the ri_TrigDesc == NULL check is necessary; removing
it causes the "Test a combination of local and remote triggers"
regression test on postgres_fdw.sql to fail.
> Alternative idea (or an improvement) would be to use the COPY method
> whenever the number of buffered tuples exceeds the threshold. It would
> cover more cases. Regarding the issue with EXPLAIN output, we could
> output both queries (INSERT and COPY) with some contexts (e.g., the
> threshold for the COPY method etc).
We could implement this as a future improvement.
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments:
v10-0001-postgres_fdw-Use-COPY-as-remote-SQL-when-possibl.patch (text/plain, 10.9K) download
From 54f1194ef096b74ecf2405cc6afff95902885189 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 28 Jan 2026 19:55:48 -0300
Subject: [PATCH v10] postgres_fdw: Use COPY as remote SQL when possible
Previously when an user execute a COPY on a foreign table, postgres_fdw
send a INSERT as a remote SQL to the foreign server. This commit
introduce the ability to use the COPY command instead.
The COPY command will only be used when an user execute a COPY on a
foreign table and also the foreign table should not have triggers
because remote triggers might modify the inserted row and since COPY
does not support a RETURNING clause, we cannot synchronize the local
TupleTableSlot with those changes for use in local AFTER triggers, so if
the foreign table has any trigger INSERT will be used.
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/deparse.c | 36 ++++
.../postgres_fdw/expected/postgres_fdw.out | 6 +-
contrib/postgres_fdw/postgres_fdw.c | 169 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
4 files changed, 206 insertions(+), 6 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index ebe2c3a596a..04829b6ee45 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2177,6 +2177,42 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 6066510c7c0..2725f067223 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9533,7 +9533,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -9690,7 +9691,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN
COPY rem2
select * from rem2;
f1 | f2
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 3572689e33b..9630bae3146 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -198,6 +201,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy;
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -545,6 +550,12 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
/*
@@ -2170,6 +2181,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2247,11 +2259,31 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if the following conditions are
+ * met: 1. The insert is not part of a partitioned table's tuple routing
+ * (ri_RootResultRelInfo == NULL). 2. There are no local triggers on the
+ * foreign table (ri_TrigDesc == NULL). Remote triggers might modify the
+ * inserted row; since COPY does not support a RETURNING clause, we cannot
+ * synchronize the local TupleTableSlot with those changes for use in
+ * local AFTER triggers. 3. There is no explicit RETURNING clause
+ * requested by the query.
+ */
+ useCopy = resultRelInfo->ri_RootResultRelInfo == NULL
+ && resultRelInfo->ri_TrigDesc == NULL
+ && resultRelInfo->ri_returningList == NIL;
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ {
+ deparseCopySql(&sql, rel, targetAttrs);
+ values_end_len = 0; /* Keep compiler quiet */
+ }
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2264,6 +2296,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4093,6 +4126,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -7886,3 +7922,128 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ Assert(fmstate->use_copy == true);
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
--
2.51.2
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-02-26 01:39 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2026-02-26 01:39 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Thu, Jan 29, 2026 at 6:02 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On Tue Jan 27, 2026 at 4:17 PM -03, Masahiko Sawada wrote:
> > On Fri, Jan 2, 2026 at 12:33 PM Matheus Alcantara
> > <matheusssilv97@gmail.com> wrote:
> >>
> >> On Fri Jan 2, 2026 at 5:15 PM -03, Masahiko Sawada wrote:
> >> > +
> >> > + /*
> >> > + * Set batch_with_copy_threshold from foreign server/table options. We do
> >> > + * this outside of create_foreign_modify() because we only want to use
> >> > + * COPY as a remote SQL when a COPY FROM on a foreign table is executed or
> >> > + * an insert is being performed on a table partition. In both cases the
> >> > + * BeginForeignInsert fdw routine is called.
> >> > + */
> >> > + fmstate->batch_with_copy_threshold = get_batch_with_copy_threshold(rel);
> >> >
> >> > Does it mean that we could end up using the COPY method not only when
> >> > executing COPY FROM but also when executing INSERT with tuple
> >> > routings? If so, how does the EXPLAIN command show the remote SQL?
> >> >
> >> It meas that we could also use the COPY method to insert rows into a
> >> specific table partition that is a foreign table.
> >>
> >> Let's say that an user execute an INSERT INTO on a partitioned table
> >> that has partitions that are postgres_fdw tables, with this patch we
> >> could use the COPY method to insert the rows on these partitions. On
> >> this scenario we would not have issue with EXPLAIN output because
> >> currently we do not show the remote SQL being executed on each partition
> >> that is involved on the INSERT statement.
> >>
> >> If an user execute an INSERT directly into a postgres_fdw table we will
> >> use the normal INSERT statement as we use today.
> >
> > I'm slightly concerned that it could be confusing for users if we use
> > the COPY method for the same table based on not only
> > batch_with_copy_threshold but also how to INSERT. For example, if we
> > insert tuples directly to a leaf partition, we always use INSERT. On
> > the other hand, if we insert tuples via its parent table, we would use
> > either COPY or INSERT based on the number of tuples and
> > batch_with_copy_threshold value. IIUC this behavior stems from FDW API
> > design (BeginForeignInsert callback is called only in cases of COPY or
> > tuple routing), which users would not be aware of in general. Also,
> > inserting tuples directly to a leaf partition is faster in general
> > than doing via the parent table, but the COPY method optimization is
> > available only in the latter case.
>
> Yeah, I agree that this patch ends up in a land that it could introduce
> more confusing than improvements for the user.
>
> > How about making use of COPY method only when users execute a COPY
> > command? Which seems more intuitive and a good start. We can
> > distinguish BeginForeignInsert called via COPY from called via INSERT
> > (tuple routing) by adding a flag to ModifyTableState or by checking if
> > the passed resultRelInfo == resultRelInfo->ri_RootResultRelInfo.
>
> This sounds a good idea, it simplify the patch scope a lot. During my
> tests I've noticed that ri_RootResultRelInfo is null when it's being
> called by CopyFrom(), so on postgresBeginForeignInsert I've included a
> check that if it's null it means that it's being executed by a COPY
> command and then we could use the COPY command as remote SQL.
Thank you for updating the patch!
> Note that using COPY as the remote SQL is not always feasible. If the
> remote table has a trigger that modifies the row, and the local foreign
> table also has an insert trigger, we cannot capture those changes. While
> postgres_fdw typically relies on INSERT ... RETURNING * to synchronize
> the TupleTableSlot with remote side effects, the COPY command does not
> support a RETURNING clause. Without this synchronization, local triggers
> would see the original data rather than the actual values inserted. This
> limitation is why the ri_TrigDesc == NULL check is necessary; removing
> it causes the "Test a combination of local and remote triggers"
> regression test on postgres_fdw.sql to fail.
Agreed. If this problem happens only when the local table has an AFTER
INSERT trigger, can we check ri_TrigDesc->trig_insert_after_row too?
Regarding the third condition, resultRelInfo->ri_returningList == NIL,
can we make it an Assert() because checking
resultRelInfo->RootResultRelInfo == NULL already checks if it's called
via COPY?
One thing it might be worth considering is to add some regression
tests verifying that COPY commands are actually being used on the
remote server in success cases. That way, we can be aware of changes
even if we change the assumption in the future that RootResultRelInfo
== NULL only when postgresBeginForeignInsert() is called via COPY. One
idea is to define a trigger on the remote server that checks if the
executed query is INSERT or COPY. For example,
create function insert_or_copy() returns trigger as $$
declare query text;
begin
query := current_query();
if query ~* '^COPY' then
raise notice 'COPY command';
elsif query ~* '^INSERT' then
raise notice 'INSERT command';
end if;
return new;
end;
$$ language plpgsql;
Note that we need to set client_min_message to 'log' so that we can
write the notice message raised via postgres_fdw.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2026-02-26 15:58 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2026-02-26 15:58 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Wed Feb 25, 2026 at 10:39 PM -03, Masahiko Sawada wrote:
>> Note that using COPY as the remote SQL is not always feasible. If the
>> remote table has a trigger that modifies the row, and the local foreign
>> table also has an insert trigger, we cannot capture those changes. While
>> postgres_fdw typically relies on INSERT ... RETURNING * to synchronize
>> the TupleTableSlot with remote side effects, the COPY command does not
>> support a RETURNING clause. Without this synchronization, local triggers
>> would see the original data rather than the actual values inserted. This
>> limitation is why the ri_TrigDesc == NULL check is necessary; removing
>> it causes the "Test a combination of local and remote triggers"
>> regression test on postgres_fdw.sql to fail.
>
> Agreed. If this problem happens only when the local table has an AFTER
> INSERT trigger, can we check ri_TrigDesc->trig_insert_after_row too?
>
Yes, it's better to only fallback to insert mode when the table have a
AFTER trigger. Fixed.
> Regarding the third condition, resultRelInfo->ri_returningList == NIL,
> can we make it an Assert() because checking
> resultRelInfo->RootResultRelInfo == NULL already checks if it's called
> via COPY?
>
Yes, souns better. Fixed.
> One thing it might be worth considering is to add some regression
> tests verifying that COPY commands are actually being used on the
> remote server in success cases. That way, we can be aware of changes
> even if we change the assumption in the future that RootResultRelInfo
> == NULL only when postgresBeginForeignInsert() is called via COPY. One
> idea is to define a trigger on the remote server that checks if the
> executed query is INSERT or COPY. For example,
>
> create function insert_or_copy() returns trigger as $$
> declare query text;
> begin
> query := current_query();
> if query ~* '^COPY' then
> raise notice 'COPY command';
> elsif query ~* '^INSERT' then
> raise notice 'INSERT command';
> end if;
> return new;
> end;
> $$ language plpgsql;
>
> Note that we need to set client_min_message to 'log' so that we can
> write the notice message raised via postgres_fdw.
>
Good, thanks for this! I've added on this new version.
Please see the new attached version. Thank you for reviewing this!
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments:
v11-0001-postgres_fdw-Use-COPY-as-remote-SQL-when-possibl.patch (text/plain, 13.7K) download
From f569b0da822e9ad0bef521b7f4f8532a45948577 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 28 Jan 2026 19:55:48 -0300
Subject: [PATCH v11] postgres_fdw: Use COPY as remote SQL when possible
Previously when an user execute a COPY on a foreign table, postgres_fdw
send a INSERT as a remote SQL to the foreign server. This commit
introduce the ability to use the COPY command instead.
The COPY command will only be used when an user execute a COPY on a
foreign table and also the foreign table should not have triggers
because remote triggers might modify the inserted row and since COPY
does not support a RETURNING clause, we cannot synchronize the local
TupleTableSlot with those changes for use in local AFTER triggers, so if
the foreign table has any trigger INSERT will be used.
Author: Matheus Alcantara <mths.dev@pm.me>
Reviewed-By: Tomas Vondra <tomas@vondra.me>
Reviewed-By: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-By: jian he <jian.universality@gmail.com>
Reviewed-By: Dewei Dai <daidewei1970@163.com>
Reviewed-By: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/deparse.c | 36 ++++
.../postgres_fdw/expected/postgres_fdw.out | 33 +++-
contrib/postgres_fdw/postgres_fdw.c | 179 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 27 +++
5 files changed, 268 insertions(+), 8 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index ebe2c3a596a..04829b6ee45 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2177,6 +2177,42 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf, quote_identifier(NameStr(attr->attname)));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 2ccb72c539a..22c2dcdb6b1 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -7603,6 +7603,27 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ if query ~* '^COPY' then
+ raise notice 'COPY command';
+ elsif query ~* '^INSERT' then
+ raise notice 'INSERT command';
+ end if;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY command
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -7620,16 +7641,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -9544,7 +9567,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -9701,7 +9725,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN
COPY rem2
select * from rem2;
f1 | f2
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 60d90329a65..6dd22e4043d 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -198,6 +201,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy;
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -545,6 +550,12 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
/*
@@ -2170,6 +2181,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2247,11 +2259,41 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if all the following conditions
+ * are met:
+ *
+ * Direct Execution: The command is a COPY FROM on the foreign table
+ * itself, not part of a partitioned table's tuple routing. (
+ * resultRelInfo->ri_RootResultRelInfo == NULL)
+ *
+ * No Local AFTER Triggers: There are no AFTER ROW triggers defined locally
+ * on the foreign table.
+ *
+ * Remote triggers might modify the inserted row. Because the COPY protocol
+ * does not support a RETURNING clause, we cannot retrieve those changes to
+ * synchronize the local TupleTableSlot required by local AFTER triggers.
+ */
+ if (resultRelInfo->ri_RootResultRelInfo == NULL)
+ {
+ /* There is no RETURNING clause on COPY */
+ Assert(resultRelInfo->ri_returningList == NIL);
+
+ useCopy = (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row);
+ }
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ {
+ deparseCopySql(&sql, rel, targetAttrs);
+ values_end_len = 0; /* Keep compiler quiet */
+ }
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2264,6 +2306,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4093,6 +4136,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -7886,3 +7932,128 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ Assert(fmstate->use_copy == true);
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 72d2d9c311b..90246ddbd02 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1929,6 +1929,33 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ if query ~* '^COPY' then
+ raise notice 'COPY command';
+ elsif query ~* '^INSERT' then
+ raise notice 'INSERT command';
+ end if;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
--
2.52.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-03-03 19:47 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2026-03-03 19:47 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Thu, Feb 26, 2026 at 7:58 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On Wed Feb 25, 2026 at 10:39 PM -03, Masahiko Sawada wrote:
> >> Note that using COPY as the remote SQL is not always feasible. If the
> >> remote table has a trigger that modifies the row, and the local foreign
> >> table also has an insert trigger, we cannot capture those changes. While
> >> postgres_fdw typically relies on INSERT ... RETURNING * to synchronize
> >> the TupleTableSlot with remote side effects, the COPY command does not
> >> support a RETURNING clause. Without this synchronization, local triggers
> >> would see the original data rather than the actual values inserted. This
> >> limitation is why the ri_TrigDesc == NULL check is necessary; removing
> >> it causes the "Test a combination of local and remote triggers"
> >> regression test on postgres_fdw.sql to fail.
> >
> > Agreed. If this problem happens only when the local table has an AFTER
> > INSERT trigger, can we check ri_TrigDesc->trig_insert_after_row too?
> >
>
> Yes, it's better to only fallback to insert mode when the table have a
> AFTER trigger. Fixed.
>
> > Regarding the third condition, resultRelInfo->ri_returningList == NIL,
> > can we make it an Assert() because checking
> > resultRelInfo->RootResultRelInfo == NULL already checks if it's called
> > via COPY?
> >
>
> Yes, souns better. Fixed.
>
> > One thing it might be worth considering is to add some regression
> > tests verifying that COPY commands are actually being used on the
> > remote server in success cases. That way, we can be aware of changes
> > even if we change the assumption in the future that RootResultRelInfo
> > == NULL only when postgresBeginForeignInsert() is called via COPY. One
> > idea is to define a trigger on the remote server that checks if the
> > executed query is INSERT or COPY. For example,
> >
> > create function insert_or_copy() returns trigger as $$
> > declare query text;
> > begin
> > query := current_query();
> > if query ~* '^COPY' then
> > raise notice 'COPY command';
> > elsif query ~* '^INSERT' then
> > raise notice 'INSERT command';
> > end if;
> > return new;
> > end;
> > $$ language plpgsql;
> >
> > Note that we need to set client_min_message to 'log' so that we can
> > write the notice message raised via postgres_fdw.
> >
>
> Good, thanks for this! I've added on this new version.
>
> Please see the new attached version. Thank you for reviewing this!
>
Thank you for updating the patch! Here are some review comments:
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ appendStringInfoString(buf,
quote_identifier(NameStr(attr->attname)));
We need to take care of the 'column_name' option here. If it's set we
should not use attr->attname as it is.
--
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+}
It might be better to explicitly specify the format 'text'.
---
+ if (useCopy)
+ {
+ deparseCopySql(&sql, rel, targetAttrs);
+ values_end_len = 0; /* Keep compiler quiet */
+ }
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel,
targetAttrs, doNothing,
+
resultRelInfo->ri_WithCheckOptions,
+
resultRelInfo->ri_returningList,
+ &retrieved_attrs,
&values_end_len);
I think we should consider whether it's okay to use the COPY command
even if resultRelInfo->ri_WithCheckOptions is non-NULL. As far as I
researched, it's okay as we currently don't support COPY to a view but
please consider it as well. We might want to explain it too in the
comment.
How about initializing values_end_len with 0 at its declaration?
---
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ if query ~* '^COPY' then
+ raise notice 'COPY command';
+ elsif query ~* '^INSERT' then
+ raise notice 'INSERT command';
+ end if;
+return new;
+end;
+$$ language plpgsql;
On second thoughts, it might be okay to output the current_query() as it is.
---
+copy grem1 from stdin;
+3
+\.
I think it would be good to have more tests, for example, checking if
the COPY command method can work properly with batch_size and
column_name options.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2026-03-04 12:17 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2026-03-04 12:17 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
Thank you for the review.
On 03/03/26 16:47, Masahiko Sawada wrote:
> Thank you for updating the patch! Here are some review comments:
>
> + Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
> +
> + if (attr->attgenerated)
> + continue;
> +
> + if (!first)
> + appendStringInfoString(buf, ", ");
> +
> + first = false;
> +
> + appendStringInfoString(buf,
> quote_identifier(NameStr(attr->attname)));
>
> We need to take care of the 'column_name' option here. If it's set we
> should not use attr->attname as it is.
>
Fixed
> --
> + }
> + if (nattrs > 0)
> + appendStringInfoString(buf, ") FROM STDIN");
> + else
> + appendStringInfoString(buf, " FROM STDIN");
> +}
>
> It might be better to explicitly specify the format 'text'.
>
Fixed
> ---
> + if (useCopy)
> + {
> + deparseCopySql(&sql, rel, targetAttrs);
> + values_end_len = 0; /* Keep compiler quiet */
> + }
> + else
> + deparseInsertSql(&sql, rte, resultRelation, rel,
> targetAttrs, doNothing,
> +
> resultRelInfo->ri_WithCheckOptions,
> +
> resultRelInfo->ri_returningList,
> + &retrieved_attrs,
> &values_end_len);
>
> I think we should consider whether it's okay to use the COPY command
> even if resultRelInfo->ri_WithCheckOptions is non-NULL. As far as I
> researched, it's okay as we currently don't support COPY to a view but
> please consider it as well. We might want to explain it too in the
> comment.
>
Good point, fixed.
> How about initializing values_end_len with 0 at its declaration?
>
Fixed
> ---
> +-- test that fdw also use COPY FROM as a remote sql
> +set client_min_messages to 'log';
> +
> +create function insert_or_copy() returns trigger as $$
> +declare query text;
> +begin
> + query := current_query();
> + if query ~* '^COPY' then
> + raise notice 'COPY command';
> + elsif query ~* '^INSERT' then
> + raise notice 'INSERT command';
> + end if;
> +return new;
> +end;
> +$$ language plpgsql;
>
> On second thoughts, it might be okay to output the current_query() as it is.
>
Fixed
> ---
> +copy grem1 from stdin;
> +3
> +\.
>
> I think it would be good to have more tests, for example, checking if
> the COPY command method can work properly with batch_size and
> column_name options.
>
I've added new test cases for these cases. To test the batch_size case
I've added an elog(DEBUG1) because using the trigger with
current_query() log an entry for each row that we send for the foreign
server, with the elog(DEBUG1) we can expect one log message for each
batch operation. Please let me know if there is better ways to do this.
Please see the new attached version.
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments:
v12-0001-postgres_fdw-Use-COPY-as-remote-SQL-when-possibl.patch (text/plain, 15.8K) download
From ebefff4fae344e4ca92ebd0aefd54a909f7e85c2 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 28 Jan 2026 19:55:48 -0300
Subject: [PATCH v12] postgres_fdw: Use COPY as remote SQL when possible
Previously when an user execute a COPY on a foreign table, postgres_fdw
send a INSERT as a remote SQL to the foreign server. This commit
introduce the ability to use the COPY command instead.
The COPY command will only be used when an user execute a COPY on a
foreign table and also the foreign table should not have triggers
because remote triggers might modify the inserted row and since COPY
does not support a RETURNING clause, we cannot synchronize the local
TupleTableSlot with those changes for use in local AFTER triggers, so if
the foreign table has any trigger INSERT will be used.
Author: Matheus Alcantara <mths.dev@pm.me>
Reviewed-By: Tomas Vondra <tomas@vondra.me>
Reviewed-By: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-By: jian he <jian.universality@gmail.com>
Reviewed-By: Dewei Dai <daidewei1970@163.com>
Reviewed-By: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/deparse.c | 56 ++++++
.../postgres_fdw/expected/postgres_fdw.out | 40 +++-
contrib/postgres_fdw/postgres_fdw.c | 185 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 49 +++++
5 files changed, 322 insertions(+), 9 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index ebe2c3a596a..c7d53a98525 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2177,6 +2177,62 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfo(buf, "(");
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+
+ appendStringInfoString(buf, " (FORMAT TEXT)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 2ccb72c539a..4a946a76afe 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -7603,6 +7603,28 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT TEXT)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -7620,16 +7642,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -7654,6 +7678,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: foreign modify with COPY batch_size: 10
+DEBUG: foreign modify with COPY batch_size: 10
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -9544,7 +9574,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -9701,7 +9732,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2
select * from rem2;
f1 | f2
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 60d90329a65..8602f67b7dc 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -63,6 +63,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -198,6 +201,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy;
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -545,6 +550,12 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
/*
@@ -2165,11 +2176,12 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
RangeTblEntry *rte;
TupleDesc tupdesc = RelationGetDescr(rel);
int attnum;
- int values_end_len;
+ int values_end_len = 0;
StringInfoData sql;
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2247,11 +2259,43 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if all the following conditions
+ * are met:
+ *
+ * Direct Execution: The command is a COPY FROM on the foreign table
+ * itself, not part of a partitioned table's tuple routing. (
+ * resultRelInfo->ri_RootResultRelInfo == NULL)
+ *
+ * No Check Options: There are no WITH CHECK OPTION constraints or
+ * Row-Level Security policies that need to be enforced locally
+ * (resultRelInfo->ri_WithCheckOptions == NIL).
+ *
+ * No Local AFTER Triggers: There are no AFTER ROW triggers defined
+ * locally on the foreign table.
+ *
+ * Remote triggers might modify the inserted row. Because the COPY
+ * protocol does not support a RETURNING clause, we cannot retrieve those
+ * changes to synchronize the local TupleTableSlot required by local AFTER
+ * triggers.
+ */
+ if (resultRelInfo->ri_RootResultRelInfo == NULL && resultRelInfo->ri_WithCheckOptions == NIL)
+ {
+ /* There is no RETURNING clause on COPY */
+ Assert(resultRelInfo->ri_returningList == NIL);
+
+ useCopy = (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row);
+ }
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2264,6 +2308,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4093,6 +4138,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -7886,3 +7934,130 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+
+ Assert(fmstate->use_copy == true);
+
+ elog(DEBUG1, "foreign modify with COPY batch_size: %d", fmstate->batch_size);
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoString(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 72d2d9c311b..1a1ecb043bf 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1929,6 +1929,37 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
+1
+\.
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -1955,6 +1986,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
--
2.52.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-03-30 19:14 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Masahiko Sawada @ 2026-03-30 19:14 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Wed, Mar 4, 2026 at 4:17 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> Thank you for the review.
>
> On 03/03/26 16:47, Masahiko Sawada wrote:
> > Thank you for updating the patch! Here are some review comments:
> >
> > + Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
> > +
> > + if (attr->attgenerated)
> > + continue;
> > +
> > + if (!first)
> > + appendStringInfoString(buf, ", ");
> > +
> > + first = false;
> > +
> > + appendStringInfoString(buf,
> > quote_identifier(NameStr(attr->attname)));
> >
> > We need to take care of the 'column_name' option here. If it's set we
> > should not use attr->attname as it is.
> >
>
> Fixed
>
> > --
> > + }
> > + if (nattrs > 0)
> > + appendStringInfoString(buf, ") FROM STDIN");
> > + else
> > + appendStringInfoString(buf, " FROM STDIN");
> > +}
> >
> > It might be better to explicitly specify the format 'text'.
> >
>
> Fixed
>
> > ---
> > + if (useCopy)
> > + {
> > + deparseCopySql(&sql, rel, targetAttrs);
> > + values_end_len = 0; /* Keep compiler quiet */
> > + }
> > + else
> > + deparseInsertSql(&sql, rte, resultRelation, rel,
> > targetAttrs, doNothing,
> > +
> > resultRelInfo->ri_WithCheckOptions,
> > +
> > resultRelInfo->ri_returningList,
> > + &retrieved_attrs,
> > &values_end_len);
> >
> > I think we should consider whether it's okay to use the COPY command
> > even if resultRelInfo->ri_WithCheckOptions is non-NULL. As far as I
> > researched, it's okay as we currently don't support COPY to a view but
> > please consider it as well. We might want to explain it too in the
> > comment.
> >
>
> Good point, fixed.
>
> > How about initializing values_end_len with 0 at its declaration?
> >
>
> Fixed
>
> > ---
> > +-- test that fdw also use COPY FROM as a remote sql
> > +set client_min_messages to 'log';
> > +
> > +create function insert_or_copy() returns trigger as $$
> > +declare query text;
> > +begin
> > + query := current_query();
> > + if query ~* '^COPY' then
> > + raise notice 'COPY command';
> > + elsif query ~* '^INSERT' then
> > + raise notice 'INSERT command';
> > + end if;
> > +return new;
> > +end;
> > +$$ language plpgsql;
> >
> > On second thoughts, it might be okay to output the current_query() as it is.
> >
>
> Fixed
>
> > ---
> > +copy grem1 from stdin;
> > +3
> > +\.
> >
> > I think it would be good to have more tests, for example, checking if
> > the COPY command method can work properly with batch_size and
> > column_name options.
> >
>
> I've added new test cases for these cases. To test the batch_size case
> I've added an elog(DEBUG1) because using the trigger with
> current_query() log an entry for each row that we send for the foreign
> server, with the elog(DEBUG1) we can expect one log message for each
> batch operation. Please let me know if there is better ways to do this.
>
> Please see the new attached version.
I've reviewed the v12 patch, and here are review comments:
The COPY data sent via postgres_fdw should properly escape the input
data. The bug I found can be reproduced with the following scenario:
-- on local server
create server remote foreign data wrapper postgres_fdw;
create foreign table t (a int, b text) server remote;
create user mapping for public server remote;
-- on remote server
create table t (a int, b text);
-- on local server
copy t(a, d) from stdin;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself, or an EOF signal.
>> 1 hello\nworld
>> \.
ERROR: invalid input syntax for type integer: "world"
CONTEXT: COPY t, line 2, column a: "world"
remote SQL command: COPY public.t(a, d) FROM STDIN (FORMAT TEXT)
---
I think the patch misses calling set_transmission_modes() and
reset_transmission_modes() when converting the tuple slots. A
problematic scenario is:
-- on local server
create server remote foreign data wrapper postgres_fdw;
create foreign table f (a float) server remote;
create user mapping for public server remote;
-- on remote server
create table f (a float);
-- on local server
set extra_float_digits = 0;
copy test_float from stdin;
1.0000000000000002
\.
select * from f;
a
----
1
(1 row)
---
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfo(buf, "(");
appendStringInfoString() or appendStringInfoChar() should be used instead.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
@ 2026-04-01 15:50 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-27 23:22 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 2 replies; 37+ messages in thread
From: Matheus Alcantara @ 2026-04-01 15:50 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Mon Mar 30, 2026 at 4:14 PM -03, Masahiko Sawada wrote:
>> Please see the new attached version.
>
> I've reviewed the v12 patch, and here are review comments:
>
> The COPY data sent via postgres_fdw should properly escape the input
> data. The bug I found can be reproduced with the following scenario:
>
> -- on local server
> create server remote foreign data wrapper postgres_fdw;
> create foreign table t (a int, b text) server remote;
> create user mapping for public server remote;
>
> -- on remote server
> create table t (a int, b text);
>
> -- on local server
> copy t(a, d) from stdin;
> Enter data to be copied followed by a newline.
> End with a backslash and a period on a line by itself, or an EOF signal.
>>> 1 hello\nworld
>>> \.
> ERROR: invalid input syntax for type integer: "world"
> CONTEXT: COPY t, line 2, column a: "world"
> remote SQL command: COPY public.t(a, d) FROM STDIN (FORMAT TEXT)
>
I think that we need something like CopyAttributeOutText() here.
To fix this I've added appendStringInfoText() which is a similar version
of CopyAttributeOutText() that works with a StringInfo. I did not find
any function that I could reuse here, if such function exists please let
me know.
I'm wondering if we should have this similar function or try to combine
both to avoid duplicated logic, although it looks complicated to me at
first look to combine these both usages.
Another option is to use the BINARY format, but it is less portable
compared to the TEXT format across machines architectures and
PostgreSQL versions [1]. For CSV we can just wrap the string into ' but
I think that we can have a performance issue. What do you think?
I've also quickly benchmarked this change and I've got very similar
execution time, with and without this change.
> ---
> I think the patch misses calling set_transmission_modes() and
> reset_transmission_modes() when converting the tuple slots. A
> problematic scenario is:
>
> -- on local server
> create server remote foreign data wrapper postgres_fdw;
> create foreign table f (a float) server remote;
> create user mapping for public server remote;
>
> -- on remote server
> create table f (a float);
>
> -- on local server
> set extra_float_digits = 0;
> copy test_float from stdin;
> 1.0000000000000002
> \.
> select * from f;
> a
> ----
> 1
> (1 row)
>
IIUC the correct output on this case should still be 1 if we did not
reset extra_float_digits, right?
postgres=# create table xxx(a float);
CREATE TABLE
postgres=# set extra_float_digits = 0;
SET
postgres=# copy xxx from stdin;
1.0000000000000002
\.
COPY 1
postgres=# select * from xxx;
a
---
1
(1 row)
postgres=# reset extra_float_digits;
RESET
postgres=# select * from xxx;
a
--------------------
1.0000000000000002
(1 row)
But you are right that set_transmission_modes() call is missing and
without it we will send "1" for the foreign server and it will lost the
actual value. I've fixed this on the new version.
> ---
> +
> + appendStringInfo(buf, "COPY ");
> + deparseRelation(buf, rel);
> + if (nattrs > 0)
> + appendStringInfo(buf, "(");
>
> appendStringInfoString() or appendStringInfoChar() should be used instead.
>
Fixed.
Thank you for reviewing this patch!
[1] https://www.postgresql.org/docs/current/sql-copy.html#SQL-COPY-FILE-FORMATS
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments:
v13-0001-postgres_fdw-Use-COPY-as-remote-SQL-when-possibl.patch (text/plain, 19.1K) download
From c344c5ba3d3d7c0e0e9f47e279442cbc7f252a6d Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 28 Jan 2026 19:55:48 -0300
Subject: [PATCH v13] postgres_fdw: Use COPY as remote SQL when possible
Previously when an user execute a COPY on a foreign table, postgres_fdw
send a INSERT as a remote SQL to the foreign server. This commit
introduce the ability to use the COPY command instead.
The COPY command will only be used when an user execute a COPY on a
foreign table and also the foreign table should not have triggers
because remote triggers might modify the inserted row and since COPY
does not support a RETURNING clause, we cannot synchronize the local
TupleTableSlot with those changes for use in local AFTER triggers, so if
the foreign table has any trigger INSERT will be used.
Author: Matheus Alcantara <mths.dev@pm.me>
Reviewed-By: Tomas Vondra <tomas@vondra.me>
Reviewed-By: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-By: jian he <jian.universality@gmail.com>
Reviewed-By: Dewei Dai <daidewei1970@163.com>
Reviewed-By: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/deparse.c | 56 ++++
.../postgres_fdw/expected/postgres_fdw.out | 64 ++++-
contrib/postgres_fdw/postgres_fdw.c | 265 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 72 +++++
5 files changed, 449 insertions(+), 9 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index c159ecd1558..9ca0b05c771 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2177,6 +2177,62 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfoChar(buf, '(');
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+
+ appendStringInfoString(buf, " (FORMAT TEXT)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 0f5271d476e..d4e3dff47a0 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -7611,6 +7611,28 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT TEXT)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -7628,16 +7650,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -7662,6 +7686,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: foreign modify with COPY batch_size: 10
+DEBUG: foreign modify with COPY batch_size: 10
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -9552,7 +9582,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -9709,7 +9740,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2
select * from rem2;
f1 | f2
@@ -9748,6 +9780,30 @@ select * from rem2;
drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+select * from rem2;
+ f1 | f2
+----+-------
+ 1 | hello+
+ | world
+(1 row)
+
+delete from rem2;
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+reset extra_float_digits;
+select * from f;
+ a
+--------------------
+ 1.0000000000000002
+(1 row)
+
+drop table f;
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 41e47cc795b..c0e7c5d6fde 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -64,6 +64,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -199,6 +202,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy;
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -546,6 +551,13 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
+static void appendStringInfoText(StringInfo buf, const char *string);
/*
@@ -2166,11 +2178,12 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
RangeTblEntry *rte;
TupleDesc tupdesc = RelationGetDescr(rel);
int attnum;
- int values_end_len;
+ int values_end_len = 0;
StringInfoData sql;
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2248,11 +2261,43 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if all the following conditions
+ * are met:
+ *
+ * Direct Execution: The command is a COPY FROM on the foreign table
+ * itself, not part of a partitioned table's tuple routing. (
+ * resultRelInfo->ri_RootResultRelInfo == NULL)
+ *
+ * No Check Options: There are no WITH CHECK OPTION constraints or
+ * Row-Level Security policies that need to be enforced locally
+ * (resultRelInfo->ri_WithCheckOptions == NIL).
+ *
+ * No Local AFTER Triggers: There are no AFTER ROW triggers defined
+ * locally on the foreign table.
+ *
+ * Remote triggers might modify the inserted row. Because the COPY
+ * protocol does not support a RETURNING clause, we cannot retrieve those
+ * changes to synchronize the local TupleTableSlot required by local AFTER
+ * triggers.
+ */
+ if (resultRelInfo->ri_RootResultRelInfo == NULL && resultRelInfo->ri_WithCheckOptions == NIL)
+ {
+ /* There is no RETURNING clause on COPY */
+ Assert(resultRelInfo->ri_returningList == NIL);
+
+ useCopy = (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row);
+ }
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2265,6 +2310,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4094,6 +4140,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -7887,3 +7936,209 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+ int nestlevel;
+
+ Assert(fmstate->use_copy == true);
+
+ elog(DEBUG1, "foreign modify with COPY batch_size: %d", fmstate->batch_size);
+
+ /* Make sure any constants in the slots are printed portably */
+ nestlevel = set_transmission_modes();
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ reset_transmission_modes(nestlevel);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ /* Escape the value if needed */
+ appendStringInfoText(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
+/*
+ * Append a string to buf, escaping special characters for COPY TEXT format.
+ * Similar to CopyAttributeOutText().
+ */
+static void
+appendStringInfoText(StringInfo buf, const char *string)
+{
+ const char *ptr;
+ const char *start;
+ char c;
+
+ ptr = string;
+ start = ptr;
+
+ while ((c = *ptr) != '\0')
+ {
+ if ((unsigned char) c < (unsigned char) 0x20)
+ {
+ /*
+ * We use C-like backslash notation for the common control
+ * characters; other control chars are passed through as-is since
+ * they won't confuse the COPY protocol.
+ */
+ switch (c)
+ {
+ case '\b':
+ c = 'b';
+ break;
+ case '\f':
+ c = 'f';
+ break;
+ case '\n':
+ c = 'n';
+ break;
+ case '\r':
+ c = 'r';
+ break;
+ case '\t':
+ c = 't';
+ break;
+ case '\v':
+ c = 'v';
+ break;
+ default:
+ /* All ASCII control chars are length 1 */
+ ptr++;
+ continue;
+ }
+ /* Dump literal portion before the control character */
+ if (ptr > start)
+ appendBinaryStringInfo(buf, start, ptr - start);
+ appendStringInfoChar(buf, '\\');
+ appendStringInfoChar(buf, c);
+ start = ++ptr;
+ }
+ else if (c == '\\')
+ {
+ /* Backslash must be escaped as \\ */
+ if (ptr > start)
+ appendBinaryStringInfo(buf, start, ptr - start);
+ appendStringInfoChar(buf, '\\');
+ start = ptr++;
+ }
+ else
+ ptr++;
+ }
+
+ /* Dump remaining literal portion */
+ if (ptr > start)
+ appendBinaryStringInfo(buf, start, ptr - start);
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 49ed797e8ef..9b0f2bc16a1 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1936,6 +1936,37 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
+1
+\.
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -1962,6 +1993,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
@@ -3026,6 +3075,29 @@ drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+1 hello\nworld
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+1.0000000000000002
+\.
+
+reset extra_float_digits;
+select * from f;
+
+drop table f;
+
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
--
2.52.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-04-07 21:00 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-07 11:38 ` Re: postgres_fdw: Use COPY to speed up batch inserts solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2026-04-07 21:00 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Wed Apr 1, 2026 at 12:50 PM -03, Matheus Alcantara wrote:
> On Mon Mar 30, 2026 at 4:14 PM -03, Masahiko Sawada wrote:
>>> Please see the new attached version.
>>
>> I've reviewed the v12 patch, and here are review comments:
>>
>> The COPY data sent via postgres_fdw should properly escape the input
>> data. The bug I found can be reproduced with the following scenario:
>>
>> -- on local server
>> create server remote foreign data wrapper postgres_fdw;
>> create foreign table t (a int, b text) server remote;
>> create user mapping for public server remote;
>>
>> -- on remote server
>> create table t (a int, b text);
>>
>> -- on local server
>> copy t(a, d) from stdin;
>> Enter data to be copied followed by a newline.
>> End with a backslash and a period on a line by itself, or an EOF signal.
>>>> 1 hello\nworld
>>>> \.
>> ERROR: invalid input syntax for type integer: "world"
>> CONTEXT: COPY t, line 2, column a: "world"
>> remote SQL command: COPY public.t(a, d) FROM STDIN (FORMAT TEXT)
>>
>
> I think that we need something like CopyAttributeOutText() here.
>
> To fix this I've added appendStringInfoText() which is a similar version
> of CopyAttributeOutText() that works with a StringInfo. I did not find
> any function that I could reuse here, if such function exists please let
> me know.
>
> I'm wondering if we should have this similar function or try to combine
> both to avoid duplicated logic, although it looks complicated to me at
> first look to combine these both usages.
>
> Another option is to use the BINARY format, but it is less portable
> compared to the TEXT format across machines architectures and
> PostgreSQL versions [1]. For CSV we can just wrap the string into ' but
> I think that we can have a performance issue. What do you think?
>
> I've also quickly benchmarked this change and I've got very similar
> execution time, with and without this change.
>
I've played with changing the format from TEXT to CSV to avoid this
duplicated code and I'm attaching a new version with the results. We
still need a special function to handle the escape but I think that it's
less complicated compared with TEXT.
I was a bit concerned about the performance so I've executed a benchmark
using pgbench that run a COPY FROM with 100 rows on a foreign table and
I've got the following results:
Command: pgbench -n -c 10 -j 10 -t 100 -f bench.sql postgres
batch_size: 10
patch tps: 5588.402946
master tps: 4691.619829
batch_size: 100
patch tps: 11834.459579
master tps: 5578.925053
batch_size: 1000
patch tps: 11181.554907
master tps: 6452.945124
The results looks good, so I think that CSV is a valid format option.
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments:
v14-0001-postgres_fdw-Use-COPY-as-remote-SQL-when-possibl.patch (text/plain, 18.7K) download
From cc0973e23013bb273e04d7966128464ac2a368d6 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 28 Jan 2026 19:55:48 -0300
Subject: [PATCH v14] postgres_fdw: Use COPY as remote SQL when possible
Previously when an user execute a COPY on a foreign table, postgres_fdw
send a INSERT as a remote SQL to the foreign server. This commit
introduce the ability to use the COPY command instead.
The COPY command will only be used when an user execute a COPY on a
foreign table and also the foreign table should not have triggers
because remote triggers might modify the inserted row and since COPY
does not support a RETURNING clause, we cannot synchronize the local
TupleTableSlot with those changes for use in local AFTER triggers, so if
the foreign table has any trigger INSERT will be used.
Author: Matheus Alcantara <mths.dev@pm.me>
Reviewed-By: Tomas Vondra <tomas@vondra.me>
Reviewed-By: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-By: jian he <jian.universality@gmail.com>
Reviewed-By: Dewei Dai <daidewei1970@163.com>
Reviewed-By: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/deparse.c | 56 +++++
.../postgres_fdw/expected/postgres_fdw.out | 75 ++++++-
contrib/postgres_fdw/postgres_fdw.c | 212 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 82 +++++++
5 files changed, 417 insertions(+), 9 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index c159ecd1558..a1e024d3f66 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2177,6 +2177,62 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the CSV format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfoChar(buf, '(');
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+
+ appendStringInfoString(buf, " (FORMAT CSV)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index cd22553236f..14fc4dfec07 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -7654,6 +7654,28 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT CSV)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -7671,16 +7693,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -7705,6 +7729,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: foreign modify with COPY batch_size: 10
+DEBUG: foreign modify with COPY batch_size: 10
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -9586,6 +9616,17 @@ select * from rem2;
2 | bar
(2 rows)
+delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 |
+ | bar
+ 3 | a"b
+(3 rows)
+
delete from rem2;
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
@@ -9595,7 +9636,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: ""-1","xyzzy""
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT CSV)
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -9752,7 +9794,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: ""-1","xyzzy""
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT CSV)
COPY rem2
select * from rem2;
f1 | f2
@@ -9791,6 +9834,30 @@ select * from rem2;
drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+select * from rem2;
+ f1 | f2
+----+-------
+ 1 | hello+
+ | world
+(1 row)
+
+delete from rem2;
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+reset extra_float_digits;
+select * from f;
+ a
+--------------------
+ 1.0000000000000002
+(1 row)
+
+drop table f;
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index cc8ec24c30e..3301d600967 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -64,6 +64,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -199,6 +202,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy;
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -546,6 +551,13 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_csv(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
+static void appendStringInfoCsv(StringInfo buf, const char *string);
/*
@@ -2166,11 +2178,12 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
RangeTblEntry *rte;
TupleDesc tupdesc = RelationGetDescr(rel);
int attnum;
- int values_end_len;
+ int values_end_len = 0;
StringInfoData sql;
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2248,11 +2261,43 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if all the following conditions
+ * are met:
+ *
+ * Direct Execution: The command is a COPY FROM on the foreign table
+ * itself, not part of a partitioned table's tuple routing. (
+ * resultRelInfo->ri_RootResultRelInfo == NULL)
+ *
+ * No Check Options: There are no WITH CHECK OPTION constraints or
+ * Row-Level Security policies that need to be enforced locally
+ * (resultRelInfo->ri_WithCheckOptions == NIL).
+ *
+ * No Local AFTER Triggers: There are no AFTER ROW triggers defined
+ * locally on the foreign table.
+ *
+ * Remote triggers might modify the inserted row. Because the COPY
+ * protocol does not support a RETURNING clause, we cannot retrieve those
+ * changes to synchronize the local TupleTableSlot required by local AFTER
+ * triggers.
+ */
+ if (resultRelInfo->ri_RootResultRelInfo == NULL && resultRelInfo->ri_WithCheckOptions == NIL)
+ {
+ /* There is no RETURNING clause on COPY */
+ Assert(resultRelInfo->ri_returningList == NIL);
+
+ useCopy = (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row);
+ }
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2265,6 +2310,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4094,6 +4140,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -7887,3 +7936,156 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+ int nestlevel;
+
+ Assert(fmstate->use_copy == true);
+
+ elog(DEBUG1, "foreign modify with COPY batch_size: %d", fmstate->batch_size);
+
+ /* Make sure any constants in the slots are printed portably */
+ nestlevel = set_transmission_modes();
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_csv(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ reset_transmission_modes(nestlevel);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_csv(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, ',');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ {
+ /* In CSV format, NULL is an empty unquoted field */
+ }
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoCsv(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
+/*
+ * Append a string to buf, with CSV escaping (quote field, double any quotes).
+ */
+static void
+appendStringInfoCsv(StringInfo buf, const char *string)
+{
+ const char *ptr;
+
+ appendStringInfoCharMacro(buf, '"');
+ for (ptr = string; *ptr; ptr++)
+ {
+ if (*ptr == '"')
+ appendStringInfoCharMacro(buf, '"');
+ appendStringInfoCharMacro(buf, *ptr);
+ }
+ appendStringInfoCharMacro(buf, '"');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 59963e298b8..1e717263aa8 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1970,6 +1970,37 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
+1
+\.
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -1996,6 +2027,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
@@ -2863,6 +2912,16 @@ select * from rem2;
delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+1 \N
+\N bar
+3 a"b
+\.
+select * from rem2;
+
+delete from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
@@ -3060,6 +3119,29 @@ drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+1 hello\nworld
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+1.0000000000000002
+\.
+
+reset extra_float_digits;
+select * from f;
+
+drop table f;
+
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
--
2.52.0
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-05-07 11:38 ` solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
2026-05-27 23:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: solaimurugan vellaipandiyan @ 2026-05-07 11:38 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
Hi all,
Thank you for the updated Patch.
On Thu, May 7, 2026 at 4:44 PM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
>
> On Wed Apr 1, 2026 at 12:50 PM -03, Matheus Alcantara wrote:
> > On Mon Mar 30, 2026 at 4:14 PM -03, Masahiko Sawada wrote:
> >>> Please see the new attached version.
> >>
> >> I've reviewed the v12 patch, and here are review comments:
> >>
> >> The COPY data sent via postgres_fdw should properly escape the input
> >> data. The bug I found can be reproduced with the following scenario:
> >>
> >> -- on local server
> >> create server remote foreign data wrapper postgres_fdw;
> >> create foreign table t (a int, b text) server remote;
> >> create user mapping for public server remote;
> >>
> >> -- on remote server
> >> create table t (a int, b text);
> >>
> >> -- on local server
> >> copy t(a, d) from stdin;
> >> Enter data to be copied followed by a newline.
> >> End with a backslash and a period on a line by itself, or an EOF signal.
> >>>> 1 hello\nworld
> >>>> \.
> >> ERROR: invalid input syntax for type integer: "world"
> >> CONTEXT: COPY t, line 2, column a: "world"
> >> remote SQL command: COPY public.t(a, d) FROM STDIN (FORMAT TEXT)
> >>
> >
> > I think that we need something like CopyAttributeOutText() here.
> >
> > To fix this I've added appendStringInfoText() which is a similar version
> > of CopyAttributeOutText() that works with a StringInfo. I did not find
> > any function that I could reuse here, if such function exists please let
> > me know.
> >
> > I'm wondering if we should have this similar function or try to combine
> > both to avoid duplicated logic, although it looks complicated to me at
> > first look to combine these both usages.
> >
> > Another option is to use the BINARY format, but it is less portable
> > compared to the TEXT format across machines architectures and
> > PostgreSQL versions [1]. For CSV we can just wrap the string into ' but
> > I think that we can have a performance issue. What do you think?
> >
> > I've also quickly benchmarked this change and I've got very similar
> > execution time, with and without this change.
> >
>
> I've played with changing the format from TEXT to CSV to avoid this
> duplicated code and I'm attaching a new version with the results. We
> still need a special function to handle the escape but I think that it's
> less complicated compared with TEXT.
>
> I was a bit concerned about the performance so I've executed a benchmark
> using pgbench that run a COPY FROM with 100 rows on a foreign table and
> I've got the following results:
>
> Command: pgbench -n -c 10 -j 10 -t 100 -f bench.sql postgres
>
> batch_size: 10
> patch tps: 5588.402946
> master tps: 4691.619829
>
> batch_size: 100
> patch tps: 11834.459579
> master tps: 5578.925053
>
> batch_size: 1000
> patch tps: 11181.554907
> master tps: 6452.945124
>
> The results looks good, so I think that CSV is a valid format option.
>
>
The patch applied successfully in my branch tree and tested the patch
by creating 2 clusters (Local cluster - as the foreign table side &
Remote cluster - as the remote table side).
I first reproduced the current behavior on an unpatched tree and observed that:
COPY ft_emp FROM stdin; resulted in row-by-row remote INSERT execution.
After applying the patch and rebuilding along with postgres_fdw, I
repeated the same test and observed the following in the remote
cluster logs as:
COPY public.emp(id, name) FROM STDIN (FORMAT CSV) that indicates the
remote COPY optimization path is being used successfully.
Local Cluster:
postgres=# COPY ft_emp FROM stdin;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself, or an EOF signal.
>> 200 A
>> 201 B
>> 202 C
>> \.
COPY 3
Remote Cluster:
postgres=# 2026-05-07 16:30:07.800 IST [210202] LOG: statement: START
TRANSACTION ISOLATION LEVEL REPEATABLE READ
2026-05-07 16:30:42.973 IST [210202] LOG: statement: COPY
public.emp(id, name) FROM STDIN (FORMAT CSV)
2026-05-07 16:30:42.976 IST [210202] LOG: statement: COPY
public.emp(id, name) FROM STDIN (FORMAT CSV)
2026-05-07 16:30:42.977 IST [210202] LOG: statement: COPY
public.emp(id, name) FROM STDIN (FORMAT CSV)
2026-05-07 16:30:42.977 IST [210202] LOG: statement: COMMIT TRANSACTION
Also I tested the behavior with a BEFORE INSERT FOR EACH ROW trigger
defined on the remote table. In my testing, one thing I observed is
that the remote COPY path was still chosen even with a BEFORE INSERT
FOR EACH ROW trigger defined on the remote table and the trigger fired
correctly during COPY execution. I wanted to confirm whether this is
the expected behavior/design for the optimization path. Otherwise the
patch looks good to me from my side.
Looking forward to more feedback on this.
Regards,
Solaimurugan V
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-07 11:38 ` Re: postgres_fdw: Use COPY to speed up batch inserts solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
@ 2026-05-27 23:25 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-08-05 21:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2026-05-27 23:25 UTC (permalink / raw)
To: solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On 07/05/26 08:38, solaimurugan vellaipandiyan wrote:
> Hi all,
>
> Thank you for the updated Patch.
>
> ...
>
> Looking forward to more feedback on this.
>
Hi,
Thank you for reviewing and testing the patch. I've just posted a new
version [1] removing the duplicated code.
[1]
https://www.postgresql.org/message-id/DITUGI3X54S8.346G3XFY7DRDK%40gmail.com
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-07 11:38 ` Re: postgres_fdw: Use COPY to speed up batch inserts solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
2026-05-27 23:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-08-05 21:33 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-09-09 20:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts Kirill Reshke <reshkekirill@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Matheus Alcantara @ 2026-08-05 21:33 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
Hi,
Attached is a new version of the patch taking a different approach from
the earlier attempts.
Off-list I discussed with Masahiko the idea of adding a dedicated COPY
API to the FDW handler, instead of trying to bolt COPY onto the existing
insert callbacks, and the attached patch implements that. The core
addition is a single new optional FdwRoutine callback:
void
ExecForeignBatchCopy(EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot **slots,
int numSlots);
When a foreign table (or a foreign-table partition) is the target of
COPY FROM and batching is enabled, CopyFrom() hands each buffered batch
of tuples to this callback instead of ExecForeignInsert /
ExecForeignBatchInsert. postgres_fdw implements it by running a
COPY ... FROM STDIN on the remote connection and streaming the rows in
text format.
I first tried the more obvious design of splitting this into
BeginForeignCopy / ExecForeignCopy / EndForeignCopy, keeping a single
COPY ... FROM STDIN open across the entire COPY command and closing it
only in EndForeignCopy. That doesn't work in general because of
connection sharing. postgres_fdw caches one connection per user
mapping, so:
- when COPY routes tuples into several foreign partitions that map to
the same remote server (the sharding case), or
- when local code runs another query on the same connection in the
middle of the COPY (e.g. a trigger or a volatile default expression
that reads a foreign table on the same server),
we would need a second command in flight on a connection that is stuck
in COPY_IN mode, which fails. Making each call self-contained avoids
this entirely, the connection is always left idle between calls, so
partitions can share it and any interleaved query just works.
The trade-off is that COPY is only used when batching is enabled
(batch_size > 1). With batching disabled we would be starting and
finishing a COPY protocol for every single row, which is slower than a
plain INSERT, so in that case COPY FROM keeps inserting rows one at a
time, exactly as today. The new callback is also independent of
ExecForeignBatchInsert, a FDW may implement either or both. COPY is
not used when the target has AFTER ROW triggers, because COPY has no
RETURNING to feed them, such cases fall back to the batch-insert path.
This is a PoC patch to experiment the idea of having a dedicated API,
but I think that the main advantage of this is that we can involve the
API to be more flexible and e.g use some user custom COPY options when
writing the remote SQL for the remote server (although this patch still
doesn't implement this) which would be hard using the ExecForeignInsert
API. Also, I think that we can have a BeginForeignCopy and
EndForeignCopy to initialize states used on ExecForeignBatchCopy (e.g
the sql COPY command).
The patch is split into three parts:
0001 - Extract CopyEscapeText() so it can be reused outside COPY TO.
0002 - Add the ExecForeignBatchCopy callback to the FDW API (copyfrom.c,
fdwapi.h and the FDW documentation).
0003 - Implement the callback in postgres_fdw.
Thoughts on the API and the self-contained approach are very welcome.
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments: download all (3 files, .tar.gz)
v16-0003-postgres_fdw-use-COPY-to-bulk-load-foreign-table.patch (application/octet-stream, 22.3K) download
From 79836b40bcdc7857badaeb8f819aca723f83d779 Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 5 Aug 2026 15:29:38 -0300
Subject: [PATCH v16 3/3] postgres_fdw: use COPY to bulk-load foreign tables in
COPY FROM
Implement the ExecForeignBatchCopy callback so that COPY FROM into a
foreign table transfers rows to the remote server with the COPY protocol
(COPY ... FROM STDIN) instead of inserting them one at a time.
deparseCopySql() builds the remote command and each row is serialized in
COPY text format using CopyEscapeText().
The callback is self-contained: it opens the COPY on the remote
connection, streams the whole batch (buffering locally and flushing once
COPYBUFSIZ bytes accumulate), and closes the COPY before returning.
Because the connection is left idle between calls, this works for a
direct COPY FROM on a foreign table as well as for foreign-table
partitions that share a cached connection.
The path is taken only when batching is enabled for the table
(batch_size > 1); with batching disabled, COPY FROM keeps inserting rows
individually, since issuing one COPY command per row would be slower.
pgfdw_abort_cleanup() is taught to terminate an in-progress COPY on the
connection before it is reused, so an error raised mid-transfer does not
leave the connection stuck in COPY_IN mode.
Author: Matheus Alcantara <mths.dev@pm.me>
Discussion: https://www.postgresql.org/message-id/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/connection.c | 41 +++++
contrib/postgres_fdw/deparse.c | 55 ++++++
.../postgres_fdw/expected/postgres_fdw.out | 92 ++++++++++-
contrib/postgres_fdw/postgres_fdw.c | 156 ++++++++++++++++++
contrib/postgres_fdw/postgres_fdw.h | 3 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 100 +++++++++++
doc/src/sgml/postgres-fdw.sgml | 12 +-
7 files changed, 451 insertions(+), 8 deletions(-)
diff --git a/contrib/postgres_fdw/connection.c b/contrib/postgres_fdw/connection.c
index 094eac2f343..81cf820ca61 100644
--- a/contrib/postgres_fdw/connection.c
+++ b/contrib/postgres_fdw/connection.c
@@ -1896,6 +1896,27 @@ pgfdw_abort_cleanup(ConnCacheEntry *entry, bool toplevel)
/* Assume we might have lost track of prepared statements */
entry->have_error = true;
+ /*
+ * If a COPY FROM STDIN started by ExecForeignBatchCopy is still open on
+ * this connection, terminate it before we can send any other commands. We
+ * send a COPY end with an error message so the remote aborts it.
+ */
+ if (entry->state.copy_in_progress)
+ {
+ if (PQputCopyEnd(entry->conn, "COPY aborted due to local error") == 1)
+ {
+ PGresult *res;
+
+ /*
+ * Consume the error result from the aborted COPY. We don't care
+ * about the specific error since we're aborting anyway.
+ */
+ res = PQgetResult(entry->conn);
+ PQclear(res);
+ }
+ entry->state.copy_in_progress = false;
+ }
+
/*
* If a command has been submitted to the remote server by using an
* asynchronous execution function, the command might not have yet
@@ -1969,6 +1990,26 @@ pgfdw_abort_cleanup_begin(ConnCacheEntry *entry, bool toplevel,
/* Assume we might have lost track of prepared statements */
entry->have_error = true;
+ /*
+ * If a COPY FROM STDIN started by ExecForeignBatchCopy is still open on
+ * this connection, terminate it before we can send any other commands.
+ */
+ if (entry->state.copy_in_progress)
+ {
+ if (PQputCopyEnd(entry->conn, "COPY aborted due to local error") == 1)
+ {
+ PGresult *res;
+
+ /*
+ * Consume the error result from the aborted COPY. We don't care
+ * about the specific error since we're aborting anyway.
+ */
+ res = PQgetResult(entry->conn);
+ PQclear(res);
+ }
+ entry->state.copy_in_progress = false;
+ }
+
/*
* If a command has been submitted to the remote server by using an
* asynchronous execution function, the command might not have yet
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index 9d1932230e7..bcbc531c241 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2197,6 +2197,61 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+
+ /*
+ * Emit the column list, skipping generated columns (which the remote
+ * server fills in itself). The opening parenthesis is emitted lazily so
+ * that a relation whose transmittable columns are all generated produces
+ * no (empty) column list at all.
+ */
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ appendStringInfoString(buf, first ? "(" : ", ");
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (!first)
+ appendStringInfoChar(buf, ')');
+
+ appendStringInfoString(buf, " FROM STDIN (FORMAT TEXT)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index d19121b05da..bcd10b6f43f 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -7829,6 +7829,32 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+alter foreign table grem1 options (add batch_size '10'); -- add batch size to enable COPY
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT TEXT)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+alter foreign table grem1 options (drop batch_size);
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+alter foreign table grem2 options (add batch_size '10'); -- enable COPY
+copy grem2 from stdin;
+alter foreign table grem2 options (drop batch_size);
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -7846,16 +7872,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -7880,6 +7908,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: postgres_fdw: COPY 10 rows into foreign table
+DEBUG: postgres_fdw: COPY 2 rows into foreign table
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -9762,6 +9796,18 @@ select * from rem2;
(2 rows)
delete from rem2;
+-- Test invalid data types to assert that COPY protocol is finished on
+-- transaction abort
+alter foreign table rem2 options (add batch_size '10'); -- enable COPY
+copy rem2 from stdin;
+ERROR: invalid input syntax for type integer: "bar"
+CONTEXT: COPY rem2, line 2, column f1: "bar"
+alter foreign table rem2 options (drop batch_size);
+select * from rem2;
+ f1 | f2
+----+----
+(0 rows)
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
@@ -9927,7 +9973,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2
select * from rem2;
f1 | f2
@@ -9966,6 +10013,45 @@ select * from rem2;
drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+alter foreign table rem2 options (add batch_size '10'); -- add batch size to enable COPY
+copy rem2 (f1, f2) from stdin;
+select * from rem2;
+ f1 | f2
+----+-------
+ 1 | hello+
+ | world
+(1 row)
+
+delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 |
+ | bar
+ 3 | a"b
+(3 rows)
+
+delete from rem2;
+alter foreign table rem2 options (drop batch_size);
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+alter foreign table f_fdw options (add batch_size '10'); -- enable COPY
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+reset extra_float_digits;
+alter foreign table f_fdw options (drop batch_size);
+select * from f;
+ a
+--------------------
+ 1.0000000000000002
+(1 row)
+
+drop table f;
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 8b660a6c02c..9b79f68f746 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -18,6 +18,7 @@
#include "access/sysattr.h"
#include "access/table.h"
#include "catalog/pg_opfamily.h"
+#include "commands/copy.h"
#include "commands/defrem.h"
#include "commands/explain_format.h"
#include "commands/explain_state.h"
@@ -26,6 +27,7 @@
#include "executor/instrument.h"
#include "foreign/fdwapi.h"
#include "funcapi.h"
+#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
@@ -69,6 +71,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data */
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -433,6 +438,12 @@ static void postgresBeginForeignInsert(ModifyTableState *mtstate,
ResultRelInfo *resultRelInfo);
static void postgresEndForeignInsert(EState *estate,
ResultRelInfo *resultRelInfo);
+
+static void postgresExecForeignBatchCopy(EState *estate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ int numSlots);
+
static int postgresIsForeignRelUpdatable(Relation rel);
static bool postgresPlanDirectModify(PlannerInfo *root,
ModifyTable *plan,
@@ -638,6 +649,9 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
/*
@@ -670,6 +684,7 @@ postgres_fdw_handler(PG_FUNCTION_ARGS)
routine->EndForeignModify = postgresEndForeignModify;
routine->BeginForeignInsert = postgresBeginForeignInsert;
routine->EndForeignInsert = postgresEndForeignInsert;
+ routine->ExecForeignBatchCopy = postgresExecForeignBatchCopy;
routine->IsForeignRelUpdatable = postgresIsForeignRelUpdatable;
routine->PlanDirectModify = postgresPlanDirectModify;
routine->BeginDirectModify = postgresBeginDirectModify;
@@ -2243,6 +2258,94 @@ postgresEndForeignModify(EState *estate,
finish_foreign_modify(fmstate);
}
+/*
+ * postgresExecForeignBatchCopy
+ * Insert a batch of tuples into a foreign table using the COPY protocol.
+ *
+ * This is a self-contained operation: it opens a COPY ... FROM STDIN on the
+ * remote connection, streams all the given tuples, and closes the COPY, all
+ * within a single call. The connection is therefore left idle again once we
+ * return, which lets several foreign partitions (or local code that runs
+ * between batches, such as triggers) safely share the same cached connection.
+ */
+static void
+postgresExecForeignBatchCopy(EState *estate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ int numSlots)
+{
+ PgFdwModifyState *fmstate = (PgFdwModifyState *) resultRelInfo->ri_FdwState;
+ StringInfoData sql;
+ StringInfoData buf;
+ PGresult *res;
+ int nestlevel;
+ int i;
+
+ elog(DEBUG1, "postgres_fdw: COPY %d rows into foreign table", numSlots);
+
+ /* Build the COPY ... FROM STDIN command for this relation. */
+ initStringInfo(&sql);
+ deparseCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+
+ /* Make sure any constants are printed portably. */
+ nestlevel = set_transmission_modes();
+
+ /* Start the COPY protocol. */
+ if (!PQsendQuery(fmstate->conn, sql.data))
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, sql.data);
+ PQclear(res);
+
+ /*
+ * The connection is now in COPY_IN mode. Record that so that, if an
+ * error is thrown before we finish, transaction abort cleanup knows it
+ * must terminate the COPY before the connection can be reused.
+ */
+ fmstate->conn_state->copy_in_progress = true;
+
+ /* Stream all the rows, flushing the buffer when it grows large. */
+ initStringInfo(&buf);
+ for (i = 0; i < numSlots; i++)
+ {
+ convert_slot_to_copy_text(&buf, fmstate, slots[i]);
+
+ if (buf.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, buf.data, buf.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+ resetStringInfo(&buf);
+ }
+ }
+ if (buf.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, buf.data, buf.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+ }
+
+ /* Close the COPY protocol and check for success. */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+
+ /*
+ * The COPY termination has been sent, so the connection is no longer in
+ * COPY_IN mode; abort cleanup no longer needs to terminate it.
+ */
+ fmstate->conn_state->copy_in_progress = false;
+
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, sql.data);
+ PQclear(res);
+
+ reset_transmission_modes(nestlevel);
+
+ pfree(buf.data);
+ pfree(sql.data);
+}
+
/*
* postgresBeginForeignInsert
* Begin an insert operation on a foreign table
@@ -8766,3 +8869,56 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ /*
+ * Append a string to buf, escaping special characters for COPY
+ * TEXT format.
+ */
+ CopyEscapeText(buf,
+ value,
+ '\t',
+ GetDatabaseEncoding(),
+ false,
+ false);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..4ded16481dc 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -137,6 +137,8 @@ typedef struct PgFdwRelationInfo
typedef struct PgFdwConnState
{
AsyncRequest *pendingAreq; /* pending async request */
+ bool copy_in_progress; /* true while a COPY FROM STDIN is open on
+ * the connection */
} PgFdwConnState;
/*
@@ -204,6 +206,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index e7019952173..d16f0a88db9 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2063,6 +2063,41 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+alter foreign table grem1 options (add batch_size '10'); -- add batch size to enable COPY
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+alter foreign table grem1 options (drop batch_size);
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+alter foreign table grem2 options (add batch_size '10'); -- enable COPY
+copy grem2 from stdin;
+1
+\.
+alter foreign table grem2 options (drop batch_size);
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -2089,6 +2124,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
@@ -2956,6 +3009,16 @@ select * from rem2;
delete from rem2;
+-- Test invalid data types to assert that COPY protocol is finished on
+-- transaction abort
+alter foreign table rem2 options (add batch_size '10'); -- enable COPY
+copy rem2 from stdin;
+1 foo
+bar 2
+\.
+alter foreign table rem2 options (drop batch_size);
+select * from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
@@ -3153,6 +3216,43 @@ drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+alter foreign table rem2 options (add batch_size '10'); -- add batch size to enable COPY
+copy rem2 (f1, f2) from stdin;
+1 hello\nworld
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+1 \N
+\N bar
+3 a"b
+\.
+select * from rem2;
+
+delete from rem2;
+
+alter foreign table rem2 options (drop batch_size);
+
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+
+alter foreign table f_fdw options (add batch_size '10'); -- enable COPY
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+1.0000000000000002
+\.
+reset extra_float_digits;
+alter foreign table f_fdw options (drop batch_size);
+select * from f;
+
+drop table f;
+
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/doc/src/sgml/postgres-fdw.sgml b/doc/src/sgml/postgres-fdw.sgml
index 8b0669f672d..9593882066e 100644
--- a/doc/src/sgml/postgres-fdw.sgml
+++ b/doc/src/sgml/postgres-fdw.sgml
@@ -467,11 +467,13 @@ OPTIONS (ADD password_required 'false');
</para>
<para>
- This option also applies when copying into foreign tables. In that case
- the actual number of rows <filename>postgres_fdw</filename> copies at
- once is determined in a similar way to the insert case, but it is
- limited to at most 1000 due to implementation restrictions of the
- <command>COPY</command> command.
+ This option also applies when copying into foreign tables. When
+ <command>COPY FROM</command> is executed on a foreign table (or routed
+ into a foreign-table partition) that has this option set greater than
+ one and has no row-level triggers, <filename>postgres_fdw</filename> uses
+ the <command>COPY</command> protocol to transfer data to the remote
+ server, which is significantly faster than inserting rows individually.
+ Otherwise the rows are inserted individually.
</para>
</listitem>
</varlistentry>
--
2.50.1 (Apple Git-155)
v16-0002-Add-COPY-FROM-bulk-load-callback-to-the-FDW-API.patch (application/octet-stream, 10.0K) download
From b74dd79c4d8653e434cb386e501e6a591a2f3a9a Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Mon, 15 Jun 2026 12:05:28 -0300
Subject: [PATCH v16 2/3] Add COPY FROM bulk-load callback to the FDW API
Add a new optional FdwRoutine callback, ExecForeignBatchCopy, that lets
a foreign data wrapper load a batch of tuples during COPY FROM using a
more efficient bulk-load protocol (such as the COPY protocol for remote
PostgreSQL servers) instead of inserting the rows one at a time.
CopyFrom() invokes the callback from the multi-insert path, so it is
used when COPY FROM targets a foreign table directly or is routed into a
foreign-table partition and batching is enabled for that table (that is,
GetForeignModifyBatchSize() reports a batch size greater than one). It
is not used when the target has AFTER ROW triggers, because the
transferred rows are not returned and so cannot feed those triggers; in
that case, and whenever the callback is not provided, COPY FROM falls
back to ExecForeignBatchInsert/ExecForeignInsert.
Each ExecForeignBatchCopy call is expected to be self-contained: the FDW
opens the bulk-load session, transfers all the given tuples, and closes
it before returning. This keeps the remote connection idle between
calls, so several foreign partitions can share one connection safely.
Setup and teardown reuse the existing BeginForeignInsert and
EndForeignInsert callbacks.
Author: Matheus Alcantara <mths.dev@pm.me>
Discussion: https://www.postgresql.org/message-id/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
doc/src/sgml/fdwhandler.sgml | 49 +++++++++++++++++++++
src/backend/commands/copyfrom.c | 77 ++++++++++++++++++++++++++++++---
src/include/foreign/fdwapi.h | 6 +++
3 files changed, 127 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml
index 0103fdacfdf..732a53947d8 100644
--- a/doc/src/sgml/fdwhandler.sgml
+++ b/doc/src/sgml/fdwhandler.sgml
@@ -908,6 +908,55 @@ EndForeignInsert(EState *estate,
<para>
<programlisting>
+void
+ExecForeignBatchCopy(EState *estate,
+ ResultRelInfo *rinfo,
+ TupleTableSlot **slots,
+ int numSlots);
+</programlisting>
+
+ Insert a batch of tuples into a foreign table during a
+ <command>COPY FROM</command> command, using an optimized bulk-load
+ protocol (such as the <command>COPY</command> protocol for remote
+ <productname>PostgreSQL</productname> servers) instead of inserting the
+ tuples one at a time.
+ <literal>estate</literal> is global execution state for the query.
+ <literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct
+ describing the target foreign table.
+ <literal>slots</literal> is an array of tuple table slots containing the
+ tuples to be inserted; <literal>numSlots</literal> is the number of tuples
+ in the array.
+ </para>
+
+ <para>
+ This callback is used only when a <command>COPY FROM</command> is executed
+ directly on a foreign table, or is routed into a foreign-table partition,
+ and batching is enabled for the table (that is, the
+ <function>GetForeignModifyBatchSize</function> callback reports a batch
+ size greater than one). It is not used when the target has
+ <literal>AFTER ROW</literal> triggers, because the transferred rows are not
+ returned to feed those triggers. In all other cases the core code falls
+ back to <function>ExecForeignBatchInsert</function> or
+ <function>ExecForeignInsert</function>.
+ </para>
+
+ <para>
+ Each call is expected to be self-contained: the callback should open the
+ bulk-load session, transfer all <literal>numSlots</literal> tuples, and
+ close the session before returning. This leaves the remote connection idle
+ between calls, which allows several foreign partitions to share one
+ connection safely.
+ </para>
+
+ <para>
+ If the <function>ExecForeignBatchCopy</function> pointer is set to
+ <literal>NULL</literal>, <command>COPY FROM</command> falls back to
+ inserting tuples through
+ <function>ExecForeignInsert</function>/<function>ExecForeignBatchInsert</function>.
+ </para>
+
+ <para>
+<programlisting>
int
IsForeignRelUpdatable(Relation rel);
</programlisting>
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 80a527ed4c6..58964c6126e 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -464,7 +464,6 @@ CopyMultiInsertBufferFlush(CopyMultiInsertInfo *miinfo,
Assert(buffer->bistate == NULL);
/* Ensure that the FDW supports batching and it's enabled */
- Assert(resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert);
Assert(batch_size > 1);
/*
@@ -474,6 +473,40 @@ CopyMultiInsertBufferFlush(CopyMultiInsertInfo *miinfo,
Assert(!cstate->relname_only);
cstate->relname_only = true;
+ if (resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy != NULL &&
+ (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row))
+ {
+ /*
+ * Send the buffered tuples to the FDW in batches of at most
+ * batch_size, as we do for ExecForeignBatchInsert. Each call is a
+ * self-contained COPY operation.
+ *
+ * COPY provides no RETURNING, so this path is only usable when
+ * there are no AFTER ROW triggers that would need the stored rows.
+ */
+ while (sent < nused)
+ {
+ int size = (batch_size < nused - sent) ? batch_size : (nused - sent);
+
+ resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy(estate,
+ resultRelInfo,
+ &slots[sent],
+ size);
+
+ sent += size;
+
+ /* Update the row counter and progress of the COPY command */
+ *processed += size;
+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
+ *processed);
+ }
+ }
+ else
+ {
+ /* Ensure that the FDW supports batching and it's enabled */
+ Assert(resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert);
+
while (sent < nused)
{
int size = (batch_size < nused - sent) ? batch_size : (nused - sent);
@@ -525,6 +558,7 @@ CopyMultiInsertBufferFlush(CopyMultiInsertInfo *miinfo,
pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
*processed);
}
+ }
for (i = 0; i < nused; i++)
ExecClearTuple(slots[i]);
@@ -774,6 +808,34 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Can this foreign result relation accept batches of tuples during COPY FROM,
+ * i.e. does CopyMultiInsertBufferFlush() have a usable path for it?
+ *
+ * The FDW can batch if it provides ExecForeignBatchInsert, or if it provides
+ * ExecForeignBatchCopy and there are no AFTER ROW triggers to feed (COPY has no
+ * RETURNING, so the stored rows are not available for them). These two
+ * callbacks are independent: an FDW may implement either or both.
+ *
+ * Callers must have already established that batching is enabled for the
+ * relation (ri_BatchSize > 1).
+ */
+static bool
+CopyFromFdwCanBatch(ResultRelInfo *resultRelInfo)
+{
+ FdwRoutine *fdwroutine = resultRelInfo->ri_FdwRoutine;
+
+ if (fdwroutine->ExecForeignBatchInsert != NULL)
+ return true;
+
+ if (fdwroutine->ExecForeignBatchCopy != NULL &&
+ (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row))
+ return true;
+
+ return false;
+}
+
/*
* Copy FROM file to relation.
*/
@@ -951,7 +1013,8 @@ CopyFrom(CopyFromState cstate)
*/
if (resultRelInfo->ri_FdwRoutine != NULL &&
resultRelInfo->ri_FdwRoutine->GetForeignModifyBatchSize &&
- resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert)
+ (resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert ||
+ resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy))
resultRelInfo->ri_BatchSize =
resultRelInfo->ri_FdwRoutine->GetForeignModifyBatchSize(resultRelInfo);
else
@@ -1007,11 +1070,14 @@ CopyFrom(CopyFromState cstate)
insertMethod = CIM_SINGLE;
}
else if (resultRelInfo->ri_FdwRoutine != NULL &&
- resultRelInfo->ri_BatchSize == 1)
+ (resultRelInfo->ri_BatchSize == 1 ||
+ !CopyFromFdwCanBatch(resultRelInfo)))
{
/*
* Can't support multi-inserts to a foreign table if the FDW does not
- * support batching, or it's disabled for the server or foreign table.
+ * support batching, or it's disabled for the server or foreign table,
+ * or the FDW only offers COPY-based batching but the table has AFTER
+ * ROW triggers to feed.
*/
insertMethod = CIM_SINGLE;
}
@@ -1235,7 +1301,8 @@ CopyFrom(CopyFromState cstate)
!has_before_insert_row_trig &&
!has_instead_insert_row_trig &&
(resultRelInfo->ri_FdwRoutine == NULL ||
- resultRelInfo->ri_BatchSize > 1);
+ (resultRelInfo->ri_BatchSize > 1 &&
+ CopyFromFdwCanBatch(resultRelInfo)));
/* Set the multi-insert buffer to use for this partition. */
if (leafpart_use_multi_insert)
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index abf59a0d8ad..0439fa8b24e 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -113,6 +113,11 @@ typedef void (*BeginForeignInsert_function) (ModifyTableState *mtstate,
typedef void (*EndForeignInsert_function) (EState *estate,
ResultRelInfo *rinfo);
+typedef void (*ExecForeignBatchCopy_function) (EState *estate,
+ ResultRelInfo *rinfo,
+ TupleTableSlot **slots,
+ int numSlots);
+
typedef int (*IsForeignRelUpdatable_function) (Relation rel);
typedef bool (*PlanDirectModify_function) (PlannerInfo *root,
@@ -241,6 +246,7 @@ typedef struct FdwRoutine
EndForeignModify_function EndForeignModify;
BeginForeignInsert_function BeginForeignInsert;
EndForeignInsert_function EndForeignInsert;
+ ExecForeignBatchCopy_function ExecForeignBatchCopy;
IsForeignRelUpdatable_function IsForeignRelUpdatable;
PlanDirectModify_function PlanDirectModify;
BeginDirectModify_function BeginDirectModify;
--
2.50.1 (Apple Git-155)
v16-0001-Extract-CopyEscapeText-for-reuse-outside-COPY-TO.patch (application/octet-stream, 6.5K) download
From 49fcedb340732b87d25014d0e6cbb7d977c3c59a Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Mon, 18 May 2026 19:23:43 -0300
Subject: [PATCH v16 1/3] Extract CopyEscapeText() for reuse outside COPY TO
Refactor CopyAttributeOutText() to extract its core text escaping logic
into a new public function CopyEscapeText() that operates on a StringInfo
buffer with explicit parameters, removing the dependency on CopyToState.
This enables other code paths, such as postgres_fdw, to reuse the COPY
text format escaping logic without needing to construct a full CopyToState.
CopyAttributeOutText() now becomes a thin wrapper that calls
CopyEscapeText() with the appropriate values from CopyToState.
Also introduce CopySendDataBuf() and CopySendCharBuf() macros as low-level
helpers that operate directly on StringInfo buffers.
Author: Matheus Alcantara <mths.dev@pm.me>
Discussion: https://www.postgresql.org/message-id/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
src/backend/commands/copyto.c | 85 ++++++++++++++++++++++++++---------
src/include/commands/copy.h | 6 +++
2 files changed, 71 insertions(+), 20 deletions(-)
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index b0bdfb58104..3e1f5be9f51 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -571,6 +571,12 @@ SendCopyEnd(CopyToState cstate)
pq_putemptymessage(PqMsg_CopyDone);
}
+#define CopySendCharBuf(buf, c) \
+ appendStringInfoCharMacro(buf, c)
+
+#define CopySendDataBuf(buf, databuf, datasize) \
+ appendBinaryStringInfo(buf, databuf, datasize)
+
/*----------
* CopySendData sends output data to the destination (file or frontend)
* CopySendString does the same for null-terminated strings
@@ -584,7 +590,7 @@ SendCopyEnd(CopyToState cstate)
static void
CopySendData(CopyToState cstate, const void *databuf, int datasize)
{
- appendBinaryStringInfo(cstate->fe_msgbuf, databuf, datasize);
+ CopySendDataBuf(cstate->fe_msgbuf, databuf, datasize);
}
static void
@@ -596,7 +602,7 @@ CopySendString(CopyToState cstate, const char *str)
static void
CopySendChar(CopyToState cstate, char c)
{
- appendStringInfoCharMacro(cstate->fe_msgbuf, c);
+ CopySendCharBuf(cstate->fe_msgbuf, c);
}
static void
@@ -1439,16 +1445,44 @@ CopyOneRowTo(CopyToState cstate, TupleTableSlot *slot)
CopySendData(cstate, start, ptr - start); \
} while (0)
-static void
-CopyAttributeOutText(CopyToState cstate, const char *string)
+/* Like above, but it works with a string buffer */
+#define DUMPSOFAR_TO_BUF() \
+ do { \
+ if (ptr > start) \
+ CopySendDataBuf(buf, start, ptr - start); \
+ } while (0)
+
+
+/*
+ * Escape a string for COPY TEXT format output
+ *
+ * Escapes control characters, backslashes, and the delimiter character
+ * according to COPY TEXT format rules. The escaped string is appended
+ * to 'buf'.
+ *
+ * Parameters:
+ * buf - StringInfo buffer to append the escaped text to
+ * string - the input string to escape
+ * delimc - the delimiter character that must be escaped
+ * file_encoding - target encoding for the output
+ * need_transcoding - if true, convert from server encoding to file_encoding
+ * encoding_embeds_ascii - if true, the encoding may have ASCII bytes as
+ * non-first bytes of multi-byte characters
+ */
+void
+CopyEscapeText(StringInfo buf,
+ const char *string,
+ char delimc,
+ int file_encoding,
+ bool need_transcoding,
+ bool encoding_embeds_ascii)
{
const char *ptr;
const char *start;
char c;
- char delimc = cstate->opts.delim[0];
- if (cstate->need_transcoding)
- ptr = pg_server_to_any(string, strlen(string), cstate->file_encoding);
+ if (need_transcoding)
+ ptr = pg_server_to_any(string, strlen(string), file_encoding);
else
ptr = string;
@@ -1466,7 +1500,7 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
* it's worth making two copies of it to get the IS_HIGHBIT_SET() test out
* of the normal safe-encoding path.
*/
- if (cstate->encoding_embeds_ascii)
+ if (encoding_embeds_ascii)
{
start = ptr;
while ((c = *ptr) != '\0')
@@ -1509,19 +1543,19 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
continue; /* fall to end of loop */
}
/* if we get here, we need to convert the control char */
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
- CopySendChar(cstate, c);
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
+ CopySendCharBuf(buf, c);
start = ++ptr; /* do not include char in next run */
}
else if (c == '\\' || c == delimc)
{
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
start = ptr++; /* we include char in next run */
}
else if (IS_HIGHBIT_SET(c))
- ptr += pg_encoding_mblen(cstate->file_encoding, ptr);
+ ptr += pg_encoding_mblen(file_encoding, ptr);
else
ptr++;
}
@@ -1569,15 +1603,15 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
continue; /* fall to end of loop */
}
/* if we get here, we need to convert the control char */
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
- CopySendChar(cstate, c);
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
+ CopySendCharBuf(buf, c);
start = ++ptr; /* do not include char in next run */
}
else if (c == '\\' || c == delimc)
{
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
start = ptr++; /* we include char in next run */
}
else
@@ -1585,7 +1619,18 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
}
}
- DUMPSOFAR();
+ DUMPSOFAR_TO_BUF();
+}
+
+static void
+CopyAttributeOutText(CopyToState cstate, const char *string)
+{
+ CopyEscapeText(cstate->fe_msgbuf,
+ string,
+ cstate->opts.delim[0],
+ cstate->file_encoding,
+ cstate->need_transcoding,
+ cstate->encoding_embeds_ascii);
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index abecfe51098..27ee58c7fdb 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -136,5 +136,11 @@ extern void EndCopyTo(CopyToState cstate);
extern uint64 DoCopyTo(CopyToState cstate);
extern List *CopyGetAttnums(TupleDesc tupDesc, Relation rel,
List *attnamelist);
+extern void CopyEscapeText(StringInfo buf,
+ const char *string,
+ char delimc,
+ int file_encoding,
+ bool need_transcoding,
+ bool encoding_embeds_ascii);
#endif /* COPY_H */
--
2.50.1 (Apple Git-155)
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-07 11:38 ` Re: postgres_fdw: Use COPY to speed up batch inserts solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
2026-05-27 23:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-08-05 21:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-09-09 20:10 ` Kirill Reshke <reshkekirill@gmail.com>
2026-09-09 20:16 ` Re: postgres_fdw: Use COPY to speed up batch inserts Kirill Reshke <reshkekirill@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Kirill Reshke @ 2026-09-09 20:10 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
On Thu, 6 Aug 2026 at 02:33, Matheus Alcantara <matheusssilv97@gmail.com> wrote:
>
> Hi,
>
> Attached is a new version of the patch taking a different approach from
> the earlier attempts.
>
> Off-list I discussed with Masahiko the idea of adding a dedicated COPY
> API to the FDW handler, instead of trying to bolt COPY onto the existing
> insert callbacks, and the attached patch implements that. The core
> addition is a single new optional FdwRoutine callback:
>
> void
> ExecForeignBatchCopy(EState *estate,
> ResultRelInfo *rinfo,
> TupleTableSlot **slots,
> int numSlots);
>
> When a foreign table (or a foreign-table partition) is the target of
> COPY FROM and batching is enabled, CopyFrom() hands each buffered batch
> of tuples to this callback instead of ExecForeignInsert /
> ExecForeignBatchInsert. postgres_fdw implements it by running a
> COPY ... FROM STDIN on the remote connection and streaming the rows in
> text format.
>
> I first tried the more obvious design of splitting this into
> BeginForeignCopy / ExecForeignCopy / EndForeignCopy, keeping a single
> COPY ... FROM STDIN open across the entire COPY command and closing it
> only in EndForeignCopy. That doesn't work in general because of
> connection sharing. postgres_fdw caches one connection per user
> mapping, so:
>
> - when COPY routes tuples into several foreign partitions that map to
> the same remote server (the sharding case), or
>
> - when local code runs another query on the same connection in the
> middle of the COPY (e.g. a trigger or a volatile default expression
> that reads a foreign table on the same server),
>
> we would need a second command in flight on a connection that is stuck
> in COPY_IN mode, which fails. Making each call self-contained avoids
> this entirely, the connection is always left idle between calls, so
> partitions can share it and any interleaved query just works.
>
> The trade-off is that COPY is only used when batching is enabled
> (batch_size > 1). With batching disabled we would be starting and
> finishing a COPY protocol for every single row, which is slower than a
> plain INSERT, so in that case COPY FROM keeps inserting rows one at a
> time, exactly as today. The new callback is also independent of
> ExecForeignBatchInsert, a FDW may implement either or both. COPY is
> not used when the target has AFTER ROW triggers, because COPY has no
> RETURNING to feed them, such cases fall back to the batch-insert path.
>
> This is a PoC patch to experiment the idea of having a dedicated API,
> but I think that the main advantage of this is that we can involve the
> API to be more flexible and e.g use some user custom COPY options when
> writing the remote SQL for the remote server (although this patch still
> doesn't implement this) which would be hard using the ExecForeignInsert
> API. Also, I think that we can have a BeginForeignCopy and
> EndForeignCopy to initialize states used on ExecForeignBatchCopy (e.g
> the sql COPY command).
>
> The patch is split into three parts:
>
> 0001 - Extract CopyEscapeText() so it can be reused outside COPY TO.
> 0002 - Add the ExecForeignBatchCopy callback to the FDW API (copyfrom.c,
> fdwapi.h and the FDW documentation).
> 0003 - Implement the callback in postgres_fdw.
>
> Thoughts on the API and the self-contained approach are very welcome.
>
> --
> Matheus Alcantara
> EDB: https://www.enterprisedb.com
Hi!
I noticed this thread and did some benchmarking. I did benchmark v14
and v16 patches.
In my benchmarks, I used a local VM setup and two VM in different AZ setup.
In the bench, I didn't measure pgbench inserts but instead I measured
COPY <foreign table> FROM <file> speed.
tldr: connection caching is very beneficial for COPY speed, CSV ->
TEXT format also did show a little improvement.
So, when I ran benches for v14/v16 I noticed that COPY speed was
drastically worse. With some gdb/strace/perf I figured out that
connection re-acquire in COPY is very very bad for my bench perf. So I
implemented caching for COPY: enter COPY IN mode in the beginning of
batch insert mode, then send only COPY DATA messages. Patches
attached, but they are very PoC, draft.
CREATE FOREIGN TABLE ft (id int, value int, note text) SERVER remote
OPTIONS (table_name 't', batch_size 'xxx');
For local VM
- 1 machine, 2 PostgreSQL instances (source:5532 with fdw, target:5533)
- source has foreign table `ft` pointing to target table `t`
- `COPY ft FROM 'data.tsv'` reads file on source, streams to target via fdw
- batch_size set via foreign table option
- HEAD = unpatched PostgreSQL (uses prepared INSERT with N VALUES per batch)
## Benchmark: 100K rows x 10KB strings (956MB total)
Long strings shows CSV vs TEXT parsing overhead.
```
HEAD v14 v16 v14+cache v16+cache
bs=100 11.8s 20.8s 18.8s 9.7s 7.6s
bs=1000 11.6s 21.1s 18.7s 9.9s 7.5s
bs=10000 11.8s 20.9s 18.5s 9.8s 7.5s
```
## Benchmark: 1M small rows (42MB total)
Small rows (3 int/text columns, ~42 bytes per row).
```
HEAD v14 v16 v14+cache v16+cache
bs=100 5.1s 6.1s 6.1s 1.0s 1.2s
bs=1000 3.5s 4.1s 4.1s 1.1s 1.1s
bs=10000 3.6s 4.2s 4.2s 1.1s 1.0s
```
## Benchmark: 10M rows (446MB, small strings)
```
HEAD v14 v16+cache
bs=100 51.3s 59.0s 9.8s
bs=1000 36.6s 42.5s 9.5s
bs=10000 36.8s — 9.4s
bs=100000 36.7s — 9.4s
```
v16+cache is faster than v14+cache on long strings (7.5s vs 9.7s) due to
TEXT format. On small rows they are similar (~1.0-1.2s).
=============
For two VM setups results are much scarier. A COPY of 1000000 rows
works ~ twice longer. With big batch sizes it would be 4.9s on HEAD
while with v14/v16. WIth batch size 100 it will be 36s vs 70s. I
didn't prepare the full result table for this case, just did a few
runs.
```
HEAD v14 v16 v16+cache
bs=100 35-36s 70s 70s 500-600ms
bs=1000 4.9s 9.8s (didn't measure) 500-600ms
````
By the way I noticed that batch_size is practically always no more
than 1000 in batch insert callback (even though configured to a bigger
value in relation options.). I didn't look for the exact reason.
Also I faced cancel query problems many times. With v14 or v16 `copy
ft from '/tmp/fdw_bench/copy_data_1000000.tsv';` is not cancellable
with postgres infinity looping here:
```
warning: 3353 ./malloc/malloc.c: No such file or directory
#0 0x000070d14aeb5643 in __GI___libc_free (mem=0x5cd07fa287a0) at
./malloc/malloc.c:3353
#1 0x000070d1375a86c6 in PQclear (res=<optimized out>) at fe-exec.c:776
#2 0x000070d14b06ec87 in libpqsrv_PQclear (bres=0x5cd07f9a57b0) at
../../src/include/libpq/libpq-be-fe.h:100
#3 libpqsrv_PQclear (bres=0x5cd07f9a57b0) at
../../src/include/libpq/libpq-be-fe.h:95
#4 pgfdw_get_cleanup_result (conn=conn@entry=0x5cd07f9faab0,
endtime=842268423525902, retrycanceltime=842268394525902,
result=result@entry=0x7ffc9a3f3b90,
timed_out=timed_out@entry=0x7ffc9a3f3b8f) at connection.c:1857
#5 0x000070d14b06ee7e in pgfdw_cancel_query_end
(conn=conn@entry=0x5cd07f9faab0,
endtime=endtime@entry=842268423525902,
retrycanceltime=retrycanceltime@entry=842268394525902,
consume_input=consume_input@entry=false) at connection.c:1642
#6 0x000070d14b06f2be in pgfdw_cancel_query (conn=0x5cd07f9faab0) at
connection.c:1594
#7 pgfdw_abort_cleanup (entry=entry@entry=0x5cd07f9401b8,
toplevel=toplevel@entry=true) at connection.c:1909
#8 0x000070d14b06fb4d in pgfdw_xact_callback (event=XACT_EVENT_ABORT,
arg=<optimized out>) at connection.c:1284
#9 0x00005cd05fd89e32 in CallXactCallbacks (event=XACT_EVENT_ABORT)
at xact.c:3900
#10 AbortTransaction () at xact.c:3015
#11 0x00005cd05fd8a6d6 in AbortCurrentTransactionInternal () at xact.c:3631
#12 AbortCurrentTransaction () at xact.c:3509
#13 0x00005cd0600aa64d in PostgresMain (dbname=<optimized out>,
username=<optimized out>) at postgres.c:4629
#14 0x00005cd0600a4dc5 in BackendMain (startup_data=<optimized out>,
startup_data_len=<optimized out>) at backend_startup.c:124
#15 0x00005cd05fffa5ab in postmaster_child_launch
(child_type=<optimized out>, child_slot=1,
startup_data=startup_data@entry=0x7ffc9a3f4140,
startup_data_len=startup_data_len@entry=24,
client_sock=client_sock@entry=0x7ffc9a3f4160) at launch_backend.c:268
#16 0x00005cd05fffde0a in BackendStartup (client_sock=0x7ffc9a3f4160)
at postmaster.c:3643
#17 ServerLoop () at postmaster.c:1729
#18 0x00005cd05ffff902 in PostmasterMain (argc=argc@entry=3,
argv=argv@entry=0x5cd07f904ec0) at postmaster.c:1416
#19 0x00005cd05fcea798 in main (argc=3, argv=0x5cd07f904ec0) at main.c:227
[Inferior 1 (process 83146) detached]
```
Looks like PostgreSQL FDW connection state housekeeping doesn't work
well with PGASYNC_COPY_IN.
with this diff
```
reshke@gp-cbdb-bench:~/postgres/contrib$ git diff postgres_fdw/connection.c
diff --git a/contrib/postgres_fdw/connection.c
b/contrib/postgres_fdw/connection.c
index edfff43426a..c34f2acd9ef 100644
inline patch 1: contrib/postgres_fdw/connection.c (+0/-0 lines) download inline-1.patch
--- a/contrib/postgres_fdw/connection.c
+++ b/contrib/postgres_fdw/connection.c
@@ -1784,20 +1784,21 @@ pgfdw_get_cleanup_result(PGconn *conn,
v16_plus_cached_full_diff.patch (application/octet-stream, 34.0K) download
diff --git a/contrib/postgres_fdw/connection.c b/contrib/postgres_fdw/connection.c
index b5d4cf3dccc..f3492cd9c85 100644
--- a/contrib/postgres_fdw/connection.c
+++ b/contrib/postgres_fdw/connection.c
@@ -1214,6 +1214,21 @@ pgfdw_xact_callback(XactEvent event, void *arg)
*/
pgfdw_reject_incomplete_xact_state_change(entry);
+ /* Close COPY if in progress before committing */
+ if (entry->state.copy_in_progress)
+ {
+ PGresult *copyres;
+
+ if (PQputCopyEnd(entry->conn, NULL) < 0 ||
+ PQflush(entry->conn))
+ pgfdw_report_error(NULL, entry->conn, NULL);
+ copyres = pgfdw_get_result(entry->conn);
+ if (PQresultStatus(copyres) != PGRES_COMMAND_OK)
+ pgfdw_report_error(copyres, entry->conn, NULL);
+ PQclear(copyres);
+ entry->state.copy_in_progress = false;
+ }
+
/* Commit all remote transactions during pre-commit */
entry->changing_xact_state = true;
if (entry->parallel_commit)
@@ -1899,6 +1914,27 @@ pgfdw_abort_cleanup(ConnCacheEntry *entry, bool toplevel)
/* Assume we might have lost track of prepared statements */
entry->have_error = true;
+ /*
+ * If a COPY FROM STDIN started by ExecForeignBatchCopy is still open on
+ * this connection, terminate it before we can send any other commands. We
+ * send a COPY end with an error message so the remote aborts it.
+ */
+ if (entry->state.copy_in_progress)
+ {
+ if (PQputCopyEnd(entry->conn, "COPY aborted due to local error") == 1)
+ {
+ PGresult *res;
+
+ /*
+ * Consume the error result from the aborted COPY. We don't care
+ * about the specific error since we're aborting anyway.
+ */
+ res = PQgetResult(entry->conn);
+ PQclear(res);
+ }
+ entry->state.copy_in_progress = false;
+ }
+
/*
* If a command has been submitted to the remote server by using an
* asynchronous execution function, the command might not have yet
@@ -1972,6 +2008,26 @@ pgfdw_abort_cleanup_begin(ConnCacheEntry *entry, bool toplevel,
/* Assume we might have lost track of prepared statements */
entry->have_error = true;
+ /*
+ * If a COPY FROM STDIN started by ExecForeignBatchCopy is still open on
+ * this connection, terminate it before we can send any other commands.
+ */
+ if (entry->state.copy_in_progress)
+ {
+ if (PQputCopyEnd(entry->conn, "COPY aborted due to local error") == 1)
+ {
+ PGresult *res;
+
+ /*
+ * Consume the error result from the aborted COPY. We don't care
+ * about the specific error since we're aborting anyway.
+ */
+ res = PQgetResult(entry->conn);
+ PQclear(res);
+ }
+ entry->state.copy_in_progress = false;
+ }
+
/*
* If a command has been submitted to the remote server by using an
* asynchronous execution function, the command might not have yet
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index 673b678826c..c73b3be2cd3 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2282,6 +2282,61 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+
+ /*
+ * Emit the column list, skipping generated columns (which the remote
+ * server fills in itself). The opening parenthesis is emitted lazily so
+ * that a relation whose transmittable columns are all generated produces
+ * no (empty) column list at all.
+ */
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ appendStringInfoString(buf, first ? "(" : ", ");
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (!first)
+ appendStringInfoChar(buf, ')');
+
+ appendStringInfoString(buf, " FROM STDIN (FORMAT TEXT)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 517d15cf1fa..c9358d7a019 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -8454,6 +8454,32 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+alter foreign table grem1 options (add batch_size '10'); -- add batch size to enable COPY
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT TEXT)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+alter foreign table grem1 options (drop batch_size);
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+alter foreign table grem2 options (add batch_size '10'); -- enable COPY
+copy grem2 from stdin;
+alter foreign table grem2 options (drop batch_size);
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -8471,16 +8497,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -8505,6 +8533,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: postgres_fdw: COPY 10 rows into foreign table
+DEBUG: postgres_fdw: COPY 2 rows into foreign table
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -10387,6 +10421,18 @@ select * from rem2;
(2 rows)
delete from rem2;
+-- Test invalid data types to assert that COPY protocol is finished on
+-- transaction abort
+alter foreign table rem2 options (add batch_size '10'); -- enable COPY
+copy rem2 from stdin;
+ERROR: invalid input syntax for type integer: "bar"
+CONTEXT: COPY rem2, line 2, column f1: "bar"
+alter foreign table rem2 options (drop batch_size);
+select * from rem2;
+ f1 | f2
+----+----
+(0 rows)
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
@@ -10552,7 +10598,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2
select * from rem2;
f1 | f2
@@ -10591,6 +10638,45 @@ select * from rem2;
drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+alter foreign table rem2 options (add batch_size '10'); -- add batch size to enable COPY
+copy rem2 (f1, f2) from stdin;
+select * from rem2;
+ f1 | f2
+----+-------
+ 1 | hello+
+ | world
+(1 row)
+
+delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 |
+ | bar
+ 3 | a"b
+(3 rows)
+
+delete from rem2;
+alter foreign table rem2 options (drop batch_size);
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+alter foreign table f_fdw options (add batch_size '10'); -- enable COPY
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+reset extra_float_digits;
+alter foreign table f_fdw options (drop batch_size);
+select * from f;
+ a
+--------------------
+ 1.0000000000000002
+(1 row)
+
+drop table f;
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 47197a733ff..7d290f1035e 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -18,6 +18,7 @@
#include "access/sysattr.h"
#include "access/table.h"
#include "catalog/pg_opfamily.h"
+#include "commands/copy.h"
#include "commands/defrem.h"
#include "commands/explain_format.h"
#include "commands/explain_state.h"
@@ -26,6 +27,7 @@
#include "executor/instrument.h"
#include "foreign/fdwapi.h"
#include "funcapi.h"
+#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
@@ -70,6 +72,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data */
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -459,6 +464,12 @@ static void postgresBeginForeignInsert(ModifyTableState *mtstate,
ResultRelInfo *resultRelInfo);
static void postgresEndForeignInsert(EState *estate,
ResultRelInfo *resultRelInfo);
+
+static void postgresExecForeignBatchCopy(EState *estate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ int numSlots);
+
static int postgresIsForeignRelUpdatable(Relation rel);
static bool postgresPlanDirectModify(PlannerInfo *root,
ModifyTable *plan,
@@ -666,6 +677,9 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
/*
@@ -698,6 +712,7 @@ postgres_fdw_handler(PG_FUNCTION_ARGS)
routine->EndForeignModify = postgresEndForeignModify;
routine->BeginForeignInsert = postgresBeginForeignInsert;
routine->EndForeignInsert = postgresEndForeignInsert;
+ routine->ExecForeignBatchCopy = postgresExecForeignBatchCopy;
routine->IsForeignRelUpdatable = postgresIsForeignRelUpdatable;
routine->PlanDirectModify = postgresPlanDirectModify;
routine->BeginDirectModify = postgresBeginDirectModify;
@@ -2370,6 +2385,80 @@ postgresEndForeignModify(EState *estate,
finish_foreign_modify(fmstate);
}
+/*
+ * postgresExecForeignBatchCopy
+ * Insert a batch of tuples into a foreign table using the COPY protocol.
+ *
+ * This is a self-contained operation: it opens a COPY ... FROM STDIN on the
+ * remote connection, streams all the given tuples, and closes the COPY, all
+ * within a single call. The connection is therefore left idle again once we
+ * return, which lets several foreign partitions (or local code that runs
+ * between batches, such as triggers) safely share the same cached connection.
+ */
+static void
+postgresExecForeignBatchCopy(EState *estate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ int numSlots)
+{
+ PgFdwModifyState *fmstate = (PgFdwModifyState *) resultRelInfo->ri_FdwState;
+ StringInfoData buf;
+ int nestlevel;
+ int i;
+
+ elog(DEBUG1, "postgres_fdw: COPY %d rows into foreign table", numSlots);
+
+ /* Make sure any constants are printed portably. */
+ nestlevel = set_transmission_modes();
+
+ /* Start COPY if not already in progress on this connection */
+ if (!fmstate->conn_state->copy_in_progress)
+ {
+ StringInfoData sql;
+ PGresult *res;
+
+ /* Build the COPY ... FROM STDIN command for this relation. */
+ initStringInfo(&sql);
+ deparseCopySql(&sql, fmstate->rel, fmstate->target_attrs);
+
+ /* Start the COPY protocol. */
+ if (!PQsendQuery(fmstate->conn, sql.data))
+ pgfdw_report_error(NULL, fmstate->conn, sql.data);
+
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, sql.data);
+ PQclear(res);
+
+ fmstate->conn_state->copy_in_progress = true;
+
+ pfree(sql.data);
+ }
+
+ /* Stream all the rows, flushing the buffer when it grows large. */
+ initStringInfo(&buf);
+ for (i = 0; i < numSlots; i++)
+ {
+ convert_slot_to_copy_text(&buf, fmstate, slots[i]);
+
+ if (buf.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, buf.data, buf.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, NULL);
+ resetStringInfo(&buf);
+ }
+ }
+ if (buf.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, buf.data, buf.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, NULL);
+ }
+
+ pfree(buf.data);
+
+ reset_transmission_modes(nestlevel);
+}
+
/*
* postgresBeginForeignInsert
* Begin an insert operation on a foreign table
@@ -9266,3 +9355,56 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ /*
+ * Append a string to buf, escaping special characters for COPY
+ * TEXT format.
+ */
+ CopyEscapeText(buf,
+ value,
+ '\t',
+ GetDatabaseEncoding(),
+ false,
+ false);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index da7da1c2ea9..3135ba8a7cf 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -147,6 +147,8 @@ typedef struct PgFdwRelationInfo
typedef struct PgFdwConnState
{
AsyncRequest *pendingAreq; /* pending async request */
+ bool copy_in_progress; /* true while a COPY FROM STDIN is open on
+ * the connection */
} PgFdwConnState;
/*
@@ -216,6 +218,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index ec766e2b28a..193ff49806b 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2337,6 +2337,41 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+alter foreign table grem1 options (add batch_size '10'); -- add batch size to enable COPY
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+alter foreign table grem1 options (drop batch_size);
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+alter foreign table grem2 options (add batch_size '10'); -- enable COPY
+copy grem2 from stdin;
+1
+\.
+alter foreign table grem2 options (drop batch_size);
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -2363,6 +2398,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
@@ -3230,6 +3283,16 @@ select * from rem2;
delete from rem2;
+-- Test invalid data types to assert that COPY protocol is finished on
+-- transaction abort
+alter foreign table rem2 options (add batch_size '10'); -- enable COPY
+copy rem2 from stdin;
+1 foo
+bar 2
+\.
+alter foreign table rem2 options (drop batch_size);
+select * from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
@@ -3427,6 +3490,43 @@ drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+alter foreign table rem2 options (add batch_size '10'); -- add batch size to enable COPY
+copy rem2 (f1, f2) from stdin;
+1 hello\nworld
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+1 \N
+\N bar
+3 a"b
+\.
+select * from rem2;
+
+delete from rem2;
+
+alter foreign table rem2 options (drop batch_size);
+
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+
+alter foreign table f_fdw options (add batch_size '10'); -- enable COPY
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+1.0000000000000002
+\.
+reset extra_float_digits;
+alter foreign table f_fdw options (drop batch_size);
+select * from f;
+
+drop table f;
+
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml
index 502441fefcb..8a15cdc4b01 100644
--- a/doc/src/sgml/fdwhandler.sgml
+++ b/doc/src/sgml/fdwhandler.sgml
@@ -908,6 +908,55 @@ EndForeignInsert(EState *estate,
<para>
<programlisting>
+void
+ExecForeignBatchCopy(EState *estate,
+ ResultRelInfo *rinfo,
+ TupleTableSlot **slots,
+ int numSlots);
+</programlisting>
+
+ Insert a batch of tuples into a foreign table during a
+ <command>COPY FROM</command> command, using an optimized bulk-load
+ protocol (such as the <command>COPY</command> protocol for remote
+ <productname>PostgreSQL</productname> servers) instead of inserting the
+ tuples one at a time.
+ <literal>estate</literal> is global execution state for the query.
+ <literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct
+ describing the target foreign table.
+ <literal>slots</literal> is an array of tuple table slots containing the
+ tuples to be inserted; <literal>numSlots</literal> is the number of tuples
+ in the array.
+ </para>
+
+ <para>
+ This callback is used only when a <command>COPY FROM</command> is executed
+ directly on a foreign table, or is routed into a foreign-table partition,
+ and batching is enabled for the table (that is, the
+ <function>GetForeignModifyBatchSize</function> callback reports a batch
+ size greater than one). It is not used when the target has
+ <literal>AFTER ROW</literal> triggers, because the transferred rows are not
+ returned to feed those triggers. In all other cases the core code falls
+ back to <function>ExecForeignBatchInsert</function> or
+ <function>ExecForeignInsert</function>.
+ </para>
+
+ <para>
+ Each call is expected to be self-contained: the callback should open the
+ bulk-load session, transfer all <literal>numSlots</literal> tuples, and
+ close the session before returning. This leaves the remote connection idle
+ between calls, which allows several foreign partitions to share one
+ connection safely.
+ </para>
+
+ <para>
+ If the <function>ExecForeignBatchCopy</function> pointer is set to
+ <literal>NULL</literal>, <command>COPY FROM</command> falls back to
+ inserting tuples through
+ <function>ExecForeignInsert</function>/<function>ExecForeignBatchInsert</function>.
+ </para>
+
+ <para>
+<programlisting>
int
IsForeignRelUpdatable(Relation rel);
</programlisting>
diff --git a/doc/src/sgml/postgres-fdw.sgml b/doc/src/sgml/postgres-fdw.sgml
index 87b1433aacb..0853c734ca5 100644
--- a/doc/src/sgml/postgres-fdw.sgml
+++ b/doc/src/sgml/postgres-fdw.sgml
@@ -467,11 +467,13 @@ OPTIONS (ADD password_required 'false');
</para>
<para>
- This option also applies when copying into foreign tables. In that case
- the actual number of rows <filename>postgres_fdw</filename> copies at
- once is determined in a similar way to the insert case, but it is
- limited to at most 1000 due to implementation restrictions of the
- <command>COPY</command> command.
+ This option also applies when copying into foreign tables. When
+ <command>COPY FROM</command> is executed on a foreign table (or routed
+ into a foreign-table partition) that has this option set greater than
+ one and has no row-level triggers, <filename>postgres_fdw</filename> uses
+ the <command>COPY</command> protocol to transfer data to the remote
+ server, which is significantly faster than inserting rows individually.
+ Otherwise the rows are inserted individually.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 3782db171be..63c5bd1501e 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -464,7 +464,6 @@ CopyMultiInsertBufferFlush(CopyMultiInsertInfo *miinfo,
Assert(buffer->bistate == NULL);
/* Ensure that the FDW supports batching and it's enabled */
- Assert(resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert);
Assert(batch_size > 1);
/*
@@ -474,6 +473,40 @@ CopyMultiInsertBufferFlush(CopyMultiInsertInfo *miinfo,
Assert(!cstate->relname_only);
cstate->relname_only = true;
+ if (resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy != NULL &&
+ (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row))
+ {
+ /*
+ * Send the buffered tuples to the FDW in batches of at most
+ * batch_size, as we do for ExecForeignBatchInsert. Each call is a
+ * self-contained COPY operation.
+ *
+ * COPY provides no RETURNING, so this path is only usable when
+ * there are no AFTER ROW triggers that would need the stored rows.
+ */
+ while (sent < nused)
+ {
+ int size = (batch_size < nused - sent) ? batch_size : (nused - sent);
+
+ resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy(estate,
+ resultRelInfo,
+ &slots[sent],
+ size);
+
+ sent += size;
+
+ /* Update the row counter and progress of the COPY command */
+ *processed += size;
+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
+ *processed);
+ }
+ }
+ else
+ {
+ /* Ensure that the FDW supports batching and it's enabled */
+ Assert(resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert);
+
while (sent < nused)
{
int size = (batch_size < nused - sent) ? batch_size : (nused - sent);
@@ -525,6 +558,7 @@ CopyMultiInsertBufferFlush(CopyMultiInsertInfo *miinfo,
pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
*processed);
}
+ }
for (i = 0; i < nused; i++)
ExecClearTuple(slots[i]);
@@ -774,6 +808,34 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Can this foreign result relation accept batches of tuples during COPY FROM,
+ * i.e. does CopyMultiInsertBufferFlush() have a usable path for it?
+ *
+ * The FDW can batch if it provides ExecForeignBatchInsert, or if it provides
+ * ExecForeignBatchCopy and there are no AFTER ROW triggers to feed (COPY has no
+ * RETURNING, so the stored rows are not available for them). These two
+ * callbacks are independent: an FDW may implement either or both.
+ *
+ * Callers must have already established that batching is enabled for the
+ * relation (ri_BatchSize > 1).
+ */
+static bool
+CopyFromFdwCanBatch(ResultRelInfo *resultRelInfo)
+{
+ FdwRoutine *fdwroutine = resultRelInfo->ri_FdwRoutine;
+
+ if (fdwroutine->ExecForeignBatchInsert != NULL)
+ return true;
+
+ if (fdwroutine->ExecForeignBatchCopy != NULL &&
+ (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row))
+ return true;
+
+ return false;
+}
+
/*
* Copy FROM file to relation.
*/
@@ -951,7 +1013,8 @@ CopyFrom(CopyFromState cstate)
*/
if (resultRelInfo->ri_FdwRoutine != NULL &&
resultRelInfo->ri_FdwRoutine->GetForeignModifyBatchSize &&
- resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert)
+ (resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert ||
+ resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy))
resultRelInfo->ri_BatchSize =
resultRelInfo->ri_FdwRoutine->GetForeignModifyBatchSize(resultRelInfo);
else
@@ -1007,11 +1070,14 @@ CopyFrom(CopyFromState cstate)
insertMethod = CIM_SINGLE;
}
else if (resultRelInfo->ri_FdwRoutine != NULL &&
- resultRelInfo->ri_BatchSize == 1)
+ (resultRelInfo->ri_BatchSize == 1 ||
+ !CopyFromFdwCanBatch(resultRelInfo)))
{
/*
* Can't support multi-inserts to a foreign table if the FDW does not
- * support batching, or it's disabled for the server or foreign table.
+ * support batching, or it's disabled for the server or foreign table,
+ * or the FDW only offers COPY-based batching but the table has AFTER
+ * ROW triggers to feed.
*/
insertMethod = CIM_SINGLE;
}
@@ -1235,7 +1301,8 @@ CopyFrom(CopyFromState cstate)
!has_before_insert_row_trig &&
!has_instead_insert_row_trig &&
(resultRelInfo->ri_FdwRoutine == NULL ||
- resultRelInfo->ri_BatchSize > 1);
+ (resultRelInfo->ri_BatchSize > 1 &&
+ CopyFromFdwCanBatch(resultRelInfo)));
/* Set the multi-insert buffer to use for this partition. */
if (leafpart_use_multi_insert)
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index 5850608a3fb..94c04baa4e7 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -571,6 +571,12 @@ SendCopyEnd(CopyToState cstate)
pq_putemptymessage(PqMsg_CopyDone);
}
+#define CopySendCharBuf(buf, c) \
+ appendStringInfoCharMacro(buf, c)
+
+#define CopySendDataBuf(buf, databuf, datasize) \
+ appendBinaryStringInfo(buf, databuf, datasize)
+
/*----------
* CopySendData sends output data to the destination (file or frontend)
* CopySendString does the same for null-terminated strings
@@ -584,7 +590,7 @@ SendCopyEnd(CopyToState cstate)
static void
CopySendData(CopyToState cstate, const void *databuf, int datasize)
{
- appendBinaryStringInfo(cstate->fe_msgbuf, databuf, datasize);
+ CopySendDataBuf(cstate->fe_msgbuf, databuf, datasize);
}
static void
@@ -596,7 +602,7 @@ CopySendString(CopyToState cstate, const char *str)
static void
CopySendChar(CopyToState cstate, char c)
{
- appendStringInfoCharMacro(cstate->fe_msgbuf, c);
+ CopySendCharBuf(cstate->fe_msgbuf, c);
}
static void
@@ -1439,16 +1445,44 @@ CopyOneRowTo(CopyToState cstate, TupleTableSlot *slot)
CopySendData(cstate, start, ptr - start); \
} while (0)
-static void
-CopyAttributeOutText(CopyToState cstate, const char *string)
+/* Like above, but it works with a string buffer */
+#define DUMPSOFAR_TO_BUF() \
+ do { \
+ if (ptr > start) \
+ CopySendDataBuf(buf, start, ptr - start); \
+ } while (0)
+
+
+/*
+ * Escape a string for COPY TEXT format output
+ *
+ * Escapes control characters, backslashes, and the delimiter character
+ * according to COPY TEXT format rules. The escaped string is appended
+ * to 'buf'.
+ *
+ * Parameters:
+ * buf - StringInfo buffer to append the escaped text to
+ * string - the input string to escape
+ * delimc - the delimiter character that must be escaped
+ * file_encoding - target encoding for the output
+ * need_transcoding - if true, convert from server encoding to file_encoding
+ * encoding_embeds_ascii - if true, the encoding may have ASCII bytes as
+ * non-first bytes of multi-byte characters
+ */
+void
+CopyEscapeText(StringInfo buf,
+ const char *string,
+ char delimc,
+ int file_encoding,
+ bool need_transcoding,
+ bool encoding_embeds_ascii)
{
const char *ptr;
const char *start;
char c;
- char delimc = cstate->opts.delim[0];
- if (cstate->need_transcoding)
- ptr = pg_server_to_any(string, strlen(string), cstate->file_encoding);
+ if (need_transcoding)
+ ptr = pg_server_to_any(string, strlen(string), file_encoding);
else
ptr = string;
@@ -1466,7 +1500,7 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
* it's worth making two copies of it to get the IS_HIGHBIT_SET() test out
* of the normal safe-encoding path.
*/
- if (cstate->encoding_embeds_ascii)
+ if (encoding_embeds_ascii)
{
start = ptr;
while ((c = *ptr) != '\0')
@@ -1509,19 +1543,19 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
continue; /* fall to end of loop */
}
/* if we get here, we need to convert the control char */
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
- CopySendChar(cstate, c);
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
+ CopySendCharBuf(buf, c);
start = ++ptr; /* do not include char in next run */
}
else if (c == '\\' || c == delimc)
{
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
start = ptr++; /* we include char in next run */
}
else if (IS_HIGHBIT_SET(c))
- ptr += pg_encoding_mblen(cstate->file_encoding, ptr);
+ ptr += pg_encoding_mblen(file_encoding, ptr);
else
ptr++;
}
@@ -1569,15 +1603,15 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
continue; /* fall to end of loop */
}
/* if we get here, we need to convert the control char */
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
- CopySendChar(cstate, c);
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
+ CopySendCharBuf(buf, c);
start = ++ptr; /* do not include char in next run */
}
else if (c == '\\' || c == delimc)
{
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
start = ptr++; /* we include char in next run */
}
else
@@ -1585,7 +1619,18 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
}
}
- DUMPSOFAR();
+ DUMPSOFAR_TO_BUF();
+}
+
+static void
+CopyAttributeOutText(CopyToState cstate, const char *string)
+{
+ CopyEscapeText(cstate->fe_msgbuf,
+ string,
+ cstate->opts.delim[0],
+ cstate->file_encoding,
+ cstate->need_transcoding,
+ cstate->encoding_embeds_ascii);
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index abecfe51098..27ee58c7fdb 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -136,5 +136,11 @@ extern void EndCopyTo(CopyToState cstate);
extern uint64 DoCopyTo(CopyToState cstate);
extern List *CopyGetAttnums(TupleDesc tupDesc, Relation rel,
List *attnamelist);
+extern void CopyEscapeText(StringInfo buf,
+ const char *string,
+ char delimc,
+ int file_encoding,
+ bool need_transcoding,
+ bool encoding_embeds_ascii);
#endif /* COPY_H */
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index abf59a0d8ad..0439fa8b24e 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -113,6 +113,11 @@ typedef void (*BeginForeignInsert_function) (ModifyTableState *mtstate,
typedef void (*EndForeignInsert_function) (EState *estate,
ResultRelInfo *rinfo);
+typedef void (*ExecForeignBatchCopy_function) (EState *estate,
+ ResultRelInfo *rinfo,
+ TupleTableSlot **slots,
+ int numSlots);
+
typedef int (*IsForeignRelUpdatable_function) (Relation rel);
typedef bool (*PlanDirectModify_function) (PlannerInfo *root,
@@ -241,6 +246,7 @@ typedef struct FdwRoutine
EndForeignModify_function EndForeignModify;
BeginForeignInsert_function BeginForeignInsert;
EndForeignInsert_function EndForeignInsert;
+ ExecForeignBatchCopy_function ExecForeignBatchCopy;
IsForeignRelUpdatable_function IsForeignRelUpdatable;
PlanDirectModify_function PlanDirectModify;
BeginDirectModify_function BeginDirectModify;
v14_plus_cached_full_diff.patch (application/octet-stream, 18.7K) download
diff --git a/contrib/postgres_fdw/connection.c b/contrib/postgres_fdw/connection.c
index b5d4cf3dccc..554c7575c71 100644
--- a/contrib/postgres_fdw/connection.c
+++ b/contrib/postgres_fdw/connection.c
@@ -1214,6 +1214,21 @@ pgfdw_xact_callback(XactEvent event, void *arg)
*/
pgfdw_reject_incomplete_xact_state_change(entry);
+ /* Close COPY if in progress before committing */
+ if (entry->state.copy_in_progress)
+ {
+ PGresult *copyres;
+
+ if (PQputCopyEnd(entry->conn, NULL) < 0 ||
+ PQflush(entry->conn))
+ pgfdw_report_error(NULL, entry->conn, NULL);
+ copyres = pgfdw_get_result(entry->conn);
+ if (PQresultStatus(copyres) != PGRES_COMMAND_OK)
+ pgfdw_report_error(copyres, entry->conn, NULL);
+ PQclear(copyres);
+ entry->state.copy_in_progress = false;
+ }
+
/* Commit all remote transactions during pre-commit */
entry->changing_xact_state = true;
if (entry->parallel_commit)
@@ -1972,6 +1987,17 @@ pgfdw_abort_cleanup_begin(ConnCacheEntry *entry, bool toplevel,
/* Assume we might have lost track of prepared statements */
entry->have_error = true;
+ /* Abort COPY if in progress */
+ if (entry->state.copy_in_progress)
+ {
+ if (PQputCopyEnd(entry->conn, "COPY aborted due to local error") == 1)
+ {
+ PGresult *copyres = PQgetResult(entry->conn);
+ PQclear(copyres);
+ }
+ entry->state.copy_in_progress = false;
+ }
+
/*
* If a command has been submitted to the remote server by using an
* asynchronous execution function, the command might not have yet
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index 673b678826c..44144973869 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2282,6 +2282,62 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the CSV format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfoChar(buf, '(');
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+
+ appendStringInfoString(buf, " (FORMAT CSV)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 517d15cf1fa..dae2f6b1f6b 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -8454,6 +8454,28 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT CSV)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -8471,16 +8493,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -8505,6 +8529,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: foreign modify with COPY batch_size: 10
+DEBUG: foreign modify with COPY batch_size: 10
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -10386,6 +10416,17 @@ select * from rem2;
2 | bar
(2 rows)
+delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 |
+ | bar
+ 3 | a"b
+(3 rows)
+
delete from rem2;
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
@@ -10395,7 +10436,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: ""-1","xyzzy""
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT CSV)
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -10552,7 +10594,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: ""-1","xyzzy""
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT CSV)
COPY rem2
select * from rem2;
f1 | f2
@@ -10591,6 +10634,30 @@ select * from rem2;
drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+select * from rem2;
+ f1 | f2
+----+-------
+ 1 | hello+
+ | world
+(1 row)
+
+delete from rem2;
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+reset extra_float_digits;
+select * from f;
+ a
+--------------------
+ 1.0000000000000002
+(1 row)
+
+drop table f;
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 47197a733ff..07b43d2e60d 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -70,6 +70,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -230,6 +233,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy;
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -666,6 +671,13 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_csv(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
+static void appendStringInfoCsv(StringInfo buf, const char *string);
/*
@@ -2386,11 +2398,12 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
RangeTblEntry *rte;
TupleDesc tupdesc = RelationGetDescr(rel);
int attnum;
- int values_end_len;
+ int values_end_len = 0;
StringInfoData sql;
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2468,11 +2481,43 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if all the following conditions
+ * are met:
+ *
+ * Direct Execution: The command is a COPY FROM on the foreign table
+ * itself, not part of a partitioned table's tuple routing. (
+ * resultRelInfo->ri_RootResultRelInfo == NULL)
+ *
+ * No Check Options: There are no WITH CHECK OPTION constraints or
+ * Row-Level Security policies that need to be enforced locally
+ * (resultRelInfo->ri_WithCheckOptions == NIL).
+ *
+ * No Local AFTER Triggers: There are no AFTER ROW triggers defined
+ * locally on the foreign table.
+ *
+ * Remote triggers might modify the inserted row. Because the COPY
+ * protocol does not support a RETURNING clause, we cannot retrieve those
+ * changes to synchronize the local TupleTableSlot required by local AFTER
+ * triggers.
+ */
+ if (resultRelInfo->ri_RootResultRelInfo == NULL && resultRelInfo->ri_WithCheckOptions == NIL)
+ {
+ /* There is no RETURNING clause on COPY */
+ Assert(resultRelInfo->ri_returningList == NIL);
+
+ useCopy = (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row);
+ }
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2485,6 +2530,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4412,6 +4458,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -9266,3 +9315,148 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+ int nestlevel;
+
+ Assert(fmstate->use_copy == true);
+
+ elog(DEBUG1, "foreign modify with COPY batch_size: %d", fmstate->batch_size);
+
+ /* Make sure any constants in the slots are printed portably */
+ nestlevel = set_transmission_modes();
+
+ /* Start COPY if not already in progress on this connection */
+ if (!fmstate->conn_state->copy_in_progress)
+ {
+ PGresult *res;
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ fmstate->conn_state->copy_in_progress = true;
+ }
+
+ /* Convert the TupleTableSlot data into a CSV-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_csv(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ reset_transmission_modes(nestlevel);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ n_rows = *numSlots;
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_csv(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, ',');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ {
+ /* In CSV format, NULL is an empty unquoted field */
+ }
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ appendStringInfoCsv(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
+/*
+ * Append a string to buf, with CSV escaping (quote field, double any quotes).
+ */
+static void
+appendStringInfoCsv(StringInfo buf, const char *string)
+{
+ const char *ptr;
+
+ appendStringInfoCharMacro(buf, '"');
+ for (ptr = string; *ptr; ptr++)
+ {
+ if (*ptr == '"')
+ appendStringInfoCharMacro(buf, '"');
+ appendStringInfoCharMacro(buf, *ptr);
+ }
+ appendStringInfoCharMacro(buf, '"');
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index da7da1c2ea9..1d3617a749d 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -147,6 +147,7 @@ typedef struct PgFdwRelationInfo
typedef struct PgFdwConnState
{
AsyncRequest *pendingAreq; /* pending async request */
+ bool copy_in_progress; /* COPY FROM STDIN is open on this connection */
} PgFdwConnState;
/*
@@ -216,6 +217,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index ec766e2b28a..3b576dd83e3 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2337,6 +2337,37 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
+1
+\.
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -2363,6 +2394,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
@@ -3230,6 +3279,16 @@ select * from rem2;
delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+1 \N
+\N bar
+3 a"b
+\.
+select * from rem2;
+
+delete from rem2;
+
-- Test check constraints
alter table loc2 add constraint loc2_f1positive check (f1 >= 0);
alter foreign table rem2 add constraint rem2_f1positive check (f1 >= 0);
@@ -3427,6 +3486,29 @@ drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+1 hello\nworld
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+1.0000000000000002
+\.
+
+reset extra_float_digits;
+select * from f;
+
+drop table f;
+
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
fdw_bench.sh (text/x-sh, 7.2K) download
#!/usr/bin/env bash
#
# postgres_fdw COPY batch insert benchmark.
#
# This script runs on the SOURCE machine. It:
# 1. Inits and starts a local PostgreSQL instance (source)
# 2. Connects to TARGET (remote host:port) via psql to create the base table
# 3. Creates a foreign table on source pointing to target (fdw on source)
# 4. Generates a TSV data file on source
# 5. Runs COPY FROM into foreign table (source pushes data to target via fdw)
# 6. Runs pgbench INSERT into foreign table
#
# The SOURCE (this machine) has the patched binaries — that's where fdw COPY lives.
# The TARGET just needs a running PostgreSQL with a table.
#
# Usage:
# ./fdw_bench.sh --target-host=127.0.0.1 --target-port=5533 [--source-port=5532] [--rows=1000000]
#
# Run twice: once with HEAD binaries, once with patched binaries. Compare.
set -euo pipefail
# --- defaults ---
SOURCE_PORT=5532
TARGET_HOST=""
TARGET_PORT=""
PGBIN="${PGBIN:-}"
PGUSER="${PGUSER:-$(whoami)}"
DATA_DIR="${DATA_DIR:-/tmp/fdw_bench}"
ROWS="${ROWS:-1000000}"
# --- parse args ---
for arg in "$@"; do
case "$arg" in
--target-host=*) TARGET_HOST="${arg#*=}" ;;
--target-port=*) TARGET_PORT="${arg#*=}" ;;
--source-port=*) SOURCE_PORT="${arg#*=}" ;;
--rows=*) ROWS="${arg#*=}" ;;
--pgbin=*) PGBIN="${arg#*=}" ;;
*) echo "Unknown arg: $arg"; exit 1 ;;
esac
done
if [ -z "$TARGET_HOST" ] || [ -z "$TARGET_PORT" ]; then
echo "Usage: $0 --target-host=HOST --target-port=PORT [--source-port=$SOURCE_PORT] [--rows=$ROWS]"
echo ""
echo "This script runs on the SOURCE machine (with patched binaries)."
echo "It initializes a local PG instance and connects to TARGET via psql"
echo "to create the base table. Then it creates a foreign table on source"
echo "pointing to target, and benchmarks COPY FROM and INSERT."
exit 1
fi
# --- resolve binaries ---
if [ -z "$PGBIN" ]; then
PGBIN=$(dirname "$(command -v initdb 2>/dev/null || echo /usr/local/pgsql/bin/initdb)")
fi
INITDB="$PGBIN/initdb"
PG_CTL="$PGBIN/pg_ctl"
PSQL="$PGBIN/psql"
PGBENCH="$PGBIN/pgbench"
# --- get source IP (best guess) ---
SOURCE_HOST=$(hostname -I 2>/dev/null | awk '{print $1}' || echo 127.0.0.1)
echo "=== Configuration ==="
echo " PGBIN: $PGBIN"
echo " SOURCE: $SOURCE_HOST:$SOURCE_PORT (local, initdb, has fdw patch)"
echo " TARGET: $TARGET_HOST:$TARGET_PORT (remote, just needs a table)"
echo " ROWS: $ROWS"
echo " PGUSER: $PGUSER"
echo " DATA_DIR: $DATA_DIR"
echo ""
SRC_DIR="$DATA_DIR/source"
mkdir -p "$DATA_DIR"
# --- helpers ---
psql_src() { "$PSQL" -h 127.0.0.1 -p "$SOURCE_PORT" -U "$PGUSER" -d postgres -v ON_ERROR_STOP=1 "$@"; }
psql_tgt() { "$PSQL" -h "$TARGET_HOST" -p "$TARGET_PORT" -U "$PGUSER" -d postgres -v ON_ERROR_STOP=1 "$@"; }
RESULTS="$DATA_DIR/results.txt"
echo "" > "$RESULTS"
# --- step 1: initdb + start source instance ---
echo "=== Initializing source instance (port $SOURCE_PORT) ==="
if [ ! -d "$SRC_DIR" ]; then
"$INITDB" -D "$SRC_DIR" --no-locale -E UTF8 -U "$PGUSER"
fi
cat > "$SRC_DIR/postgresql.conf" <<CONF
listen_addresses = '*'
port = $SOURCE_PORT
max_wal_size = 4GB
shared_buffers = 256MB
max_connections = 300
CONF
# Allow connections from target (trust auth for benchmark)
if ! grep -q "0.0.0.0/0 trust" "$SRC_DIR/pg_hba.conf" 2>/dev/null; then
echo "host all all 0.0.0.0/0 trust" >> "$SRC_DIR/pg_hba.conf"
fi
"$PG_CTL" -D "$SRC_DIR" -l "$DATA_DIR/source.log" start
sleep 3
echo " Source started on port $SOURCE_PORT"
echo ""
# --- step 2: create table on TARGET (the remote side) ---
echo "=== Creating table on target ($TARGET_HOST:$TARGET_PORT) ==="
psql_tgt -c "DROP TABLE IF EXISTS t CASCADE; CREATE TABLE t (id int, value int, note text);"
echo ""
# --- step 3: setup foreign table on SOURCE (local, pointing to target) ---
echo "=== Setting up foreign table on source (fdw -> target) ==="
psql_src -c "CREATE EXTENSION IF NOT EXISTS postgres_fdw;" 2>/dev/null || true
psql_src -c "DROP FOREIGN TABLE IF EXISTS ft; DROP SERVER IF EXISTS remote CASCADE; DROP USER MAPPING IF EXISTS $PGUSER SERVER remote;" 2>/dev/null || true
psql_src -c "CREATE SERVER remote FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '$TARGET_HOST', port '$TARGET_PORT', dbname 'postgres');"
psql_src -c "CREATE USER MAPPING FOR $PGUSER SERVER remote OPTIONS (user '$PGUSER');"
psql_src -c "CREATE FOREIGN TABLE ft (id int, value int, note text) SERVER remote OPTIONS (table_name 't', batch_size '1000');"
# Verify connectivity
echo " Testing connectivity..."
psql_src -c "INSERT INTO ft VALUES (0, 0, 'connectivity test'); SELECT count(*) FROM ft;"
psql_tgt -c "TRUNCATE t;"
echo ""
# --- step 4: generate data file on source ---
DATA_FILE="$DATA_DIR/copy_data_${ROWS}.tsv"
if [ ! -f "$DATA_FILE" ]; then
echo "=== Generating $ROWS rows of TSV data ==="
seq 0 $((ROWS - 1)) | awk -v OFS='\t' '{print $1, $1*100, "batch copy test data " $1}' > "$DATA_FILE"
echo " File: $DATA_FILE ($(du -h "$DATA_FILE" | cut -f1))"
fi
echo ""
# --- step 5: COPY FROM benchmarks (run on source, data file is local) ---
echo "=== COPY FROM $ROWS rows (local file -> fdw -> target) ==="
echo "=== COPY FROM $ROWS rows ===" >> "$RESULTS"
for BS in 100 1000 10000 100000; do
psql_tgt -c "TRUNCATE t;" 2>/dev/null
psql_src -c "DROP FOREIGN TABLE ft; CREATE FOREIGN TABLE ft (id int, value int, note text) SERVER remote OPTIONS (table_name 't', batch_size '$BS');" 2>/dev/null
sleep 1
echo -n " batch_size=$BS: "
START=$(date +%s%N)
psql_src -c "COPY ft FROM '$DATA_FILE'" > /dev/null 2>&1
END=$(date +%s%N)
ELAPSED_MS=$(( (END - START) / 1000000 ))
ELAPSED_S=$(echo "scale=2; $ELAPSED_MS / 1000" | bc 2>/dev/null || echo "${ELAPSED_MS}ms")
COUNT=$(psql_tgt -tA -c "SELECT count(*) FROM t;" 2>/dev/null || echo "ERROR")
echo "${ELAPSED_S}s (${COUNT} rows)"
echo " batch_size=$BS: ${ELAPSED_S}s (${COUNT} rows)" >> "$RESULTS"
done
echo ""
# --- step 6: pgbench INSERT benchmarks (run on source) ---
echo "=== pgbench INSERT ==="
echo "=== pgbench INSERT ===" >> "$RESULTS"
BENCH_SQL="$DATA_DIR/fdw_bench.sql"
cat > "$BENCH_SQL" <<SQLEOF
INSERT INTO ft (id, value, note)
SELECT generate_series(1, $((ROWS / 10))),
random() * 100000,
'batch insert test data' || generate_series(1, $((ROWS / 10)));
SQLEOF
for BS in 1 10 100 1000; do
psql_tgt -c "TRUNCATE t;" 2>/dev/null
psql_src -c "DROP FOREIGN TABLE ft; CREATE FOREIGN TABLE ft (id int, value int, note text) SERVER remote OPTIONS (table_name 't', batch_size '$BS');" 2>/dev/null
sleep 1
echo -n " batch_size=$BS: "
TPS=$("$PGBENCH" -n -c 10 -j 10 -t 10 -f "$BENCH_SQL" -h 127.0.0.1 -p "$SOURCE_PORT" -U "$PGUSER" -d postgres 2>&1 | grep "tps =" | sed 's/.*tps = //;s/ .*//')
echo "TPS=$TPS"
echo " batch_size=$BS: TPS=$TPS" >> "$RESULTS"
done
echo ""
# --- results ---
echo "============================================"
echo "=== RESULTS ==="
echo "============================================"
cat "$RESULTS"
echo ""
echo "Results saved to: $RESULTS"
# --- cleanup ---
psql_src -c "DROP FOREIGN TABLE ft; DROP SERVER remote CASCADE;" 2>/dev/null || true
psql_tgt -c "DROP TABLE t;" 2>/dev/null || true
"$PG_CTL" -D "$SRC_DIR" stop -m immediate 2>/dev/null || true
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-07 11:38 ` Re: postgres_fdw: Use COPY to speed up batch inserts solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
2026-05-27 23:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-08-05 21:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-09-09 20:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts Kirill Reshke <reshkekirill@gmail.com>
@ 2026-09-09 20:16 ` Kirill Reshke <reshkekirill@gmail.com>
0 siblings, 0 replies; 37+ messages in thread
From: Kirill Reshke @ 2026-09-09 20:16 UTC (permalink / raw)
To: Matheus Alcantara <matheusssilv97@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
On Thu, 10 Sept 2026 at 01:10, I <reshkekirill@gmail.com> wrote:
>
>
> By the way I noticed that batch_size is practically always no more
> than 1000 in batch insert callback (even though configured to a bigger
> value in relation options.). I didn't look for the exact reason.
>
I did actually take a fast look:
```
/*
* Send the buffered tuples to the FDW in batches of at most
* batch_size, as we do for ExecForeignBatchInsert. Each call is a
* self-contained COPY operation.
*
* COPY provides no RETURNING, so this path is only usable when
* there are no AFTER ROW triggers that would need the stored rows.
*/
while (sent < nused)
{
int size = (batch_size < nused - sent) ? batch_size : (nused - sent);
resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy(estate,
resultRelInfo,
&slots[sent],
size);
sent += size;
/* Update the row counter and progress of the COPY command */
*processed += size;
pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
*processed);
}
```
In this place, big batch_size (1e5-1e6) is always capped with 1000 by
CopyMultiInsertBuffer logic. At least for me this is the case. Maybe
this is another place for adjusting in sight of this feature.
--
Best regards,
Kirill Reshke
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: postgres_fdw: Use COPY to speed up batch inserts
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Tomas Vondra <tomas@vondra.me>
2025-10-21 14:25 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` Re: postgres_fdw: Use COPY to speed up batch inserts jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Re: postgres_fdw: Use COPY to speed up batch inserts Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Re: postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
@ 2026-05-27 23:22 ` Matheus Alcantara <matheusssilv97@gmail.com>
1 sibling, 0 replies; 37+ messages in thread
From: Matheus Alcantara @ 2026-05-27 23:22 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; jian he <jian.universality@gmail.com>; Tomas Vondra <tomas@vondra.me>; pgsql-hackers
On Wed Apr 1, 2026 at 12:50 PM -03, Matheus Alcantara wrote:
>> -- on local server
>> copy t(a, d) from stdin;
>> Enter data to be copied followed by a newline.
>> End with a backslash and a period on a line by itself, or an EOF signal.
>>>> 1 hello\nworld
>>>> \.
>> ERROR: invalid input syntax for type integer: "world"
>> CONTEXT: COPY t, line 2, column a: "world"
>> remote SQL command: COPY public.t(a, d) FROM STDIN (FORMAT TEXT)
>>
>
> I think that we need something like CopyAttributeOutText() here.
>
> To fix this I've added appendStringInfoText() which is a similar version
> of CopyAttributeOutText() that works with a StringInfo. I did not find
> any function that I could reuse here, if such function exists please let
> me know.
>
> I'm wondering if we should have this similar function or try to combine
> both to avoid duplicated logic, although it looks complicated to me at
> first look to combine these both usages.
>
I've spent some time hacking into this and actually I don't think that
it's too complicated. Please see the attached patchset.
0001: Extract CopyAttributeOutText escape logic into a reusable function
that can be used by postgres_fdw and by COPY TO routine.
0002: Is the main feature implementation using the TEXT format reusing
the refactor from 0001 avoiding all the duplicated code introduced on
this previous patch version. I've also removed the
resultRelInfo->ri_WithCheckOptions == NIL check that I've introduced
again by mistake since it was already commented at [1] that it should be
removed.
[1] https://www.postgresql.org/message-id/72ec1708-0c02-4ae9-b4f5-ee2bac5fd2f3%40gmail.com
--
Matheus Alcantara
EDB: https://www.enterprisedb.com
Attachments: download all (2 files, .tar.gz)
v15-0001-Extract-CopyEscapeText-for-reuse-outside-COPY-TO.patch (text/plain, 6.5K) download
From e02ecd11c93306a0160b7ffb87a00e1422b2bd8a Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Mon, 18 May 2026 19:23:43 -0300
Subject: [PATCH v15 1/2] Extract CopyEscapeText() for reuse outside COPY TO
Refactor CopyAttributeOutText() to extract its core text escaping logic
into a new public function CopyEscapeText() that operates on a StringInfo
buffer with explicit parameters, removing the dependency on CopyToState.
This enables other code paths, such as postgres_fdw, to reuse the COPY
text format escaping logic without needing to construct a full CopyToState.
CopyAttributeOutText() now becomes a thin wrapper that calls
CopyEscapeText() with the appropriate values from CopyToState.
Also introduce CopySendDataBuf() and CopySendCharBuf() macros as low-level
helpers that operate directly on StringInfo buffers.
Author: Matheus Alcantara <mths.dev@pm.me>
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
src/backend/commands/copyto.c | 85 ++++++++++++++++++++++++++---------
src/include/commands/copy.h | 6 +++
2 files changed, 71 insertions(+), 20 deletions(-)
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index ffed63a2986..ceee0014cfc 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -553,6 +553,12 @@ SendCopyEnd(CopyToState cstate)
pq_putemptymessage(PqMsg_CopyDone);
}
+#define CopySendCharBuf(buf, c) \
+ appendStringInfoCharMacro(buf, c)
+
+#define CopySendDataBuf(buf, databuf, datasize) \
+ appendBinaryStringInfo(buf, databuf, datasize)
+
/*----------
* CopySendData sends output data to the destination (file or frontend)
* CopySendString does the same for null-terminated strings
@@ -566,7 +572,7 @@ SendCopyEnd(CopyToState cstate)
static void
CopySendData(CopyToState cstate, const void *databuf, int datasize)
{
- appendBinaryStringInfo(cstate->fe_msgbuf, databuf, datasize);
+ CopySendDataBuf(cstate->fe_msgbuf, databuf, datasize);
}
static void
@@ -578,7 +584,7 @@ CopySendString(CopyToState cstate, const char *str)
static void
CopySendChar(CopyToState cstate, char c)
{
- appendStringInfoCharMacro(cstate->fe_msgbuf, c);
+ CopySendCharBuf(cstate->fe_msgbuf, c);
}
static void
@@ -1417,16 +1423,44 @@ CopyOneRowTo(CopyToState cstate, TupleTableSlot *slot)
CopySendData(cstate, start, ptr - start); \
} while (0)
-static void
-CopyAttributeOutText(CopyToState cstate, const char *string)
+/* Like above, but it works with a string buffer */
+#define DUMPSOFAR_TO_BUF() \
+ do { \
+ if (ptr > start) \
+ CopySendDataBuf(buf, start, ptr - start); \
+ } while (0)
+
+
+/*
+ * Escape a string for COPY TEXT format output
+ *
+ * Escapes control characters, backslashes, and the delimiter character
+ * according to COPY TEXT format rules. The escaped string is appended
+ * to 'buf'.
+ *
+ * Parameters:
+ * buf - StringInfo buffer to append the escaped text to
+ * string - the input string to escape
+ * delimc - the delimiter character that must be escaped
+ * file_encoding - target encoding for the output
+ * need_transcoding - if true, convert from server encoding to file_encoding
+ * encoding_embeds_ascii - if true, the encoding may have ASCII bytes as
+ * non-first bytes of multi-byte characters
+ */
+void
+CopyEscapeText(StringInfo buf,
+ const char *string,
+ char delimc,
+ int file_encoding,
+ bool need_transcoding,
+ bool encoding_embeds_ascii)
{
const char *ptr;
const char *start;
char c;
- char delimc = cstate->opts.delim[0];
- if (cstate->need_transcoding)
- ptr = pg_server_to_any(string, strlen(string), cstate->file_encoding);
+ if (need_transcoding)
+ ptr = pg_server_to_any(string, strlen(string), file_encoding);
else
ptr = string;
@@ -1444,7 +1478,7 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
* it's worth making two copies of it to get the IS_HIGHBIT_SET() test out
* of the normal safe-encoding path.
*/
- if (cstate->encoding_embeds_ascii)
+ if (encoding_embeds_ascii)
{
start = ptr;
while ((c = *ptr) != '\0')
@@ -1487,19 +1521,19 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
continue; /* fall to end of loop */
}
/* if we get here, we need to convert the control char */
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
- CopySendChar(cstate, c);
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
+ CopySendCharBuf(buf, c);
start = ++ptr; /* do not include char in next run */
}
else if (c == '\\' || c == delimc)
{
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
start = ptr++; /* we include char in next run */
}
else if (IS_HIGHBIT_SET(c))
- ptr += pg_encoding_mblen(cstate->file_encoding, ptr);
+ ptr += pg_encoding_mblen(file_encoding, ptr);
else
ptr++;
}
@@ -1547,15 +1581,15 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
continue; /* fall to end of loop */
}
/* if we get here, we need to convert the control char */
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
- CopySendChar(cstate, c);
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
+ CopySendCharBuf(buf, c);
start = ++ptr; /* do not include char in next run */
}
else if (c == '\\' || c == delimc)
{
- DUMPSOFAR();
- CopySendChar(cstate, '\\');
+ DUMPSOFAR_TO_BUF();
+ CopySendCharBuf(buf, '\\');
start = ptr++; /* we include char in next run */
}
else
@@ -1563,7 +1597,18 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
}
}
- DUMPSOFAR();
+ DUMPSOFAR_TO_BUF();
+}
+
+static void
+CopyAttributeOutText(CopyToState cstate, const char *string)
+{
+ CopyEscapeText(cstate->fe_msgbuf,
+ string,
+ cstate->opts.delim[0],
+ cstate->file_encoding,
+ cstate->need_transcoding,
+ cstate->encoding_embeds_ascii);
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index abecfe51098..27ee58c7fdb 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -136,5 +136,11 @@ extern void EndCopyTo(CopyToState cstate);
extern uint64 DoCopyTo(CopyToState cstate);
extern List *CopyGetAttnums(TupleDesc tupDesc, Relation rel,
List *attnamelist);
+extern void CopyEscapeText(StringInfo buf,
+ const char *string,
+ char delimc,
+ int file_encoding,
+ bool need_transcoding,
+ bool encoding_embeds_ascii);
#endif /* COPY_H */
--
2.50.1 (Apple Git-155)
v15-0002-postgres_fdw-Use-COPY-as-remote-SQL-when-possibl.patch (text/plain, 18.4K) download
From a4ce235442d38ebf751c4711bdd4b685db0c35ef Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <mths.dev@pm.me>
Date: Wed, 28 Jan 2026 19:55:48 -0300
Subject: [PATCH v15 2/2] postgres_fdw: Use COPY as remote SQL when possible
When a user executes COPY on a foreign table, postgres_fdw previously
sent INSERT statements to the foreign server. This commit enables using
the COPY protocol instead, which is significantly faster for bulk inserts.
The COPY protocol is used when copying data to a foreign table that has
no triggers. Triggers are excluded because they might modify the inserted
row, and since COPY does not support a RETURNING clause, we cannot
synchronize the local TupleTableSlot with those changes for use in local
AFTER triggers.
This uses the CopyEscapeText() function introduced on 051b56b4f95 to
properly escape text values for the COPY protocol.
Author: Matheus Alcantara <mths.dev@pm.me>
Reviewed-By: Tomas Vondra <tomas@vondra.me>
Reviewed-By: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-By: jian he <jian.universality@gmail.com>
Reviewed-By: Dewei Dai <daidewei1970@163.com>
Reviewed-By: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/DDIZJ217OUDK.2R5WE4OGL5PTY%40gmail.com
---
contrib/postgres_fdw/deparse.c | 56 +++++
.../postgres_fdw/expected/postgres_fdw.out | 75 ++++++-
contrib/postgres_fdw/postgres_fdw.c | 202 +++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.h | 1 +
contrib/postgres_fdw/sql/postgres_fdw.sql | 82 +++++++
5 files changed, 407 insertions(+), 9 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index 2dcc6c8af1b..b34792e7dc1 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2177,6 +2177,62 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * Build a COPY FROM STDIN statement using the TEXT format
+ */
+void
+deparseCopySql(StringInfo buf, Relation rel, List *target_attrs)
+{
+ Oid relid = RelationGetRelid(rel);
+ TupleDesc tupdesc = RelationGetDescr(rel);
+ bool first = true;
+ int nattrs = list_length(target_attrs);
+
+ appendStringInfo(buf, "COPY ");
+ deparseRelation(buf, rel);
+ if (nattrs > 0)
+ appendStringInfoChar(buf, '(');
+
+ foreach_int(attnum, target_attrs)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, attnum - 1);
+ char *colname;
+ List *options;
+ ListCell *lc;
+
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoString(buf, ", ");
+
+ first = false;
+
+ /* Use attribute name or column_name option. */
+ colname = NameStr(attr->attname);
+ options = GetForeignColumnOptions(relid, attnum);
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "column_name") == 0)
+ {
+ colname = defGetString(def);
+ break;
+ }
+ }
+
+ appendStringInfoString(buf, quote_identifier(colname));
+ }
+ if (nattrs > 0)
+ appendStringInfoString(buf, ") FROM STDIN");
+ else
+ appendStringInfoString(buf, " FROM STDIN");
+
+ appendStringInfoString(buf, " (FORMAT TEXT)");
+}
+
+
/*
* rebuild remote INSERT statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index e90289e4ab1..ae655545074 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -7654,6 +7654,28 @@ select * from grem1;
(2 rows)
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+copy grem1 from stdin;
+LOG: received message via remote connection: NOTICE: COPY public.gloc1(a) FROM STDIN (FORMAT TEXT)
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -7671,16 +7693,18 @@ insert into grem1 (a) values (1), (2);
select * from gloc1;
a | b | c
---+---+---
+ 3 | 6 |
1 | 2 |
2 | 4 |
-(2 rows)
+(3 rows)
select * from grem1;
a | b | c
---+---+---
+ 3 | 6 | 9
1 | 2 | 3
2 | 4 | 6
-(2 rows)
+(3 rows)
delete from grem1;
-- batch insert with foreign partitions.
@@ -7705,6 +7729,12 @@ select count(*) from tab_batch_sharded;
drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+DEBUG: foreign modify with COPY batch_size: 10
+DEBUG: foreign modify with COPY batch_size: 10
+reset client_min_messages;
alter server loopback options (drop batch_size);
-- ===================================================================
-- test local triggers
@@ -9595,7 +9625,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2, line 1: "-1 xyzzy"
select * from rem2;
f1 | f2
@@ -9752,7 +9783,8 @@ copy rem2 from stdin;
copy rem2 from stdin; -- ERROR
ERROR: new row for relation "loc2" violates check constraint "loc2_f1positive"
DETAIL: Failing row contains (-1, xyzzy).
-CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2)
+CONTEXT: COPY loc2, line 1: "-1 xyzzy"
+remote SQL command: COPY public.loc2(f1, f2) FROM STDIN (FORMAT TEXT)
COPY rem2
select * from rem2;
f1 | f2
@@ -9791,6 +9823,41 @@ select * from rem2;
drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+select * from rem2;
+ f1 | f2
+----+-------
+ 1 | hello+
+ | world
+(1 row)
+
+delete from rem2;
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+select * from rem2;
+ f1 | f2
+----+-----
+ 1 |
+ | bar
+ 3 | a"b
+(3 rows)
+
+delete from rem2;
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+reset extra_float_digits;
+select * from f;
+ a
+--------------------
+ 1.0000000000000002
+(1 row)
+
+drop table f;
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 0a589f8db74..7bde40ffe92 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -18,6 +18,7 @@
#include "access/sysattr.h"
#include "access/table.h"
#include "catalog/pg_opfamily.h"
+#include "commands/copy.h"
#include "commands/defrem.h"
#include "commands/explain_format.h"
#include "commands/explain_state.h"
@@ -27,6 +28,7 @@
#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "funcapi.h"
+#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
@@ -67,6 +69,9 @@ PG_MODULE_MAGIC_EXT(
/* If no remote estimates, assume a sort costs 20% extra */
#define DEFAULT_FDW_SORT_MULTIPLIER 1.2
+/* Buffer size to send COPY IN data*/
+#define COPYBUFSIZ 8192
+
/*
* Indexes of FDW-private information stored in fdw_private lists.
*
@@ -202,6 +207,8 @@ typedef struct PgFdwModifyState
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
+ bool use_copy; /* use COPY protocol for ExecForeignInsert? */
+
/* info about parameters for prepared statement */
AttrNumber ctidAttno; /* attnum of input resjunk ctid column */
int p_nums; /* number of parameters to transmit */
@@ -760,6 +767,13 @@ static void merge_fdw_options(PgFdwRelationInfo *fpinfo,
const PgFdwRelationInfo *fpinfo_o,
const PgFdwRelationInfo *fpinfo_i);
static int get_batch_size_option(Relation rel);
+static TupleTableSlot **execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots);
+static void convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot);
+static void appendStringInfoText(StringInfo buf, const char *string);
/*
@@ -2381,11 +2395,12 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
RangeTblEntry *rte;
TupleDesc tupdesc = RelationGetDescr(rel);
int attnum;
- int values_end_len;
+ int values_end_len = 0;
StringInfoData sql;
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ bool useCopy = false;
/*
* If the foreign table we are about to insert routed rows into is also an
@@ -2463,11 +2478,38 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
rte = exec_rt_fetch(resultRelation, estate);
}
+ /*
+ * We can use COPY for remote inserts only if all the following conditions
+ * are met:
+ *
+ * Direct Execution: The command is a COPY FROM on the foreign table
+ * itself, not part of a partitioned table's tuple routing.
+ *
+ * No Local AFTER Triggers: There are no AFTER ROW triggers defined
+ * locally on the foreign table.
+ *
+ * Remote triggers might modify the inserted row. Because the COPY
+ * protocol does not support a RETURNING clause, we cannot retrieve those
+ * changes to synchronize the local TupleTableSlot required by local AFTER
+ * triggers.
+ */
+ if (resultRelInfo->ri_RootResultRelInfo == NULL)
+ {
+ /* There is no RETURNING clause on COPY */
+ Assert(resultRelInfo->ri_returningList == NIL);
+
+ useCopy = (resultRelInfo->ri_TrigDesc == NULL ||
+ !resultRelInfo->ri_TrigDesc->trig_insert_after_row);
+ }
+
/* Construct the SQL command string. */
- deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
- resultRelInfo->ri_WithCheckOptions,
- resultRelInfo->ri_returningList,
- &retrieved_attrs, &values_end_len);
+ if (useCopy)
+ deparseCopySql(&sql, rel, targetAttrs);
+ else
+ deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
+ resultRelInfo->ri_WithCheckOptions,
+ resultRelInfo->ri_returningList,
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2480,6 +2522,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
+ fmstate->use_copy = useCopy;
/*
* If the given resultRelInfo already has PgFdwModifyState set, it means
@@ -4309,6 +4352,9 @@ execute_foreign_modify(EState *estate,
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ if (fmstate->use_copy)
+ return execute_foreign_modify_using_copy(fmstate, slots, numSlots);
+
/* First, process a pending asynchronous request, if any. */
if (fmstate->conn_state->pendingAreq)
process_pending_request(fmstate->conn_state->pendingAreq);
@@ -8835,3 +8881,149 @@ get_batch_size_option(Relation rel)
return batch_size;
}
+
+/*
+ * execute_foreign_modify_using_copy
+ * Perform foreign-table modification using the COPY command.
+ */
+static TupleTableSlot **
+execute_foreign_modify_using_copy(PgFdwModifyState *fmstate,
+ TupleTableSlot **slots,
+ int *numSlots)
+{
+ PGresult *res;
+ StringInfoData copy_data;
+ int n_rows;
+ int i;
+ int nestlevel;
+
+ Assert(fmstate->use_copy == true);
+
+ elog(DEBUG1, "foreign modify with COPY batch_size: %d", fmstate->batch_size);
+
+ /* Make sure any constants in the slots are printed portably */
+ nestlevel = set_transmission_modes();
+
+ /* Send COPY command */
+ if (!PQsendQuery(fmstate->conn, fmstate->query))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /* get the COPY result */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ /* Clean up the COPY command result */
+ PQclear(res);
+
+ /* Convert the TupleTableSlot data into a TEXT-formatted line */
+ initStringInfo(©_data);
+ for (i = 0; i < *numSlots; i++)
+ {
+ convert_slot_to_copy_text(©_data, fmstate, slots[i]);
+
+ /*
+ * Send initial COPY data if the buffer reaches the limit to avoid
+ * large memory usage.
+ */
+ if (copy_data.len >= COPYBUFSIZ)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ resetStringInfo(©_data);
+ }
+ }
+
+ /* Send the remaining COPY data */
+ if (copy_data.len > 0)
+ {
+ if (PQputCopyData(fmstate->conn, copy_data.data, copy_data.len) <= 0)
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+ }
+
+ pfree(copy_data.data);
+
+ /* End the COPY operation */
+ if (PQputCopyEnd(fmstate->conn, NULL) < 0 || PQflush(fmstate->conn))
+ pgfdw_report_error(NULL, fmstate->conn, fmstate->query);
+
+ /*
+ * Get the result, and check for success.
+ */
+ res = pgfdw_get_result(fmstate->conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(res, fmstate->conn, fmstate->query);
+
+ n_rows = atoi(PQcmdTuples(res));
+
+ /* And clean up */
+ PQclear(res);
+
+ reset_transmission_modes(nestlevel);
+
+ MemoryContextReset(fmstate->temp_cxt);
+
+ *numSlots = n_rows;
+
+ /*
+ * Return NULL if nothing was inserted on the remote end
+ */
+ return (n_rows > 0) ? slots : NULL;
+}
+
+/*
+ * Write target attribute values from fmstate into buf buffer to be sent as
+ * COPY FROM STDIN data
+ */
+static void
+convert_slot_to_copy_text(StringInfo buf,
+ PgFdwModifyState *fmstate,
+ TupleTableSlot *slot)
+{
+ TupleDesc tupdesc = RelationGetDescr(fmstate->rel);
+ bool first = true;
+ int i = 0;
+
+ foreach_int(attnum, fmstate->target_attrs)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(tupdesc, attnum - 1);
+ Datum datum;
+ bool isnull;
+
+ /* Ignore generated columns; they are set to DEFAULT */
+ if (attr->attgenerated)
+ continue;
+
+ if (!first)
+ appendStringInfoCharMacro(buf, '\t');
+ first = false;
+
+ datum = slot_getattr(slot, attnum, &isnull);
+
+ if (isnull)
+ appendStringInfoString(buf, "\\N");
+ else
+ {
+ const char *value = OutputFunctionCall(&fmstate->p_flinfo[i],
+ datum);
+
+ /* Escape the value if needed */
+ appendStringInfoText(buf, value);
+ }
+ i++;
+ }
+
+ appendStringInfoCharMacro(buf, '\n');
+}
+
+/* Append a string to buf, escaping special characters for COPY TEXT format. */
+static void
+appendStringInfoText(StringInfo buf, const char *string)
+{
+ CopyEscapeText(buf,
+ string,
+ '\t',
+ PG_UTF8,
+ false,
+ false);
+}
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index a2bb1ff352c..fc6922ddd4f 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -204,6 +204,7 @@ extern void rebuildInsertSql(StringInfo buf, Relation rel,
char *orig_query, List *target_attrs,
int values_end_len, int num_params,
int num_rows);
+extern void deparseCopySql(StringInfo buf, Relation rel, List *target_attrs);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index dfc58beb0d2..db95b148ca8 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1970,6 +1970,37 @@ select * from gloc1;
select * from grem1;
delete from grem1;
+-- test that fdw also use COPY FROM as a remote sql
+set client_min_messages to 'log';
+
+create function insert_or_copy() returns trigger as $$
+declare query text;
+begin
+ query := current_query();
+ raise notice '%', query;
+return new;
+end;
+$$ language plpgsql;
+
+CREATE TRIGGER trig_row_before
+BEFORE INSERT OR UPDATE OR DELETE ON gloc1
+FOR EACH ROW EXECUTE PROCEDURE insert_or_copy();
+
+copy grem1 from stdin;
+3
+\.
+
+drop trigger trig_row_before on gloc1;
+reset client_min_messages;
+
+-- test that copy does not fail with column_name alias
+create table gloc2(xxx int);
+create foreign table grem2(a int) server loopback options(table_name 'gloc2');
+alter foreign table grem2 alter column a options (column_name 'xxx');
+copy grem2 from stdin;
+1
+\.
+
-- test batch insert
alter server loopback options (add batch_size '10');
explain (verbose, costs off)
@@ -1996,6 +2027,24 @@ drop table tab_batch_local;
drop table tab_batch_sharded;
drop table tab_batch_sharded_p1_remote;
+-- test batch insert using copy
+set client_min_messages to 'debug1';
+copy grem1 from stdin;
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+\.
+reset client_min_messages;
+
alter server loopback options (drop batch_size);
-- ===================================================================
@@ -3060,6 +3109,39 @@ drop trigger trig_null on loc2;
delete from rem2;
+-- Test COPY FROM with column list and special characters
+copy rem2 (f1, f2) from stdin;
+1 hello\nworld
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test COPY with NULL and special characters
+copy rem2 from stdin;
+1 \N
+\N bar
+3 a"b
+\.
+select * from rem2;
+
+delete from rem2;
+
+-- Test that float numbers do not loose precision when sending to the foreign
+-- server
+create table f(a float);
+create foreign table f_fdw(a float) server loopback options(table_name 'f');
+
+set extra_float_digits = 0;
+copy f_fdw from stdin;
+1.0000000000000002
+\.
+
+reset extra_float_digits;
+select * from f;
+
+drop table f;
+
-- Check with zero-column foreign table; batch insert will be disabled
alter table loc2 drop column f1;
alter table loc2 drop column f2;
--
2.50.1 (Apple Git-155)
end of thread, other threads:[~2026-09-09 20:16 UTC | newest]
Thread overview: 37+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-10-15 15:02 postgres_fdw: Use COPY to speed up batch inserts Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-16 20:39 ` Tomas Vondra <tomas@vondra.me>
2025-10-17 09:28 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-10-21 14:26 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-21 14:25 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 00:00 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-23 09:49 ` jian he <jian.universality@gmail.com>
2025-10-24 18:27 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-29 03:10 ` jian he <jian.universality@gmail.com>
2025-10-30 00:28 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-10-30 16:32 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-31 19:02 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-06 23:49 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-18 02:03 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-18 22:13 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-11-19 23:32 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-11-26 20:51 ` Matheus Alcantara <matheusssilv97@gmail.com>
2025-12-11 12:03 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-02 20:15 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-02 20:33 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-03 12:45 ` Dewei Dai <daidewei1970@163.com>
2026-01-05 13:43 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-01-27 19:17 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-29 14:02 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-02-26 01:39 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-02-26 15:58 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-03 19:47 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-03-04 12:17 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-03-30 19:14 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-04-01 15:50 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-04-07 21:00 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-05-07 11:38 ` solaimurugan vellaipandiyan <drsolaimurugan.v@gmail.com>
2026-05-27 23:25 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-08-05 21:33 ` Matheus Alcantara <matheusssilv97@gmail.com>
2026-09-09 20:10 ` Kirill Reshke <reshkekirill@gmail.com>
2026-09-09 20:16 ` Kirill Reshke <reshkekirill@gmail.com>
2026-05-27 23:22 ` Matheus Alcantara <matheusssilv97@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox