Subject: Re: Bug in asynchronous Append

From: Gleb Kashkin <[email-masked]>
Date: Fri, 10 Jul 2026 13:56:46 +0300

Alexander Korotkov писал(а) 2026-07-04 01:00:
> Hi!
> 
> ExecReScanAppend() unconditionally resets callback_pending for all
> AsyncRequests.  The problem is that postgres_fdw keeps its own
> knowledge for the same fact: PgFdwConnState.pendingAreq – a pointer
> to "pending async request" for a given connection.  That connection
> can be shared by several partitions/foreign tables (postgres_fdw
> caches one connection per server+usermapping pair). The blind reset in
> nodeAppend.c only touches the local AsyncRequest.callback_pending; it
> never touches PgFdwConnState.pendingAreq, which correctly points to
> the still-dangling request.
> 
> Later, when another partition sharing that same connection gets its
> own ReScan (for instance, its chgParam changed because of the LATERAL
> parameter, and it already has a cursor open), it sends "CLOSE cursor"
> via pgfdw_exec_query().  Before sending any new command on the
> connection, that function first drains whatever request is still
> outstanding on it:
> 
> if (state && state->pendingAreq)
>     process_pending_request(state->pendingAreq);
> 
> And process_pending_request() starts with:
> 
> Assert(areq->callback_pending);
> 
> – which fails, because the flag was corrupted some rounds earlier.
> 
> The attached patch contains both the reproduction case and the fix.
> The fix postpones the reset of the callback_pending flag to
> ExecAppendAsyncBegin().  ExecAppendAsyncBegin() performs this cleanup
> along with ExecReScan(), which completes the async fetch.
> 
> ------
> Regards,
> Alexander Korotkov
> Supabase

Hi!

It seems, that there is a major issue with the patch, the draining 
doesn't
throw away the result in cases when it is required. Consider this case:

-- Expose stale rows from a pending async request that was valid for a
-- previous rescan but is pruned out for the current one.
CREATE VIEW base_tbl2_slow AS
  SELECT t.a, t.b, t.c
  FROM base_tbl2 t, LATERAL pg_sleep(0.2);
ALTER FOREIGN TABLE async_p2 OPTIONS (SET table_name 'base_tbl2_slow');
SELECT o.x, s.a
FROM (VALUES (2505), (3505)) o(x),
  LATERAL (
    SELECT a
    FROM async_pt
    WHERE a = o.x OR (o.x = 2505 AND a = 1505)
    LIMIT 1
  ) s
ORDER BY o.x;
  x   |  a   
------+------
  2505 | 1505
  3505 | 2505
(2 rows)

1) Append executes two async fscans, one of which is slow
2) The first fscan finishes and the append execution stops due to LIMIT 
1
3) Second fscan async request is still pending
4) Next append rescan prunes async_p2 for new outer value
5) Old async request for async_p2 is allowed to drain (receive a 
callback)
6) The stale tuple from async_p2 is accepted by Append, no 
pruning/filter check
7) We get a row that should be impossible (3505 | 2505) in this query

See full reproducer in the attached patch.

diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index e90289e4ab1..eebdf1c4ad8 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -11661,6 +11661,22 @@ SELECT * FROM result_tbl ORDER BY a;
 (3 rows)
 
 DELETE FROM result_tbl;
+-- Check that a pending async request on a shared connection isn't corrupted
+-- when the Append is rescanned with the subplan holding it pruned out.  Here
+-- async_p2 and async_p3 share a connection, and per outer row exactly one of
+-- them is pruned while async_p1 (a different connection) always matches; the
+-- inner LIMIT leaves the other shared-connection request outstanding across
+-- the rescan, and reusing that connection for the next round must drain it
+-- rather than trip over stale state.
+SELECT o.x FROM (VALUES (2505), (3505)) o(x),
+  LATERAL (SELECT a FROM async_pt WHERE a = 1505 OR a = o.x LIMIT 1) s
+ORDER BY o.x;
+  x   
+------
+ 2505
+ 3505
+(2 rows)
+
 -- Test COPY TO when foreign table is partition
 COPY async_pt TO stdout; --error
 ERROR:  cannot copy from foreign table "async_p1"
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index dfc58beb0d2..c18b54f1756 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -3997,6 +3997,38 @@ INSERT INTO result_tbl SELECT * FROM async_pt WHERE b === 505;
 SELECT * FROM result_tbl ORDER BY a;
 DELETE FROM result_tbl;
 
+-- Check that a pending async request on a shared connection isn't corrupted
+-- when the Append is rescanned with the subplan holding it pruned out.  Here
+-- async_p2 and async_p3 share a connection, and per outer row exactly one of
+-- them is pruned while async_p1 (a different connection) always matches; the
+-- inner LIMIT leaves the other shared-connection request outstanding across
+-- the rescan, and reusing that connection for the next round must drain it
+-- rather than trip over stale state.
+SELECT o.x FROM (VALUES (2505), (3505)) o(x),
+  LATERAL (SELECT a FROM async_pt WHERE a = 1505 OR a = o.x LIMIT 1) s
+ORDER BY o.x;
+
+-- Expose stale rows from a pending async request that was valid for a
+-- previous rescan but is pruned out for the current one.
+CREATE VIEW base_tbl2_slow AS
+  SELECT t.a, t.b, t.c
+  FROM base_tbl2 t, LATERAL pg_sleep(0.2);
+
+ALTER FOREIGN TABLE async_p2 OPTIONS (SET table_name 'base_tbl2_slow');
+
+SELECT o.x, s.a
+FROM (VALUES (2505), (3505)) o(x),
+  LATERAL (
+    SELECT a
+    FROM async_pt
+    WHERE a = o.x OR (o.x = 2505 AND a = 1505)
+    LIMIT 1
+  ) s
+ORDER BY o.x;
+
+ALTER FOREIGN TABLE async_p2 OPTIONS (SET table_name 'base_tbl2');
+DROP VIEW base_tbl2_slow;
+
 -- Test COPY TO when foreign table is partition
 COPY async_pt TO stdout; --error
 
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index 987358e27fa..6a5a14cd576 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -469,7 +469,21 @@ ExecReScanAppend(AppendState *node)
 		{
 			AsyncRequest *areq = node->as_asyncrequests[i];
 
-			areq->callback_pending = false;
+			/*
+			 * Leave a request that is still marked as pending a callback
+			 * alone: it may genuinely still be in flight, or it may have an
+			 * unconsumed result already sitting on a connection shared with
+			 * another subplan (as can happen with postgres_fdw).  Blindly
+			 * clearing callback_pending here would desync our bookkeeping
+			 * from the async-capable node's own, which can lead it to
+			 * mishandle that connection later (e.g. postgres_fdw asserts that
+			 * a request it still considers in-process has callback_pending
+			 * set).  Such a request is instead drained lazily, right before
+			 * it would be reused, in ExecAppendAsyncBegin().
+			 */
+			if (areq->callback_pending)
+				continue;
+
 			areq->request_complete = false;
 			areq->result = NULL;
 		}
@@ -915,7 +929,25 @@ ExecAppendAsyncBegin(AppendState *node)
 		AsyncRequest *areq = node->as_asyncrequests[i];
 
 		Assert(areq->request_index == i);
-		Assert(!areq->callback_pending);
+
+		/*
+		 * This request may still be marked as pending a callback, if
+		 * ExecReScanAppend() left it alone because it might have been
+		 * genuinely in flight (or had an unconsumed result waiting on a
+		 * connection shared with another subplan).  Drain it now, before
+		 * reusing it: ExecReScan() lets the async-capable node settle any
+		 * such outstanding state (e.g. postgres_fdw's
+		 * postgresReScanForeignScan() will wait for an in-progress request on
+		 * its own connection and consume its result), after which it's safe
+		 * to reset our own bookkeeping and issue a fresh request.
+		 */
+		if (areq->callback_pending)
+		{
+			ExecReScan(node->appendplans[i]);
+			areq->callback_pending = false;
+			areq->request_complete = false;
+			areq->result = NULL;
+		}
 
 		/* Do the actual work. */
 		ExecAsyncRequest(areq);