public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 3/4] WIP: v16 docs: Add enable_presorted_aggregate GUC
7+ messages / 3 participants
[nested] [flat]
* [PATCH 3/4] WIP: v16 docs: Add enable_presorted_aggregate GUC
@ 2023-03-19 02:41 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Justin Pryzby @ 2023-03-19 02:41 UTC (permalink / raw)
3226f47282a05979483475d1e4a11aab8c1bfc39
---
doc/src/sgml/config.sgml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 20b8b5ae1de..68e344c7bf0 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5440,15 +5440,15 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</term>
<listitem>
<para>
- Controls if the query planner will produce a plan which will provide
- rows which are presorted in the order required for the query's
+ Enables or disables the query planner's use of plans which presort
+ rows in the order required for the query's
<literal>ORDER BY</literal> / <literal>DISTINCT</literal> aggregate
functions. When disabled, the query planner will produce a plan which
will always require the executor to perform a sort before performing
aggregation of each aggregate function containing an
<literal>ORDER BY</literal> or <literal>DISTINCT</literal> clause.
When enabled, the planner will try to produce a more efficient plan
- which provides input to the aggregate functions which is presorted in
+ which provides input to the aggregate functions in
the order they require for aggregation. The default value is
<literal>on</literal>.
</para>
--
2.34.1
--Zw+DqMmNEUQwlI/M
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-WIP-v16-docs-Add-more-details-about-pg_stat_get_xact.patch"
^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH] Add RetrieveInstrumentation hook for CustomScan providers
@ 2026-02-18 09:39 Siddharth Kothari <[email protected]>
2026-04-08 05:27 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Siddharth Kothari @ 2026-02-18 09:39 UTC (permalink / raw)
To: [email protected]; +Cc: Vaibhav Jain <[email protected]>; Madhukar <[email protected]>
Dear PostgreSQL Hackers,
This email proposes a patch to enhance the CustomScan provider interface.
The patch file,
0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch, is
attached.
*Problem:*
CustomScan providers currently lack a standard method to aggregate
instrumentation data from parallel workers back to the leader process
before the Dynamic Shared Memory (DSM) segment is unlinked. This makes it
difficult to gather comprehensive performance metrics from parallelized
custom scans.
*Solution:*
This patch introduces a new optional hook, RetrieveInstrumentationCustomScan,
to the CustomExecMethods struct. This hook allows custom scan providers to
implement logic to collect and consolidate instrumentation from shared
memory or worker states during the parallel query cleanup phase. This hook
is invoked via the new ExecCustomScanRetrieveInstrumentation function,
called from ExecParallelRetrieveInstrumentation for T_CustomScanState nodes.
Since the hook is optional (checked for NULL before calling), it maintains
full backward compatibility.
*Testing & Compatibility:*
- The patch compiles and passes all core regression tests (make
check-world) on my x86_64 instance.
- The changes are not platform-specific.
- Regression Tests: This patch provides a new *capability* for custom
scan providers. Since the hook's functionality is only realized when
implemented by an extension, specific tests would naturally reside within
that extension rather than in the core regression suite.
This patch does not directly address a specific item on the official TODO
list but enhances the extensibility framework.
I believe this patch is complete and ready for review. I look forward to
any feedback and am happy to make revisions. I will also add this patch to
the next CommitFest.
Thank you,
Siddharth Kothari
Attachments:
[application/x-patch] 0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch (3.3K, ../../CAGCUe0Kp7kTUWFjHhiE-xjZ4=UDe2fUdDVajKcjV+BT+T5d21A@mail.gmail.com/3-0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch)
download | inline diff:
From ac6049e8b0af21666314edb73018fd99940d0731 Mon Sep 17 00:00:00 2001
From: Siddharth Kothari <[email protected]>
Date: Tue, 17 Feb 2026 12:23:56 +0000
Subject: [PATCH] Add RetrieveInstrumentationCustomScan hook for CustomScan
providers
CustomScan providers currently lack a standard method to aggregate
instrumentation data from parallel workers back to the leader process
before the Dynamic Shared Memory segment is destroyed.
This patch introduces an optional RetrieveInstrumentationCustomScan
callback to the CustomExecMethods struct. This allows custom scan
providers to implement logic to collect and consolidate instrumentation
from shared memory.
The new hook is called in ExecRetrieveInstrumentation for CustomScanState
nodes during the parallel query cleanup phase.
---
src/backend/executor/execParallel.c | 3 +++
src/backend/executor/nodeCustom.c | 9 +++++++++
src/include/executor/nodeCustom.h | 1 +
src/include/nodes/extensible.h | 3 +++
4 files changed, 16 insertions(+)
diff --git a/src/backend/executor/execParallel.c b/src/backend/executor/execParallel.c
index f87978c137e..c2416a98170 100644
--- a/src/backend/executor/execParallel.c
+++ b/src/backend/executor/execParallel.c
@@ -1118,6 +1118,9 @@ ExecParallelRetrieveInstrumentation(PlanState *planstate,
case T_BitmapHeapScanState:
ExecBitmapHeapRetrieveInstrumentation((BitmapHeapScanState *) planstate);
break;
+ case T_CustomScanState:
+ ExecCustomScanRetrieveInstrumentation((CustomScanState *) planstate);
+ break;
default:
break;
}
diff --git a/src/backend/executor/nodeCustom.c b/src/backend/executor/nodeCustom.c
index a9ad5af6a98..f027d24993f 100644
--- a/src/backend/executor/nodeCustom.c
+++ b/src/backend/executor/nodeCustom.c
@@ -217,6 +217,15 @@ ExecCustomScanInitializeWorker(CustomScanState *node,
}
}
+void
+ExecCustomScanRetrieveInstrumentation(CustomScanState *node)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->RetrieveInstrumentationCustomScan)
+ methods->RetrieveInstrumentationCustomScan(node);
+}
+
void
ExecShutdownCustomScan(CustomScanState *node)
{
diff --git a/src/include/executor/nodeCustom.h b/src/include/executor/nodeCustom.h
index fb0acc6e414..0a9cfb40381 100644
--- a/src/include/executor/nodeCustom.h
+++ b/src/include/executor/nodeCustom.h
@@ -37,6 +37,7 @@ extern void ExecCustomScanReInitializeDSM(CustomScanState *node,
ParallelContext *pcxt);
extern void ExecCustomScanInitializeWorker(CustomScanState *node,
ParallelWorkerContext *pwcxt);
+extern void ExecCustomScanRetrieveInstrumentation(CustomScanState *node);
extern void ExecShutdownCustomScan(CustomScanState *node);
#endif /* NODECUSTOM_H */
diff --git a/src/include/nodes/extensible.h b/src/include/nodes/extensible.h
index 517db95c4a3..cda478b538f 100644
--- a/src/include/nodes/extensible.h
+++ b/src/include/nodes/extensible.h
@@ -151,6 +151,9 @@ typedef struct CustomExecMethods
void *coordinate);
void (*ShutdownCustomScan) (CustomScanState *node);
+ /* Optional: retrieve parallel instrumentation */
+ void (*RetrieveInstrumentationCustomScan) (CustomScanState *node);
+
/* Optional: print additional information in EXPLAIN */
void (*ExplainCustomScan) (CustomScanState *node,
List *ancestors,
--
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers
2026-02-18 09:39 [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
@ 2026-04-08 05:27 ` Siddharth Kothari <[email protected]>
2026-06-23 09:08 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Siddharth Kothari @ 2026-04-08 05:27 UTC (permalink / raw)
To: [email protected]; +Cc: Vaibhav Jain <[email protected]>; Madhukar <[email protected]>
Hi everyone,
I’m just checking in to see if anyone has had a chance to look at this or
if there’s any further information I should provide to help with the
review. I have also added the patch to PG20-1 CF queue, the link is
https://commitfest.postgresql.org/patch/6524/.
Thanks,
Siddharth
On Wed, Feb 18, 2026 at 3:09 PM Siddharth Kothari <[email protected]> wrote:
> Dear PostgreSQL Hackers,
>
> This email proposes a patch to enhance the CustomScan provider interface.
> The patch file,
> 0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch, is
> attached.
>
> *Problem:*
>
> CustomScan providers currently lack a standard method to aggregate
> instrumentation data from parallel workers back to the leader process
> before the Dynamic Shared Memory (DSM) segment is unlinked. This makes it
> difficult to gather comprehensive performance metrics from parallelized
> custom scans.
>
> *Solution:*
>
> This patch introduces a new optional hook,
> RetrieveInstrumentationCustomScan, to the CustomExecMethods struct. This
> hook allows custom scan providers to implement logic to collect and
> consolidate instrumentation from shared memory or worker states during the
> parallel query cleanup phase. This hook is invoked via the new
> ExecCustomScanRetrieveInstrumentation function, called from
> ExecParallelRetrieveInstrumentation for T_CustomScanState nodes. Since
> the hook is optional (checked for NULL before calling), it maintains full
> backward compatibility.
>
> *Testing & Compatibility:*
>
> - The patch compiles and passes all core regression tests (make
> check-world) on my x86_64 instance.
> - The changes are not platform-specific.
> - Regression Tests: This patch provides a new *capability* for custom
> scan providers. Since the hook's functionality is only realized when
> implemented by an extension, specific tests would naturally reside within
> that extension rather than in the core regression suite.
>
> This patch does not directly address a specific item on the official TODO
> list but enhances the extensibility framework.
>
> I believe this patch is complete and ready for review. I look forward to
> any feedback and am happy to make revisions. I will also add this patch to
> the next CommitFest.
>
> Thank you,
>
> Siddharth Kothari
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers
2026-02-18 09:39 [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-04-08 05:27 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
@ 2026-06-23 09:08 ` Siddharth Kothari <[email protected]>
2026-07-08 16:52 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Siddharth Kothari @ 2026-06-23 09:08 UTC (permalink / raw)
To: [email protected]; +Cc: Vaibhav Jain <[email protected]>; Madhukar <[email protected]>
Hi hackers,
Bumping this thread as this small patch is passing all CF checks but needs
a reviewer.
*The Problem:* Right now, if you write a parallel CustomScan extension, it
is impossible to get performance metrics from parallel workers. The data is
destroyed during DSM unlinking before the leader can grab it.
*The Fix:* This patch adds an optional RetrieveInstrumentation hook to
CustomExecMethods. It has zero overhead for existing extensions (it's a
simple NULL check) but unlocks metric aggregation for parallel custom scans.
Please let me know your thoughts.
Thanks,
Siddharth
On Wed, Apr 8, 2026 at 10:57 AM Siddharth Kothari <[email protected]> wrote:
> Hi everyone,
>
> I’m just checking in to see if anyone has had a chance to look at this or
> if there’s any further information I should provide to help with the
> review. I have also added the patch to PG20-1 CF queue, the link is
> https://commitfest.postgresql.org/patch/6524/.
>
> Thanks,
> Siddharth
>
> On Wed, Feb 18, 2026 at 3:09 PM Siddharth Kothari <[email protected]>
> wrote:
>
>> Dear PostgreSQL Hackers,
>>
>> This email proposes a patch to enhance the CustomScan provider interface.
>> The patch file,
>> 0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch, is
>> attached.
>>
>> *Problem:*
>>
>> CustomScan providers currently lack a standard method to aggregate
>> instrumentation data from parallel workers back to the leader process
>> before the Dynamic Shared Memory (DSM) segment is unlinked. This makes it
>> difficult to gather comprehensive performance metrics from parallelized
>> custom scans.
>>
>> *Solution:*
>>
>> This patch introduces a new optional hook,
>> RetrieveInstrumentationCustomScan, to the CustomExecMethods struct. This
>> hook allows custom scan providers to implement logic to collect and
>> consolidate instrumentation from shared memory or worker states during the
>> parallel query cleanup phase. This hook is invoked via the new
>> ExecCustomScanRetrieveInstrumentation function, called from
>> ExecParallelRetrieveInstrumentation for T_CustomScanState nodes. Since
>> the hook is optional (checked for NULL before calling), it maintains full
>> backward compatibility.
>>
>> *Testing & Compatibility:*
>>
>> - The patch compiles and passes all core regression tests (make
>> check-world) on my x86_64 instance.
>> - The changes are not platform-specific.
>> - Regression Tests: This patch provides a new *capability* for custom
>> scan providers. Since the hook's functionality is only realized when
>> implemented by an extension, specific tests would naturally reside within
>> that extension rather than in the core regression suite.
>>
>> This patch does not directly address a specific item on the official TODO
>> list but enhances the extensibility framework.
>>
>> I believe this patch is complete and ready for review. I look forward to
>> any feedback and am happy to make revisions. I will also add this patch to
>> the next CommitFest.
>>
>> Thank you,
>>
>> Siddharth Kothari
>>
>
Attachments:
[application/octet-stream] 0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch (3.3K, ../../CAGCUe0+DV0Tkd19rX7HrQikyuWTF0uQS-OeikeMOye+CzSvxgQ@mail.gmail.com/3-0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch)
download | inline diff:
From ac6049e8b0af21666314edb73018fd99940d0731 Mon Sep 17 00:00:00 2001
From: Siddharth Kothari <[email protected]>
Date: Tue, 17 Feb 2026 12:23:56 +0000
Subject: [PATCH] Add RetrieveInstrumentationCustomScan hook for CustomScan
providers
CustomScan providers currently lack a standard method to aggregate
instrumentation data from parallel workers back to the leader process
before the Dynamic Shared Memory segment is destroyed.
This patch introduces an optional RetrieveInstrumentationCustomScan
callback to the CustomExecMethods struct. This allows custom scan
providers to implement logic to collect and consolidate instrumentation
from shared memory.
The new hook is called in ExecRetrieveInstrumentation for CustomScanState
nodes during the parallel query cleanup phase.
---
src/backend/executor/execParallel.c | 3 +++
src/backend/executor/nodeCustom.c | 9 +++++++++
src/include/executor/nodeCustom.h | 1 +
src/include/nodes/extensible.h | 3 +++
4 files changed, 16 insertions(+)
diff --git a/src/backend/executor/execParallel.c b/src/backend/executor/execParallel.c
index f87978c137e..c2416a98170 100644
--- a/src/backend/executor/execParallel.c
+++ b/src/backend/executor/execParallel.c
@@ -1118,6 +1118,9 @@ ExecParallelRetrieveInstrumentation(PlanState *planstate,
case T_BitmapHeapScanState:
ExecBitmapHeapRetrieveInstrumentation((BitmapHeapScanState *) planstate);
break;
+ case T_CustomScanState:
+ ExecCustomScanRetrieveInstrumentation((CustomScanState *) planstate);
+ break;
default:
break;
}
diff --git a/src/backend/executor/nodeCustom.c b/src/backend/executor/nodeCustom.c
index a9ad5af6a98..f027d24993f 100644
--- a/src/backend/executor/nodeCustom.c
+++ b/src/backend/executor/nodeCustom.c
@@ -217,6 +217,15 @@ ExecCustomScanInitializeWorker(CustomScanState *node,
}
}
+void
+ExecCustomScanRetrieveInstrumentation(CustomScanState *node)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->RetrieveInstrumentationCustomScan)
+ methods->RetrieveInstrumentationCustomScan(node);
+}
+
void
ExecShutdownCustomScan(CustomScanState *node)
{
diff --git a/src/include/executor/nodeCustom.h b/src/include/executor/nodeCustom.h
index fb0acc6e414..0a9cfb40381 100644
--- a/src/include/executor/nodeCustom.h
+++ b/src/include/executor/nodeCustom.h
@@ -37,6 +37,7 @@ extern void ExecCustomScanReInitializeDSM(CustomScanState *node,
ParallelContext *pcxt);
extern void ExecCustomScanInitializeWorker(CustomScanState *node,
ParallelWorkerContext *pwcxt);
+extern void ExecCustomScanRetrieveInstrumentation(CustomScanState *node);
extern void ExecShutdownCustomScan(CustomScanState *node);
#endif /* NODECUSTOM_H */
diff --git a/src/include/nodes/extensible.h b/src/include/nodes/extensible.h
index 517db95c4a3..cda478b538f 100644
--- a/src/include/nodes/extensible.h
+++ b/src/include/nodes/extensible.h
@@ -151,6 +151,9 @@ typedef struct CustomExecMethods
void *coordinate);
void (*ShutdownCustomScan) (CustomScanState *node);
+ /* Optional: retrieve parallel instrumentation */
+ void (*RetrieveInstrumentationCustomScan) (CustomScanState *node);
+
/* Optional: print additional information in EXPLAIN */
void (*ExplainCustomScan) (CustomScanState *node,
List *ancestors,
--
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers
2026-02-18 09:39 [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-04-08 05:27 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-06-23 09:08 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
@ 2026-07-08 16:52 ` Siddharth Kothari <[email protected]>
2026-07-08 18:40 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Tomas Vondra <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Siddharth Kothari @ 2026-07-08 16:52 UTC (permalink / raw)
To: [email protected]; +Cc: Vaibhav Jain <[email protected]>; Madhukar <[email protected]>; [email protected]; [email protected]
Hi hackers,
Adding Melanie and Tomas to this thread. I noticed you both recently
committed similar changes to handle shared memory instrumentation
separation and retrieval for IndexScan/IndexOnlyScan (dd78e69cfc
<https://github.com/postgres/postgres/commit/dd78e69cfc337f93cfc0303ddf660262d7f1237e;)
and SeqScan (3b1117d6e2
<https://github.com/postgres/postgres/commit/3b1117d6e2e47d86cdbd978b79434c630cb0ef52;
).
My patch implements the exact same pattern for CustomScan states. It adds
an optional RetrieveInstrumentationCustomScan hook so that extensions can
aggregate their worker metrics before the DSM gets unlinked, bringing
custom scans to parity with the recent core scan instrumentation
improvements.
Could you please help review?
The CommitFest entry is also updated here:
https://commitfest.postgresql.org/patch/6524/
Thanks, Siddharth
On Tue, Jun 23, 2026 at 2:38 PM Siddharth Kothari <[email protected]> wrote:
> Hi hackers,
>
> Bumping this thread as this small patch is passing all CF checks but needs
> a reviewer.
>
> *The Problem:* Right now, if you write a parallel CustomScan extension,
> it is impossible to get performance metrics from parallel workers. The data
> is destroyed during DSM unlinking before the leader can grab it.
>
> *The Fix:* This patch adds an optional RetrieveInstrumentation hook to
> CustomExecMethods. It has zero overhead for existing extensions (it's a
> simple NULL check) but unlocks metric aggregation for parallel custom scans.
>
> Please let me know your thoughts.
>
> Thanks,
>
> Siddharth
>
> On Wed, Apr 8, 2026 at 10:57 AM Siddharth Kothari <[email protected]>
> wrote:
>
>> Hi everyone,
>>
>> I’m just checking in to see if anyone has had a chance to look at this or
>> if there’s any further information I should provide to help with the
>> review. I have also added the patch to PG20-1 CF queue, the link is
>> https://commitfest.postgresql.org/patch/6524/.
>>
>> Thanks,
>> Siddharth
>>
>> On Wed, Feb 18, 2026 at 3:09 PM Siddharth Kothari <[email protected]>
>> wrote:
>>
>>> Dear PostgreSQL Hackers,
>>>
>>> This email proposes a patch to enhance the CustomScan provider
>>> interface. The patch file,
>>> 0001-Add-RetrieveInstrumentationCustomScan-hook-for-Custo.patch, is
>>> attached.
>>>
>>> *Problem:*
>>>
>>> CustomScan providers currently lack a standard method to aggregate
>>> instrumentation data from parallel workers back to the leader process
>>> before the Dynamic Shared Memory (DSM) segment is unlinked. This makes it
>>> difficult to gather comprehensive performance metrics from parallelized
>>> custom scans.
>>>
>>> *Solution:*
>>>
>>> This patch introduces a new optional hook,
>>> RetrieveInstrumentationCustomScan, to the CustomExecMethods struct.
>>> This hook allows custom scan providers to implement logic to collect and
>>> consolidate instrumentation from shared memory or worker states during the
>>> parallel query cleanup phase. This hook is invoked via the new
>>> ExecCustomScanRetrieveInstrumentation function, called from
>>> ExecParallelRetrieveInstrumentation for T_CustomScanState nodes. Since
>>> the hook is optional (checked for NULL before calling), it maintains full
>>> backward compatibility.
>>>
>>> *Testing & Compatibility:*
>>>
>>> - The patch compiles and passes all core regression tests (make
>>> check-world) on my x86_64 instance.
>>> - The changes are not platform-specific.
>>> - Regression Tests: This patch provides a new *capability* for
>>> custom scan providers. Since the hook's functionality is only realized when
>>> implemented by an extension, specific tests would naturally reside within
>>> that extension rather than in the core regression suite.
>>>
>>> This patch does not directly address a specific item on the official
>>> TODO list but enhances the extensibility framework.
>>>
>>> I believe this patch is complete and ready for review. I look forward to
>>> any feedback and am happy to make revisions. I will also add this patch to
>>> the next CommitFest.
>>>
>>> Thank you,
>>>
>>> Siddharth Kothari
>>>
>>
Attachments:
[application/x-patch] v2-0001-Add-RetrieveInstrumentationCustomScan-hook-for-Cu.patch (3.3K, ../../CAGCUe0JRZP-=rOefQz0AYLPpeg=EN=a=TcD9=ZSDup1Mn+Sanw@mail.gmail.com/3-v2-0001-Add-RetrieveInstrumentationCustomScan-hook-for-Cu.patch)
download | inline diff:
From 297ea1876dd6aa77ac59f1c9dbf5bd84eb726fb5 Mon Sep 17 00:00:00 2001
From: Siddharth Kothari <[email protected]>
Date: Tue, 17 Feb 2026 12:23:56 +0000
Subject: [PATCH v2] Add RetrieveInstrumentationCustomScan hook for CustomScan
providers
CustomScan providers currently lack a standard method to aggregate
instrumentation data from parallel workers back to the leader process
before the Dynamic Shared Memory segment is destroyed.
This patch introduces an optional RetrieveInstrumentationCustomScan
callback to the CustomExecMethods struct. This allows custom scan
providers to implement logic to collect and consolidate instrumentation
from shared memory.
The new hook is called in ExecRetrieveInstrumentation for CustomScanState
nodes during the parallel query cleanup phase.
---
src/backend/executor/execParallel.c | 3 +++
src/backend/executor/nodeCustom.c | 9 +++++++++
src/include/executor/nodeCustom.h | 1 +
src/include/nodes/extensible.h | 3 +++
4 files changed, 16 insertions(+)
diff --git a/src/backend/executor/execParallel.c b/src/backend/executor/execParallel.c
index 81b87d82fab..f9fe5347761 100644
--- a/src/backend/executor/execParallel.c
+++ b/src/backend/executor/execParallel.c
@@ -1166,6 +1166,9 @@ ExecParallelRetrieveInstrumentation(PlanState *planstate,
case T_TidRangeScanState:
ExecTidRangeScanRetrieveInstrumentation((TidRangeScanState *) planstate);
break;
+ case T_CustomScanState:
+ ExecCustomScanRetrieveInstrumentation((CustomScanState *) planstate);
+ break;
default:
break;
}
diff --git a/src/backend/executor/nodeCustom.c b/src/backend/executor/nodeCustom.c
index b7cc890cd20..bdb25d334a1 100644
--- a/src/backend/executor/nodeCustom.c
+++ b/src/backend/executor/nodeCustom.c
@@ -217,6 +217,15 @@ ExecCustomScanInitializeWorker(CustomScanState *node,
}
}
+void
+ExecCustomScanRetrieveInstrumentation(CustomScanState *node)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->RetrieveInstrumentationCustomScan)
+ methods->RetrieveInstrumentationCustomScan(node);
+}
+
void
ExecShutdownCustomScan(CustomScanState *node)
{
diff --git a/src/include/executor/nodeCustom.h b/src/include/executor/nodeCustom.h
index fb0acc6e414..0a9cfb40381 100644
--- a/src/include/executor/nodeCustom.h
+++ b/src/include/executor/nodeCustom.h
@@ -37,6 +37,7 @@ extern void ExecCustomScanReInitializeDSM(CustomScanState *node,
ParallelContext *pcxt);
extern void ExecCustomScanInitializeWorker(CustomScanState *node,
ParallelWorkerContext *pwcxt);
+extern void ExecCustomScanRetrieveInstrumentation(CustomScanState *node);
extern void ExecShutdownCustomScan(CustomScanState *node);
#endif /* NODECUSTOM_H */
diff --git a/src/include/nodes/extensible.h b/src/include/nodes/extensible.h
index 517db95c4a3..cda478b538f 100644
--- a/src/include/nodes/extensible.h
+++ b/src/include/nodes/extensible.h
@@ -151,6 +151,9 @@ typedef struct CustomExecMethods
void *coordinate);
void (*ShutdownCustomScan) (CustomScanState *node);
+ /* Optional: retrieve parallel instrumentation */
+ void (*RetrieveInstrumentationCustomScan) (CustomScanState *node);
+
/* Optional: print additional information in EXPLAIN */
void (*ExplainCustomScan) (CustomScanState *node,
List *ancestors,
--
2.55.0.795.g602f6c329a-goog
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers
2026-02-18 09:39 [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-04-08 05:27 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-06-23 09:08 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-07-08 16:52 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
@ 2026-07-08 18:40 ` Tomas Vondra <[email protected]>
2026-07-09 08:07 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Tomas Vondra @ 2026-07-08 18:40 UTC (permalink / raw)
To: Siddharth Kothari <[email protected]>; [email protected]; +Cc: Vaibhav Jain <[email protected]>; Madhukar <[email protected]>; Melanie Plageman <[email protected]>
Hi Siddharth,
On 7/8/26 18:52, Siddharth Kothari wrote:
> Hi hackers,
>
> Adding Melanie and Tomas to this thread. I noticed you both recently
> committed similar changes to handle shared memory instrumentation
> separation and retrieval for IndexScan/IndexOnlyScan (dd78e69cfc
> <https://github.com/postgres/postgres/commit/
> dd78e69cfc337f93cfc0303ddf660262d7f1237e>) and SeqScan (3b1117d6e2
> <https://github.com/postgres/postgres/
> commit/3b1117d6e2e47d86cdbd978b79434c630cb0ef52>).
>
> My patch implements the exact same pattern for CustomScan states. It
> adds an optional RetrieveInstrumentationCustomScan hook so that
> extensions can aggregate their worker metrics before the DSM gets
> unlinked, bringing custom scans to parity with the recent core scan
> instrumentation improvements.
>
> Could you please help review?
>
> The CommitFest entry is also updated here: https://
> commitfest.postgresql.org/patch/6524/ <https://
> commitfest.postgresql.org/patch/6524/>
>
Thanks for the patch, and sorry for not responding earlier. The last
couple months were incredibly busy, both because of the last commitfest
and personal reasons.
I think the patch looks generally OK, except for two things:
1) It needs to add the new callback to doc/src/sgml/custom-scan.sgml,
with similar documentation as for the other callbacks.
2) While looking at execParallel.c it occurred to me the CustomScan may
have the same issue as described in [1] (and a couple messages after
that). The scan may in the parallel part of a plan, but not necessarily
parallel-aware. But a lot of the initialization is gated by
if (planstate->plan->parallel_aware)
...
All the other scans initialize some of the instrumentation always, in
blocks like this:
/* even when not parallel-aware, for EXPLAIN ANALYZE */
And BHS had this issue until 9c18b47e610, but we didn't do the same
thing for CustomScan. I guess we should.
Of course, it's not the fault of this patch, it's just something I
noticed while looking at the code. OTOH it probably also makes sense for
CustomScan to handle the instrumentation just like the other scans, i.e.
by allocating a separate DSM for the instrumentation. AFAICS that'll
require a couple more optional callbacks in CustomExecMethods:
ExecCustomScanInstrumentEstimate
ExecCustomScanInstrumentInitDSM
ExecCustomScanInstrumentInitWorker
But I think that's OK.
[1]
https://www.postgresql.org/message-id/3bdbc70d-ad44-494a-8aab-868b5066fe8b%40vondra.me
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers
2026-02-18 09:39 [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-04-08 05:27 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-06-23 09:08 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-07-08 16:52 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-07-08 18:40 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Tomas Vondra <[email protected]>
@ 2026-07-09 08:07 ` Siddharth Kothari <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Siddharth Kothari @ 2026-07-09 08:07 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: [email protected]; Vaibhav Jain <[email protected]>; Madhukar <[email protected]>; Melanie Plageman <[email protected]>
Hi Tomas,
Thanks for the review, and no worries.
I completely agree with both of your suggestions.
I added the new missing callbacks:
ExecCustomScanInstrumentEstimate
ExecCustomScanInstrumentInitDSM
ExecCustomScanInstrumentInitWorker
along with the existing
ExecCustomScanRetrieveInstrumentation
and extracted the custom scan instrumentation tracking out of the
parallel_aware guard block in execParallel.c. The core execution
orchestrator will now correctly invoke these instrumentation callbacks
even when a CustomScan node is part of a parallel chunk but not
strictly parallel_aware itself.
I also updated the Custom scan documentation to properly document the
new callbacks.
I've attached the v3 patch with these changes included. Let me know if
everything looks good or if there is anything else I should adjust.
Regards,
Siddharth
On Thu, Jul 9, 2026 at 12:10 AM Tomas Vondra <[email protected]> wrote:
>
> Hi Siddharth,
>
> On 7/8/26 18:52, Siddharth Kothari wrote:
> > Hi hackers,
> >
> > Adding Melanie and Tomas to this thread. I noticed you both recently
> > committed similar changes to handle shared memory instrumentation
> > separation and retrieval for IndexScan/IndexOnlyScan (dd78e69cfc
> > <https://github.com/postgres/postgres/commit/
> > dd78e69cfc337f93cfc0303ddf660262d7f1237e>) and SeqScan (3b1117d6e2
> > <https://github.com/postgres/postgres/
> > commit/3b1117d6e2e47d86cdbd978b79434c630cb0ef52>).
> >
> > My patch implements the exact same pattern for CustomScan states. It
> > adds an optional RetrieveInstrumentationCustomScan hook so that
> > extensions can aggregate their worker metrics before the DSM gets
> > unlinked, bringing custom scans to parity with the recent core scan
> > instrumentation improvements.
> >
> > Could you please help review?
> >
> > The CommitFest entry is also updated here: https://
> > commitfest.postgresql.org/patch/6524/ <https://
> > commitfest.postgresql.org/patch/6524/>
> >
>
> Thanks for the patch, and sorry for not responding earlier. The last
> couple months were incredibly busy, both because of the last commitfest
> and personal reasons.
>
> I think the patch looks generally OK, except for two things:
>
> 1) It needs to add the new callback to doc/src/sgml/custom-scan.sgml,
> with similar documentation as for the other callbacks.
>
> 2) While looking at execParallel.c it occurred to me the CustomScan may
> have the same issue as described in [1] (and a couple messages after
> that). The scan may in the parallel part of a plan, but not necessarily
> parallel-aware. But a lot of the initialization is gated by
>
> if (planstate->plan->parallel_aware)
> ...
>
> All the other scans initialize some of the instrumentation always, in
> blocks like this:
>
> /* even when not parallel-aware, for EXPLAIN ANALYZE */
>
> And BHS had this issue until 9c18b47e610, but we didn't do the same
> thing for CustomScan. I guess we should.
>
> Of course, it's not the fault of this patch, it's just something I
> noticed while looking at the code. OTOH it probably also makes sense for
> CustomScan to handle the instrumentation just like the other scans, i.e.
> by allocating a separate DSM for the instrumentation. AFAICS that'll
> require a couple more optional callbacks in CustomExecMethods:
>
> ExecCustomScanInstrumentEstimate
> ExecCustomScanInstrumentInitDSM
> ExecCustomScanInstrumentInitWorker
>
> But I think that's OK.
>
>
> [1]
> https://www.postgresql.org/message-id/3bdbc70d-ad44-494a-8aab-868b5066fe8b%40vondra.me
>
>
> regards
>
> --
> Tomas Vondra
>
Attachments:
[application/x-patch] v3-0001-Add-parallel-instrumentation-hooks-for-CustomScan.patch (8.9K, ../../CAGCUe0J7m3SYhvfJ1216Ts=quPXJEmtw=QXh6AvaMWKU5JUeDQ@mail.gmail.com/2-v3-0001-Add-parallel-instrumentation-hooks-for-CustomScan.patch)
download | inline diff:
From d57d42e876ab87085593738066f70cbad546aa73 Mon Sep 17 00:00:00 2001
From: Siddharth Kothari <[email protected]>
Date: Tue, 17 Feb 2026 12:23:56 +0000
Subject: [PATCH v3] Add parallel instrumentation hooks for CustomScan
providers
CustomScan providers currently lack a standard method to aggregate
instrumentation data from parallel workers back to the leader process
before the Dynamic Shared Memory segment is destroyed. Furthermore,
unlike built-in scans, they are missing the callbacks required to allocate
instrumentation DSM when they are in the parallel part of a plan but
not necessarily parallel-aware.
This patch introduces four optional callbacks to CustomExecMethods:
- EstimateDSMCustomScanInstrumentation
- InitializeDSMCustomScanInstrumentation
- InitializeWorkerCustomScanInstrumentation
- RetrieveInstrumentationCustomScan
This allows custom scan providers to fully support parallel EXPLAIN ANALYZE
metrics and consolidate instrumentation from shared memory similarly to
built-in scan nodes.
---
doc/src/sgml/custom-scan.sgml | 43 +++++++++++++++++++++++++++++
src/backend/executor/execParallel.c | 12 ++++++++
src/backend/executor/nodeCustom.c | 37 +++++++++++++++++++++++++
src/include/executor/nodeCustom.h | 7 +++++
src/include/nodes/extensible.h | 13 +++++++++
5 files changed, 112 insertions(+)
diff --git a/doc/src/sgml/custom-scan.sgml b/doc/src/sgml/custom-scan.sgml
index a200d502cdd..8c2a4f82041 100644
--- a/doc/src/sgml/custom-scan.sgml
+++ b/doc/src/sgml/custom-scan.sgml
@@ -409,6 +409,49 @@ void (*ShutdownCustomScan) (CustomScanState *node);
<para>
<programlisting>
+void (*EstimateDSMCustomScanInstrumentation) (CustomScanState *node,
+ ParallelContext *pcxt);
+</programlisting>
+ Estimate the amount of dynamic shared memory that will be required
+ for parallel operation's instrumentation (for <literal>EXPLAIN ANALYZE</literal>).
+ This callback is optional, and need only be supplied if this custom
+ scan provider supports parallel execution and needs custom instrumentation.
+ </para>
+
+ <para>
+<programlisting>
+void (*InitializeDSMCustomScanInstrumentation) (CustomScanState *node,
+ ParallelContext *pcxt);
+</programlisting>
+ Initialize the dynamic shared memory that will be required for parallel
+ operation's instrumentation.
+ This callback is optional, and need only be supplied if this custom
+ scan provider supports parallel execution and needs custom instrumentation.
+ </para>
+
+ <para>
+<programlisting>
+void (*InitializeWorkerCustomScanInstrumentation) (CustomScanState *node,
+ ParallelWorkerContext *pwcxt);
+</programlisting>
+ Initialize a parallel worker's local state based on the shared state
+ set up by the leader during <function>InitializeDSMCustomScanInstrumentation</function>.
+ This callback is optional, and need only be supplied if this custom
+ scan provider supports parallel execution and needs custom instrumentation.
+ </para>
+
+ <para>
+<programlisting>
+void (*RetrieveInstrumentationCustomScan) (CustomScanState *node);
+</programlisting>
+ Retrieve parallel instrumentation data from dynamic shared memory into local
+ memory when the custom-scan plan node finishes execution.
+ This callback is optional, and need only be supplied if this custom
+ scan provider supports parallel execution and needs custom instrumentation.
+ </para>
+
+ <para>
+<programlisting>
void (*ExplainCustomScan) (CustomScanState *node,
List *ancestors,
ExplainState *es);
diff --git a/src/backend/executor/execParallel.c b/src/backend/executor/execParallel.c
index 81b87d82fab..7983ac1024e 100644
--- a/src/backend/executor/execParallel.c
+++ b/src/backend/executor/execParallel.c
@@ -304,6 +304,9 @@ ExecParallelEstimate(PlanState *planstate, ExecParallelEstimateContext *e)
if (planstate->plan->parallel_aware)
ExecCustomScanEstimate((CustomScanState *) planstate,
e->pcxt);
+ /* even when not parallel-aware, for EXPLAIN ANALYZE */
+ ExecCustomScanInstrumentEstimate((CustomScanState *) planstate,
+ e->pcxt);
break;
case T_BitmapHeapScanState:
if (planstate->plan->parallel_aware)
@@ -552,6 +555,9 @@ ExecParallelInitializeDSM(PlanState *planstate,
if (planstate->plan->parallel_aware)
ExecCustomScanInitializeDSM((CustomScanState *) planstate,
d->pcxt);
+ /* even when not parallel-aware, for EXPLAIN ANALYZE */
+ ExecCustomScanInstrumentInitDSM((CustomScanState *) planstate,
+ d->pcxt);
break;
case T_BitmapHeapScanState:
if (planstate->plan->parallel_aware)
@@ -1166,6 +1172,9 @@ ExecParallelRetrieveInstrumentation(PlanState *planstate,
case T_TidRangeScanState:
ExecTidRangeScanRetrieveInstrumentation((TidRangeScanState *) planstate);
break;
+ case T_CustomScanState:
+ ExecCustomScanRetrieveInstrumentation((CustomScanState *) planstate);
+ break;
default:
break;
}
@@ -1451,6 +1460,9 @@ ExecParallelInitializeWorker(PlanState *planstate, ParallelWorkerContext *pwcxt)
if (planstate->plan->parallel_aware)
ExecCustomScanInitializeWorker((CustomScanState *) planstate,
pwcxt);
+ /* even when not parallel-aware, for EXPLAIN ANALYZE */
+ ExecCustomScanInstrumentInitWorker((CustomScanState *) planstate,
+ pwcxt);
break;
case T_BitmapHeapScanState:
if (planstate->plan->parallel_aware)
diff --git a/src/backend/executor/nodeCustom.c b/src/backend/executor/nodeCustom.c
index b7cc890cd20..04a043543d1 100644
--- a/src/backend/executor/nodeCustom.c
+++ b/src/backend/executor/nodeCustom.c
@@ -217,6 +217,43 @@ ExecCustomScanInitializeWorker(CustomScanState *node,
}
}
+void
+ExecCustomScanInstrumentEstimate(CustomScanState *node, ParallelContext *pcxt)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->EstimateDSMCustomScanInstrumentation)
+ methods->EstimateDSMCustomScanInstrumentation(node, pcxt);
+}
+
+void
+ExecCustomScanInstrumentInitDSM(CustomScanState *node, ParallelContext *pcxt)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->InitializeDSMCustomScanInstrumentation)
+ methods->InitializeDSMCustomScanInstrumentation(node, pcxt);
+}
+
+void
+ExecCustomScanInstrumentInitWorker(CustomScanState *node,
+ ParallelWorkerContext *pwcxt)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->InitializeWorkerCustomScanInstrumentation)
+ methods->InitializeWorkerCustomScanInstrumentation(node, pwcxt);
+}
+
+void
+ExecCustomScanRetrieveInstrumentation(CustomScanState *node)
+{
+ const CustomExecMethods *methods = node->methods;
+
+ if (methods->RetrieveInstrumentationCustomScan)
+ methods->RetrieveInstrumentationCustomScan(node);
+}
+
void
ExecShutdownCustomScan(CustomScanState *node)
{
diff --git a/src/include/executor/nodeCustom.h b/src/include/executor/nodeCustom.h
index fb0acc6e414..f702ef01709 100644
--- a/src/include/executor/nodeCustom.h
+++ b/src/include/executor/nodeCustom.h
@@ -37,6 +37,13 @@ extern void ExecCustomScanReInitializeDSM(CustomScanState *node,
ParallelContext *pcxt);
extern void ExecCustomScanInitializeWorker(CustomScanState *node,
ParallelWorkerContext *pwcxt);
+extern void ExecCustomScanInstrumentEstimate(CustomScanState *node,
+ ParallelContext *pcxt);
+extern void ExecCustomScanInstrumentInitDSM(CustomScanState *node,
+ ParallelContext *pcxt);
+extern void ExecCustomScanInstrumentInitWorker(CustomScanState *node,
+ ParallelWorkerContext *pwcxt);
+extern void ExecCustomScanRetrieveInstrumentation(CustomScanState *node);
extern void ExecShutdownCustomScan(CustomScanState *node);
#endif /* NODECUSTOM_H */
diff --git a/src/include/nodes/extensible.h b/src/include/nodes/extensible.h
index 517db95c4a3..69f5a627039 100644
--- a/src/include/nodes/extensible.h
+++ b/src/include/nodes/extensible.h
@@ -151,6 +151,19 @@ typedef struct CustomExecMethods
void *coordinate);
void (*ShutdownCustomScan) (CustomScanState *node);
+ /* Optional: estimate parallel instrumentation size */
+ void (*EstimateDSMCustomScanInstrumentation) (CustomScanState *node,
+ ParallelContext *pcxt);
+ /* Optional: initialize parallel instrumentation DSM */
+ void (*InitializeDSMCustomScanInstrumentation) (CustomScanState *node,
+ ParallelContext *pcxt);
+ /* Optional: initialize parallel instrumentation worker */
+ void (*InitializeWorkerCustomScanInstrumentation) (CustomScanState *node,
+ ParallelWorkerContext *pwcxt);
+
+ /* Optional: retrieve parallel instrumentation */
+ void (*RetrieveInstrumentationCustomScan) (CustomScanState *node);
+
/* Optional: print additional information in EXPLAIN */
void (*ExplainCustomScan) (CustomScanState *node,
List *ancestors,
--
2.55.0.795.g602f6c329a-goog
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-07-09 08:07 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-03-19 02:41 [PATCH 3/4] WIP: v16 docs: Add enable_presorted_aggregate GUC Justin Pryzby <[email protected]>
2026-02-18 09:39 [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-04-08 05:27 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-06-23 09:08 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-07-08 16:52 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
2026-07-08 18:40 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Tomas Vondra <[email protected]>
2026-07-09 08:07 ` Re: [PATCH] Add RetrieveInstrumentation hook for CustomScan providers Siddharth Kothari <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox