Subject: Relids instead of Bitmapset * in plannode.h

From: Ashutosh Bapat <[email-masked]>
Date: Tue, 31 Oct 2023 11:41:45 +0530

Hi,
For some reason plannode.h has declared variable to hold RTIs as
Bitmapset * instead of Relids like other places. Here's patch to fix
it. This is superficial change as Relids is typedefed to Bitmapset *.
Build succeeds for me and also make check passes.

-- 
Best Wishes,
Ashutosh Bapat
From 1458cbd5dade42670b9a0499b65213758790b71f Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email-masked]>
Date: Tue, 31 Oct 2023 11:29:29 +0530
Subject: [PATCH] Use Relids instead of Bitmapset * in plannode.h

Most of the planner code uses Relids instead of Bitmapset * to declare a
bitmapset of RTIs. Fix plannode.h to do the same.

Ashutosh Bapat
---
 src/include/nodes/plannodes.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 1b787fe031..7b30a41a62 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -18,9 +18,9 @@
 #include "access/stratnum.h"
 #include "common/relpath.h"
 #include "lib/stringinfo.h"
-#include "nodes/bitmapset.h"
 #include "nodes/lockoptions.h"
 #include "nodes/parsenodes.h"
+#include "nodes/pathnodes.h"
 #include "nodes/primnodes.h"

@@ -263,7 +263,7 @@ struct PartitionPruneInfo;		/* forward reference to struct below */
 typedef struct Append
 {
 	Plan		plan;
-	Bitmapset  *apprelids;		/* RTIs of appendrel(s) formed by this node */
+	Relids		apprelids;		/* RTIs of appendrel(s) formed by this node */
 	List	   *appendplans;
 	int			nasyncplans;	/* # of asynchronous plans */
 
@@ -287,7 +287,7 @@ typedef struct MergeAppend
 	Plan		plan;
 
 	/* RTIs of appendrel(s) formed by this node */
-	Bitmapset  *apprelids;
+	Relids		apprelids;
 
 	List	   *mergeplans;
 
@@ -714,8 +714,8 @@ typedef struct ForeignScan
 	List	   *fdw_private;	/* private data for FDW */
 	List	   *fdw_scan_tlist; /* optional tlist describing scan tuple */
 	List	   *fdw_recheck_quals;	/* original quals not in scan.plan.qual */
-	Bitmapset  *fs_relids;		/* base+OJ RTIs generated by this scan */
-	Bitmapset  *fs_base_relids; /* base RTIs generated by this scan */
+	Relids		fs_relids;		/* base+OJ RTIs generated by this scan */
+	Relids		fs_base_relids; /* base RTIs generated by this scan */
 	bool		fsSystemCol;	/* true if any "system column" is needed */
 } ForeignScan;
 
@@ -743,7 +743,7 @@ typedef struct CustomScan
 	List	   *custom_exprs;	/* expressions that custom code may evaluate */
 	List	   *custom_private; /* private data for custom code */
 	List	   *custom_scan_tlist;	/* optional tlist describing scan tuple */
-	Bitmapset  *custom_relids;	/* RTIs generated by this scan */
+	Relids		custom_relids;	/* RTIs generated by this scan */
 
 	/*
 	 * NOTE: The method field of CustomScan is required to be a pointer to a
-- 
2.25.1

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 7 Nov 2023 12:06:28 +0100

Hello,

> Build succeeds for me and also make check passes.

I think the reason for having done it this way, was mostly to avoid
including pathnodes.h in plannodes.h.  Did you explore what the
consequences are?  Starting here:
https://doxygen.postgresql.org/plannodes_8h.html

While looking at it, I noticed that tcopprot.h includes both plannodes.h
and parsenodes.h, but there's no reason to include the latter (or at
least headerscheck doesn't complain after removing it), so I propose to
remove it, per 0001 here.  There's a couple of files that need to be
repaired for this change.  windowfuncs.c is a no-brainer.  However,
having to edit bootstrap.h is a bit surprising -- I think before
dac048f71ebb ("Build all Flex files standalone") this inclusion wasn't
necessary, because the .y file already includes parsenodes.h; but after
that commit, bootparse.h needs parsenodes.h to declare YYSTYPE, per
comments in bootscanner.l.  Anyway, this seems a good change.

I also noticed while looking that I messed up in commit 7103ebb7aae8
("Add support for MERGE SQL command") on this point, because I added
#include parsenodes.h to plannodes.h.  This is because MergeAction,
which is in parsenodes.h, is also needed by some executor code.  But the
real way to fix that is to define that struct in primnodes.h.  0002 does
that.  (I'm forced to also move enum OverridingKind there, which is a
bit annoying.)

0003 here is your patch, which I include because of conflicts with my
0002.  After my 0002, plannodes.h is pretty slim, so I'd be hesitant to
include pathnodes.h just to be able to change the Bitmapset * to Relids.
But on the other hand, it doesn't seem to have too bad an effect overall
(if only because plannodes.h is included by rather few files), so +0.1
on doing this.  I would be more at ease if we didn't have to include
parsenodes.h in pathnodes.h, though, but that looks more difficult to
achieve.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
>From dee38d19b281586a17788856cf169b8344c30da7 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email-masked]>
Date: Mon, 6 Nov 2023 18:27:42 +0100
Subject: [PATCH v2 1/3] Don't include parsenodes.h in tcopprot.h

---
 src/backend/utils/adt/windowfuncs.c | 1 +
 src/include/bootstrap/bootstrap.h   | 1 +
 src/include/tcop/tcopprot.h         | 1 -
 3 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..cbdfba6994 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -14,6 +14,7 @@
 #include "postgres.h"
 
 #include "nodes/supportnodes.h"
+#include "nodes/parsenodes.h"
 #include "utils/builtins.h"
 #include "windowapi.h"
 
diff --git a/src/include/bootstrap/bootstrap.h b/src/include/bootstrap/bootstrap.h
index e1cb73c5f2..6a212122a3 100644
--- a/src/include/bootstrap/bootstrap.h
+++ b/src/include/bootstrap/bootstrap.h
@@ -15,6 +15,7 @@
 #define BOOTSTRAP_H
 
 #include "nodes/execnodes.h"
+#include "nodes/parsenodes.h"

 /*
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index 6c49db3bb7..b694d85974 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -15,7 +15,6 @@
 #define TCOPPROT_H
 
 #include "nodes/params.h"
-#include "nodes/parsenodes.h"
 #include "nodes/plannodes.h"
 #include "storage/procsignal.h"
 #include "utils/guc.h"
-- 
2.39.2

>From 870a853434b56d1fecd1a5fe85a56a2f60ff4ca9 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email-masked]>
Date: Mon, 6 Nov 2023 18:28:08 +0100
Subject: [PATCH v2 2/3] Remove parsenodes.h from plannodes.h

This mistake was made because MergeAction was added to parsenodes.h
instead of primnodes.h, where it should have been.  Put it there.
---
 src/include/nodes/parsenodes.h | 24 ------------------------
 src/include/nodes/plannodes.h  |  1 -
 src/include/nodes/primnodes.h  | 27 +++++++++++++++++++++++++++
 3 files changed, 27 insertions(+), 25 deletions(-)

diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index cf7e79062e..e494309da8 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -30,13 +30,6 @@
 #include "partitioning/partdefs.h"

-typedef enum OverridingKind
-{
-	OVERRIDING_NOT_SET = 0,
-	OVERRIDING_USER_VALUE,
-	OVERRIDING_SYSTEM_VALUE,
-} OverridingKind;
-
 /* Possible sources of a Query */
 typedef enum QuerySource
 {
@@ -1681,23 +1674,6 @@ typedef struct MergeWhenClause
 	List	   *values;			/* VALUES to INSERT, or NULL */
 } MergeWhenClause;
 
-/*
- * MergeAction -
- *		Transformed representation of a WHEN clause in a MERGE statement
- */
-typedef struct MergeAction
-{
-	NodeTag		type;
-	bool		matched;		/* true=MATCHED, false=NOT MATCHED */
-	CmdType		commandType;	/* INSERT/UPDATE/DELETE/DO NOTHING */
-	/* OVERRIDING clause */
-	OverridingKind override pg_node_attr(query_jumble_ignore);
-	Node	   *qual;			/* transformed WHEN conditions */
-	List	   *targetList;		/* the target list (of TargetEntry) */
-	/* target attribute numbers of an UPDATE */
-	List	   *updateColnos pg_node_attr(query_jumble_ignore);
-} MergeAction;
-
 /*
  * TriggerTransition -
  *	   representation of transition row or table naming clause
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 24d46c76dc..d40af8e59f 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -20,7 +20,6 @@
 #include "lib/stringinfo.h"
 #include "nodes/bitmapset.h"
 #include "nodes/lockoptions.h"
-#include "nodes/parsenodes.h"
 #include "nodes/primnodes.h"

diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index ab6d7fdc6d..bb930afb52 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -22,6 +22,14 @@
 #include "nodes/pg_list.h"

+typedef enum OverridingKind
+{
+	OVERRIDING_NOT_SET = 0,
+	OVERRIDING_USER_VALUE,
+	OVERRIDING_SYSTEM_VALUE,
+} OverridingKind;
+
+
 /* ----------------------------------------------------------------
  *						node definitions
  * ----------------------------------------------------------------
@@ -1718,6 +1726,25 @@ typedef struct BooleanTest
 	int			location;		/* token location, or -1 if unknown */
 } BooleanTest;
 
+
+/*
+ * MergeAction
+ *
+ * Transformed representation of a WHEN clause in a MERGE statement
+ */
+typedef struct MergeAction
+{
+	NodeTag		type;
+	bool		matched;		/* true=MATCHED, false=NOT MATCHED */
+	CmdType		commandType;	/* INSERT/UPDATE/DELETE/DO NOTHING */
+	/* OVERRIDING clause */
+	OverridingKind override pg_node_attr(query_jumble_ignore);
+	Node	   *qual;			/* transformed WHEN conditions */
+	List	   *targetList;		/* the target list (of TargetEntry) */
+	/* target attribute numbers of an UPDATE */
+	List	   *updateColnos pg_node_attr(query_jumble_ignore);
+} MergeAction;
+
 /*
  * CoerceToDomain
  *
-- 
2.39.2

>From 02987af92cea736dfe6819200304f7e122c842f5 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email-masked]>
Date: Tue, 31 Oct 2023 11:29:29 +0530
Subject: [PATCH v2 3/3] Use Relids instead of Bitmapset * in plannode.h

Ashutosh Bapat
---

index d40af8e59f..c835a4ec16 100644
@@ -18,8 +18,8 @@

 
 
 	List	   *mergeplans;

 
 	/*
2.39.2

---

From: Tom Lane <[email-masked]>
Date: Tue, 07 Nov 2023 10:24:32 -0500

> including pathnodes.h in plannodes.h.

Yes, I'm pretty sure that's exactly the reason, and I'm strongly
against the initially-proposed patch.  The include footprint of
pathnodes.h would be greatly expanded, for no real benefit.
Among other things, that fuzzes the distinction between planner
modules and non-planner modules.

> remove it, per 0001 here.

0001 is ok, except check #include alphabetization.

> that.  (I'm forced to also move enum OverridingKind there, which is a

This seems OK.  It seems to me that parsenodes.h has been required
by plannodes.h for a long time, but if we can decouple them, all
the better.

> 0002.

Still don't like it.

> ... I would be more at ease if we didn't have to include

Yeah, that dependency has been there a long time too.  I'm not too
fussed by dependencies on parsenodes.h, because anything involved
with either planning or execution will certainly be looking at
query trees too.  But I don't want to add dependencies that tie
planning and execution together.

			regards, tom lane

---

From: Ashutosh Bapat <[email-masked]>
Date: Thu, 9 Nov 2023 11:12:28 +0530

> modules and non-planner modules.

As I mentioned in [1] the Bitmapset implementation is not space
efficient to be used as Relids when there are thousands of partitions.
I was assessing all usages of Bitmapset to find if there are other
places where this is an issue. That's when I noticed this. At some
point in future (possibly quite near) when queries will involved
thousands of relations (partitions or otherwise) we will need to
implement Relids in more space efficient way. Having all Relids usages
of Bitmapset labelled as Relids will help us then. If we don't want to
add pathnodes.h to plannodes.h there will be more work to identify
Relids usage. That shouldn't be a couple of days work, so it's ok.

Other possibilities are
1. Define Relids in bitmapset.h itself and use Relids everywhere
Bitmapset * is really Relids. Wherever Relids is used bitmapset.h must
have been included one or other other way. That's a bigger churn.

2. Replace RTIs with Relids in the comments and add the following
comment somewhere near the #include section. "The Relids members in
various structures in this file have been declared as Bitmapset * to
avoid including pathnodes.h in this file. This include has greatly
expanded footprint for no real benefit.".

3. Do nothing right now. If and when we implement Relids as a separate
datastructure, it will get its own module. We will be able to place it
somewhere properly.

I have no additional comments on other patches.

[1] https://www.postgresql.org/message-id/CAExHW5s4EqY43oB%3Dne6B2%3D-xLgrs9ZGeTr1NXwkGFt2j-OmaQQ%40mail.gmail.com

-- 
Best Wishes,
Ashutosh Bapat

---

From: Tom Lane <[email-masked]>
Date: Thu, 09 Nov 2023 01:08:41 -0500

> efficient to be used as Relids when there are thousands of partitions.

TBH, I'd be very strongly against "optimizing" that case by adopting a
data structure that is less efficient for typical rangetable sizes.
I continue to think that anybody who is using that many partitions
is Doing It Wrong and has no excuse for thinking it'll be free.
Moreover, the size of their relid sets is pretty unlikely to be
their worst pain point.

In any case, that is a poor argument for weakening the separation
between planner and executor.  When and if somebody comes up with
a credible replacement for bitmapsets here, we can consider what
we want to do in terms of header-file organization --- but I do
not think including pathnodes.h into executor files will be it.

			regards, tom lane