Subject: Re: POC, WIP: OR-clause support for indexes

From: Andrei Lepikhov <[email-masked]>
Date: Mon, 19 Feb 2024 15:35:19 +0700

On 16/2/2024 19:54, jian he wrote:
> After setting these parameters, overall enable_or_transformation ON is
> performance better.
> sorry for the noise.
Don't worry, at least we know a weak point of partial paths estimation.
> so now I didn't find any corner case where enable_or_transformation is
> ON peforms worse than when it's OFF.
> 
> +typedef struct OrClauseGroupEntry
> +{
> + OrClauseGroupKey key;
> +
> + Node   *node;
> + List   *consts;
> + Oid scalar_type;
> + List   *exprs;
> +} OrClauseGroupEntry;
> 
> I found that the field `scalar_type` was never used.
Thanks, fixed.
In attachment - v17 for both patches. As I see it, the only general 
explanation of the idea is not addressed. I'm not sure how deeply we 
should explain it.

-- 
regards,
Andrei Lepikhov
Postgres Professional
From 3a3b6aa36320a06b64f2f608e3526255e53ed655 Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clause to ANY expressions.

Replace (expr op C1) OR (expr op C2) ... with expr op ANY(C1, C2, ...) on the
preliminary stage of optimization when we are still working with the tree
expression.
Here C<X> is a constant expression, 'expr' is non-constant expression, 'op' is
an operator which returns boolean result and has a commuter (for the case of
reverse order of constant and non-constant parts of the expression,
like 'CX op expr').
Sometimes it can lead to not optimal plan. But we think it is better to have
array of elements instead of a lot of OR clauses. Here is a room for further
optimizations on decomposing that array into more optimal parts.
Authors: Alena Rybakina <[email-masked]>, Andrey Lepikhov <[email-masked]>
Reviewed-by: Peter Geoghegan <[email-masked]>, Ranier Vilela <[email-masked]>
Reviewed-by: Alexander Korotkov <[email-masked]>, Robert Haas <[email-masked]>
Reviewed-by: jian he <[email-masked]>
---
 .../postgres_fdw/expected/postgres_fdw.out    |  16 +-
 src/backend/nodes/queryjumblefuncs.c          |  27 ++
 src/backend/parser/parse_expr.c               | 326 +++++++++++++++++-
 src/backend/utils/misc/guc_tables.c           |  11 +
 src/backend/utils/misc/postgresql.conf.sample |   1 +
 src/bin/pg_dump/t/002_pg_dump.pl              |   6 +-
 src/include/nodes/queryjumble.h               |   1 +
 src/include/optimizer/optimizer.h             |   1 +
 src/test/regress/expected/create_index.out    | 156 ++++++++-
 src/test/regress/expected/inherit.out         |   2 +-
 src/test/regress/expected/join.out            |  62 +++-
 src/test/regress/expected/partition_prune.out | 219 ++++++++++--
 src/test/regress/expected/rules.out           |   4 +-
 src/test/regress/expected/stats_ext.out       |  12 +-
 src/test/regress/expected/sysviews.out        |   3 +-
 src/test/regress/expected/tidscan.out         |  23 +-
 src/test/regress/sql/create_index.sql         |  35 ++
 src/test/regress/sql/join.sql                 |  10 +
 src/test/regress/sql/partition_prune.sql      |  22 ++
 src/test/regress/sql/tidscan.sql              |   6 +
 src/tools/pgindent/typedefs.list              |   2 +
 21 files changed, 875 insertions(+), 70 deletions(-)

diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index c355e8f3f7..0523bbd8f7 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1349,7 +1349,7 @@ SELECT t1.c1, t1.c2, t2.c1, t2.c2 FROM ft4 t1 LEFT JOIN (SELECT * FROM ft5 WHERE
  Foreign Scan
    Output: t1.c1, t1.c2, ft5.c1, ft5.c2
    Relations: (public.ft4 t1) LEFT JOIN (public.ft5)
-   Remote SQL: SELECT r1.c1, r1.c2, r4.c1, r4.c2 FROM ("S 1"."T 3" r1 LEFT JOIN "S 1"."T 4" r4 ON (((r1.c1 = r4.c1)) AND ((r4.c1 < 10)))) WHERE (((r4.c1 < 10) OR (r4.c1 IS NULL))) AND ((r1.c1 < 10))
+   Remote SQL: SELECT r1.c1, r1.c2, r4.c1, r4.c2 FROM ("S 1"."T 3" r1 LEFT JOIN "S 1"."T 4" r4 ON (((r1.c1 = r4.c1)) AND ((r4.c1 < 10)))) WHERE (((r4.c1 IS NULL) OR (r4.c1 < 10))) AND ((r1.c1 < 10))
 (4 rows)
 
 SELECT t1.c1, t1.c2, t2.c1, t2.c2 FROM ft4 t1 LEFT JOIN (SELECT * FROM ft5 WHERE c1 < 10) t2 ON (t1.c1 = t2.c1)
@@ -3105,7 +3105,7 @@ select array_agg(distinct (t1.c1)%5) from ft4 t1 full join ft5 t2 on (t1.c1 = t2
  Foreign Scan
    Output: (array_agg(DISTINCT (t1.c1 % 5))), ((t2.c1 % 3))
    Relations: Aggregate on ((public.ft4 t1) FULL JOIN (public.ft5 t2))
-   Remote SQL: SELECT array_agg(DISTINCT (r1.c1 % 5)), (r2.c1 % 3) FROM ("S 1"."T 3" r1 FULL JOIN "S 1"."T 4" r2 ON (((r1.c1 = r2.c1)))) WHERE (((r1.c1 < 20) OR ((r1.c1 IS NULL) AND (r2.c1 < 5)))) GROUP BY 2 ORDER BY array_agg(DISTINCT (r1.c1 % 5)) ASC NULLS LAST
+   Remote SQL: SELECT array_agg(DISTINCT (r1.c1 % 5)), (r2.c1 % 3) FROM ("S 1"."T 3" r1 FULL JOIN "S 1"."T 4" r2 ON (((r1.c1 = r2.c1)))) WHERE ((((r1.c1 IS NULL) AND (r2.c1 < 5)) OR (r1.c1 < 20))) GROUP BY 2 ORDER BY array_agg(DISTINCT (r1.c1 % 5)) ASC NULLS LAST
 (4 rows)
 
 select array_agg(distinct (t1.c1)%5) from ft4 t1 full join ft5 t2 on (t1.c1 = t2.c1) where t1.c1 < 20 or (t1.c1 is null and t2.c1 < 5) group by (t2.c1)%3 order by 1;
@@ -3123,7 +3123,7 @@ select array_agg(distinct (t1.c1)%5 order by (t1.c1)%5) from ft4 t1 full join ft
  Foreign Scan
    Output: (array_agg(DISTINCT (t1.c1 % 5) ORDER BY (t1.c1 % 5))), ((t2.c1 % 3))
    Relations: Aggregate on ((public.ft4 t1) FULL JOIN (public.ft5 t2))
-   Remote SQL: SELECT array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) ASC NULLS LAST), (r2.c1 % 3) FROM ("S 1"."T 3" r1 FULL JOIN "S 1"."T 4" r2 ON (((r1.c1 = r2.c1)))) WHERE (((r1.c1 < 20) OR ((r1.c1 IS NULL) AND (r2.c1 < 5)))) GROUP BY 2 ORDER BY array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) ASC NULLS LAST) ASC NULLS LAST
+   Remote SQL: SELECT array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) ASC NULLS LAST), (r2.c1 % 3) FROM ("S 1"."T 3" r1 FULL JOIN "S 1"."T 4" r2 ON (((r1.c1 = r2.c1)))) WHERE ((((r1.c1 IS NULL) AND (r2.c1 < 5)) OR (r1.c1 < 20))) GROUP BY 2 ORDER BY array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) ASC NULLS LAST) ASC NULLS LAST
 (4 rows)
 
 select array_agg(distinct (t1.c1)%5 order by (t1.c1)%5) from ft4 t1 full join ft5 t2 on (t1.c1 = t2.c1) where t1.c1 < 20 or (t1.c1 is null and t2.c1 < 5) group by (t2.c1)%3 order by 1;
@@ -3140,7 +3140,7 @@ select array_agg(distinct (t1.c1)%5 order by (t1.c1)%5 desc nulls last) from ft4
  Foreign Scan
    Output: (array_agg(DISTINCT (t1.c1 % 5) ORDER BY (t1.c1 % 5) DESC NULLS LAST)), ((t2.c1 % 3))
    Relations: Aggregate on ((public.ft4 t1) FULL JOIN (public.ft5 t2))
-   Remote SQL: SELECT array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) DESC NULLS LAST), (r2.c1 % 3) FROM ("S 1"."T 3" r1 FULL JOIN "S 1"."T 4" r2 ON (((r1.c1 = r2.c1)))) WHERE (((r1.c1 < 20) OR ((r1.c1 IS NULL) AND (r2.c1 < 5)))) GROUP BY 2 ORDER BY array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) DESC NULLS LAST) ASC NULLS LAST
+   Remote SQL: SELECT array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) DESC NULLS LAST), (r2.c1 % 3) FROM ("S 1"."T 3" r1 FULL JOIN "S 1"."T 4" r2 ON (((r1.c1 = r2.c1)))) WHERE ((((r1.c1 IS NULL) AND (r2.c1 < 5)) OR (r1.c1 < 20))) GROUP BY 2 ORDER BY array_agg(DISTINCT (r1.c1 % 5) ORDER BY ((r1.c1 % 5)) DESC NULLS LAST) ASC NULLS LAST
 (4 rows)
 
 select array_agg(distinct (t1.c1)%5 order by (t1.c1)%5 desc nulls last) from ft4 t1 full join ft5 t2 on (t1.c1 = t2.c1) where t1.c1 < 20 or (t1.c1 is null and t2.c1 < 5) group by (t2.c1)%3 order by 1;
@@ -8797,18 +8797,18 @@ insert into utrtest values (2, 'qux');
 -- Check case where the foreign partition is a subplan target rel
 explain (verbose, costs off)
 update utrtest set a = 1 where a = 1 or a = 2 returning *;
-                                             QUERY PLAN                                             
-----------------------------------------------------------------------------------------------------
+                                                  QUERY PLAN                                                  
+--------------------------------------------------------------------------------------------------------------
  Update on public.utrtest
    Output: utrtest_1.a, utrtest_1.b
    Foreign Update on public.remp utrtest_1
    Update on public.locp utrtest_2
    ->  Append
          ->  Foreign Update on public.remp utrtest_1
-               Remote SQL: UPDATE public.loct SET a = 1 WHERE (((a = 1) OR (a = 2))) RETURNING a, b
+               Remote SQL: UPDATE public.loct SET a = 1 WHERE ((a = ANY ('{1,2}'::integer[]))) RETURNING a, b
          ->  Seq Scan on public.locp utrtest_2
                Output: 1, utrtest_2.tableoid, utrtest_2.ctid, NULL::record
-               Filter: ((utrtest_2.a = 1) OR (utrtest_2.a = 2))
+               Filter: (utrtest_2.a = ANY ('{1,2}'::integer[]))
 (10 rows)
 
 -- The new values are concatenated with ' triggered !'
diff --git a/src/backend/nodes/queryjumblefuncs.c b/src/backend/nodes/queryjumblefuncs.c
index 82f725baaa..b203746be4 100644
--- a/src/backend/nodes/queryjumblefuncs.c
+++ b/src/backend/nodes/queryjumblefuncs.c
@@ -141,6 +141,33 @@ JumbleQuery(Query *query)
 	return jstate;
 }
 
+JumbleState *
+JumbleExpr(Expr *expr, uint64 *queryId)
+{
+	JumbleState *jstate = NULL;
+
+	Assert(queryId != NULL);
+
+	jstate = (JumbleState *) palloc(sizeof(JumbleState));
+
+	/* Set up workspace for query jumbling */
+	jstate->jumble = (unsigned char *) palloc(JUMBLE_SIZE);
+	jstate->jumble_len = 0;
+	jstate->clocations_buf_size = 32;
+	jstate->clocations = (LocationLen *)
+		palloc(jstate->clocations_buf_size * sizeof(LocationLen));
+	jstate->clocations_count = 0;
+	jstate->highest_extern_param_id = 0;
+
+	/* Compute query ID */
+	_jumbleNode(jstate, (Node *) expr);
+	*queryId = DatumGetUInt64(hash_any_extended(jstate->jumble,
+												jstate->jumble_len,
+												0));
+
+	return jstate;
+}
+
 /*
  * Enables query identifier computation.
  *
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 9300c7b9ab..279713d5ae 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -16,12 +16,14 @@
 #include "postgres.h"
 
 #include "catalog/pg_aggregate.h"
+#include "catalog/pg_operator.h"
 #include "catalog/pg_proc.h"
 #include "catalog/pg_type.h"
 #include "commands/dbcommands.h"
 #include "miscadmin.h"
 #include "nodes/makefuncs.h"
 #include "nodes/nodeFuncs.h"
+#include "nodes/queryjumble.h"
 #include "optimizer/optimizer.h"
 #include "parser/analyze.h"
 #include "parser/parse_agg.h"
@@ -38,11 +40,13 @@
 #include "utils/date.h"
 #include "utils/fmgroids.h"
 #include "utils/lsyscache.h"
+#include "utils/syscache.h"
 #include "utils/timestamp.h"
 #include "utils/xml.h"
 
 /* GUC parameters */
 bool		Transform_null_equals = false;
+bool		enable_or_transformation = true;

 static Node *transformExprRecurse(ParseState *pstate, Node *expr);
@@ -99,6 +103,326 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname,
 static Node *make_nulltest_from_distinct(ParseState *pstate,
 										 A_Expr *distincta, Node *arg);
 
+typedef struct OrClauseGroupKey
+{
+	Expr   *expr; /* Pointer to the expression tree which has been a source for
+					the hashkey value */
+	Oid		opno;
+	Oid		exprtype;
+} OrClauseGroupKey;
+
+typedef struct OrClauseGroupEntry
+{
+	OrClauseGroupKey key;
+
+	Node		   *node;
+	List		   *consts;
+	List 		   *exprs;
+} OrClauseGroupEntry;
+
+/*
+ * Hash function to find candidate clauses.
+ */
+static uint32
+orclause_hash(const void *data, Size keysize)
+{
+	OrClauseGroupKey   *key = (OrClauseGroupKey *) data;
+	uint64				hash;
+
+	Assert(keysize == sizeof(OrClauseGroupKey));
+
+	(void) JumbleExpr(key->expr, &hash);
+	hash += ((uint64) key->opno + (uint64) key->exprtype);
+	return (uint32) (hash % UINT32_MAX);
+}
+
+static void *
+orclause_keycopy(void *dest, const void *src, Size keysize)
+{
+	OrClauseGroupKey *src_key = (OrClauseGroupKey *) src;
+	OrClauseGroupKey *dst_key = (OrClauseGroupKey *) dest;
+
+	Assert(sizeof(OrClauseGroupKey) == keysize);
+
+	dst_key->expr = src_key->expr;
+	dst_key->opno = src_key->opno;
+	dst_key->exprtype = src_key->exprtype;
+	return dst_key;
+}
+
+/*
+ * Dynahash match function to use in guc_hashtab
+ */
+static int
+orclause_match(const void *data1, const void *data2, Size keysize)
+{
+	OrClauseGroupKey   *key1 = (OrClauseGroupKey *) data1;
+	OrClauseGroupKey   *key2 = (OrClauseGroupKey *) data2;
+
+	Assert(sizeof(OrClauseGroupKey) == keysize);
+
+	if (key1->opno == key2->opno && key1->exprtype == key2->exprtype &&
+		equal(key1->expr, key2->expr))
+		return 0;
+
+	return 1;
+}
+
+static Node *
+transformBoolExprOr(ParseState *pstate, BoolExpr *expr)
+{
+	List				   *or_list = NIL;
+	List				   *entries = NIL;
+	ListCell			   *lc;
+	HASHCTL					info;
+	HTAB 				   *or_group_htab = NULL;
+	int 					len_ors = list_length(expr->args);
+	OrClauseGroupEntry	   *entry = NULL;
+
+	/* If this is not an 'OR' expression, skip the transformation */
+	if (!enable_or_transformation || expr->boolop != OR_EXPR || len_ors < 2)
+		return transformBoolExpr(pstate, (BoolExpr *) expr);
+
+	MemSet(&info, 0, sizeof(info));
+	info.keysize = sizeof(OrClauseGroupKey);
+	info.entrysize = sizeof(OrClauseGroupEntry);
+	info.hash = orclause_hash;
+	info.keycopy = orclause_keycopy;
+	info.match = orclause_match;
+	or_group_htab = hash_create("OR Groups",
+								len_ors,
+								&info,
+								HASH_ELEM | HASH_FUNCTION | HASH_COMPARE | HASH_KEYCOPY);
+
+	foreach(lc, expr->args)
+	{
+		Node				   *arg = lfirst(lc);
+		Node				   *orqual;
+		Node				   *const_expr;
+		Node				   *nconst_expr;
+		OrClauseGroupKey		hashkey;
+		bool					found;
+		Oid						opno;
+		Oid						exprtype;
+		Node				   *leftop, *rightop;
+
+		/* At first, transform the arg and evaluate constant expressions. */
+		orqual = transformExprRecurse(pstate, (Node *) arg);
+		orqual = coerce_to_boolean(pstate, orqual, "OR");
+
+		if (!IsA(orqual, OpExpr))
+		{
+			or_list = lappend(or_list, orqual);
+			continue;
+		}
+
+		opno = ((OpExpr *) orqual)->opno;
+		if (get_op_rettype(opno) != BOOLOID)
+		{
+			/* Only operator returning boolean suits OR -> ANY transformation */
+			or_list = lappend(or_list, orqual);
+			continue;
+		}
+
+		/*
+		 * Detect the constant side of the clause. Recall non-constant
+		 * expression can be made not only with Vars, but also with Params,
+		 * which is not bonded with any relation. Thus, we detect the const
+		 * side - if another side is constant too, the orqual couldn't be
+		 * an OpExpr.
+		 * Get pointers to constant and expression sides of the qual.
+		 */
+		leftop = get_leftop(orqual);
+		if (IsA(leftop, RelabelType))
+			leftop = (Node *) ((RelabelType *) leftop)->arg;
+		rightop = get_rightop(orqual);
+		if (IsA(rightop, RelabelType))
+			rightop = (Node *) ((RelabelType *) rightop)->arg;
+
+		if (IsA(leftop, Const))
+		{
+			opno = get_commutator(opno);
+
+			if (!OidIsValid(opno))
+			{
+				/* commutator doesn't exist, we can't reverse the order */
+				or_list = lappend(or_list, orqual);
+				continue;
+			}
+
+			nconst_expr = get_rightop(orqual);
+			const_expr = get_leftop(orqual);
+		}
+		else if (IsA(rightop, Const))
+		{
+			const_expr = get_rightop(orqual);
+			nconst_expr = get_leftop(orqual);
+		}
+		else
+		{
+			or_list = lappend(or_list, orqual);
+			continue;
+		}
+
+		/*
+		 * Transformation only works with both side type is not
+		 * { array | composite | domain | record }.
+		 * Also, forbid it for volatile expressions.
+		 */
+		exprtype = exprType(nconst_expr);
+		if (type_is_rowtype(exprType(const_expr)) ||
+			type_is_rowtype(exprtype) ||
+			contain_volatile_functions((Node *) nconst_expr))
+		{
+			or_list = lappend(or_list, orqual);
+			continue;
+		}
+
+		/*
+		* At this point we definitely have a transformable clause.
+		* Classify it and add into specific group of clauses, or create new
+		* group.
+		*/
+		hashkey.expr = (Expr *) nconst_expr;
+		hashkey.opno = opno;
+		hashkey.exprtype = exprtype;
+		entry = hash_search(or_group_htab, &hashkey, HASH_ENTER, &found);
+
+		if (unlikely(found))
+		{
+			entry->consts = lappend(entry->consts, const_expr);
+			entry->exprs = lappend(entry->exprs, orqual);
+		}
+		else
+		{
+			entry->node = nconst_expr;
+			entry->consts = list_make1(const_expr);
+			entry->exprs = list_make1(orqual);
+
+			/*
+			 * Add the entry to the list. It is needed exclusively to manage the
+			 * problem with the order of transformed clauses in explain.
+			 * Hash value can depend on the platform and version. Hence,
+			 * sequental scan of the hash table would prone to change the order
+			 * of clauses in lists and, as a result, break regression tests
+			 * accidentially.
+			 */
+			entries = lappend(entries, entry);
+		}
+	}
+
+	/* Let's convert each group of clauses to an IN operation. */
+
+	/*
+	 * Go through the list of groups and convert each, where number of
+	 * consts more than 1. trivial groups move to OR-list again
+	 */
+
+	foreach (lc, entries)
+	{
+		Oid				    scalar_type;
+		Oid					array_type;
+
+		entry = (OrClauseGroupEntry *) lfirst(lc);
+
+		Assert(list_length(entry->consts) > 0);
+		Assert(list_length(entry->exprs) == list_length(entry->consts));
+
+		if (list_length(entry->consts) == 1)
+		{
+			/*
+			 * Only one element in the class. Return origin expression into
+			 * the BoolExpr args list unchanged.
+			 */
+			list_free(entry->consts);
+			or_list = list_concat(or_list, entry->exprs);
+			continue;
+		}
+
+		/*
+		 * Do the transformation.
+		 */
+
+		scalar_type = entry->key.exprtype;
+		array_type = OidIsValid(scalar_type) ? get_array_type(scalar_type) :
+											   InvalidOid;
+
+		if (OidIsValid(array_type))
+		{
+			/*
+			 * OK: coerce all the right-hand non-Var inputs to the common
+			 * type and build an ArrayExpr for them.
+			 */
+			List			   *aexprs = NIL;
+			ArrayExpr		   *newa = NULL;
+			ScalarArrayOpExpr  *saopexpr = NULL;
+			HeapTuple			opertup;
+			Form_pg_operator	operform;
+			List			   *namelist = NIL;
+			ListCell		   *lc1;
+
+			foreach(lc1, entry->consts)
+			{
+				Node   *rexpr = (Node *) lfirst(lc1);
+
+				rexpr = coerce_to_common_type(pstate, rexpr,
+											  scalar_type,
+											  "IN");
+				aexprs = lappend(aexprs, rexpr);
+			}
+
+			newa = makeNode(ArrayExpr);
+			/* array_collid will be set by parse_collate.c */
+			newa->element_typeid = scalar_type;
+			newa->array_typeid = array_type;
+			newa->multidims = false;
+			newa->elements = aexprs;
+			newa->location = -1;
+
+			opertup = SearchSysCache1(OPEROID,
+									  ObjectIdGetDatum(entry->key.opno));
+			if (!HeapTupleIsValid(opertup))
+				elog(ERROR, "cache lookup failed for operator %u",
+					 entry->key.opno);
+
+			operform = (Form_pg_operator) GETSTRUCT(opertup);
+			if (!OperatorIsVisible(entry->key.opno))
+				namelist = lappend(namelist, makeString(get_namespace_name(operform->oprnamespace)));
+
+			namelist = lappend(namelist, makeString(pstrdup(NameStr(operform->oprname))));
+			ReleaseSysCache(opertup);
+
+			saopexpr =
+				(ScalarArrayOpExpr *)
+					make_scalar_array_op(pstate,
+										 namelist,
+										 true,
+										 entry->node,
+										 (Node *) newa,
+										 -1);
+
+			or_list = lappend(or_list, (void *) saopexpr);
+		}
+		else
+		{
+			/*
+			 * If the const node (right side of operator expression) 's type
+			 *  don't have “true” array type, then we cannnot do the transformation.
+			 * We simply concatenate the expression node.
+			 *
+			 */
+			list_free(entry->consts);
+			or_list = list_concat(or_list, entry->exprs);
+		}
+	}
+	hash_destroy(or_group_htab);
+	list_free(entries);
+
+	/* One more trick: assemble correct clause */
+	return (Node *) ((list_length(or_list) > 1) ?
+					 makeBoolExpr(OR_EXPR, or_list, expr->location) :
+					 linitial(or_list));
+}
 
 /*
  * transformExpr -
@@ -212,7 +536,7 @@ transformExprRecurse(ParseState *pstate, Node *expr)
 			}
 
 		case T_BoolExpr:
-			result = transformBoolExpr(pstate, (BoolExpr *) expr);
+			result = transformBoolExprOr(pstate, (BoolExpr *) expr);
 			break;
 
 		case T_FuncCall:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 70652f0a3f..f42711f336 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1071,6 +1071,17 @@ struct config_bool ConfigureNamesBool[] =
 		true,
 		NULL, NULL, NULL
 	},
+	{
+		{"enable_or_transformation", PGC_USERSET, QUERY_TUNING_OTHER,
+			gettext_noop("Transform a sequence of OR clauses to an IN expression."),
+			gettext_noop("The planner will replace clauses like 'x=c1 OR x=c2 .."
+						 "to the clause 'x IN (c1,c2,...)'"),
+			GUC_EXPLAIN
+		},
+		&enable_or_transformation,
+		true,
+		NULL, NULL, NULL
+	},
 	{
 		/*
 		 * Not for general use --- used by SET SESSION AUTHORIZATION and SET
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e10755972a..f8058e05f5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -379,6 +379,7 @@
 # - Planner Method Configuration -
 
 #enable_async_append = on
+#enable_or_transformation = on
 #enable_bitmapscan = on
 #enable_gathermerge = on
 #enable_hashagg = on
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 00b5092713..d28bf617db 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2095,9 +2095,9 @@ my %tests = (
 		regexp => qr/^
 			\QCREATE DOMAIN dump_test.us_postal_code AS text COLLATE pg_catalog."C" DEFAULT '10014'::text\E\n\s+
 			\QCONSTRAINT us_postal_code_check CHECK \E
-			\Q(((VALUE ~ '^\d{5}\E
-			\$\Q'::text) OR (VALUE ~ '^\d{5}-\d{4}\E\$
-			\Q'::text)));\E(.|\n)*
+			\Q((VALUE ~ ANY (ARRAY['^\d{5}\E
+			\$\Q'::text, '^\d{5}-\d{4}\E\$
+			\Q'::text])));\E(.|\n)*
 			\QCOMMENT ON CONSTRAINT us_postal_code_check ON DOMAIN dump_test.us_postal_code IS 'check it';\E
 			/xm,
 		like =>
diff --git a/src/include/nodes/queryjumble.h b/src/include/nodes/queryjumble.h
index f1c55c8067..a9ae048af5 100644
--- a/src/include/nodes/queryjumble.h
+++ b/src/include/nodes/queryjumble.h
@@ -65,6 +65,7 @@ extern PGDLLIMPORT int compute_query_id;
 
 extern const char *CleanQuerytext(const char *query, int *location, int *len);
 extern JumbleState *JumbleQuery(Query *query);
+extern JumbleState *JumbleExpr(Expr *expr, uint64 *queryId);
 extern void EnableQueryId(void);
 
 extern PGDLLIMPORT bool query_id_enabled;
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
index 7b63c5cf71..35ab577501 100644
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -50,6 +50,7 @@ struct PlannedStmt;
 struct ParamListInfoData;
 struct HeapTupleData;
 
+extern PGDLLIMPORT bool enable_or_transformation;
 
 /* in path/clausesel.c: */
 
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 79fa117cb5..b8653c09ea 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1838,18 +1838,50 @@ DROP TABLE onek_with_null;
 EXPLAIN (COSTS OFF)
 SELECT * FROM tenk1
   WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
-                                                               QUERY PLAN                                                                
------------------------------------------------------------------------------------------------------------------------------------------
- Bitmap Heap Scan on tenk1
-   Recheck Cond: (((thousand = 42) AND (tenthous = 1)) OR ((thousand = 42) AND (tenthous = 3)) OR ((thousand = 42) AND (tenthous = 42)))
-   ->  BitmapOr
-         ->  Bitmap Index Scan on tenk1_thous_tenthous
-               Index Cond: ((thousand = 42) AND (tenthous = 1))
-         ->  Bitmap Index Scan on tenk1_thous_tenthous
-               Index Cond: ((thousand = 42) AND (tenthous = 3))
-         ->  Bitmap Index Scan on tenk1_thous_tenthous
-               Index Cond: ((thousand = 42) AND (tenthous = 42))
-(9 rows)
+                                  QUERY PLAN                                  
+------------------------------------------------------------------------------
+ Index Scan using tenk1_thous_tenthous on tenk1
+   Index Cond: ((thousand = 42) AND (tenthous = ANY ('{1,3,42}'::integer[])))
+(2 rows)
+
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+ unique1 | unique2 | two | four | ten | twenty | hundred | thousand | twothousand | fivethous | tenthous | odd | even | stringu1 | stringu2 | string4 
+---------+---------+-----+------+-----+--------+---------+----------+-------------+-----------+----------+-----+------+----------+----------+---------
+      42 |    5530 |   0 |    2 |   2 |      2 |      42 |       42 |          42 |        42 |       42 |  84 |   85 | QBAAAA   | SEIAAA   | OOOOxx
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand = ANY ('{42,99}'::integer[])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = ANY ('{42,99}'::integer[]))
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+ count 
+-------
+    10
+(1 row)
+
+SET enable_or_transformation = on;
+EXPLAIN (COSTS OFF)
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+                                  QUERY PLAN                                  
+------------------------------------------------------------------------------
+ Index Scan using tenk1_thous_tenthous on tenk1
+   Index Cond: ((thousand = 42) AND (tenthous = ANY ('{1,3,42}'::integer[])))
+(2 rows)
 
 SELECT * FROM tenk1
   WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
@@ -1861,28 +1893,116 @@ SELECT * FROM tenk1
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
-                                   QUERY PLAN                                    
----------------------------------------------------------------------------------
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand = ANY ('{42,99}'::integer[])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = ANY ('{42,99}'::integer[]))
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+ count 
+-------
+    10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand < 42 OR thousand < 99 OR 43 > thousand OR 42 > thousand);
+                                        QUERY PLAN                                        
+------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand < ANY ('{42,99,43,42}'::integer[])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand < ANY ('{42,99,43,42}'::integer[]))
+(8 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+                                               QUERY PLAN                                               
+--------------------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: (((thousand = 42) AND (tenthous = ANY ('{1,3}'::integer[]))) OR (thousand = 41))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: ((thousand = 42) AND (tenthous = ANY ('{1,3}'::integer[])))
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = 41)
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+ count 
+-------
+    10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+                                                         QUERY PLAN                                                          
+-----------------------------------------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: (((hundred = 42) AND ((thousand = ANY ('{42,99}'::integer[])) OR (tenthous < 2))) OR (thousand = 41))
+         ->  BitmapOr
+               ->  BitmapAnd
+                     ->  Bitmap Index Scan on tenk1_hundred
+                           Index Cond: (hundred = 42)
+                     ->  BitmapOr
+                           ->  Bitmap Index Scan on tenk1_thous_tenthous
+                                 Index Cond: (thousand = ANY ('{42,99}'::integer[]))
+                           ->  Bitmap Index Scan on tenk1_thous_tenthous
+                                 Index Cond: (tenthous < 2)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = 41)
+(14 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+ count 
+-------
+    20
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
+                                                          QUERY PLAN                                                          
+------------------------------------------------------------------------------------------------------------------------------
  Aggregate
    ->  Bitmap Heap Scan on tenk1
-         Recheck Cond: ((hundred = 42) AND ((thousand = 42) OR (thousand = 99)))
+         Recheck Cond: ((hundred = 42) AND (((thousand = 99) AND (tenthous = 2)) OR (thousand = ANY ('{42,41}'::integer[]))))
          ->  BitmapAnd
                ->  Bitmap Index Scan on tenk1_hundred
                      Index Cond: (hundred = 42)
                ->  BitmapOr
                      ->  Bitmap Index Scan on tenk1_thous_tenthous
-                           Index Cond: (thousand = 42)
+                           Index Cond: ((thousand = 99) AND (tenthous = 2))
                      ->  Bitmap Index Scan on tenk1_thous_tenthous
-                           Index Cond: (thousand = 99)
+                           Index Cond: (thousand = ANY ('{42,41}'::integer[]))
 (11 rows)
 
 SELECT count(*) FROM tenk1
-  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
  count 
 -------
     10
 (1 row)
 
+RESET enable_or_transformation;
 --
 -- Check behavior with duplicate index column contents
 --
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
index 1e6cc8e9ba..7cdf9d6412 100644
--- a/src/test/regress/expected/inherit.out
+++ b/src/test/regress/expected/inherit.out
@@ -2600,7 +2600,7 @@ explain (costs off) select * from list_parted where a = 'ab' or a in (null, 'cd'
                                    QUERY PLAN                                    
 ---------------------------------------------------------------------------------
  Seq Scan on part_ab_cd list_parted
-   Filter: (((a)::text = 'ab'::text) OR ((a)::text = ANY ('{NULL,cd}'::text[])))
+   Filter: (((a)::text = ANY ('{NULL,cd}'::text[])) OR ((a)::text = 'ab'::text))
 (2 rows)
 
 explain (costs off) select * from list_parted where a = 'ab';
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 0c2cba8921..82a5f689a1 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -4210,10 +4210,10 @@ explain (costs off)
 select * from tenk1 a join tenk1 b on
   (a.unique1 = 1 and b.unique1 = 2) or
   ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
-                                                      QUERY PLAN                                                      
-----------------------------------------------------------------------------------------------------------------------
+                                                       QUERY PLAN                                                       
+------------------------------------------------------------------------------------------------------------------------
  Nested Loop
-   Join Filter: (((a.unique1 = 1) AND (b.unique1 = 2)) OR (((a.unique2 = 3) OR (a.unique2 = 7)) AND (b.hundred = 4)))
+   Join Filter: (((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY ('{3,7}'::integer[])) AND (b.hundred = 4)))
    ->  Bitmap Heap Scan on tenk1 b
          Recheck Cond: ((unique1 = 2) OR (hundred = 4))
          ->  BitmapOr
@@ -4223,16 +4223,64 @@ select * from tenk1 a join tenk1 b on
                      Index Cond: (hundred = 4)
    ->  Materialize
          ->  Bitmap Heap Scan on tenk1 a
-               Recheck Cond: ((unique1 = 1) OR (unique2 = 3) OR (unique2 = 7))
+               Recheck Cond: ((unique1 = 1) OR (unique2 = ANY ('{3,7}'::integer[])))
                ->  BitmapOr
                      ->  Bitmap Index Scan on tenk1_unique1
                            Index Cond: (unique1 = 1)
                      ->  Bitmap Index Scan on tenk1_unique2
-                           Index Cond: (unique2 = 3)
+                           Index Cond: (unique2 = ANY ('{3,7}'::integer[]))
+(17 rows)
+
+SET enable_or_transformation = on;
+explain (costs off)
+select * from tenk1 a join tenk1 b on
+  (a.unique1 = 1 and b.unique1 = 2) or
+  ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+                                                       QUERY PLAN                                                       
+------------------------------------------------------------------------------------------------------------------------
+ Nested Loop
+   Join Filter: (((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY ('{3,7}'::integer[])) AND (b.hundred = 4)))
+   ->  Bitmap Heap Scan on tenk1 b
+         Recheck Cond: ((unique1 = 2) OR (hundred = 4))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on tenk1_unique1
+                     Index Cond: (unique1 = 2)
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 4)
+   ->  Materialize
+         ->  Bitmap Heap Scan on tenk1 a
+               Recheck Cond: ((unique1 = 1) OR (unique2 = ANY ('{3,7}'::integer[])))
+               ->  BitmapOr
+                     ->  Bitmap Index Scan on tenk1_unique1
+                           Index Cond: (unique1 = 1)
                      ->  Bitmap Index Scan on tenk1_unique2
-                           Index Cond: (unique2 = 7)
-(19 rows)
+                           Index Cond: (unique2 = ANY ('{3,7}'::integer[]))
+(17 rows)
+
+explain (costs off)
+select * from tenk1 a join tenk1 b on
+  (a.unique1 < 20 or a.unique1 = 3 or a.unique1 = 1 and b.unique1 = 2) or
+  ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+                                                                          QUERY PLAN                                                                           
+---------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Nested Loop
+   Join Filter: (((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY ('{3,7}'::integer[])) AND (b.hundred = 4)) OR (a.unique1 < 20) OR (a.unique1 = 3))
+   ->  Seq Scan on tenk1 b
+   ->  Materialize
+         ->  Bitmap Heap Scan on tenk1 a
+               Recheck Cond: ((unique1 = 1) OR (unique2 = ANY ('{3,7}'::integer[])) OR (unique1 < 20) OR (unique1 = 3))
+               ->  BitmapOr
+                     ->  Bitmap Index Scan on tenk1_unique1
+                           Index Cond: (unique1 = 1)
+                     ->  Bitmap Index Scan on tenk1_unique2
+                           Index Cond: (unique2 = ANY ('{3,7}'::integer[]))
+                     ->  Bitmap Index Scan on tenk1_unique1
+                           Index Cond: (unique1 < 20)
+                     ->  Bitmap Index Scan on tenk1_unique1
+                           Index Cond: (unique1 = 3)
+(15 rows)
 
+RESET enable_or_transformation;
 --
 -- test placement of movable quals in a parameterized join tree
 --
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 9a4c48c055..3423af4768 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -82,25 +82,47 @@ explain (costs off) select * from lp where a is null;
 (2 rows)
 
 explain (costs off) select * from lp where a = 'a' or a = 'c';
-                        QUERY PLAN                        
-----------------------------------------------------------
+                  QUERY PLAN                   
+-----------------------------------------------
  Append
    ->  Seq Scan on lp_ad lp_1
-         Filter: ((a = 'a'::bpchar) OR (a = 'c'::bpchar))
+         Filter: (a = ANY ('{a,c}'::bpchar[]))
    ->  Seq Scan on lp_bc lp_2
-         Filter: ((a = 'a'::bpchar) OR (a = 'c'::bpchar))
+         Filter: (a = ANY ('{a,c}'::bpchar[]))
 (5 rows)
 
 explain (costs off) select * from lp where a is not null and (a = 'a' or a = 'c');
-                                   QUERY PLAN                                   
---------------------------------------------------------------------------------
+                             QUERY PLAN                              
+---------------------------------------------------------------------
  Append
    ->  Seq Scan on lp_ad lp_1
-         Filter: ((a IS NOT NULL) AND ((a = 'a'::bpchar) OR (a = 'c'::bpchar)))
+         Filter: ((a IS NOT NULL) AND (a = ANY ('{a,c}'::bpchar[])))
    ->  Seq Scan on lp_bc lp_2
-         Filter: ((a IS NOT NULL) AND ((a = 'a'::bpchar) OR (a = 'c'::bpchar)))
+         Filter: ((a IS NOT NULL) AND (a = ANY ('{a,c}'::bpchar[])))
 (5 rows)
 
+SET enable_or_transformation = on;
+explain (costs off) select * from lp where a = 'a' or a = 'c';
+                  QUERY PLAN                   
+-----------------------------------------------
+ Append
+   ->  Seq Scan on lp_ad lp_1
+         Filter: (a = ANY ('{a,c}'::bpchar[]))
+   ->  Seq Scan on lp_bc lp_2
+         Filter: (a = ANY ('{a,c}'::bpchar[]))
+(5 rows)
+
+explain (costs off) select * from lp where a is not null and (a = 'a' or a = 'c');
+                             QUERY PLAN                              
+---------------------------------------------------------------------
+ Append
+   ->  Seq Scan on lp_ad lp_1
+         Filter: ((a IS NOT NULL) AND (a = ANY ('{a,c}'::bpchar[])))
+   ->  Seq Scan on lp_bc lp_2
+         Filter: ((a IS NOT NULL) AND (a = ANY ('{a,c}'::bpchar[])))
+(5 rows)
+
+RESET enable_or_transformation;
 explain (costs off) select * from lp where a <> 'g';
              QUERY PLAN             
 ------------------------------------
@@ -515,10 +537,10 @@ explain (costs off) select * from rlp where a <= 31;
 (27 rows)
 
 explain (costs off) select * from rlp where a = 1 or a = 7;
-           QUERY PLAN           
---------------------------------
+                QUERY PLAN                
+------------------------------------------
  Seq Scan on rlp2 rlp
-   Filter: ((a = 1) OR (a = 7))
+   Filter: (a = ANY ('{1,7}'::integer[]))
 (2 rows)
 
 explain (costs off) select * from rlp where a = 1 or b = 'ab';
@@ -589,20 +611,20 @@ explain (costs off) select * from rlp where a < 1 or (a > 20 and a < 25);
 ------------------------------------------------------
  Append
    ->  Seq Scan on rlp1 rlp_1
-         Filter: ((a < 1) OR ((a > 20) AND (a < 25)))
+         Filter: (((a > 20) AND (a < 25)) OR (a < 1))
    ->  Seq Scan on rlp4_1 rlp_2
-         Filter: ((a < 1) OR ((a > 20) AND (a < 25)))
+         Filter: (((a > 20) AND (a < 25)) OR (a < 1))
 (5 rows)
 
 -- where clause contradicts sub-partition's constraint
 explain (costs off) select * from rlp where a = 20 or a = 40;
-               QUERY PLAN               
-----------------------------------------
+                    QUERY PLAN                    
+--------------------------------------------------
  Append
    ->  Seq Scan on rlp4_1 rlp_1
-         Filter: ((a = 20) OR (a = 40))
+         Filter: (a = ANY ('{20,40}'::integer[]))
    ->  Seq Scan on rlp5_default rlp_2
-         Filter: ((a = 20) OR (a = 40))
+         Filter: (a = ANY ('{20,40}'::integer[]))
 (5 rows)
 
 explain (costs off) select * from rlp3 where a = 20;   /* empty */
@@ -671,6 +693,163 @@ explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a =
          Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
 (11 rows)
 
+SET enable_or_transformation = on;
+explain (costs off) select * from rlp where a = 1 or a = 7;
+                QUERY PLAN                
+------------------------------------------
+ Seq Scan on rlp2 rlp
+   Filter: (a = ANY ('{1,7}'::integer[]))
+(2 rows)
+
+explain (costs off) select * from rlp where a = 1 or b = 'ab';
+                      QUERY PLAN                       
+-------------------------------------------------------
+ Append
+   ->  Seq Scan on rlp1 rlp_1
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp2 rlp_2
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp3abcd rlp_3
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp4_1 rlp_4
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp4_2 rlp_5
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp4_default rlp_6
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp5_1 rlp_7
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp5_default rlp_8
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp_default_10 rlp_9
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp_default_30 rlp_10
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp_default_null rlp_11
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+   ->  Seq Scan on rlp_default_default rlp_12
+         Filter: ((a = 1) OR ((b)::text = 'ab'::text))
+(25 rows)
+
+explain (costs off) select * from rlp where a > 20 and a < 27;
+               QUERY PLAN                
+-----------------------------------------
+ Append
+   ->  Seq Scan on rlp4_1 rlp_1
+         Filter: ((a > 20) AND (a < 27))
+   ->  Seq Scan on rlp4_2 rlp_2
+         Filter: ((a > 20) AND (a < 27))
+(5 rows)
+
+explain (costs off) select * from rlp where a = 29;
+          QUERY PLAN          
+------------------------------
+ Seq Scan on rlp4_default rlp
+   Filter: (a = 29)
+(2 rows)
+
+explain (costs off) select * from rlp where a >= 29;
+                 QUERY PLAN                  
+---------------------------------------------
+ Append
+   ->  Seq Scan on rlp4_default rlp_1
+         Filter: (a >= 29)
+   ->  Seq Scan on rlp5_1 rlp_2
+         Filter: (a >= 29)
+   ->  Seq Scan on rlp5_default rlp_3
+         Filter: (a >= 29)
+   ->  Seq Scan on rlp_default_30 rlp_4
+         Filter: (a >= 29)
+   ->  Seq Scan on rlp_default_default rlp_5
+         Filter: (a >= 29)
+(11 rows)
+
+explain (costs off) select * from rlp where a < 1 or (a > 20 and a < 25);
+                      QUERY PLAN                      
+------------------------------------------------------
+ Append
+   ->  Seq Scan on rlp1 rlp_1
+         Filter: (((a > 20) AND (a < 25)) OR (a < 1))
+   ->  Seq Scan on rlp4_1 rlp_2
+         Filter: (((a > 20) AND (a < 25)) OR (a < 1))
+(5 rows)
+
+explain (costs off) select * from rlp where a = 20 or a = 40;
+                    QUERY PLAN                    
+--------------------------------------------------
+ Append
+   ->  Seq Scan on rlp4_1 rlp_1
+         Filter: (a = ANY ('{20,40}'::integer[]))
+   ->  Seq Scan on rlp5_default rlp_2
+         Filter: (a = ANY ('{20,40}'::integer[]))
+(5 rows)
+
+explain (costs off) select * from rlp3 where a = 20;   /* empty */
+        QUERY PLAN        
+--------------------------
+ Result
+   One-Time Filter: false
+(2 rows)
+
+explain (costs off) select * from rlp where a > 1 and a = 10;	/* only default */
+            QUERY PLAN            
+----------------------------------
+ Seq Scan on rlp_default_10 rlp
+   Filter: ((a > 1) AND (a = 10))
+(2 rows)
+
+explain (costs off) select * from rlp where a > 1 and a >=15;	/* rlp3 onwards, including default */
+                  QUERY PLAN                  
+----------------------------------------------
+ Append
+   ->  Seq Scan on rlp3abcd rlp_1
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp3efgh rlp_2
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp3nullxy rlp_3
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp3_default rlp_4
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp4_1 rlp_5
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp4_2 rlp_6
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp4_default rlp_7
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp5_1 rlp_8
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp5_default rlp_9
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp_default_30 rlp_10
+         Filter: ((a > 1) AND (a >= 15))
+   ->  Seq Scan on rlp_default_default rlp_11
+         Filter: ((a > 1) AND (a >= 15))
+(23 rows)
+
+explain (costs off) select * from rlp where a = 1 and a = 3;	/* empty */
+        QUERY PLAN        
+--------------------------
+ Result
+   One-Time Filter: false
+(2 rows)
+
+explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a = 15);
+                            QUERY PLAN                             
+-------------------------------------------------------------------
+ Append
+   ->  Seq Scan on rlp2 rlp_1
+         Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
+   ->  Seq Scan on rlp3abcd rlp_2
+         Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
+   ->  Seq Scan on rlp3efgh rlp_3
+         Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
+   ->  Seq Scan on rlp3nullxy rlp_4
+         Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
+   ->  Seq Scan on rlp3_default rlp_5
+         Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
+(11 rows)
+
+RESET enable_or_transformation;
 -- multi-column keys
 create table mc3p (a int, b int, c int) partition by range (a, abs(b), c);
 create table mc3p_default partition of mc3p default;
@@ -1933,10 +2112,10 @@ explain (costs off) select * from hp where a = 1 and b = 'abcde';
 
 explain (costs off) select * from hp where a = 1 and b = 'abcde' and
   (c = 2 or c = 3);
-                              QUERY PLAN                              
-----------------------------------------------------------------------
+                                   QUERY PLAN                                   
+--------------------------------------------------------------------------------
  Seq Scan on hp2 hp
-   Filter: ((a = 1) AND (b = 'abcde'::text) AND ((c = 2) OR (c = 3)))
+   Filter: ((c = ANY ('{2,3}'::integer[])) AND (a = 1) AND (b = 'abcde'::text))
 (2 rows)
 
 drop table hp2;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index b7488d760e..b424ad3154 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2501,7 +2501,7 @@ pg_stats| SELECT n.nspname AS schemaname,
      JOIN pg_class c ON ((c.oid = s.starelid)))
      JOIN pg_attribute a ON (((c.oid = a.attrelid) AND (a.attnum = s.staattnum))))
      LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
-  WHERE ((NOT a.attisdropped) AND has_column_privilege(c.oid, a.attnum, 'select'::text) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid))));
+  WHERE ((NOT a.attisdropped) AND has_column_privilege(c.oid, a.attnum, 'select'::text) AND ((NOT row_security_active(c.oid)) OR (c.relrowsecurity = false)));
 pg_stats_ext| SELECT cn.nspname AS schemaname,
     c.relname AS tablename,
     sn.nspname AS statistics_schemaname,
@@ -2532,7 +2532,7 @@ pg_stats_ext| SELECT cn.nspname AS schemaname,
   WHERE ((NOT (EXISTS ( SELECT 1
            FROM (unnest(s.stxkeys) k(k)
              JOIN pg_attribute a ON (((a.attrelid = s.stxrelid) AND (a.attnum = k.k))))
-          WHERE (NOT has_column_privilege(c.oid, a.attnum, 'select'::text))))) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid))));
+          WHERE (NOT has_column_privilege(c.oid, a.attnum, 'select'::text))))) AND ((NOT row_security_active(c.oid)) OR (c.relrowsecurity = false)));
 pg_stats_ext_exprs| SELECT cn.nspname AS schemaname,
     c.relname AS tablename,
     sn.nspname AS statistics_schemaname,
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 10903bdab0..6f55b9e3ec 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1322,19 +1322,19 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE
 SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE (a = 1 OR a = 51) AND b = ''1''');
  estimated | actual 
 -----------+--------
-        99 |    100
+       100 |    100
 (1 row)
 
 SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE (a = 1 OR a = 51) AND (b = ''1'' OR b = ''2'')');
  estimated | actual 
 -----------+--------
-        99 |    100
+       100 |    100
 (1 row)
 
 SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE (a = 1 OR a = 2 OR a = 51 OR a = 52) AND (b = ''1'' OR b = ''2'')');
  estimated | actual 
 -----------+--------
-       197 |    200
+       200 |    200
 (1 row)
 
 -- OR clauses referencing different attributes are incompatible
@@ -1664,19 +1664,19 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE
 SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE ((a * 2) = 2 OR (a * 2) = 102) AND upper(b) = ''1''');
  estimated | actual 
 -----------+--------
-        99 |    100
+       100 |    100
 (1 row)
 
 SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE ((a * 2) = 2 OR (a * 2) = 102) AND (upper(b) = ''1'' OR upper(b) = ''2'')');
  estimated | actual 
 -----------+--------
-        99 |    100
+       100 |    100
 (1 row)
 
 SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE ((a * 2) = 2 OR (a * 2) = 4 OR (a * 2) = 102 OR (a * 2) = 104) AND (upper(b) = ''1'' OR upper(b) = ''2'')');
  estimated | actual 
 -----------+--------
-       197 |    200
+       200 |    200
 (1 row)
 
 -- OR clauses referencing different attributes
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 9be7aca2b8..1f9029b5b2 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -124,6 +124,7 @@ select name, setting from pg_settings where name like 'enable%';
  enable_memoize                 | on
  enable_mergejoin               | on
  enable_nestloop                | on
+ enable_or_transformation       | on
  enable_parallel_append         | on
  enable_parallel_hash           | on
  enable_partition_pruning       | on
@@ -134,7 +135,7 @@ select name, setting from pg_settings where name like 'enable%';
  enable_seqscan                 | on
  enable_sort                    | on
  enable_tidscan                 | on
-(23 rows)
+(24 rows)
 
 -- There are always wait event descriptions for various types.
 select type, count(*) > 0 as ok FROM pg_wait_events
diff --git a/src/test/regress/expected/tidscan.out b/src/test/regress/expected/tidscan.out
index f133b5a4ac..2a079e996b 100644
--- a/src/test/regress/expected/tidscan.out
+++ b/src/test/regress/expected/tidscan.out
@@ -43,10 +43,26 @@ SELECT ctid, * FROM tidscan WHERE '(0,1)' = ctid;
 -- OR'd clauses
 EXPLAIN (COSTS OFF)
 SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
-                          QUERY PLAN                          
---------------------------------------------------------------
+                      QUERY PLAN                       
+-------------------------------------------------------
+ Tid Scan on tidscan
+   TID Cond: (ctid = ANY ('{"(0,2)","(0,1)"}'::tid[]))
+(2 rows)
+
+SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
+ ctid  | id 
+-------+----
+ (0,1) |  1
+ (0,2) |  2
+(2 rows)
+
+SET enable_or_transformation = on;
+EXPLAIN (COSTS OFF)
+SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
+                      QUERY PLAN                       
+-------------------------------------------------------
  Tid Scan on tidscan
-   TID Cond: ((ctid = '(0,2)'::tid) OR ('(0,1)'::tid = ctid))
+   TID Cond: (ctid = ANY ('{"(0,2)","(0,1)"}'::tid[]))
 (2 rows)
 
 SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
@@ -56,6 +72,7 @@ SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
  (0,2) |  2
 (2 rows)
 
+RESET enable_or_transformation;
 -- ctid = ScalarArrayOp - implemented as tidscan
 EXPLAIN (COSTS OFF)
 SELECT ctid, * FROM tidscan WHERE ctid = ANY(ARRAY['(0,1)', '(0,2)']::tid[]);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index d49ce9f300..56fde15bc1 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -737,6 +737,41 @@ SELECT count(*) FROM tenk1
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
 
+SET enable_or_transformation = on;
+EXPLAIN (COSTS OFF)
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand < 42 OR thousand < 99 OR 43 > thousand OR 42 > thousand);
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
+RESET enable_or_transformation;
+
 --
 -- Check behavior with duplicate index column contents
 --
diff --git a/src/test/regress/sql/join.sql b/src/test/regress/sql/join.sql
index 257f727a2b..0d1427c115 100644
--- a/src/test/regress/sql/join.sql
+++ b/src/test/regress/sql/join.sql
@@ -1408,6 +1408,16 @@ explain (costs off)
 select * from tenk1 a join tenk1 b on
   (a.unique1 = 1 and b.unique1 = 2) or
   ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+SET enable_or_transformation = on;
+explain (costs off)
+select * from tenk1 a join tenk1 b on
+  (a.unique1 = 1 and b.unique1 = 2) or
+  ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+explain (costs off)
+select * from tenk1 a join tenk1 b on
+  (a.unique1 < 20 or a.unique1 = 3 or a.unique1 = 1 and b.unique1 = 2) or
+  ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+RESET enable_or_transformation;
 
 --
 -- test placement of movable quals in a parameterized join tree
diff --git a/src/test/regress/sql/partition_prune.sql b/src/test/regress/sql/partition_prune.sql
index 7bf3920827..1e270ae9c0 100644
--- a/src/test/regress/sql/partition_prune.sql
+++ b/src/test/regress/sql/partition_prune.sql
@@ -21,6 +21,12 @@ explain (costs off) select * from lp where a is not null;
 explain (costs off) select * from lp where a is null;
 explain (costs off) select * from lp where a = 'a' or a = 'c';
 explain (costs off) select * from lp where a is not null and (a = 'a' or a = 'c');
+
+SET enable_or_transformation = on;
+explain (costs off) select * from lp where a = 'a' or a = 'c';
+explain (costs off) select * from lp where a is not null and (a = 'a' or a = 'c');
+RESET enable_or_transformation;
+
 explain (costs off) select * from lp where a <> 'g';
 explain (costs off) select * from lp where a <> 'a' and a <> 'd';
 explain (costs off) select * from lp where a not in ('a', 'd');
@@ -99,6 +105,22 @@ explain (costs off) select * from rlp where a > 1 and a >=15;	/* rlp3 onwards, i
 explain (costs off) select * from rlp where a = 1 and a = 3;	/* empty */
 explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a = 15);
 
+
+SET enable_or_transformation = on;
+explain (costs off) select * from rlp where a = 1 or a = 7;
+explain (costs off) select * from rlp where a = 1 or b = 'ab';
+explain (costs off) select * from rlp where a > 20 and a < 27;
+explain (costs off) select * from rlp where a = 29;
+explain (costs off) select * from rlp where a >= 29;
+explain (costs off) select * from rlp where a < 1 or (a > 20 and a < 25);
+explain (costs off) select * from rlp where a = 20 or a = 40;
+explain (costs off) select * from rlp3 where a = 20;   /* empty */
+explain (costs off) select * from rlp where a > 1 and a = 10;	/* only default */
+explain (costs off) select * from rlp where a > 1 and a >=15;	/* rlp3 onwards, including default */
+explain (costs off) select * from rlp where a = 1 and a = 3;	/* empty */
+explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a = 15);
+RESET enable_or_transformation;
+
 -- multi-column keys
 create table mc3p (a int, b int, c int) partition by range (a, abs(b), c);
 create table mc3p_default partition of mc3p default;
diff --git a/src/test/regress/sql/tidscan.sql b/src/test/regress/sql/tidscan.sql
index 313e0fb9b6..0499bedb9e 100644
--- a/src/test/regress/sql/tidscan.sql
+++ b/src/test/regress/sql/tidscan.sql
@@ -22,6 +22,12 @@ EXPLAIN (COSTS OFF)
 SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
 SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
 
+SET enable_or_transformation = on;
+EXPLAIN (COSTS OFF)
+SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
+SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
+RESET enable_or_transformation;
+
 -- ctid = ScalarArrayOp - implemented as tidscan
 EXPLAIN (COSTS OFF)
 SELECT ctid, * FROM tidscan WHERE ctid = ANY(ARRAY['(0,1)', '(0,2)']::tid[]);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d808aad8b0..f8d5871153 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1654,6 +1654,8 @@ NumericVar
 OM_uint32
 OP
 OSAPerGroupState
+OrClauseGroupEntry
+OrClauseGroupKey
 OSAPerQueryState
 OSInfo
 OSSLCipher
-- 
2.43.0

From 82a14d2adbe9638abf13e4518b0836141ff46452 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Wed, 24 Jan 2024 14:07:17 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

Likewise OR clauses, discover SAOP array and try to split its elements
between smaller sized arrays to fit a set of partial indexes.
---
 src/backend/optimizer/path/indxpath.c     | 301 ++++++++++++++++++----
 src/backend/optimizer/util/predtest.c     |  46 ++++
 src/backend/optimizer/util/restrictinfo.c |  13 +
 src/include/optimizer/optimizer.h         |  16 ++
 src/include/optimizer/restrictinfo.h      |   1 +
 src/test/regress/expected/select.out      | 282 ++++++++++++++++++++
 src/test/regress/sql/select.sql           |  82 ++++++
 src/tools/pgindent/typedefs.list          |   1 +
 8 files changed, 689 insertions(+), 53 deletions(-)

diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 32c6a8bbdc..56b04541db 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -32,6 +32,7 @@
 #include "optimizer/paths.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "utils/array.h"
 #include "utils/lsyscache.h"
 #include "utils/selfuncs.h"
 
@@ -1220,11 +1221,174 @@ build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel,
 	return result;
 }
 
+/*
+ * Building index paths over SAOP clause differs from the logic of OR clauses.
+ * Here we iterate across all the array elements and split them to SAOPs,
+ * corresponding to different indexes. We must match each element to an index.
+ */
+static List *
+build_paths_for_SAOP(PlannerInfo *root, RelOptInfo *rel, RestrictInfo *rinfo,
+					 List *other_clauses)
+{
+	List			   *result = NIL;
+	List			   *predicate_lists = NIL;
+	ListCell		   *lc;
+	PredicatesData	   *pd;
+	ScalarArrayOpExpr  *saop = (ScalarArrayOpExpr *) rinfo->clause;
+
+	Assert(IsA(saop, ScalarArrayOpExpr) && saop->useOr);
+
+	if (!IsA(lsecond(saop->args), Const))
+		/*
+		 * Has it practical outcome to merge arrays which couldn't constantified
+		 * before that step?
+		 */
+		return NIL;
+
+	/* Collect predicates */
+	foreach(lc, rel->indexlist)
+	{
+		IndexOptInfo *index = (IndexOptInfo *) lfirst(lc);
+
+		/* Take into consideration partial indexes supporting bitmap scans */
+		if (!index->amhasgetbitmap || index->indpred == NIL || index->predOK)
+			continue;
+
+		pd = palloc0(sizeof(PredicatesData));
+		pd->id = foreach_current_index(lc);
+		/* The trick with reducing recursion is stolen from predicate_implied_by */
+		pd->predicate = list_length(index->indpred) == 1 ?
+										(Node *) linitial(index->indpred) :
+										(Node *) index->indpred;
+		predicate_lists = lappend(predicate_lists, (void *) pd);
+	}
+
+	/* Split the array data according to index predicates. */
+	if (predicate_lists == NIL ||
+		!saop_covered_by_predicates(saop, predicate_lists))
+		return NIL;
+
+	other_clauses = list_delete_ptr(other_clauses, rinfo);
+
+	/*
+	 * Having incoming SAOP split to set of smaller SAOPs which can be applied
+	 * to partial indexes, generate paths for each one.
+	 */
+	foreach(lc, predicate_lists)
+	{
+		IndexOptInfo	   *index;
+		IndexClauseSet		clauseset;
+		List			   *indexpaths;
+		RestrictInfo	   *rinfo1 = NULL;
+		Expr			   *clause;
+		ArrayType		   *arrayval = NULL;
+		ArrayExpr		   *arr = NULL;
+		Const			   *arrayconst = lsecond_node(Const, saop->args);
+		ScalarArrayOpExpr  *dest = makeNode(ScalarArrayOpExpr);
+
+		pd = (PredicatesData *) lfirst(lc);
+		if (pd->elems == NIL)
+			/* The index doesn't participate in this operation */
+			continue;
+
+		/* Make up new array */
+		arrayval = DatumGetArrayTypeP(arrayconst->constvalue);
+		arr = makeNode(ArrayExpr);
+		arr->array_collid = arrayconst->constcollid;
+		arr->array_typeid = arrayconst->consttype;
+		arr->element_typeid = arrayval->elemtype;
+		arr->elements = pd->elems;
+		arr->location = -1;
+		arr->multidims = false;
+
+		/* Compose new SAOP, partially covering the source one */
+		memcpy(dest, saop, sizeof(ScalarArrayOpExpr));
+		dest->args = list_make2(linitial(saop->args), arr);
+
+		clause = (Expr *) estimate_expression_value(root, (Node *) dest);
+
+		/*
+		 * Create new RestrictInfo. It maybe more heavy than just copy node,
+		 * but remember some internals: the serial number, selectivity
+		 * cache etc.
+		 */
+		rinfo1 = make_restrictinfo(root, clause,
+								   rinfo->is_pushed_down,
+								   rinfo->has_clone,
+								   rinfo->is_clone,
+								   rinfo->pseudoconstant,
+								   rinfo->security_level,
+								   rinfo->required_relids,
+								   rinfo->incompatible_relids,
+								   rinfo->outer_relids);
+
+		index = list_nth(rel->indexlist, pd->id);
+		Assert(predicate_implied_by(index->indpred, list_make1(rinfo1), true));
+
+		/* Excluding partial indexes with predOK we make this statement false */
+		Assert(!predicate_implied_by(index->indpred, other_clauses, false));
+
+		/* Time to generate index paths */
+
+		MemSet(&clauseset, 0, sizeof(clauseset));
+		match_clauses_to_index(root, list_make1(rinfo1), index, &clauseset);
+		match_clauses_to_index(root, other_clauses, index, &clauseset);
+
+		/* Predicate has found already. So, it is ok for the empty match */
+
+		indexpaths = build_index_paths(root, rel,
+									   index, &clauseset,
+									   true,
+									   ST_BITMAPSCAN,
+									   NULL,
+									   NULL);
+		Assert(indexpaths != NIL);
+		result = lappend(result, indexpaths);
+	}
+	return result;
+}
+
+static List *
+generate_saop_pathlist(PlannerInfo *root, RelOptInfo *rel,
+					   RestrictInfo *rinfo, List *all_clauses)
+{
+	List	   *pathlist = NIL;
+	Path	   *bitmapqual;
+	List	   *indlist;
+	ListCell   *lc;
+
+	if (!enable_or_transformation)
+		return NIL;
+
+	/*
+	 * We must be able to match at least one index to each element of
+	 * the array, else we can't use it.
+	 */
+	indlist = build_paths_for_SAOP(root, rel, rinfo, all_clauses);
+	if (indlist == NIL)
+		return NIL;
+
+	/*
+	 * OK, pick the most promising AND combination, and add it to
+	 * pathlist.
+	 */
+	foreach (lc, indlist)
+	{
+		List *plist = lfirst_node(List, lc);
+
+		bitmapqual = choose_bitmap_and(root, rel, plist);
+		pathlist = lappend(pathlist, bitmapqual);
+	}
+
+	return pathlist;
+}
+
 /*
  * generate_bitmap_or_paths
- *		Look through the list of clauses to find OR clauses, and generate
- *		a BitmapOrPath for each one we can handle that way.  Return a list
- *		of the generated BitmapOrPaths.
+ *		Look through the list of clauses to find OR and SAOP clauses, and
+ *		Each saop clause are splitted to be covered by partial indexes.
+ *		generate a BitmapOrPath for each one we can handle that way.
+ *		Return a list of the generated BitmapOrPaths.
  *
  * other_clauses is a list of additional clauses that can be assumed true
  * for the purpose of generating indexquals, but are not to be searched for
@@ -1247,68 +1411,99 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel,
 	foreach(lc, clauses)
 	{
 		RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc);
-		List	   *pathlist;
+		List	   *pathlist = NIL;
 		Path	   *bitmapqual;
 		ListCell   *j;
 
-		/* Ignore RestrictInfos that aren't ORs */
-		if (!restriction_is_or_clause(rinfo))
+		if (restriction_is_saop_clause(rinfo))
+		{
+			pathlist = generate_saop_pathlist(root, rel, rinfo,
+											  all_clauses);
+		}
+		else if (!restriction_is_or_clause(rinfo))
+			/* Ignore RestrictInfos that aren't ORs */
 			continue;
-
-		/*
-		 * We must be able to match at least one index to each of the arms of
-		 * the OR, else we can't use it.
-		 */
-		pathlist = NIL;
-		foreach(j, ((BoolExpr *) rinfo->orclause)->args)
+		else
 		{
-			Node	   *orarg = (Node *) lfirst(j);
-			List	   *indlist;
-
-			/* OR arguments should be ANDs or sub-RestrictInfos */
-			if (is_andclause(orarg))
+			/*
+			 * We must be able to match at least one index to each of the arms of
+			 * the OR, else we can't use it.
+			 */
+			foreach(j, ((BoolExpr *) rinfo->orclause)->args)
 			{
-				List	   *andargs = ((BoolExpr *) orarg)->args;
+				Node	   *orarg = (Node *) lfirst(j);
+				List	   *indlist;
 
-				indlist = build_paths_for_OR(root, rel,
-											 andargs,
-											 all_clauses);
+				/* OR arguments should be ANDs or sub-RestrictInfos */
+				if (is_andclause(orarg))
+				{
+					List	   *andargs = ((BoolExpr *) orarg)->args;
 
-				/* Recurse in case there are sub-ORs */
-				indlist = list_concat(indlist,
-									  generate_bitmap_or_paths(root, rel,
-															   andargs,
-															   all_clauses));
-			}
-			else
-			{
-				RestrictInfo *ri = castNode(RestrictInfo, orarg);
-				List	   *orargs;
+					indlist = build_paths_for_OR(root, rel,
+												 andargs,
+												 all_clauses);
 
-				Assert(!restriction_is_or_clause(ri));
-				orargs = list_make1(ri);
+					/* Recurse in case there are sub-ORs */
+					indlist = list_concat(indlist,
+										  generate_bitmap_or_paths(root, rel,
+																   andargs,
+																   all_clauses));
+				}
+				else
+				{
+					RestrictInfo *ri = castNode(RestrictInfo, orarg);
+					List	   *orargs;
 
-				indlist = build_paths_for_OR(root, rel,
-											 orargs,
-											 all_clauses);
-			}
+					Assert(!restriction_is_or_clause(ri));
 
-			/*
-			 * If nothing matched this arm, we can't do anything with this OR
-			 * clause.
-			 */
-			if (indlist == NIL)
-			{
-				pathlist = NIL;
-				break;
-			}
+					orargs = list_make1(ri);
 
-			/*
-			 * OK, pick the most promising AND combination, and add it to
-			 * pathlist.
-			 */
-			bitmapqual = choose_bitmap_and(root, rel, indlist);
-			pathlist = lappend(pathlist, bitmapqual);
+					if (restriction_is_saop_clause(ri))
+					{
+						List *paths;
+
+						paths = generate_saop_pathlist(root, rel, ri,
+														 all_clauses);
+
+						if (paths != NIL)
+						{
+							/*
+							 * Add paths to pathlist and immediately jump to the
+							 * next element of the OR clause.
+							 */
+							pathlist = list_concat(pathlist, paths);
+							continue;
+						}
+
+						/*
+						 * Pass down out of this if construction:
+						 * If saop isn't covered by partial indexes, try to
+						 * build scan path for the saop as a whole.
+						 */
+					}
+
+					indlist = build_paths_for_OR(root, rel,
+												 orargs,
+												 all_clauses);
+				}
+
+				/*
+				 * If nothing matched this arm, we can't do anything with this OR
+				 * clause.
+				 */
+				if (indlist == NIL)
+				{
+					pathlist = NIL;
+					break;
+				}
+
+				/*
+				 * OK, pick the most promising AND combination, and add it to
+				 * pathlist.
+				 */
+				bitmapqual = choose_bitmap_and(root, rel, indlist);
+				pathlist = lappend(pathlist, bitmapqual);
+			}
 		}
 
 		/*
diff --git a/src/backend/optimizer/util/predtest.c b/src/backend/optimizer/util/predtest.c
index c37b416e24..8ed80a78b4 100644
--- a/src/backend/optimizer/util/predtest.c
+++ b/src/backend/optimizer/util/predtest.c
@@ -112,6 +112,52 @@ static Oid	get_btree_test_op(Oid pred_op, Oid clause_op, bool refute_it);
 static void InvalidateOprProofCacheCallBack(Datum arg, int cacheid, uint32 hashvalue);

+/*
+ * Could this ANY () expression can be split into a set of ANYs over partial
+ * indexes? If yes, return these saops in the PredicatesData structure.
+ */
+bool
+saop_covered_by_predicates(ScalarArrayOpExpr *saop, List *predicate_lists)
+{
+	ListCell		   *lc;
+	PredIterInfoData	clause_info;
+	bool				result = false;
+
+	if (predicate_classify((Node *) saop, &clause_info) != CLASS_OR)
+		return false;
+
+	iterate_begin(pitem, (Node *) saop, clause_info)
+	{
+		result = false;
+
+		foreach(lc, predicate_lists)
+		{
+			PredicatesData *pd = (PredicatesData *) lfirst(lc);
+
+			if (!predicate_implied_by_recurse(pitem, pd->predicate, false))
+				continue;
+
+			/* Predicate is found. Add the elem to the saop clause */
+			Assert(IsA(pitem, OpExpr));
+
+			/* Extract constant from the expression */
+			pd->elems = lappend(pd->elems,
+					copyObject(lsecond_node(Const, ((OpExpr *) pitem)->args)));
+			result = true;
+			break;
+		}
+
+		if (!result)
+			/*
+			 * The element doesn't fit any index. Interrupt the process immediately
+			 */
+			break;
+	}
+	iterate_end(clause_info);
+
+	return result;
+}
+
 /*
  * predicate_implied_by
  *	  Recursively checks whether the clauses in clause_list imply that the
diff --git a/src/backend/optimizer/util/restrictinfo.c b/src/backend/optimizer/util/restrictinfo.c
index 0b406e9334..1dad1dc654 100644
--- a/src/backend/optimizer/util/restrictinfo.c
+++ b/src/backend/optimizer/util/restrictinfo.c
@@ -421,6 +421,19 @@ restriction_is_or_clause(RestrictInfo *restrictinfo)
 		return false;
 }
 
+bool
+restriction_is_saop_clause(RestrictInfo *restrictinfo)
+{
+	if (restrictinfo->clause && IsA(restrictinfo->clause, ScalarArrayOpExpr))
+	{
+		ScalarArrayOpExpr *saop = (ScalarArrayOpExpr *) restrictinfo->clause;
+
+		if (saop->useOr)
+			return true;
+	}
+	return false;
+}
+
 /*
  * restriction_is_securely_promotable
  *
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
index 35ab577501..232afcd00c 100644
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -160,6 +160,22 @@ extern List *expand_function_arguments(List *args, bool include_out_arguments,
 
 /* in util/predtest.c: */
 
+/*
+ * Contains information needed to extract from saop a set of elements which can
+ * be covered by the partial index:
+ * id - caller's identification of the index.
+ * predicate - predicate expression of the index
+ * elems - returning list of array elements which corresponds to this predicate
+ */
+typedef struct PredicatesData
+{
+	int		id;
+	Node   *predicate;
+	List   *elems;
+} PredicatesData;
+
+extern bool saop_covered_by_predicates(ScalarArrayOpExpr *saop,
+									  List *predicate_lists);
 extern bool predicate_implied_by(List *predicate_list, List *clause_list,
 								 bool weak);
 extern bool predicate_refuted_by(List *predicate_list, List *clause_list,
diff --git a/src/include/optimizer/restrictinfo.h b/src/include/optimizer/restrictinfo.h
index 1b42c832c5..2cd5fbf943 100644
--- a/src/include/optimizer/restrictinfo.h
+++ b/src/include/optimizer/restrictinfo.h
@@ -34,6 +34,7 @@ extern RestrictInfo *make_restrictinfo(PlannerInfo *root,
 									   Relids outer_relids);
 extern RestrictInfo *commute_restrictinfo(RestrictInfo *rinfo, Oid comm_op);
 extern bool restriction_is_or_clause(RestrictInfo *restrictinfo);
+extern bool restriction_is_saop_clause(RestrictInfo *restrictinfo);
 extern bool restriction_is_securely_promotable(RestrictInfo *restrictinfo,
 											   RelOptInfo *rel);
 extern List *get_actual_clauses(List *restrictinfo_list);
diff --git a/src/test/regress/expected/select.out b/src/test/regress/expected/select.out
index 33a6dceb0e..070202b0f0 100644
--- a/src/test/regress/expected/select.out
+++ b/src/test/regress/expected/select.out
@@ -907,6 +907,288 @@ select unique1, unique2 from onek2
        0 |     998
 (2 rows)
 
+SET enable_seqscan TO off;
+SET enable_indexscan TO off; -- Only BitmapScan is a subject matter here
+SET enable_or_transformation = 'off';
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 = 'A' OR stringu1 = 'J';
+                                QUERY PLAN                                
+--------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 < 'B'::name) OR (stringu1 = 'J'::name))
+         Filter: ((stringu1 = 'A'::name) OR (stringu1 = 'J'::name))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_u2_prtl
+               ->  Bitmap Index Scan on onek2_stu1_prtl
+                     Index Cond: (stringu1 = 'J'::name)
+(8 rows)
+
+-- Without the transformation only seqscan possible here
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique2 < 1 AND stringu1 IN ('A','J') AND stringu1 < 'Z';
+                                         QUERY PLAN                                          
+---------------------------------------------------------------------------------------------
+ Seq Scan on onek2
+   Filter: ((unique2 < 1) AND (stringu1 = ANY ('{A,J}'::name[])) AND (stringu1 < 'Z'::name))
+(2 rows)
+
+-- Use partial indexes
+explain (costs off)
+SELECT count(*) FROM onek2
+WHERE stringu1 IN ('B','J') AND (stringu1 = 'A' OR unique1 = 1);
+                                             QUERY PLAN                                             
+----------------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 < 'B'::name) OR (unique1 = 1))
+         Filter: ((stringu1 = ANY ('{B,J}'::name[])) AND ((stringu1 = 'A'::name) OR (unique1 = 1)))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_u2_prtl
+               ->  Bitmap Index Scan on onek2_u1_prtl
+                     Index Cond: (unique1 = 1)
+(8 rows)
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 < 1 OR (stringu1 = 'A' OR stringu1 = 'J');
+                                     QUERY PLAN                                      
+-------------------------------------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: ((unique1 < 1) OR (stringu1 < 'B'::name) OR (stringu1 = 'J'::name))
+   Filter: ((unique1 < 1) OR (stringu1 = 'A'::name) OR (stringu1 = 'J'::name))
+   ->  BitmapOr
+         ->  Bitmap Index Scan on onek2_u1_prtl
+               Index Cond: (unique1 < 1)
+         ->  Bitmap Index Scan on onek2_u2_prtl
+         ->  Bitmap Index Scan on onek2_stu1_prtl
+               Index Cond: (stringu1 = 'J'::name)
+(9 rows)
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 = 1 OR unique2 = PI()::integer;
+                 QUERY PLAN                 
+--------------------------------------------
+ Seq Scan on onek2
+   Filter: ((unique1 = 1) OR (unique2 = 3))
+(2 rows)
+
+RESET enable_or_transformation;
+-- OR <-> ANY transformation must find a path with partial indexes scan
+-- regardless the clause representation.
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 = 'A' OR stringu1 = 'J';
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name))
+         Filter: (stringu1 = ANY ('{A,J}'::name[]))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_stu1_prtl
+                     Index Cond: (stringu1 = ANY ('{J}'::name[]))
+               ->  Bitmap Index Scan on onek2_u2_prtl
+(8 rows)
+
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 IN ('A','J');
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name))
+         Filter: (stringu1 = ANY ('{A,J}'::name[]))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_stu1_prtl
+                     Index Cond: (stringu1 = ANY ('{J}'::name[]))
+               ->  Bitmap Index Scan on onek2_u2_prtl
+(8 rows)
+
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 = ANY ('{A,J}');
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name))
+         Filter: (stringu1 = ANY ('{A,J}'::name[]))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_stu1_prtl
+                     Index Cond: (stringu1 = ANY ('{J}'::name[]))
+               ->  Bitmap Index Scan on onek2_u2_prtl
+(8 rows)
+
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 IN ('A') OR stringu1 IN ('J');
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name))
+         Filter: (stringu1 = ANY ('{A,J}'::name[]))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_stu1_prtl
+                     Index Cond: (stringu1 = ANY ('{J}'::name[]))
+               ->  Bitmap Index Scan on onek2_u2_prtl
+(8 rows)
+
+-- Don't scan partial indexes because of extra value.
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 IN ('A', 'J', 'C');
+                      QUERY PLAN                      
+------------------------------------------------------
+ Aggregate
+   ->  Seq Scan on onek2
+         Filter: (stringu1 = ANY ('{A,J,C}'::name[]))
+(3 rows)
+
+explain (costs off)
+SELECT unique2 FROM onek2
+WHERE stringu1 IN ('A', 'A') AND (stringu1 = 'A' OR stringu1 = 'A');
+                                      QUERY PLAN                                       
+---------------------------------------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: (stringu1 < 'B'::name)
+   Filter: ((stringu1 = ANY ('{A,A}'::name[])) AND (stringu1 = ANY ('{A,A}'::name[])))
+   ->  Bitmap Index Scan on onek2_u2_prtl
+(4 rows)
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique2 < 1 AND stringu1 IN ('A','J') AND stringu1 < 'Z';
+                                                          QUERY PLAN                                                           
+-------------------------------------------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: (((stringu1 = ANY ('{J}'::name[])) AND (stringu1 < 'Z'::name)) OR ((unique2 < 1) AND (stringu1 < 'B'::name)))
+   Filter: ((unique2 < 1) AND (stringu1 = ANY ('{A,J}'::name[])))
+   ->  BitmapOr
+         ->  Bitmap Index Scan on onek2_stu1_prtl
+               Index Cond: ((stringu1 = ANY ('{J}'::name[])) AND (stringu1 < 'Z'::name))
+         ->  Bitmap Index Scan on onek2_u2_prtl
+               Index Cond: (unique2 < 1)
+(8 rows)
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 = 1 OR unique1 = PI()::integer;
+                    QUERY PLAN                    
+--------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: ((unique1 = 3) OR (unique1 = 1))
+   ->  BitmapOr
+         ->  Bitmap Index Scan on onek2_u1_prtl
+               Index Cond: (unique1 = 3)
+         ->  Bitmap Index Scan on onek2_u1_prtl
+               Index Cond: (unique1 = 1)
+(7 rows)
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 IN (1, PI()::integer); -- TODO: why it is differs from previous example?
+                        QUERY PLAN                        
+----------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: (unique1 = ANY ('{1,3}'::integer[]))
+   ->  Bitmap Index Scan on onek2_u1_prtl
+         Index Cond: (unique1 = ANY ('{1,3}'::integer[]))
+(4 rows)
+
+-- Don't apply the optimization to clauses, containing volatile functions
+EXPLAIN (COSTS OFF)
+SELECT unique2,stringu1 FROM onek2
+WHERE unique1 = (random()*2)::integer OR unique1 = (random()*3)::integer;
+                                                             QUERY PLAN                                                             
+------------------------------------------------------------------------------------------------------------------------------------
+ Seq Scan on onek2
+   Filter: ((unique1 = ((random() * '2'::double precision))::integer) OR (unique1 = ((random() * '3'::double precision))::integer))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT unique2,stringu1 FROM onek2
+WHERE unique1 IN ((random()*2)::integer, (random()*3)::integer);
+                                                           QUERY PLAN                                                            
+---------------------------------------------------------------------------------------------------------------------------------
+ Seq Scan on onek2
+   Filter: (unique1 = ANY (ARRAY[((random() * '2'::double precision))::integer, ((random() * '3'::double precision))::integer]))
+(2 rows)
+
+-- Combine different saops. Soe of them doesnt' fit a set of partial indexes,
+-- but other fits.
+-- Unfortunately, we don't combine saop and OR clauses so far.
+EXPLAIN (COSTS OFF)
+SELECT unique2,stringu1 FROM onek2
+WHERE
+  unique1 IN (1,2,21) AND
+  (stringu1 IN ('A','J') OR unique1 IN (3,4) OR stringu1 = 'J');
+                                                                                          QUERY PLAN                                                                                          
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name) OR ((unique1 = ANY ('{3,4}'::integer[])) AND (unique1 = ANY ('{1,2,21}'::integer[]))) OR (stringu1 = 'J'::name))
+   Filter: ((unique1 = ANY ('{1,2,21}'::integer[])) AND ((stringu1 = ANY ('{A,J}'::name[])) OR (unique1 = ANY ('{3,4}'::integer[])) OR (stringu1 = 'J'::name)))
+   ->  BitmapOr
+         ->  Bitmap Index Scan on onek2_stu1_prtl
+               Index Cond: (stringu1 = ANY ('{J}'::name[]))
+         ->  Bitmap Index Scan on onek2_u2_prtl
+         ->  Bitmap Index Scan on onek2_u1_prtl
+               Index Cond: ((unique1 = ANY ('{3,4}'::integer[])) AND (unique1 = ANY ('{1,2,21}'::integer[])))
+         ->  Bitmap Index Scan on onek2_stu1_prtl
+               Index Cond: (stringu1 = 'J'::name)
+(11 rows)
+
+-- Check recursive combination of OR and SAOP expressions
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 < 1 OR (stringu1 = 'A' OR stringu1 = 'J');
+                                          QUERY PLAN                                           
+-----------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name) OR (unique1 < 1))
+   Filter: ((stringu1 = ANY ('{A,J}'::name[])) OR (unique1 < 1))
+   ->  BitmapOr
+         ->  Bitmap Index Scan on onek2_stu1_prtl
+               Index Cond: (stringu1 = ANY ('{J}'::name[]))
+         ->  Bitmap Index Scan on onek2_u2_prtl
+         ->  Bitmap Index Scan on onek2_u1_prtl
+               Index Cond: (unique1 < 1)
+(9 rows)
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE (unique1 < 1 OR stringu1 IN ('A','J'));
+                                          QUERY PLAN                                           
+-----------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on onek2
+   Recheck Cond: ((stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name) OR (unique1 < 1))
+   Filter: ((stringu1 = ANY ('{A,J}'::name[])) OR (unique1 < 1))
+   ->  BitmapOr
+         ->  Bitmap Index Scan on onek2_stu1_prtl
+               Index Cond: (stringu1 = ANY ('{J}'::name[]))
+         ->  Bitmap Index Scan on onek2_u2_prtl
+         ->  Bitmap Index Scan on onek2_u1_prtl
+               Index Cond: (unique1 < 1)
+(9 rows)
+
+-- Although SAOP doesn't fit partial indexes fully, we can use anded OR clause
+-- to scan another couple of partial indexes.
+explain (costs off)
+SELECT count(*) FROM onek2
+WHERE stringu1 IN ('B','J') AND (stringu1 = 'A' OR unique1 = 1);
+                                             QUERY PLAN                                             
+----------------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on onek2
+         Recheck Cond: ((stringu1 < 'B'::name) OR (unique1 = 1))
+         Filter: ((stringu1 = ANY ('{B,J}'::name[])) AND ((stringu1 = 'A'::name) OR (unique1 = 1)))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on onek2_u2_prtl
+               ->  Bitmap Index Scan on onek2_u1_prtl
+                     Index Cond: (unique1 = 1)
+(8 rows)
+
+RESET enable_indexscan;
+RESET enable_seqscan;
 --
 -- Test some corner cases that have been known to confuse the planner
 --
diff --git a/src/test/regress/sql/select.sql b/src/test/regress/sql/select.sql
index 019f1e7673..0e650a2301 100644
--- a/src/test/regress/sql/select.sql
+++ b/src/test/regress/sql/select.sql
@@ -234,6 +234,88 @@ select unique1, unique2 from onek2
 select unique1, unique2 from onek2
   where (unique2 = 11 and stringu1 < 'B') or unique1 = 0;
 
+SET enable_seqscan TO off;
+SET enable_indexscan TO off; -- Only BitmapScan is a subject matter here
+SET enable_or_transformation = 'off';
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 = 'A' OR stringu1 = 'J';
+-- Without the transformation only seqscan possible here
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique2 < 1 AND stringu1 IN ('A','J') AND stringu1 < 'Z';
+-- Use partial indexes
+explain (costs off)
+SELECT count(*) FROM onek2
+WHERE stringu1 IN ('B','J') AND (stringu1 = 'A' OR unique1 = 1);
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 < 1 OR (stringu1 = 'A' OR stringu1 = 'J');
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 = 1 OR unique2 = PI()::integer;
+RESET enable_or_transformation;
+
+-- OR <-> ANY transformation must find a path with partial indexes scan
+-- regardless the clause representation.
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 = 'A' OR stringu1 = 'J';
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 IN ('A','J');
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 = ANY ('{A,J}');
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 IN ('A') OR stringu1 IN ('J');
+
+-- Don't scan partial indexes because of extra value.
+explain (costs off)
+SELECT count(*) FROM onek2 WHERE stringu1 IN ('A', 'J', 'C');
+explain (costs off)
+SELECT unique2 FROM onek2
+WHERE stringu1 IN ('A', 'A') AND (stringu1 = 'A' OR stringu1 = 'A');
+
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique2 < 1 AND stringu1 IN ('A','J') AND stringu1 < 'Z';
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 = 1 OR unique1 = PI()::integer;
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 IN (1, PI()::integer); -- TODO: why it is differs from previous example?
+
+-- Don't apply the optimization to clauses, containing volatile functions
+EXPLAIN (COSTS OFF)
+SELECT unique2,stringu1 FROM onek2
+WHERE unique1 = (random()*2)::integer OR unique1 = (random()*3)::integer;
+EXPLAIN (COSTS OFF)
+SELECT unique2,stringu1 FROM onek2
+WHERE unique1 IN ((random()*2)::integer, (random()*3)::integer);
+
+-- Combine different saops. Soe of them doesnt' fit a set of partial indexes,
+-- but other fits.
+-- Unfortunately, we don't combine saop and OR clauses so far.
+EXPLAIN (COSTS OFF)
+SELECT unique2,stringu1 FROM onek2
+WHERE
+  unique1 IN (1,2,21) AND
+  (stringu1 IN ('A','J') OR unique1 IN (3,4) OR stringu1 = 'J');
+
+-- Check recursive combination of OR and SAOP expressions
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE unique1 < 1 OR (stringu1 = 'A' OR stringu1 = 'J');
+explain (costs off)
+SELECT unique2, stringu1 FROM onek2
+WHERE (unique1 < 1 OR stringu1 IN ('A','J'));
+-- Although SAOP doesn't fit partial indexes fully, we can use anded OR clause
+-- to scan another couple of partial indexes.
+explain (costs off)
+SELECT count(*) FROM onek2
+WHERE stringu1 IN ('B','J') AND (stringu1 = 'A' OR unique1 = 1);
+
+RESET enable_indexscan;
+RESET enable_seqscan;
+
 --
 -- Test some corner cases that have been known to confuse the planner
 --
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f8d5871153..3cb45efe60 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2138,6 +2138,7 @@ PredIterInfoData
 PredXactList
 PredicateLockData
 PredicateLockTargetType
+PredicatesData
 PrefetchBufferResult
 PrepParallelRestorePtrType
 PrepareStmt
-- 
2.43.0

---

From: Ranier Vilela <[email-masked]>
Date: Mon, 19 Feb 2024 09:53:26 -0300

> > I found that the field `scalar_type` was never used.
Not that it will make a big difference, but it would be good to avoid, I
think.

v17-0002
1) move the vars *arrayconst and *dest, to after if, to avoid makeNode
(palloc).
+ Const   *arrayconst;
+ ScalarArrayOpExpr  *dest;
+
+ pd = (PredicatesData *) lfirst(lc);
+ if (pd->elems == NIL)
+ /* The index doesn't participate in this operation */
+ continue;

+ arrayconst = lsecond_node(Const, saop->args);
+ dest = makeNode(ScalarArrayOpExpr);

best regards,
Ranier Vilela

---

From: Andrei Lepikhov <[email-masked]>
Date: Tue, 20 Feb 2024 10:18:23 +0700

> + dest = makeNode(ScalarArrayOpExpr);
Thanks for the review!
I'm not sure I understand you clearly. Does the patch in attachment fix 
the issue you raised?

-- 
regards,
Andrei Lepikhov
Postgres Professional
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 56b04541db..1545876e2f 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -1284,7 +1284,7 @@ build_paths_for_SAOP(PlannerInfo *root, RelOptInfo *rel, RestrictInfo *rinfo,
 		ArrayType		   *arrayval = NULL;
 		ArrayExpr		   *arr = NULL;
 		Const			   *arrayconst = lsecond_node(Const, saop->args);
-		ScalarArrayOpExpr  *dest = makeNode(ScalarArrayOpExpr);
+		ScalarArrayOpExpr	dest;
 
 		pd = (PredicatesData *) lfirst(lc);
 		if (pd->elems == NIL)
@@ -1302,10 +1302,10 @@ build_paths_for_SAOP(PlannerInfo *root, RelOptInfo *rel, RestrictInfo *rinfo,
 		arr->multidims = false;
 
 		/* Compose new SAOP, partially covering the source one */
-		memcpy(dest, saop, sizeof(ScalarArrayOpExpr));
-		dest->args = list_make2(linitial(saop->args), arr);
+		memcpy(&dest, saop, sizeof(ScalarArrayOpExpr));
+		dest.args = list_make2(linitial(saop->args), arr);
 
-		clause = (Expr *) estimate_expression_value(root, (Node *) dest);
+		clause = (Expr *) estimate_expression_value(root, (Node *) &dest);
 
 		/*
 		 * Create new RestrictInfo. It maybe more heavy than just copy node,

---

From: jian he <[email-masked]>
Date: Tue, 20 Feb 2024 12:03:31 +0800

> On Tue, Nov 28, 2023 at 5:04 AM Robert Haas <[email-masked]> wrote:
>
> On Mon, Nov 27, 2023 at 3:02 AM Andrei Lepikhov
> <[email-masked]> wrote:
> > On 25/11/2023 08:23, Alexander Korotkov wrote:
> > > I think patch certainly gets better in this aspect.  One thing I can't
> > > understand is why do we use home-grown code for resolving
> > > hash-collisions.  You can just define custom hash and match functions
> > > in HASHCTL.  Even if we need to avoid repeated JumbleExpr calls, we
> > > still can save pre-calculated hash value into hash entry and use
> > > custom hash and match.  This doesn't imply us to write our own
> > > collision-resolving code.
> >
> > Thanks, it was an insightful suggestion.
> > I implemented it, and the code has become shorter (see attachment).
>
> Neither the code comments nor the commit message really explain the
> design idea here. That's unfortunate, principally because it makes
> review difficult.
>
> I'm very skeptical about the idea of using JumbleExpr for any part of
> this. It seems fairly expensive, and it might produce false matches.
> If expensive is OK, then why not just use equal()? If it's not, then
> this probably isn't really OK either. But in any case there should be
> comments explaining why this strategy was chosen.

The above message
(https://postgr.es/m/CA%2BTgmoZCgP6FrBQEusn4yaWm02XU8OPeoEMk91q7PRBgwaAkFw%40mail.gmail.com)
seems still not answered.
How can we evaluate whether JumbleExpr is expensive or not?
I used this naive script to test, but didn't find a big difference
when enable_or_transformation is ON or OFF.

`
create table test_1_100 as (select (random()*1000)::int x,
(random()*1000) y from generate_series(1,1_000_000) i);
explain(costs off, analyze)
select * from test
where x = 1 or x + 2= 3 or x + 3= 4 or x + 4= 5
or x + 5= 6 or x + 6= 7 or x + 7= 8 or x + 8= 9 or x + 9=10
or x + 10= 11 or x + 11= 12 or x + 12= 13 or x + 13= 14
or x + 14= 15 or x + 15= 16 or x + 16= 17 or x + 17= 18
or x + 18=19 or x + 19= 20 or x + 20= 21 or x + 21= 22
or x + 22= 23 or x + 23= 24 or x + 24= 25 or x + 25= 26
or x + 26= 27 or x + 27=28 or x + 28= 29 or x + 29= 30
or x + 30= 31 \watch i=0.1 c=10
`

`leftop operator rightop`
the operator can also be volatile.
Do we need to check (op_volatile(opno) == PROVOLATILE_VOLATILE) within
transformBoolExprOr?

---

From: Andrei Lepikhov <[email-masked]>
Date: Tue, 20 Feb 2024 11:29:36 +0700

> when enable_or_transformation is ON or OFF.
First, I am open to discussion here. But IMO, equal() operation is quite 
expensive by itself. We should use the hash table approach to avoid 
quadratic behaviour when looking for similar clauses in the 'OR' list.
Moreover, we use equal() in many places: selectivity estimations, 
proving of fitting the index, predtest, etc. So, by reducing the clause 
list, we eliminate many calls of the equal() routine, too.

> Do we need to check (op_volatile(opno) == PROVOLATILE_VOLATILE) within
As usual, could you provide a test case to discuss it more objectively?

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: Ranier Vilela <[email-masked]>
Date: Tue, 20 Feb 2024 08:17:07 -0300

> I'm not sure I understand you clearly. Does the patch in attachment fix
Sorry if I wasn't clear.
What I meant is to move the initializations of the variables *arrayconst*
and *dest*
for after the test (if (pd->elems == NIL)
To avoid unnecessary initialization if the test fails.

best regards,
Ranier Vilela

---

From: jian he <[email-masked]>
Date: Sat, 24 Feb 2024 19:28:01 +0800

Hi.
I wrote the first draft patch of the documentation.
it's under the section: Planner Method Configuration (runtime-config-query.html)
but this feature's main meat is in src/backend/parser/parse_expr.c
so it may be slightly inconsistent, as mentioned by others.

You can further furnish it.

---

From: Alena Rybakina <[email-masked]>
Date: Mon, 26 Feb 2024 07:10:47 +0300

> You can further furnish it.

Thank you for your work. I found a few spelling mistakes - I fixed this 
and added some information about generating a partial index plan. I 
attached it.

-- 
Regards,
Alena Rybakina
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
From e3a0e01c43a70099f6870a468d0cc3a8bdcb2775 Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Mon, 26 Feb 2024 06:37:36 +0300
Subject: [PATCH] doc1

---
 doc/src/sgml/config.sgml | 74 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 36a2a5ce431..47f82ca2dc9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5294,6 +5294,80 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
       </listitem>
      </varlistentry>
 
+     <varlistentry id="guc-enable-or-transformation" xreflabel="enable_or_transformation">
+      <term><varname>enable_or_transformation</varname> (<type>boolean</type>)
+       <indexterm>
+        <primary><varname>enable_or_transformation</varname> configuration parameter</primary>
+       </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Enables or disables in query planner's ability to transform multiple expressions in (<xref linkend="sql-expressions"/>)
+        <xref linkend="sql-where"/> to a ANY expression (<xref linkend="functions-comparisons-any-some"/>).
+        This transformations only apply under the following condition:
+        <itemizedlist>
+          <listitem>
+          <para>
+            Each expression should be formed as: <replaceable>expression</replaceable> <replaceable>operator</replaceable>  (<replaceable>expression</replaceable>).
+            The right-hand side of the operator should be just a plain constant.
+            The left-hand side of these expressions should remain unchanged.
+          </para>
+          </listitem>
+        </itemizedlist>
+        <itemizedlist>
+          <listitem>
+          <para>
+            At the stage of index formation, a check is made on the restructuring of the plan using partial indexes or the formation of expressions combined by the "OR" <replaceable>operator</replaceable>.
+          </para>
+          </listitem>
+        </itemizedlist>
+            <itemizedlist>
+              <listitem>
+                <para>
+                  Each expression form should return Boolean (true/false) result.
+                </para>
+              </listitem>
+            </itemizedlist>
+            <itemizedlist>
+              <listitem>
+                <para>
+                These expressions are logically linked in a OR condition.
+                </para>
+              </listitem>
+            </itemizedlist>
+          The default is <literal>on</literal>.
+          </para>
+        <para>
+          For example, the following query without setting <varname>enable_or_transformation</varname> is usually applied to the three filtering conditions separately,
+          but if we set <varname>enable_or_transformation</varname>, we combine the three expressions into only one expression: <literal>unique1 = ANY ('{1,2,3}'::integer[]) </literal>.
+          <programlisting>
+          EXPLAIN SELECT * FROM tenk1 WHERE unique1 = 1 or unique1 = 2 or unique1 = 3;
+
+                                  QUERY PLAN
+          -------------------------------------------------------------
+          Seq Scan on tenk1  (cost=0.00..482.50 rows=3 width=244)
+            Filter: (unique1 = ANY ('{1,2,3}'::integer[]))
+          </programlisting>
+        </para>
+        <para>
+          Another example is the following query with a given <varname>enable_or_transformation</varname> value, but we have generated a plan with partial indexes.
+          <programlisting>
+          EXPLAIN SELECT unique2, stringu1 FROM onek2 WHERE unique1 = 1 OR unique1 = PI()::integer;
+                              QUERY PLAN                    
+          --------------------------------------------------
+          Bitmap Heap Scan on onek2
+            Recheck Cond: ((unique1 = 3) OR (unique1 = 1))
+            ->  BitmapOr
+                  ->  Bitmap Index Scan on onek2_u1_prtl
+                        Index Cond: (unique1 = 3)
+                  ->  Bitmap Index Scan on onek2_u1_prtl
+                        Index Cond: (unique1 = 1)
+          (7 rows)
+          </programlisting>
+        </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry id="guc-enable-parallel-append" xreflabel="enable_parallel_append">
       <term><varname>enable_parallel_append</varname> (<type>boolean</type>)
       <indexterm>
-- 
2.34.1

---

From: Andrei Lepikhov <[email-masked]>
Date: Wed, 28 Feb 2024 11:18:59 +0700

> and added some information about generating a partial index plan. I 
Thanks Jian and Alena,
It is a good start for the documentation. But I think the runtime-config 
section needs only a condensed description of a feature underlying the 
GUC. The explanations in this section look a bit awkward.
Having looked through the documentation for a better place for a 
detailed explanation, I found array.sgml as a candidate. Also, we have 
the parser's short overview section. I'm unsure about the best place but 
it is better when the server config section.
What's more, there are some weak points in the documentation:
1. We choose constant and variable parts of an expression and don't 
require the constant to be on the right side.
2. We should describe the second part of the feature, where the 
optimiser can split an array to fit the optimal BitmapOr scan path.

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: jian he <[email-masked]>
Date: Wed, 28 Feb 2024 18:07:21 +0800

> it is better when the server config section.

doc/src/sgml/array.sgml corresponds to
https://www.postgresql.org/docs/current/arrays.html.
 this GUC is related to parser|optimzier.
adding a GUC to array.sgml seems strange. (I think).

> optimiser can split an array to fit the optimal BitmapOr scan path.

we can add a sentence explaining that:
it may not do the expression transformation when the original
expression can be utilized by index mechanism.
I am not sure how to rephrase it.

---

From: Alena Rybakina <[email-masked]>
Date: Wed, 28 Feb 2024 13:27:41 +0300

> adding a GUC to array.sgml seems strange. (I think).

I suggest describing our feature in array.sgml and mentioning a GUC there.

We can describe a GUC in config.sgml.

> I am not sure how to rephrase it.

Maybe like that:

It also considers the way to generate a path using BitmapScan indexes, 
converting the transformed expression into expressions separated by "OR" 
operations, and if it turns out to be the best and finally selects the 
best one.

---

From: Andrei Lepikhov <[email-masked]>
Date: Thu, 29 Feb 2024 08:50:06 +0700

> adding a GUC to array.sgml seems strange. (I think).
Maybe. In that case, I suggest adding extended comments to functions 
transformBoolExprOr and generate_saop_pathlist (including 
cross-referencing each other). These are starting points to understand 
the transformation and, therefore, a good place for a detailed explanation.

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: Andrei Lepikhov <[email-masked]>
Date: Thu, 29 Feb 2024 15:59:23 +0700

> operations, and if it turns out to be the best and finally selects the 
Thanks,
I spent some time describing the feature with documentation.
A condensed description of the GUC is in the runtime-config file.
Feature description has spread between TransformOrExprToANY and 
generate_saop_pathlist routines.
Also, I've made tiny changes in the code to look more smoothly.
All modifications are integrated into the two new patches.

Feel free to add, change or totally rewrite these changes.

-- 
regards,
Andrei Lepikhov
Postgres Professional
From 015a564cc784139c806a7004f25bf5f7a4b4a29d Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clause to ANY expressions.

 doc/src/sgml/config.sgml                      |  18 +
 src/backend/parser/parse_expr.c               | 340 ++++++++++++++++++
 src/test/regress/expected/create_index.out    | 156 +++++++-
 src/test/regress/expected/partition_prune.out | 219 +++++++++--
 22 files changed, 908 insertions(+), 69 deletions(-)

 

 
 -- The new values are concatenated with ' triggered !'
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 43b1a132a2..abad02f92c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5433,6 +5433,24 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
       </listitem>
      </varlistentry>
 
+     <varlistentry id="guc-enable-or-transformation" xreflabel="enable_or_transformation">
+      <term><varname>enable_or_transformation</varname> (<type>boolean</type>)
+       <indexterm>
+        <primary><varname>enable_or_transformation</varname> configuration parameter</primary>
+       </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Enables or disables the query planner's ability to lookup and group multiple
+        similar OR expressions to ANY (<xref linkend="functions-comparisons-any-some"/>) expressions.
+        The grouping technique of this transformation is based on the similarity of variable sides.
+        It applies to equality expressions only. One side of such an expression
+        must be a constant clause, and the other must contain a variable clause.
+        The default is <literal>on</literal>.
+        </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry id="guc-enable-parallel-append" xreflabel="enable_parallel_append">
       <term><varname>enable_parallel_append</varname> (<type>boolean</type>)
       <indexterm>
diff --git a/src/backend/nodes/queryjumblefuncs.c b/src/backend/nodes/queryjumblefuncs.c
index 82f725baaa..b203746be4 100644
--- a/src/backend/nodes/queryjumblefuncs.c
+++ b/src/backend/nodes/queryjumblefuncs.c
@@ -141,6 +141,33 @@ JumbleQuery(Query *query)
 	return jstate;
 }
 
+JumbleState *
index 9300c7b9ab..10ceccc80c 100644

 /* GUC parameters */

 static Node *transformExprRecurse(ParseState *pstate, Node *expr);
@@ -99,6 +103,337 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname,
 static Node *make_nulltest_from_distinct(ParseState *pstate,
 										 A_Expr *distincta, Node *arg);
 
+ * TransformOrExprToANY -
+ *	  Discover the args of an OR expression and try to group similar OR
+ *	  expressions to an ANY operation.
+ *	  Transformation must be already done on input args list before the call.
+ *	  Transformation groups two-sided equality operations. One side of such an
+ *	  operation must be plain constant or constant expression. The other side of
+ *	  the clause must be a variable expression without volatile functions.
+ *	  The grouping technique is based on the similarity of variable sides of the
+ *	  expression: using queryId and equal() routine, it groups constant sides of
+ *	  similar clauses into an array. After the grouping procedure, each couple
+ *	  ('variable expression' and 'constant array') form a new SAOP operation,
+ *	  which is added to the args list of the returning expression.
+ *	  NOTE: function returns OR BoolExpr if more than one clause are detected in
+ *	  the final args list, or ScalarArrayOpExpr if all args were grouped into
+ *	  the single SAOP expression.
+TransformOrExprToANY(ParseState *pstate, List *args, int location)
+	int 					len_ors = list_length(args);
+	Assert(enable_or_transformation && len_ors > 1);
+	foreach(lc, args)
+		Node				   *orqual = lfirst(lc);
+					 makeBoolExpr(OR_EXPR, or_list, location) :
+					 linitial(or_list));
+}
 
 /*
  * transformExpr -
@@ -1386,6 +1721,11 @@ transformBoolExpr(ParseState *pstate, BoolExpr *a)
 		args = lappend(args, arg);
 	}
 
+	/* Make an attempt to group similar OR clauses into ANY operation */
+	if (enable_or_transformation && a->boolop == OR_EXPR &&
+		list_length(args) > 1)
+		return TransformOrExprToANY(pstate, args, a->location);
+
 	return (Node *) makeBoolExpr(a->boolop, args, a->location);
 }
 
index 93ded31ed9..7d3a1ca238 100644
@@ -1026,6 +1026,17 @@ struct config_bool ConfigureNamesBool[] =
index edcc0282b2..d30dc6d51c 100644
@@ -389,6 +389,7 @@

 

 
 
 SELECT * FROM tenk1

index 130a924228..684886336c 100644
 
index 9605400021..cd2d78a636 100644
 
index b41950d923..26b05d7a5a 100644

 

 

@@ -2036,10 +2215,10 @@ explain (costs off) select * from hp where a = 1 and b = 'abcde';

 drop table hp2;

 

 

 
 
index c4c6c7b8ba..1663608043 100644
 
index 7ba6a9ff37..31ad4166d5 100644
 
+
 
index fc8b15d0cf..5cf444e95b 100644
@@ -1656,6 +1656,8 @@ NumericVar
 OM_uint32
 OP
 OSAPerGroupState
+OrClauseGroupEntry
+OrClauseGroupKey
 OSAPerQueryState
 OSInfo
 OSSLCipher
-- 
2.43.2

From 2810be3413696796e9f2b88a8dd77801c0088139 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Wed, 24 Jan 2024 14:07:17 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

Likewise OR clauses, discover SAOP array and try to split its elements
between smaller sized arrays to fit a set of partial indexes.
---
 doc/src/sgml/config.sgml                  |   3 +
 src/backend/optimizer/path/indxpath.c     | 315 ++++++++++++++++++----
 src/backend/optimizer/util/predtest.c     |  46 ++++
 src/backend/optimizer/util/restrictinfo.c |  13 +
 src/include/optimizer/optimizer.h         |  16 ++
 src/include/optimizer/restrictinfo.h      |   1 +
 src/test/regress/expected/select.out      | 282 +++++++++++++++++++
 src/test/regress/sql/select.sql           |  82 ++++++
 src/tools/pgindent/typedefs.list          |   1 +
 9 files changed, 706 insertions(+), 53 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index abad02f92c..e046b6bbca 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5447,6 +5447,9 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
         It applies to equality expressions only. One side of such an expression
         must be a constant clause, and the other must contain a variable clause.
         The default is <literal>on</literal>.
+        Also, during BitmapScan paths generation it enables analysis of elements
+        of IN or ANY constant arrays to cover such clause with BitmapOr set of
+        partial index scans.
         </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 32c6a8bbdc..47bf53820d 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -32,6 +32,7 @@
 #include "optimizer/paths.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "utils/array.h"
 #include "utils/lsyscache.h"
 #include "utils/selfuncs.h"
 
@@ -1220,11 +1221,188 @@ build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel,
 	return result;
 }
 
+/*
+ * Analyse incoming SAOP node to cover it by partial indexes.
+ * The returning pathlist must be ANDed to the final BitmapScan path.
+ * The function returns NULL when an array element cannot be fitted with some
+ * partial index. The Rationale for such an operation is that when schema
+ * contains many partial indexes, the SAOP clause may be effectively fulfilled
+ * by appending results of scanning some minimal set of tiny partial indexes.
+ * Working jointly with the TransformOrExprToANY routine, it provides a user
+ * with some sort of independence of the query plan from the approach to writing
+ * alternatives for the same entity in the WHERE section.
@@ -1247,68 +1425,99 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel,

 

 
-			/*
 
-			/*
 
 		/*

+/*
 
+bool

+/*

index 5cf444e95b..698e460840 100644
@@ -2141,6 +2141,7 @@ PredIterInfoData
2.43.2

---

From: Alexander Korotkov <[email-masked]>
Date: Fri, 1 Mar 2024 17:33:05 +0200

Hi, Andrei,
Hi, Alena!

> Feel free to add, change or totally rewrite these changes.

I'm going to review and revise the patch.

One question I have yet.

>        /*
>         * Transformation only works with both side type is not
>         * { array | composite | domain | record }.

Why do we limit transformation for these types?  Also, it doesn't seem the
current code restricts anything except composite/record.

------
Regards,
Alexander Korotkov

---

From: Alena Rybakina <[email-masked]>
Date: Sun, 3 Mar 2024 12:26:19 +0300

I found that it was mentioned here - 
https://www.postgresql.org/message-id/CACJufxFrZS07oBHMk1_c8P3A84VZ3ysXiZV8NeU6gAnvu%2BHsVA%40mail.gmail.com.

To be honest, I couldn't find any explanation for that.

On 01.03.2024 18:33, Alexander Korotkov wrote:
> Hi, Andrei,
> Hi, Alena!
>
> On Thu, Feb 29, 2024 at 10:59 AM Andrei Lepikhov 
> Why do we limit transformation for these types?  Also, it doesn't seem 
> the current code restricts anything except composite/record.
>
> ------
> Regards,
> Alexander Korotkov

-- 
Regards,
Alena Rybakina
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company

---

From: Alena Rybakina <[email-masked]>
Date: Sun, 3 Mar 2024 12:48:19 +0300

Sorry, I found explanation - 
https://www.postgresql.org/message-id/CACJufxFS-xcjaWq2Du2OyJUjRAyqCk12Q_zGOPxv61sgrdpw9w%40mail.gmail.com

On 03.03.2024 12:26, Alena Rybakina wrote:
>> Why do we limit transformation for these types?  Also, it doesn't 
>> seem the current code restricts anything except composite/record.
>>
>> ------
>> Regards,
>> Alexander Korotkov
> -- 
> Regards,
> Alena Rybakina

---

From: jian he <[email-masked]>
Date: Mon, 4 Mar 2024 10:26:48 +0800

> Feel free to add, change or totally rewrite these changes.

diff --git a/src/backend/utils/misc/guc_tables.c
b/src/backend/utils/misc/guc_tables.c

I think it should be something like:
+ gettext_noop("Transform a sequence of OR expressions to an array
expression."),
+ gettext_noop("The planner will replace expression like 'x=c1 OR x=c2 "
+ "to the expression 'x = ANY( ARRAY[c1,c2])''"

+JumbleState *
queryId may not be a good variable name here?

comment `/* Compute query ID */`
seems not correct, here we are just hashing the expression?

+/*
the above comments seem not correct?

       <para>
        Enables or disables the query planner's ability to lookup and
group multiple
        similar OR expressions to ANY (<xref
linkend="functions-comparisons-any-some"/>) expressions.
        The grouping technique of this transformation is based on the
similarity of variable sides.
        It applies to equality expressions only. One side of such an expression
        must be a constant clause, and the other must contain a variable clause.
        The default is <literal>on</literal>.
        Also, during BitmapScan paths generation it enables analysis of elements
        of IN or ANY constant arrays to cover such clause with BitmapOr set of
        partial index scans.
        </para>
` It applies to equality expressions only.` seems not correct?
`select * from tenk1 where unique1 < 1 or unique1 < 2; ` can also do
the transformation.
`similarity of variable sides.` seems not correct,
should  it be 'sameness of the variable sides`?

in [1], we can get:
expression IN (value [, ...])
is equivalent to
expression = value1
OR
expression = value2
OR

in [2], we can get:
SOME is a synonym for ANY. IN is equivalent to = ANY.

but still transforming OR to ANY is not intuitive.
a normal user may not know what is "transforming OR to ANY".
so maybe adding a simple example at
<varlistentry id="guc-enable-or-transformation"
xreflabel="enable_or_transformation">
would be great. which, I did at previous thread.

I also did some refactoring based on v18, attached.

[1] https://www.postgresql.org/docs/current/functions-comparisons.html#FUNCTIONS-COMPARISONS-IN-SCALAR
[2] https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-ANY-SOME

---

From: Andrei Lepikhov <[email-masked]>
Date: Tue, 5 Mar 2024 12:30:25 +0700

>> Feel free to add, change or totally rewrite these changes.
On replacement of static ScalarArrayOpExpr dest with dynamic allocated one:
After discussion [1] I agree with that replacement.

Some style (and language) changes in comments I haven't applied because 
it looks debatable for me.

> + "to the expression 'x = ANY( ARRAY[c1,c2])''"
Fixed

> queryId may not be a good variable name here?
Not sure. QueryId is a concept, part of queryjumble technique and can be 
used by other tools. It just tells the developer what it is the same 
thing as Query Jumbling but for a separate expression.
At least you don't insist on removing of JumbleState return pointer that 
looks strange for a simple hash ...
> seems not correct, here we are just hashing the expression?
The same as above.
> the above comments seem not correct?
Yes, fixed.

> `select * from tenk1 where unique1 < 1 or unique1 < 2; ` can also do
Yes, I forgot it.
> should  it be 'sameness of the variable sides`?
The term 'equivalence' looks better *).

> would be great. which, I did at previous thread.
Not sure. Examples in that section are unusual things. What's more, 
should a user who doesn't know what it means to change this setting? 
Let's wait for other opinions.

[1] https://www.postgresql.org/message-id/[email-masked]

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: Andrei Lepikhov <[email-masked]>
Date: Tue, 5 Mar 2024 14:59:16 +0700

> On 4/3/2024 09:26, jian he wrote:
... and the new version of the patchset is attached.

-- 
regards,
Andrei Lepikhov
Postgres Professional
From 1c3ac3e006cd66ff40f1ddaaa09e3fc0f3a75ca5 Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clauses to ANY expression.

Replace (expr op C1) OR (expr op C2) ... with expr op ANY(ARRAY[C1, C2, ...]) on the
preliminary stage of optimization when we are still working with the
expression tree.
 doc/src/sgml/config.sgml                      |  17 +
 src/backend/parser/parse_expr.c               | 339 ++++++++++++++++++
 22 files changed, 906 insertions(+), 69 deletions(-)

 

 
 -- The new values are concatenated with ' triggered !'
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..1fdfffd79b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5433,6 +5433,23 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
       </listitem>
      </varlistentry>
 
+        The grouping technique of this transformation is based on the equivalence of variable sides.
+        One side of such an expression must be a constant clause, and the other must contain a variable clause.
index 426112fa37..c4c4cab8ed 100644
 
+JumbleState *
index 9300c7b9ab..73298ad621 100644

 /* GUC parameters */

 static Node *transformExprRecurse(ParseState *pstate, Node *expr);
@@ -99,6 +103,336 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname,
 static Node *make_nulltest_from_distinct(ParseState *pstate,
 										 A_Expr *distincta, Node *arg);
 
+ * Dynahash match function to use in or_group_htab
+ *	  The grouping technique is based on an equivalence of variable sides of the
 
 /*
  * transformExpr -
@@ -1386,6 +1720,11 @@ transformBoolExpr(ParseState *pstate, BoolExpr *a)
 		args = lappend(args, arg);
 	}

diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 543a87c659..e82f6b7f42 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1026,6 +1026,17 @@ struct config_bool ConfigureNamesBool[] =
 		true,
 		NULL, NULL, NULL
 	},
+	{
+		{"enable_or_transformation", PGC_USERSET, QUERY_TUNING_OTHER,
+			gettext_noop("Transform a sequence of OR clauses to an array expression."),
+			gettext_noop("The planner will replace expression like 'x=c1 OR x=c2 ..'"
+						 "to the expression 'x = ANY(ARRAY[c1,c2,..])'"),
+			GUC_EXPLAIN
+		},
+		&enable_or_transformation,
+		true,
+		NULL, NULL, NULL
+	},
 	{
 		/*
 		 * Not for general use --- used by SET SESSION AUTHORIZATION and SET
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..d30dc6d51c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -389,6 +389,7 @@
 # - Planner Method Configuration -

 

 
 
 SELECT * FROM tenk1

 
 
index bf0657b9f2..58a96a29a6 100644

 

 

@@ -2072,10 +2251,10 @@ explain (costs off) select * from hp where a = 1 and b = 'abcde';

 drop table hp2;
index 0cd2c64fca..0378280c7d 100644

 

 

 

index a09b27d820..9717c8c835 100644
 
+
 
index 782b7d7b1c..c81f6caba8 100644
@@ -1657,6 +1657,8 @@ NumericVar
 OM_uint32
 OP
 OSAPerGroupState
+OrClauseGroupEntry
+OrClauseGroupKey
 OSAPerQueryState
 OSInfo
 OSSLCipher
-- 
2.43.2

From 99bdb199a240bd7b9d6e300bedddc9084c53b973 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Tue, 5 Mar 2024 13:29:46 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 1fdfffd79b..8008b05327 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5446,6 +5446,9 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
         The grouping technique of this transformation is based on the equivalence of variable sides.
         One side of such an expression must be a constant clause, and the other must contain a variable clause.
         The default is <literal>on</literal>.
+        Also, during BitmapScan paths generation it enables analysis of elements
+        of IN or ANY constant arrays to cover such clause with BitmapOr set of
+        partial index scans.
         </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 32c6a8bbdc..f92a47c3d5 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -32,6 +32,7 @@
 #include "optimizer/paths.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "utils/array.h"
 #include "utils/lsyscache.h"
 #include "utils/selfuncs.h"

+/*

 

 
-			/*
 
-			/*
 
 		/*

+/*
 
+bool

+/*
index 33a6dceb0e..768c9b0780 100644
 
index 019f1e7673..223f55af4e 100644
 
index c81f6caba8..dade69f47a 100644
@@ -2142,6 +2142,7 @@ PredIterInfoData

---

From: Alexander Korotkov <[email-masked]>
Date: Thu, 7 Mar 2024 16:51:43 +0200

Hi!

> ... and the new version of the patchset is attached.

I made some revisions for the patchset.
1) Use hash_combine() to combine hash values.
2) Upper limit the number of array elements by MAX_SAOP_ARRAY_SIZE.
3) Better save the original order of clauses by putting hash entries and
untransformable clauses to the same list.  A lot of differences in
regression tests output have gone.

One important issue I found.

# create table t as (select i::int%100 i from generate_series(1,10000) i);
# analyze t;
# explain select * from t where i = 1 or i = 1;
                     QUERY PLAN
-----------------------------------------------------
 Seq Scan on t  (cost=0.00..189.00 rows=200 width=4)
   Filter: (i = ANY ('{1,1}'::integer[]))
(2 rows)

# set enable_or_transformation = false;
SET
# explain select * from t where i = 1 or i = 1;
                     QUERY PLAN
-----------------------------------------------------
 Seq Scan on t  (cost=0.00..189.00 rows=100 width=4)
   Filter: (i = 1)
(2 rows)

We don't make array values unique.  That might make query execution
performance somewhat worse, and also makes selectivity estimation worse.  I
suggest Andrei and/or Alena should implement making array values unique.

------
Regards,
Alexander Korotkov

---

From: Alena Rybakina <[email-masked]>
Date: Thu, 7 Mar 2024 23:28:59 +0300

Hi!

On 07.03.2024 17:51, Alexander Korotkov wrote:
> Hi!
>
> On Tue, Mar 5, 2024 at 9:59 AM Andrei Lepikhov 
> 3) Better save the original order of clauses by putting hash entries 
> and untransformable clauses to the same list.  A lot of differences in 
Thank you for your changes. I agree with them.
> performance somewhat worse, and also makes selectivity estimation 
> worse.  I suggest Andrei and/or Alena should implement making array 
> values unique.
>
>
I have corrected this and some spelling mistakes. The 
unique_any_elements_change.no-cfbot file contains changes.

While I was correcting the test results caused by such changes, I 
noticed that the same behavior was when converting the IN expression,
and this can be seen in the result of the regression test:

  EXPLAIN (COSTS OFF)
  SELECT unique2 FROM onek2
  WHERE stringu1 IN ('A', 'A') AND (stringu1 = 'A' OR stringu1 = 'A');
                                 QUERY PLAN
---------------------------------------------------------------------------
   Bitmap Heap Scan on onek2
     Recheck Cond: (stringu1 < 'B'::name)
    Filter: ((stringu1 = ANY ('{A,A}'::name[])) AND (stringu1 = 'A'::name))
     ->  Bitmap Index Scan on onek2_u2_prtl
  (4 rows)

-- 
Regards,
Alena Rybakina
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index baeb83e58b..e9813ef6ab 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -312,8 +312,8 @@ TransformOrExprToANY(ParseState *pstate, List *args, int location)
 
 		if (unlikely(found))
 		{
-			entry->consts = lappend(entry->consts, const_expr);
-			entry->exprs = lappend(entry->exprs, orqual);
+			entry->consts = list_append_unique(entry->consts, const_expr);
+			entry->exprs = list_append_unique(entry->exprs, orqual);
 		}
 		else
 		{
@@ -352,7 +352,6 @@ TransformOrExprToANY(ParseState *pstate, List *args, int location)
 		entry = (OrClauseGroupEntry *) lfirst(lc);
 
 		Assert(list_length(entry->consts) > 0);
-		Assert(list_length(entry->exprs) == list_length(entry->consts));
 
 		if (list_length(entry->consts) == 1 ||
 			list_length(entry->consts) > MAX_SAOP_ARRAY_SIZE)
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 7b721bac71..606a4399f9 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1915,16 +1915,16 @@ SELECT count(*) FROM tenk1
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand < 42 OR thousand < 99 OR 43 > thousand OR 42 > thousand);
-                                        QUERY PLAN                                        
-------------------------------------------------------------------------------------------
+                                      QUERY PLAN                                       
+---------------------------------------------------------------------------------------
  Aggregate
    ->  Bitmap Heap Scan on tenk1
-         Recheck Cond: ((hundred = 42) AND (thousand < ANY ('{42,99,43,42}'::integer[])))
+         Recheck Cond: ((hundred = 42) AND (thousand < ANY ('{42,99,43}'::integer[])))
          ->  BitmapAnd
                ->  Bitmap Index Scan on tenk1_hundred
                      Index Cond: (hundred = 42)
                ->  Bitmap Index Scan on tenk1_thous_tenthous
-                     Index Cond: (thousand < ANY ('{42,99,43,42}'::integer[]))
+                     Index Cond: (thousand < ANY ('{42,99,43}'::integer[]))
 (8 rows)
 
 EXPLAIN (COSTS OFF)
diff --git a/src/test/regress/expected/select.out b/src/test/regress/expected/select.out
index 0ebaf002e8..f4f5493c43 100644
--- a/src/test/regress/expected/select.out
+++ b/src/test/regress/expected/select.out
@@ -1047,11 +1047,11 @@ SELECT count(*) FROM onek2 WHERE stringu1 IN ('A', 'J', 'C');
 EXPLAIN (COSTS OFF)
 SELECT unique2 FROM onek2
 WHERE stringu1 IN ('A', 'A') AND (stringu1 = 'A' OR stringu1 = 'A');
-                                      QUERY PLAN                                       
----------------------------------------------------------------------------------------
+                                QUERY PLAN                                 
+---------------------------------------------------------------------------
  Bitmap Heap Scan on onek2
    Recheck Cond: (stringu1 < 'B'::name)
-   Filter: ((stringu1 = ANY ('{A,A}'::name[])) AND (stringu1 = ANY ('{A,A}'::name[])))
+   Filter: ((stringu1 = ANY ('{A,A}'::name[])) AND (stringu1 = 'A'::name))
    ->  Bitmap Index Scan on onek2_u2_prtl
 (4 rows)
 
@@ -1114,7 +1114,7 @@ WHERE unique1 IN ((random()*2)::integer, (random()*3)::integer);
    Filter: (unique1 = ANY (ARRAY[((random() * '2'::double precision))::integer, ((random() * '3'::double precision))::integer]))
 (2 rows)
 
--- Combine different saops. Soe of them doesnt' fit a set of partial indexes,
+-- Combine different saops. Some of them doesnt' fit a set of partial indexes,
 -- but other fits.
 -- Unfortunately, we don't combine saop and OR clauses so far.
 EXPLAIN (COSTS OFF)
@@ -1170,7 +1170,7 @@ WHERE (unique1 < 1 OR stringu1 IN ('A','J'));
          ->  Bitmap Index Scan on onek2_u2_prtl
 (9 rows)
 
--- Although SAOP doesn't fit partial indexes fully, we can use anded OR clause
+-- Although SAOP doesn't fit partial indexes fully, we can use added OR clause
 -- to scan another couple of partial indexes.
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM onek2
@@ -1188,6 +1188,17 @@ WHERE stringu1 IN ('B','J') AND (stringu1 = 'A' OR unique1 = 1);
 (8 rows)
 
 RESET enable_indexscan;
+--Check the case when we construct ANY list with unique elements
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM onek2
+WHERE unique1 = 1 OR unique1 = 1;
+                     QUERY PLAN                     
+----------------------------------------------------
+ Aggregate
+   ->  Index Only Scan using onek2_u1_prtl on onek2
+         Index Cond: (unique1 = 1)
+(3 rows)
+
 RESET enable_seqscan;
 --
 -- Test some corner cases that have been known to confuse the planner
diff --git a/src/test/regress/sql/select.sql b/src/test/regress/sql/select.sql
index 223f55af4e..b8cee41658 100644
--- a/src/test/regress/sql/select.sql
+++ b/src/test/regress/sql/select.sql
@@ -291,7 +291,7 @@ EXPLAIN (COSTS OFF)
 SELECT unique2,stringu1 FROM onek2
 WHERE unique1 IN ((random()*2)::integer, (random()*3)::integer);
 
--- Combine different saops. Soe of them doesnt' fit a set of partial indexes,
+-- Combine different saops. Some of them doesnt' fit a set of partial indexes,
 -- but other fits.
 -- Unfortunately, we don't combine saop and OR clauses so far.
 EXPLAIN (COSTS OFF)
@@ -307,13 +307,18 @@ WHERE unique1 < 1 OR (stringu1 = 'A' OR stringu1 = 'J');
 EXPLAIN (COSTS OFF)
 SELECT unique2, stringu1 FROM onek2
 WHERE (unique1 < 1 OR stringu1 IN ('A','J'));
--- Although SAOP doesn't fit partial indexes fully, we can use anded OR clause
+-- Although SAOP doesn't fit partial indexes fully, we can use added OR clause
 -- to scan another couple of partial indexes.
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM onek2
 WHERE stringu1 IN ('B','J') AND (stringu1 = 'A' OR unique1 = 1);
 
 RESET enable_indexscan;
+--Check the case when we construct ANY list with unique elements
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM onek2
+WHERE unique1 = 1 OR unique1 = 1;
+
 RESET enable_seqscan;
 
 --
From 9721f3e8ecda972241d9bedbfb915fd1514cb9d7 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Tue, 5 Mar 2024 13:29:46 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

Likewise OR clauses, discover SAOP array and try to split its elements
between smaller sized arrays to fit a set of partial indexes.
---
 doc/src/sgml/config.sgml                  |   3 +
 src/backend/optimizer/path/indxpath.c     | 315 ++++++++++++++++++----
 src/backend/optimizer/util/predtest.c     |  46 ++++
 src/backend/optimizer/util/restrictinfo.c |  13 +
 src/include/optimizer/optimizer.h         |  16 ++
 src/include/optimizer/restrictinfo.h      |   1 +
 src/test/regress/expected/select.out      | 293 ++++++++++++++++++++
 src/test/regress/sql/select.sql           |  87 ++++++
 src/tools/pgindent/typedefs.list          |   1 +
 9 files changed, 722 insertions(+), 53 deletions(-)

 
+/*

 

 
-			/*
 
-			/*
 
 		/*
diff --git a/src/backend/optimizer/util/predtest.c b/src/backend/optimizer/util/predtest.c
index b76896dfbf..a3a6d0024b 100644
--- a/src/backend/optimizer/util/predtest.c
+++ b/src/backend/optimizer/util/predtest.c
@@ -103,6 +103,52 @@ static Oid	get_btree_test_op(Oid pred_op, Oid clause_op, bool refute_it);
 static void InvalidateOprProofCacheCallBack(Datum arg, int cacheid, uint32 hashvalue);

+/*
 
+bool
+restriction_is_saop_clause(RestrictInfo *restrictinfo)
+{
+	if (restrictinfo->clause && IsA(restrictinfo->clause, ScalarArrayOpExpr))
+	{
+		ScalarArrayOpExpr *saop = (ScalarArrayOpExpr *) restrictinfo->clause;
+
+		if (saop->useOr)
+			return true;
+	}
+	return false;
+}
+
 /*
  * restriction_is_securely_promotable
  *
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
index 0ee7154e08..4029cf07be 100644
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -160,6 +160,20 @@ extern List *expand_function_arguments(List *args, bool include_out_arguments,

+/*
+ * Contains information needed to extract from saop a set of elements which can
+ * be covered by the partial index:
+ * id - caller's identification of the index.
+ * predicate - predicate expression of the index
+ * elems - returning list of array elements which corresponds to this predicate
+ */
+typedef struct PredicatesData
+{
+	int		id;
+	Node   *predicate;
+	List   *elems;
+} PredicatesData;
+
 /*
  * Proof attempts involving large arrays in ScalarArrayOpExpr nodes are
  * likely to require O(N^2) time, and more often than not fail anyway.
@@ -169,6 +183,8 @@ extern List *expand_function_arguments(List *args, bool include_out_arguments,
  */
 #define MAX_SAOP_ARRAY_SIZE		100
 
index 33a6dceb0e..f4f5493c43 100644
@@ -907,6 +907,299 @@ select unique1, unique2 from onek2
        0 |     998
 (2 rows)
 
+---------------------------------------------------------------------------
+   Filter: ((stringu1 = ANY ('{A,A}'::name[])) AND (stringu1 = 'A'::name))
+   Recheck Cond: ((unique1 = 1) OR (unique1 = 3))
+-- Combine different saops. Some of them doesnt' fit a set of partial indexes,
+   Recheck Cond: ((unique1 < 1) OR (stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name))
+   Filter: ((unique1 < 1) OR (stringu1 = ANY ('{A,J}'::name[])))
+   Recheck Cond: ((unique1 < 1) OR (stringu1 = ANY ('{J}'::name[])) OR (stringu1 < 'B'::name))
+   Filter: ((unique1 < 1) OR (stringu1 = ANY ('{A,J}'::name[])))
+-- Although SAOP doesn't fit partial indexes fully, we can use added OR clause
+--Check the case when we construct ANY list with unique elements
+WHERE unique1 = 1 OR unique1 = 1;
+----------------------------------------------------
+   ->  Index Only Scan using onek2_u1_prtl on onek2
index 019f1e7673..b8cee41658 100644
@@ -234,6 +234,93 @@ select unique1, unique2 from onek2
 
+-- Combine different saops. Some of them doesnt' fit a set of partial indexes,
+-- Although SAOP doesn't fit partial indexes fully, we can use added OR clause
+--Check the case when we construct ANY list with unique elements
+WHERE unique1 = 1 OR unique1 = 1;
index f4b6fee893..7930a17e81 100644

From af4592111fcd1f5fb85216c7d56f19ed0ffde38a Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clauses to ANY expression.

Replace (expr op C1) OR (expr op C2) ... with expr op ANY(ARRAY[C1, C2, ...]) on the
preliminary stage of optimization when we are still working with the
expression tree.
Here C<X> is a constant expression, 'expr' is non-constant expression, 'op' is
an operator which returns boolean result and has a commuter (for the case of
reverse order of constant and non-constant parts of the expression,
like 'CX op expr').
Sometimes it can lead to not optimal plan. But we think it is better to have
array of elements instead of a lot of OR clauses. Here is a room for further
optimizations on decomposing that array into more optimal parts.
Authors: Alena Rybakina <[email-masked]>, Andrey Lepikhov <[email-masked]>
Reviewed-by: Peter Geoghegan <[email-masked]>, Ranier Vilela <[email-masked]>
Reviewed-by: Alexander Korotkov <[email-masked]>, Robert Haas <[email-masked]>
Reviewed-by: jian he <[email-masked]>
---
 .../postgres_fdw/expected/postgres_fdw.out    |   8 +-
 doc/src/sgml/config.sgml                      |  17 +
 src/backend/nodes/queryjumblefuncs.c          |  27 ++
 src/backend/optimizer/util/predtest.c         |   9 -
 src/backend/parser/parse_expr.c               | 356 ++++++++++++++++++
 src/backend/utils/misc/guc_tables.c           |  11 +
 src/backend/utils/misc/postgresql.conf.sample |   1 +
 src/bin/pg_dump/t/002_pg_dump.pl              |   6 +-
 src/include/nodes/queryjumble.h               |   1 +
 src/include/optimizer/optimizer.h             |  10 +
 src/test/regress/expected/create_index.out    | 156 +++++++-
 src/test/regress/expected/join.out            |  62 ++-
 src/test/regress/expected/partition_prune.out | 215 ++++++++++-
 src/test/regress/expected/stats_ext.out       |  12 +-
 src/test/regress/expected/sysviews.out        |   3 +-
 src/test/regress/expected/tidscan.out         |  23 +-
 src/test/regress/sql/create_index.sql         |  35 ++
 src/test/regress/sql/join.sql                 |  10 +
 src/test/regress/sql/partition_prune.sql      |  22 ++
 src/test/regress/sql/tidscan.sql              |   6 +
 src/tools/pgindent/typedefs.list              |   2 +
 21 files changed, 923 insertions(+), 69 deletions(-)

index c355e8f3f7..48685bf6ff 100644

index 2c116c8728..0c5b4a011b 100644
 
+JumbleState *
index c37b416e24..b76896dfbf 100644
@@ -30,15 +30,6 @@
 #include "utils/syscache.h"

-/*
- * Proof attempts involving large arrays in ScalarArrayOpExpr nodes are
- * likely to require O(N^2) time, and more often than not fail anyway.
- * So we set an arbitrary limit on the number of array elements that
- * we will allow to be treated as an AND or OR clause.
- * XXX is it worth exposing this as a GUC knob?
- */
-#define MAX_SAOP_ARRAY_SIZE		100
-
 /*
  * To avoid redundant coding in predicate_implied_by_recurse and
  * predicate_refuted_by_recurse, we need to abstract out the notion of
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 9300c7b9ab..e9813ef6ab 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -16,12 +16,15 @@
 #include "postgres.h"
 
+#include "common/hashfn.h"
@@ -38,11 +41,13 @@
 
 /* GUC parameters */

 static Node *transformExprRecurse(ParseState *pstate, Node *expr);
@@ -99,6 +104,352 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname,
 static Node *make_nulltest_from_distinct(ParseState *pstate,
 										 A_Expr *distincta, Node *arg);
 
+	NodeTag	type;
+	uint64				exprHash;
+	Assert(IsA(data, Invalid));
+	(void) JumbleExpr(key->expr, &exprHash);
+	return hash_combine((uint32) exprHash,
+						hash_combine((uint32) key->opno,
+									 (uint32) key->exprtype));
+	Assert(IsA(src, Invalid));
+	dst_key->type = T_Invalid;
+	Assert(IsA(key1, Invalid));
+	Assert(IsA(key2, Invalid));
+	if (key1->opno == key2->opno &&
+		key1->exprtype == key2->exprtype &&
+			entries = lappend(entries, orqual);
+			entries = lappend(entries, orqual);
+				entries = lappend(entries, orqual);
+			entries = lappend(entries, orqual);
+			entries = lappend(entries, orqual);
+		hashkey.type = T_Invalid;
+			entry->consts = list_append_unique(entry->consts, const_expr);
+			entry->exprs = list_append_unique(entry->exprs, orqual);
+		if (!IsA(lfirst(lc), Invalid))
+			or_list = lappend(or_list, lfirst(lc));
+		if (list_length(entry->consts) == 1 ||
+			list_length(entry->consts) > MAX_SAOP_ARRAY_SIZE)
+			 * Only one element or more than MAX_SAOP_ARRAY_SIZE elements in
+			 * the class Return origin expression into.
+										 (Node *) entry->key.expr,
 
 /*
  * transformExpr -
@@ -1386,6 +1737,11 @@ transformBoolExpr(ParseState *pstate, BoolExpr *a)
 		args = lappend(args, arg);
 	}

index 45013582a7..c0ea53495b 100644

 
index 7b63c5cf71..0ee7154e08 100644

 
@@ -159,6 +160,15 @@ extern List *expand_function_arguments(List *args, bool include_out_arguments,

+/*
+ * Proof attempts involving large arrays in ScalarArrayOpExpr nodes are
+ * likely to require O(N^2) time, and more often than not fail anyway.
+ * So we set an arbitrary limit on the number of array elements that
+ * we will allow to be treated as an AND or OR clause.
+ * XXX is it worth exposing this as a GUC knob?
+ */
+#define MAX_SAOP_ARRAY_SIZE		100
+
index 79fa117cb5..606a4399f9 100644
 
 SELECT * FROM tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand < ANY ('{42,99,43}'::integer[])))
+                     Index Cond: (thousand < ANY ('{42,99,43}'::integer[]))
+         Recheck Cond: ((hundred = 42) AND ((thousand = ANY ('{42,41}'::integer[])) OR ((thousand = 99) AND (tenthous = 2))))

index 9605400021..d8018bef4f 100644
+   Join Filter: ((a.unique1 < 20) OR (a.unique1 = 3) OR ((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY ('{3,7}'::integer[])) AND (b.hundred = 4)))
+               Recheck Cond: ((unique1 < 20) OR (unique1 = 3) OR (unique1 = 1) OR (unique2 = ANY ('{3,7}'::integer[])))
 
index bf0657b9f2..1e153c3bb5 100644

 

 explain (costs off) select * from rlp where a = 1 or b = 'ab';
@@ -596,13 +618,13 @@ explain (costs off) select * from rlp where a < 1 or (a > 20 and a < 25);

 
+         Filter: ((a < 1) OR ((a > 20) AND (a < 25)))
+         Filter: ((a < 1) OR ((a > 20) AND (a < 25)))

 drop table hp2;

 

 

 

 
+
 
index cc3611e606..f4b6fee893 100644

---

From: jian he <[email-masked]>
Date: Sat, 9 Mar 2024 08:00:00 +0800

+ if (!IsA(lfirst(lc), Invalid))
+ {
+ or_list = lappend(or_list, lfirst(lc));
+ continue;
+ }
 Currently `IsA(lfirst(lc)` works.
but is this generally OK? I didn't find any other examples.
do you need do cast, like `(Node *) lfirst(lc);`

If I understand the logic correctly:
In `foreach(lc, args) ` if everything goes well, it will reach
`hashkey.type = T_Invalid;`
which will make `IsA(lfirst(lc), Invalid)` be true.
adding some comments to the above code would be great.

---

From: Andrei Lepikhov <[email-masked]>
Date: Mon, 11 Mar 2024 12:13:02 +0700

On 7/3/2024 21:51, Alexander Korotkov wrote:
> <[email-masked] <mailto:[email-masked]>> wrote:
Great!
> 1) Use hash_combine() to combine hash values.
Looks better
> 2) Upper limit the number of array elements by MAX_SAOP_ARRAY_SIZE.

I'm not convinced about this limit. The initial reason was to combine 
long lists of ORs into the array because such a transformation made at 
an early stage increases efficiency.
I understand the necessity of this limit in the array decomposition 
routine but not in the creation one.

> regression tests output have gone.
I agree that reducing the number of changes in regression tests looks 
better. But to achieve this, you introduced a hack that increases the 
complexity of the code. Is it worth it? Maybe it would be better to make 
one-time changes in tests instead of getting this burden on board. Or 
have you meant something more introducing the node type?

> worse.  I suggest Andrei and/or Alena should implement making array 
The fix Alena has made looks correct. But I urge you to think twice:
The optimizer doesn't care about duplicates, so why do we do it?
What's more, this optimization is intended to speed up queries with long 
OR lists. Using the list_append_unique() comparator on such lists could 
impact performance. I suggest sticking to the common rule and leaving 
the responsibility on the user's shoulders.
At least, we should do this optimization later, in one pass, with 
sorting elements before building the array. But what if we don't have a 
sort operator for the type?

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: Alexander Korotkov <[email-masked]>
Date: Mon, 11 Mar 2024 13:31:12 +0200

Hi Andrei,

Thank you for your response.

> routine but not in the creation one.

The comment near MAX_SAOP_ARRAY_SIZE says that this limit is because
N^2 algorithms could be applied to arrays.  Are you sure that's not
true for our case?

> have you meant something more introducing the node type?

For me the reason is not just a regression test.  The current code
keeps the original order of quals as much as possible.  The OR
transformation code reorders quals even in cases when it doesn't
eventually apply any optimization.  I don't think that's acceptable.
However, less hackery ways for this is welcome for sure.

> the responsibility on the user's shoulders.

I don't see why the optimizer doesn't care about duplicates for OR
lists.  As I showed before in an example, it successfully removes the
duplicate.  So, currently OR transformation clearly introduces a
regression in terms of selectivity estimation.  I think we should
evade that.

> sort operator for the type?

It was probably discussed before, but can we do our work later?  There
is a canonicalize_qual() which calls find_duplicate_ors().  This is
the place where currently duplicate OR clauses are removed.  Could our
OR-to-ANY transformation be just another call from
canonicalize_qual()?

------
Regards,
Alexander Korotkov

---

From: Andrei Lepikhov <[email-masked]>
Date: Mon, 11 Mar 2024 19:43:39 +0700

> N^2 algorithms could be applied to arrays.  Are you sure that's not
When you operate an array, indeed. But when we transform ORs to an 
array, not. Just check all the places in the optimizer and even the 
executor where we would pass along the list of ORs. This is why I think 
we should use this optimization even more intensively for huge numbers 
of ORs in an attempt to speed up the overall query.

> However, less hackery ways for this is welcome for sure.
Why is it unacceptable? Can the user implement some order-dependent 
logic with clauses, and will it be correct?
Otherwise, it is a matter of taste, and generally, this decision is up 
to you.
> regression in terms of selectivity estimation.  I think we should
I think you are right. It is probably a better place than any other to 
remove duplicates in an array. I just think we should sort and remove 
duplicates from entry->consts in one pass. Thus, this optimisation 
should be applied to sortable constants.

> OR-to-ANY transformation be just another call from
Hmm, we already tried to do it at that point. I vaguely recall some 
issues caused by this approach. Anyway, it should be done as quickly as 
possible to increase the effect of the optimization.

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: Alexander Korotkov <[email-masked]>
Date: Tue, 12 Mar 2024 17:20:12 +0200

> of ORs in an attempt to speed up the overall query.

Ok.

> Otherwise, it is a matter of taste, and generally, this decision is up

I think this is an important property that the user sees the quals in
the plan in the same order as they were in the query.  And if some
transformations are applied, then the order is saved as much as
possible.  I don't think we should sacrifice this property without
strong reasons.  A bit of code complexity is definitely not that
reason for me.

> should be applied to sortable constants.

Ok.

> possible to increase the effect of the optimization.

I think there were provided quite strong reasons why this shouldn't be
implemented at the parse analysis stage [1], [2], [3].  The
canonicalize_qual() looks quite appropriate place for that since it
does similar transformations.

Links.
1. https://www.postgresql.org/message-id/CA%2BTgmoZCgP6FrBQEusn4yaWm02XU8OPeoEMk91q7PRBgwaAkFw%40mail.gmail.com
2. https://www.postgresql.org/message-id/CAH2-Wzm2%3Dnf_JhiM3A2yetxRs8Nd2NuN3JqH%3Dfm_YWYd1oYoPg%40mail.gmail.com
3. https://www.postgresql.org/message-id/CA%2BTgmoaOiwMXBBTYknczepoZzKTp-Zgk5ss1%2BCuVQE-eFTqBmA%40mail.gmail.com

------
Regards,
Alexander Korotkov

---

From: Andrei Lepikhov <[email-masked]>
Date: Wed, 13 Mar 2024 12:52:50 +0700

>> should be applied to sortable constants.
New version of the patch set implemented all we have agreed on for now. 
We can return MAX_SAOP_ARRAY_SIZE constraint and Alena's approach to 
duplicates deletion for non-sortable cases at the end.
> does similar transformations.
Ok. Let's discuss these reasons. In Robert's opinion [1,3], we should do 
the transformation based on the cost model. But in the canonicalize_qual 
routine, we still make the transformation blindly. Moreover, the second 
patch reduces the weight of this reason, doesn't it? Maybe we shouldn't 
think about that as about optimisation but some 'general form of 
expression'?
Peter [2] worries about the possible transformation outcomes at this 
stage. But remember, we already transform clauses like ROW() IN (...) to 
a series of ORs here, so it is not an issue. Am I wrong?
Why did we discard the attempt with canonicalize_qual on the previous 
iteration? - The stage of parsing is much more native for building SAOP 
quals. We can reuse make_scalar_array_op and other stuff, for example.
During the optimisation stage, the only list partitioning machinery 
creates SAOP based on a list of constants. So, in theory, it is possible 
to implement. But do we really need to make the code more complex?
	
-- 
regards,
Andrei Lepikhov
Postgres Professional
From 86d969f2598a03b1eba84c0c064707de34ee60ac Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clauses to ANY expression.

 src/backend/parser/parse_expr.c               | 416 ++++++++++++++++++
 src/test/regress/expected/create_index.out    | 156 ++++++-
 src/test/regress/expected/partition_prune.out | 215 ++++++++-
 src/test/regress/sql/partition_prune.sql      |  22 +
 20 files changed, 974 insertions(+), 60 deletions(-)

index 58a603ac56..a965b43cc6 100644
@@ -8838,18 +8838,18 @@ insert into utrtest values (2, 'qux');
 
 -- The new values are concatenated with ' triggered !'
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 65a6e6c408..2de6ae301a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5472,6 +5472,23 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
       </listitem>
      </varlistentry>

+JumbleState *
index 9300c7b9ab..84038ab68f 100644

 /* GUC parameters */

 static Node *transformExprRecurse(ParseState *pstate, Node *expr);
@@ -99,6 +104,412 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname,
 static Node *make_nulltest_from_distinct(ParseState *pstate,
 										 A_Expr *distincta, Node *arg);
 
+	FunctionCallInfo	fcinfo;
+static FunctionCallInfo locfcinfo = NULL;
+saop_const_comparator(const ListCell *a, const ListCell *b)
+	Node   *leftop = (Node *) lfirst(a);
+	Node   *rightop = (Node *) lfirst(b);
+	Assert(IsA(leftop, Const) && IsA(rightop, Const));
+	locfcinfo->args[0].value = ((Const *) leftop)->constvalue;
+	locfcinfo->args[0].isnull = false;
+	locfcinfo->args[1].value = ((Const *) rightop)->constvalue;
+	locfcinfo->args[1].isnull = false;
+	return DatumGetInt32(FunctionCallInvoke(locfcinfo));
+saop_delete_duplicates(ParseState *pstate, OrClauseGroupEntry *entry,
+					   Oid scalar_type)
+	List	   *result;
+	locfcinfo = entry->fcinfo;
+	list_sort(entry->consts, saop_const_comparator);
+	result = list_make1(linitial(entry->consts));
+	for_each_from(lc, entry->consts, 1)
+		Node *node = (Node *) lfirst(lc);
+		if (equal(llast(result), node))
+		node = coerce_to_common_type(pstate, node, scalar_type, "IN");
+		result = lappend(result, node);
+		TypeCacheEntry		   *typentry;
+		typentry = lookup_type_cache(exprtype, TYPECACHE_CMP_PROC_FINFO);
+		 * Within this transformation we remove duplicates. To avoid
+		 * quadratic behavior we need to sort clauses after making a list of
+		 * elements. So, need sort operator to implement that.
+		if (!OidIsValid(typentry->cmp_proc_finfo.fn_oid))
+			/* Prepare funcall for sort comparator */
+			entry->fcinfo =
+						(FunctionCallInfo) palloc(SizeForFunctionCallInfo(2));
+			InitFunctionCallInfoData(*(entry->fcinfo),
+									 &typentry->cmp_proc_finfo, 2,
+									 typentry->typcollation, NULL, NULL);
+			 * Only one element returns origin expression into the BoolExpr args
+			 * list unchanged.
+			aexprs = saop_delete_duplicates(pstate, entry, scalar_type);
 
 /*
  * transformExpr -
@@ -1386,6 +1797,11 @@ transformBoolExpr(ParseState *pstate, BoolExpr *a)
 		args = lappend(args, arg);
 	}

index d77214795d..0c3f278420 100644
index 2244ee52f7..524516593f 100644
@@ -391,6 +391,7 @@

 

 
index 79fa117cb5..4a4dfb3bb5 100644
 
 SELECT * FROM tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand < ANY ('{42,43,99}'::integer[])))
+                     Index Cond: (thousand < ANY ('{42,43,99}'::integer[]))
+         Recheck Cond: ((hundred = 42) AND ((thousand = ANY ('{41,42}'::integer[])) OR ((thousand = 99) AND (tenthous = 2))))
+                           Index Cond: (thousand = ANY ('{41,42}'::integer[]))

 

 

 

 
 
 drop table hp2;

 

 
 
index f133b5a4ac..ad91abff7b 100644
+   TID Cond: (ctid = ANY ('{"(0,1)","(0,2)"}'::tid[]))
+   TID Cond: (ctid = ANY ('{"(0,1)","(0,2)"}'::tid[]))
 (2 rows)

 

+
 
index aa7a25b8f8..c6027c588d 100644
2.44.0

From c53b658521f9112fe4d9b7e9ccaba978fe422828 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Wed, 13 Mar 2024 12:26:02 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

index 2de6ae301a..0df56f44e3 100644
@@ -5485,6 +5485,9 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="

+/*

 

 
-			/*
 
-			/*
 
 		/*

+/*
 
+bool
index 35ab577501..72cf88dcbd 100644

+/*
index 33a6dceb0e..81399ec5b1 100644
 
+   Filter: ((stringu1 = ANY ('{A}'::name[])) AND (stringu1 = ANY ('{A,A}'::name[])))
 
index c6027c588d..bac4e3c2c8 100644
@@ -2143,6 +2143,7 @@ PredIterInfoData
2.44.0

---

From: Alexander Korotkov <[email-masked]>
Date: Wed, 13 Mar 2024 13:05:41 +0200

> to implement. But do we really need to make the code more complex?

As we currently do OR-to-ANY transformation at the parse stage, the
system catalog (including views, inheritance clauses, partial and
expression indexes, and others) would have a form depending on
enable_or_transformation at the moment of DDL execution.  I think this
is rather wrong.  The enable_or_transformation should be run-time
optimization which affects the resulting query plan, its result
shouldn't be persistent.

Regarding the ROW() IN (...) precedent.

1. AFAICS, this is not exactly an optimization.  This transformation
allows us to perform type matching individually for every value.
Therefore it allows the execute some queries which otherwise would end
up with error.
2. I don't think this is a sample of good design.  This is rather
hack, which is historically here, but we don't want to replicate this
experience.

Given all of the above, I think moving transformation to the
canonicalize_qual() would be the right way to go.

------
Regards,
Alexander Korotkov

---

From: Andrei Lepikhov <[email-masked]>
Date: Wed, 13 Mar 2024 19:16:09 +0700

> canonicalize_qual() would be the right way to go.
Ok, I will try to move the code.
I have no idea about the timings so far. I recall the last time I got 
bogged down in tons of duplicated code. I hope with an almost-ready 
sketch, it will be easier.

-- 
regards,
Andrei Lepikhov
Postgres Professional

---

From: Alexander Korotkov <[email-masked]>
Date: Thu, 14 Mar 2024 11:31:58 +0200

> sketch, it will be easier.

Thank you!  I'll be looking forward to the updated patch.

I also have notes about the bitmap patch.

/*
 * Building index paths over SAOP clause differs from the logic of OR
clauses.
 * Here we iterate across all the array elements and split them to SAOPs,
 * corresponding to different indexes. We must match each element to an
index.
 */

This covers the case I posted before.  But in order to fix all possible
cases we probably need to handle the SAOP clause in the same way as OR
clauses.  Check also this case.

Setup
create table t (a int not null, b int not null, c int not null);
insert into t (select 1, 1, i from generate_series(1,10000) i);
insert into t (select i, 2, 2 from generate_series(1,10000) i);
create index t_a_b_idx on t (a, b);
create statistics t_a_b_stat (mcv) on a, b from t;
create statistics t_b_c_stat (mcv) on b, c from t;
vacuum analyze t;

Plan with enable_or_transformation = on:
# explain select * from t where a = 1 and (b = 1 or b = 2) and c = 2;
                                  QUERY PLAN
------------------------------------------------------------------------------
 Bitmap Heap Scan on t  (cost=156.55..440.56 rows=5001 width=12)
   Recheck Cond: (a = 1)
   Filter: ((b = ANY ('{1,2}'::integer[])) AND (c = 2))
   ->  Bitmap Index Scan on t_a_b_idx  (cost=0.00..155.29 rows=10001
width=0)
         Index Cond: (a = 1)
(5 rows)

Plan with enable_or_transformation = off:
# explain select * from t where a = 1 and (b = 1 or b = 2) and c = 2;
                                  QUERY PLAN
------------------------------------------------------------------------------
 Bitmap Heap Scan on t  (cost=11.10..18.32 rows=5001 width=12)
   Recheck Cond: (((b = 1) AND (c = 2)) OR ((a = 1) AND (b = 2)))
   Filter: ((a = 1) AND (c = 2))
   ->  BitmapOr  (cost=11.10..11.10 rows=2 width=0)
         ->  Bitmap Index Scan on t_b_c_idx  (cost=0.00..4.30 rows=1
width=0)
               Index Cond: ((b = 1) AND (c = 2))
         ->  Bitmap Index Scan on t_a_b_idx  (cost=0.00..4.30 rows=1
width=0)
               Index Cond: ((a = 1) AND (b = 2))
(8 rows)

As you can see this case is not related to partial indexes.  Just no index
selective for the whole query.  However, splitting scan by the OR qual lets
use a combination of two selective indexes.

------
Regards,
Alexander Korotkov

---

From: Andrei Lepikhov <[email-masked]>
Date: Thu, 14 Mar 2024 17:11:15 +0700

On 14/3/2024 16:31, Alexander Korotkov wrote:
> On Wed, Mar 13, 2024 at 2:16 PM Andrei Lepikhov 
Okay, I moved the 0001-* patch to the prepqual.c module. See it in the 
attachment. I treat it as a transient patch.
It has positive outcomes as well as negative ones.
The most damaging result you can see in the partition_prune test:
partition pruning, in some cases, moved to the executor initialization 
stage. I guess, we should avoid it somehow in the next version.

-- 
regards,
Andrei Lepikhov
Postgres Professional
From 170f6871540025d0d1683750442e7af902b11a40 Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clauses to ANY expression.

 src/backend/optimizer/prep/prepqual.c         | 371 +++++++++++++++++-
 src/include/optimizer/optimizer.h             |   2 +
 src/test/regress/expected/partition_prune.out | 235 +++++++++--
 src/test/regress/expected/tidscan.out         |  19 +-
 19 files changed, 939 insertions(+), 61 deletions(-)

 
 
+JumbleState *
+JumbleExpr(Expr *expr, uint64 *queryId)
+{
+	JumbleState *jstate = NULL;
+
+	Assert(queryId != NULL);
+
+	jstate = (JumbleState *) palloc(sizeof(JumbleState));
+
+	/* Set up workspace for query jumbling */
+	jstate->jumble = (unsigned char *) palloc(JUMBLE_SIZE);
+	jstate->jumble_len = 0;
+	jstate->clocations_buf_size = 32;
+	jstate->clocations = (LocationLen *)
+		palloc(jstate->clocations_buf_size * sizeof(LocationLen));
+	jstate->clocations_count = 0;
+	jstate->highest_extern_param_id = 0;
+
+	/* Compute query ID */
+	_jumbleNode(jstate, (Node *) expr);
+	*queryId = DatumGetUInt64(hash_any_extended(jstate->jumble,
+												jstate->jumble_len,
+												0));
+
+	return jstate;
+}
+
 /*
  * Enables query identifier computation.
  *
diff --git a/src/backend/optimizer/prep/prepqual.c b/src/backend/optimizer/prep/prepqual.c
index cbcf83f847..7ee585dc89 100644
--- a/src/backend/optimizer/prep/prepqual.c
+++ b/src/backend/optimizer/prep/prepqual.c
@@ -31,16 +31,25 @@
 
 #include "postgres.h"
 
+#include "catalog/namespace.h"
+#include "catalog/pg_operator.h"
+#include "common/hashfn.h"
 #include "nodes/makefuncs.h"
 #include "nodes/nodeFuncs.h"
+#include "nodes/queryjumble.h"
 #include "optimizer/optimizer.h"
+#include "parser/parse_coerce.h"
+#include "parser/parse_oper.h"
 #include "utils/lsyscache.h"
+#include "utils/syscache.h"

 static List *pull_ands(List *andlist);
 static List *pull_ors(List *orlist);
 static Expr *find_duplicate_ors(Expr *qual, bool is_check);
 static Expr *process_duplicate_ors(List *orlist);
+static List *or_transformation(List *orlist);

 /*
@@ -266,6 +275,354 @@ negate_clause(Node *node)
 	return (Node *) make_notclause((Expr *) node);
 }
 
+	Oid		consttype;
+	Oid		inputcollid; /* XXX: Could we lookup for common collation? */
+							hash_combine((uint32) key->consttype,
+										 (uint32) key->inputcollid)));
+	dst_key->consttype = src_key->consttype;
+	dst_key->inputcollid = src_key->inputcollid;
+		key1->consttype == key2->consttype &&
+		key1->inputcollid == key2->inputcollid &&
+ * or_transformation -
+or_transformation(List *orlist)
+	List				   *neworlist = NIL;
+	int 					len_ors = list_length(orlist);
+	foreach(lc, orlist)
+		Oid						consttype;
+		consttype = exprType(const_expr);
+			type_is_rowtype(consttype) ||
+		hashkey.consttype = consttype;
+		hashkey.inputcollid = exprCollation(const_expr);
+			neworlist = lappend(neworlist, lfirst(lc));
+			neworlist = list_concat(neworlist, entry->exprs);
+		scalar_type = entry->key.consttype;
+			foreach(lc, entry->consts)
+				node = coerce_to_common_type(NULL, node, scalar_type,
+											 "OR ANY Transformation");
+				aexprs = lappend(aexprs, node);
+					make_scalar_array_op(NULL,
+			saopexpr->inputcollid = entry->key.inputcollid;
+			neworlist = lappend(neworlist, (void *) saopexpr);
+			neworlist = list_concat(neworlist, entry->exprs);
+	return neworlist;
+}
 
 /*
  * canonicalize_qual
@@ -604,7 +961,15 @@ process_duplicate_ors(List *orlist)
 	 * If no winners, we can't transform the OR
 	 */
 	if (winners == NIL)
-		return make_orclause(orlist);
+	{
+		/* Make an attempt to group similar OR clauses into ANY operation */
+		if (enable_or_transformation && list_length(orlist) > 1)
+			orlist = or_transformation(orlist);
+
+		/* Transformation could group all OR clauses to a single SAOP */
+		return (list_length(orlist) == 1) ?
+							(Expr *) linitial(orlist) : make_orclause(orlist);
+	}
 
 	/*
 	 * Generate new OR list consisting of the remaining sub-clauses.
@@ -651,6 +1016,10 @@ process_duplicate_ors(List *orlist)
 		}
 	}
 
+	/* Make an attempt to group similar OR clauses into ANY operation */
+	if (enable_or_transformation && list_length(neworlist) > 1)
+		neworlist = or_transformation(neworlist);
+
 	/*
 	 * Append reduced OR to the winners list, if it's not degenerate, handling
 	 * the special case of one element correctly (can that really happen?).
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 57d9de4dd9..d4f082ec62 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1026,6 +1026,17 @@ struct config_bool ConfigureNamesBool[] =
 		true,
 		NULL, NULL, NULL
 	},
+	{
+		{"enable_or_transformation", PGC_USERSET, QUERY_TUNING_OTHER,
+			gettext_noop("Transform a sequence of OR clauses to an array expression."),
+			gettext_noop("The planner will replace expression like 'x=c1 OR x=c2 ..'"
+						 "to the expression 'x = ANY(ARRAY[c1,c2,..])'"),
+			GUC_EXPLAIN
+		},
+		&enable_or_transformation,
+		true,
+		NULL, NULL, NULL
+	},
 	{
 		/*
 		 * Not for general use --- used by SET SESSION AUTHORIZATION and SET
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2244ee52f7..524516593f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -391,6 +391,7 @@
 # - Planner Method Configuration -

 
 extern PGDLLIMPORT bool query_id_enabled;
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
index 7b63c5cf71..32eec0b27c 100644
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -133,6 +133,8 @@ extern void extract_query_dependencies(Node *query,
 
 /* in prep/prepqual.c: */
 
+extern PGDLLIMPORT bool enable_or_transformation;
+
 extern Node *negate_clause(Node *node);
 extern Expr *canonicalize_qual(Expr *qual, bool is_check);
 
index 70ab47a92f..d144aeec2a 100644
+------------------------------------------------------------------------
+   Index Cond: ((thousand = 42) AND (tenthous = ANY (ARRAY[1, 3, 42])))
+-----------------------------------------------------------------------------
+         Recheck Cond: ((hundred = 42) AND (thousand = ANY (ARRAY[42, 99])))
+                     Index Cond: (thousand = ANY (ARRAY[42, 99]))
+------------------------------------------------------------------------
+   Index Cond: ((thousand = 42) AND (tenthous = ANY (ARRAY[1, 3, 42])))
+(2 rows)
 
 SELECT * FROM tenk1
   WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
@@ -1861,28 +1893,116 @@ SELECT * FROM tenk1
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
-                                   QUERY PLAN                                    
----------------------------------------------------------------------------------
+                                 QUERY PLAN                                  
+-----------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand = ANY (ARRAY[42, 99])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = ANY (ARRAY[42, 99]))
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+ count 
+-------
+    10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand < 42 OR thousand < 99 OR 43 > thousand OR 42 > thousand);
+                                     QUERY PLAN                                      
+-------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand < ANY (ARRAY[42, 99, 43, 42])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand < ANY (ARRAY[42, 99, 43, 42]))
+(8 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+                                           QUERY PLAN                                            
+-------------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: (((thousand = 42) AND (tenthous = ANY (ARRAY[1, 3]))) OR (thousand = 41))
+         ->  BitmapOr
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: ((thousand = 42) AND (tenthous = ANY (ARRAY[1, 3])))
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = 41)
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+ count 
+-------
+    10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+                                                      QUERY PLAN                                                      
+----------------------------------------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: (((hundred = 42) AND ((thousand = ANY (ARRAY[42, 99])) OR (tenthous < 2))) OR (thousand = 41))
+         ->  BitmapOr
+               ->  BitmapAnd
+                     ->  Bitmap Index Scan on tenk1_hundred
+                           Index Cond: (hundred = 42)
+                     ->  BitmapOr
+                           ->  Bitmap Index Scan on tenk1_thous_tenthous
+                                 Index Cond: (thousand = ANY (ARRAY[42, 99]))
+                           ->  Bitmap Index Scan on tenk1_thous_tenthous
+                                 Index Cond: (tenthous < 2)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = 41)
+(14 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+ count 
+-------
+    20
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
+                                                      QUERY PLAN                                                       
+-----------------------------------------------------------------------------------------------------------------------
  Aggregate
    ->  Bitmap Heap Scan on tenk1
-         Recheck Cond: ((hundred = 42) AND ((thousand = 42) OR (thousand = 99)))
+         Recheck Cond: ((hundred = 42) AND ((thousand = ANY (ARRAY[42, 41])) OR ((thousand = 99) AND (tenthous = 2))))
          ->  BitmapAnd
                ->  Bitmap Index Scan on tenk1_hundred
                      Index Cond: (hundred = 42)
                ->  BitmapOr
                      ->  Bitmap Index Scan on tenk1_thous_tenthous
-                           Index Cond: (thousand = 42)
+                           Index Cond: (thousand = ANY (ARRAY[42, 41]))
                      ->  Bitmap Index Scan on tenk1_thous_tenthous
-                           Index Cond: (thousand = 99)
+                           Index Cond: ((thousand = 99) AND (tenthous = 2))
 (11 rows)

index 9605400021..3368629c9c 100644
+-----------------------------------------------------------------------------------------------------------------
+   Join Filter: (((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY (ARRAY[3, 7])) AND (b.hundred = 4)))
+               Recheck Cond: ((unique1 = 1) OR (unique2 = ANY (ARRAY[3, 7])))
+                           Index Cond: (unique2 = ANY (ARRAY[3, 7]))
+-----------------------------------------------------------------------------------------------------------------
+   Join Filter: (((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY (ARRAY[3, 7])) AND (b.hundred = 4)))
+               Recheck Cond: ((unique1 = 1) OR (unique2 = ANY (ARRAY[3, 7])))
+                           Index Cond: (unique2 = ANY (ARRAY[3, 7]))
+--------------------------------------------------------------------------------------------------------------------------------------------------------
+   Join Filter: ((a.unique1 < 20) OR (a.unique1 = 3) OR ((a.unique1 = 1) AND (b.unique1 = 2)) OR ((a.unique2 = ANY (ARRAY[3, 7])) AND (b.hundred = 4)))
+               Recheck Cond: ((unique1 < 20) OR (unique1 = 3) OR (unique1 = 1) OR (unique2 = ANY (ARRAY[3, 7])))
+                           Index Cond: (unique2 = ANY (ARRAY[3, 7]))
+(15 rows)
 
+RESET enable_or_transformation;
 --
 -- test placement of movable quals in a parameterized join tree
 --
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index bf0657b9f2..ce6d6f40fb 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -82,25 +82,51 @@ explain (costs off) select * from lp where a is null;
 (2 rows)
 
+   Subplans Removed: 4
-(5 rows)
+(6 rows)
 
+   Subplans Removed: 3
-(5 rows)
+(6 rows)
+   Subplans Removed: 4
+(6 rows)
 
+   Subplans Removed: 3
+(6 rows)
@@ -515,11 +541,13 @@ explain (costs off) select * from rlp where a <= 31;
 (27 rows)
 
- Seq Scan on rlp2 rlp
-(2 rows)
+------------------------------------------------
+   Subplans Removed: 1
 
 explain (costs off) select * from rlp where a = 1 or b = 'ab';
                       QUERY PLAN                       
@@ -596,14 +624,15 @@ explain (costs off) select * from rlp where a < 1 or (a > 20 and a < 25);
 
+   Subplans Removed: 2
-(5 rows)
+(6 rows)
 
 explain (costs off) select * from rlp3 where a = 20;   /* empty */
         QUERY PLAN        
@@ -671,6 +700,166 @@ explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a =
          Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
 (11 rows)
 
+------------------------------------------------
+   Subplans Removed: 1
+   Subplans Removed: 2
+(6 rows)
@@ -2072,10 +2261,10 @@ explain (costs off) select * from hp where a = 1 and b = 'abcde';

 drop table hp2;

 

 
 
 -- There are always wait event descriptions for various types.
 select type, count(*) > 0 as ok FROM pg_wait_events
diff --git a/src/test/regress/expected/tidscan.out b/src/test/regress/expected/tidscan.out
index f133b5a4ac..4782cb3e06 100644
--- a/src/test/regress/expected/tidscan.out
+++ b/src/test/regress/expected/tidscan.out
@@ -46,7 +46,7 @@ SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
                           QUERY PLAN                          
 --------------------------------------------------------------
  Tid Scan on tidscan
-   TID Cond: ((ctid = '(0,2)'::tid) OR ('(0,1)'::tid = ctid))
+   TID Cond: (ctid = ANY (ARRAY['(0,2)'::tid, '(0,1)'::tid]))
 (2 rows)
 
 SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
@@ -56,6 +56,23 @@ SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
  (0,2) |  2
 (2 rows)
 
+--------------------------------------------------------------
+   TID Cond: (ctid = ANY (ARRAY['(0,2)'::tid, '(0,1)'::tid]))

 
+

---

From: Alexander Korotkov <[email-masked]>
Date: Thu, 14 Mar 2024 12:39:42 +0200

> stage. I guess, we should avoid it somehow in the next version.

Thank you, Andrei.  Looks like a very undesirable side effect.  Do you
have any idea why it happens?  Partition pruning should work correctly
for both transformed and non-transformed quals, why does
transformation hurt it?

------
Regards,
Alexander Korotkov

---

From: Andrei Lepikhov <[email-masked]>
Date: Fri, 15 Mar 2024 12:05:02 +0700

> for both transformed and non-transformed quals, why does
Now we have the v23-0001-* patch with all issues resolved. The last one 
which caused execution stage pruning was about necessity to evaluate 
SAOP expression right after transformation. In previous version the core 
executed it on transformed expressions.

 > As you can see this case is not related to partial indexes.  Just no
 > index selective for the whole query.  However, splitting scan by the
 > OR qual lets use a combination of two selective indexes.
Thanks for the case. I will try to resolve it.

-- 
regards,
Andrei Lepikhov
Postgres Professional
From 156c00c820a38e5e1856f07363af87b3109b5d77 Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clauses to ANY expression.

 src/backend/optimizer/prep/prepqual.c         | 374 +++++++++++++++++-
 src/test/regress/expected/partition_prune.out | 215 +++++++++-
 19 files changed, 929 insertions(+), 58 deletions(-)

 
 
+JumbleState *
index cbcf83f847..a127dafec7 100644
 
 #include "postgres.h"

 

 /*
@@ -266,6 +275,357 @@ negate_clause(Node *node)
 	return (Node *) make_notclause((Expr *) node);
 }
 
+ *	  expressions to an SAOP operation.
+ *	  To group quals, inputcollid, opno and constype of the OR OpExpr quals must
+ *	  be equal too.
+			 * Try to cast this expression to Const. Due to current strict
+			 * transformation rules it should be done [almost] every time.
+			newa = (ArrayExpr *) eval_const_expressions(NULL, (Node *) newa);
 
 /*
  * canonicalize_qual
@@ -604,7 +964,15 @@ process_duplicate_ors(List *orlist)
 
 	/*
 	 * Generate new OR list consisting of the remaining sub-clauses.
@@ -651,6 +1019,10 @@ process_duplicate_ors(List *orlist)
 		}
 	}

 

 
 
index 70ab47a92f..66e9a395e0 100644
 
 SELECT * FROM tenk1

 

 

 

 
 
 drop table hp2;

 

 

 

 
+

---

From: Andrei Lepikhov <[email-masked]>
Date: Tue, 19 Mar 2024 12:16:59 +0700

On 14/3/2024 16:31, Alexander Korotkov wrote:
> index selective for the whole query.  However, splitting scan by the OR 
> qual lets use a combination of two selective indexes.
I have rewritten the 0002-* patch according to your concern. A candidate 
and some thoughts are attached.
As I see, we have a problem here: expanding each array and trying to 
apply an element to each index can result in a lengthy planning stage.
Also, an index scan with the SAOP may potentially be more effective than 
with the list of OR clauses.
Originally, the transformation's purpose was to reduce a query's 
complexity and the number of optimization ways to speed up planning and 
(sometimes) execution. Here, we reduce planning complexity only in the 
case of an array size larger than MAX_SAOP_ARRAY_SIZE.
Maybe we can fall back to the previous version of the second patch, 
keeping in mind that someone who wants to get maximum profit from the 
BitmapOr scan of multiple indexes can just disable this optimization, 
enabling deep search of the most optimal scanning way?
As a compromise solution, I propose adding one more option to the 
previous version: if an element doesn't fit any partial index, try to 
cover it with a plain index.
In this case, we still do not guarantee the most optimal fit of elements 
to the set of indexes, but we speed up planning. Does that make sense?

-- 
regards,
Andrei Lepikhov
Postgres Professional
From d2d8944fc83ccd090653c1b15703a2c3ba096fa9 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Wed, 13 Mar 2024 12:26:02 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

Likewise OR clauses, discover SAOP array and try to split its elements
between smaller sized arrays to fit a set of partial indexes.
---
 doc/src/sgml/config.sgml                   |   3 +
 src/backend/optimizer/path/indxpath.c      |  74 +++++-
 src/backend/optimizer/util/predtest.c      |  37 +++
 src/backend/optimizer/util/restrictinfo.c  |  13 +
 src/include/optimizer/optimizer.h          |   3 +
 src/include/optimizer/restrictinfo.h       |   1 +
 src/test/regress/expected/create_index.out |  24 +-
 src/test/regress/expected/select.out       | 280 +++++++++++++++++++++
 src/test/regress/sql/select.sql            |  82 ++++++
 9 files changed, 500 insertions(+), 17 deletions(-)

index 32c6a8bbdc..96685429de 100644
 
@@ -1220,11 +1221,66 @@ build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel,
 	return result;
 }
 
+/*
+ * Expand SAOP node to use it in bitmapscan path building routine.
+ *
+ * If RestrictInfo is an OR bool expression, extract each SAOP from the list of
+ * arguments, if possible.
+ * Working jointly with the TransformOrExprToANY routine, it provides a user
+ * with some sort of independence of the query plan from the approach to writing
+ * alternatives for the same entity in the WHERE section.
+ */
+static List *
+extract_saop_ors(PlannerInfo *root, RestrictInfo *rinfo)
+{
+	List		   *orlist = NIL;
+	List		   *result = NIL;
+	ListCell	   *lc;
+
+	Assert(IsA(rinfo, RestrictInfo));
+
+	if (restriction_is_or_clause(rinfo))
+		orlist = ((BoolExpr *) rinfo->orclause)->args;
+
+	if (!enable_or_transformation)
+		return orlist;
+
+	if (restriction_is_saop_clause(rinfo))
+	{
+		result = transform_saop_to_ors(root, rinfo);
+		return (result == NIL) ? list_make1(rinfo) : result;
+	}
+
+	foreach(lc, orlist)
+	{
+		Expr *expr = (Expr *) lfirst(lc);
+
+		if (IsA(expr, RestrictInfo) && restriction_is_saop_clause((RestrictInfo *) expr))
+		{
+			List *sublist;
+
+			sublist = extract_saop_ors(root, (RestrictInfo *) lfirst(lc));
+			if (sublist != NIL)
+			{
+				result = list_concat(result, sublist);
+				continue;
+			}
+
+			/* Need to return expr to the result list */
+		}
+
+		result = lappend(result, expr);
+	}
+
+	return result;
+}
+
 /*
  * generate_bitmap_or_paths
- *		Look through the list of clauses to find OR clauses, and generate
- *		a BitmapOrPath for each one we can handle that way.  Return a list
- *		of the generated BitmapOrPaths.
+ *		Look through the list of clauses to find OR and SAOP clauses, and
+ *		Each saop clause are splitted to be covered by partial indexes.
+ *		generate a BitmapOrPath for each one we can handle that way.
+ *		Return a list of the generated BitmapOrPaths.
  *
  * other_clauses is a list of additional clauses that can be assumed true
  * for the purpose of generating indexquals, but are not to be searched for
@@ -1247,20 +1303,24 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel,
 	foreach(lc, clauses)
 	{
 		RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc);
-		List	   *pathlist;
+		List	   *pathlist = NIL;
 		Path	   *bitmapqual;
 		ListCell   *j;
+		List	   *orlist = NIL;
 
-		/* Ignore RestrictInfos that aren't ORs */
-		if (!restriction_is_or_clause(rinfo))
+		orlist = extract_saop_ors(root, rinfo);
+		if (orlist == NIL)
+			/* Ignore RestrictInfo that doesn't provide ORs */
 			continue;
 
+		all_clauses = list_delete(all_clauses, rinfo);
+
 		/*
 		 * We must be able to match at least one index to each of the arms of
 		 * the OR, else we can't use it.
 		 */
 		pathlist = NIL;
-		foreach(j, ((BoolExpr *) rinfo->orclause)->args)
+		foreach(j, orlist)
 		{
 			Node	   *orarg = (Node *) lfirst(j);
 			List	   *indlist;
diff --git a/src/backend/optimizer/util/predtest.c b/src/backend/optimizer/util/predtest.c
index c37b416e24..e159493a21 100644
--- a/src/backend/optimizer/util/predtest.c
+++ b/src/backend/optimizer/util/predtest.c
@@ -111,6 +111,43 @@ static bool operator_same_subexprs_lookup(Oid pred_op, Oid clause_op,
 static Oid	get_btree_test_op(Oid pred_op, Oid clause_op, bool refute_it);
 static void InvalidateOprProofCacheCallBack(Datum arg, int cacheid, uint32 hashvalue);
 
+/*
+ * Expand a SAOP operation into the list of OR expressions
+ */
+List *
+transform_saop_to_ors(PlannerInfo *root, RestrictInfo *rinfo)
+{
+	PredIterInfoData	clause_info;
+	List			   *orlist = NIL;
+	Node			   *saop = (Node *) rinfo->clause;
+
+	Assert(IsA(saop, ScalarArrayOpExpr));
+
+	if (predicate_classify(saop, &clause_info) != CLASS_OR)
+		return NIL;
+
+	iterate_begin(pitem, saop, clause_info)
+	{
+		RestrictInfo   *rinfo1;
+
+		/* Predicate is found. Add the elem to the saop clause */
+		Assert(IsA(pitem, OpExpr));
+
+		/* Extract constant from the expression */
+		rinfo1 = make_restrictinfo(root, (Expr *) copyObject(pitem),
+								   rinfo->is_pushed_down,
+								   rinfo->has_clone, rinfo->is_clone,
+								   rinfo->pseudoconstant,
+								   rinfo->security_level,
+								   rinfo->required_relids,
+								   rinfo->incompatible_relids,
+								   rinfo->outer_relids);
+		orlist = lappend(orlist, rinfo1);
+	}
+	iterate_end(clause_info);
+
+	return orlist;
+}
 
 /*
  * predicate_implied_by
 
+bool
index 32eec0b27c..492368d9e2 100644
@@ -23,6 +23,7 @@
 #define OPTIMIZER_H
 
 #include "nodes/parsenodes.h"
+#include "optimizer/restrictinfo.h"
 
 /*
  * We don't want to include nodes/pathnodes.h here, because non-planner
@@ -161,6 +162,8 @@ extern List *expand_function_arguments(List *args, bool include_out_arguments,

+
+extern List *transform_saop_to_ors(PlannerInfo *root, RestrictInfo *rinfo);
 extern bool predicate_implied_by(List *predicate_list, List *clause_list,
 								 bool weak);
 extern bool predicate_refuted_by(List *predicate_list, List *clause_list,
diff --git a/src/include/optimizer/restrictinfo.h b/src/include/optimizer/restrictinfo.h
index 1b42c832c5..2cd5fbf943 100644
--- a/src/include/optimizer/restrictinfo.h
+++ b/src/include/optimizer/restrictinfo.h
@@ -34,6 +34,7 @@ extern RestrictInfo *make_restrictinfo(PlannerInfo *root,
 									   Relids outer_relids);
 extern RestrictInfo *commute_restrictinfo(RestrictInfo *rinfo, Oid comm_op);
 extern bool restriction_is_or_clause(RestrictInfo *restrictinfo);
+extern bool restriction_is_saop_clause(RestrictInfo *restrictinfo);
 extern bool restriction_is_securely_promotable(RestrictInfo *restrictinfo,
 											   RelOptInfo *rel);
 extern List *get_actual_clauses(List *restrictinfo_list);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 66e9a395e0..3f57d5c968 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1952,23 +1952,25 @@ SELECT count(*) FROM tenk1
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
-                                                         QUERY PLAN                                                          
------------------------------------------------------------------------------------------------------------------------------
+                                                       QUERY PLAN                                                       
+------------------------------------------------------------------------------------------------------------------------
  Aggregate
    ->  Bitmap Heap Scan on tenk1
-         Recheck Cond: (((hundred = 42) AND ((thousand = ANY ('{42,99}'::integer[])) OR (tenthous < 2))) OR (thousand = 41))
+         Recheck Cond: (((hundred = 42) AND ((thousand = 42) OR (thousand = 99) OR (tenthous < 2))) OR (thousand = 41))
          ->  BitmapOr
                ->  BitmapAnd
                      ->  Bitmap Index Scan on tenk1_hundred
                            Index Cond: (hundred = 42)
                      ->  BitmapOr
                            ->  Bitmap Index Scan on tenk1_thous_tenthous
-                                 Index Cond: (thousand = ANY ('{42,99}'::integer[]))
+                                 Index Cond: (thousand = 42)
+                           ->  Bitmap Index Scan on tenk1_thous_tenthous
+                                 Index Cond: (thousand = 99)
                            ->  Bitmap Index Scan on tenk1_thous_tenthous
                                  Index Cond: (tenthous < 2)
                ->  Bitmap Index Scan on tenk1_thous_tenthous
                      Index Cond: (thousand = 41)
-(14 rows)
+(16 rows)
 
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
@@ -1980,20 +1982,22 @@ SELECT count(*) FROM tenk1
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
-                                                          QUERY PLAN                                                          
-------------------------------------------------------------------------------------------------------------------------------
+                                                       QUERY PLAN                                                        
+-------------------------------------------------------------------------------------------------------------------------
  Aggregate
    ->  Bitmap Heap Scan on tenk1
-         Recheck Cond: ((hundred = 42) AND ((thousand = ANY ('{42,41}'::integer[])) OR ((thousand = 99) AND (tenthous = 2))))
+         Recheck Cond: ((hundred = 42) AND ((thousand = 42) OR (thousand = 41) OR ((thousand = 99) AND (tenthous = 2))))
          ->  BitmapAnd
                ->  Bitmap Index Scan on tenk1_hundred
                      Index Cond: (hundred = 42)
                ->  BitmapOr
                      ->  Bitmap Index Scan on tenk1_thous_tenthous
-                           Index Cond: (thousand = ANY ('{42,41}'::integer[]))
+                           Index Cond: (thousand = 42)
+                     ->  Bitmap Index Scan on tenk1_thous_tenthous
+                           Index Cond: (thousand = 41)
                      ->  Bitmap Index Scan on tenk1_thous_tenthous
                            Index Cond: ((thousand = 99) AND (tenthous = 2))
-(11 rows)
+(13 rows)
 
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
diff --git a/src/test/regress/expected/select.out b/src/test/regress/expected/select.out
index 33a6dceb0e..d0352a054c 100644
--- a/src/test/regress/expected/select.out
+++ b/src/test/regress/expected/select.out
@@ -907,6 +907,286 @@ select unique1, unique2 from onek2
        0 |     998
 (2 rows)
 
+---------------------------------------------------------------------------------------------------------------------
+   Recheck Cond: (((unique2 < 1) AND (stringu1 < 'B'::name)) OR ((stringu1 = 'J'::name) AND (stringu1 < 'Z'::name)))
+               Index Cond: ((stringu1 = 'J'::name) AND (stringu1 < 'Z'::name))
+WHERE unique1 IN (1, PI()::integer);
+----------------------------------------------------------------------------------------------------------------------------------------------------------------
+   Recheck Cond: ((stringu1 < 'B'::name) OR (stringu1 = 'J'::name) OR (unique1 = 3) OR (unique1 = 4) OR (stringu1 = 'J'::name))
+               Index Cond: (unique1 = 4)
+(13 rows)
index 019f1e7673..37685694e6 100644
 
+WHERE unique1 IN (1, PI()::integer);

---

From: Alexander Korotkov <[email-masked]>
Date: Thu, 28 Mar 2024 15:54:57 +0200

> to the set of indexes, but we speed up planning. Does that make sense?

Thank you for your research Andrei.  Now things get more clear on the
advantages and disadvantages of this transformation.

The current patch has a boolean guc enable_or_transformation.
However, when we have just a few ORs to be transformated, then we
should get less performance gain from the transformation and higher
chances to lose a good bitmap scan plan from that.  When there is a
huge list of ORs to be transformed, then the performance gain is
greater and it is less likely we could lose a good bitmap scan plan.

What do you think about introducing a GUC threshold value: the minimum
size of list to do OR-to-ANY transformation?
min_list_or_transformation or something.

------
Regards,
Alexander Korotkov

---

From: Andrei Lepikhov <[email-masked]>
Date: Mon, 1 Apr 2024 13:37:58 +0700

> min_list_or_transformation or something.
I labelled it or_transformation_limit (see in attachment). Feel free to 
rename it.
It's important to note that the limiting GUC doesn't operate 
symmetrically for forward, OR -> SAOP, and backward SAOP -> OR 
operations. In the forward case, it functions as you've proposed. 
However, in the backward case, we only check whether the feature is 
enabled or not. This is due to our existing limitation, 
MAX_SAOP_ARRAY_SIZE, and the fact that we can't match the length of the 
original OR list with the sizes of the resulting SAOPs. For instance, a 
lengthy OR list with 100 elements can be transformed into 3 SAOPs, each 
with a size of around 30 elements.
One aspect that requires attention is the potential inefficiency of our 
OR -> ANY transformation when we have a number of elements less than 
MAX_SAOP_ARRAY_SIZE. This is because we perform a reverse transformation 
ANY -> OR at the stage of generating bitmap scans. If the BitmapScan 
path dominates, we may have done unnecessary work. Is this an occurrence 
that we should address?
But the concern above may just be a point of improvement later: We can 
add one more strategy to the optimizer: testing each array element as an 
OR clause; we can also provide a BitmapOr path, where SAOP is covered 
with a minimal number of partial indexes (likewise, previous version).

-- 
regards,
Andrei Lepikhov
Postgres Professional
From e42a7111a12ef82eecdb2e692d65e7ba6e43ad79 Mon Sep 17 00:00:00 2001
From: Alena Rybakina <[email-masked]>
Date: Fri, 2 Feb 2024 22:01:09 +0300
Subject: [PATCH 1/2] Transform OR clauses to ANY expression.

Replace (expr op C1) OR (expr op C2) ... with expr op ANY(ARRAY[C1, C2, ...]) on the
preliminary stage of optimization when we are still working with the
expression tree.
Here C<X> is a constant expression, 'expr' is non-constant expression, 'op' is
an operator which returns boolean result and has a commuter (for the case of
reverse order of constant and non-constant parts of the expression,
like 'CX op expr').
Sometimes it can lead to not optimal plan. But we think it is better to have
array of elements instead of a lot of OR clauses. Here is a room for further
optimizations on decomposing that array into more optimal parts.
Authors: Alena Rybakina <[email-masked]>, Andrey Lepikhov <[email-masked]>
Reviewed-by: Peter Geoghegan <[email-masked]>, Ranier Vilela <[email-masked]>
Reviewed-by: Alexander Korotkov <[email-masked]>, Robert Haas <[email-masked]>
Reviewed-by: jian he <[email-masked]>
---
 .../postgres_fdw/expected/postgres_fdw.out    |   8 +-
 doc/src/sgml/config.sgml                      |  18 +
 src/backend/nodes/queryjumblefuncs.c          |  27 ++
 src/backend/optimizer/prep/prepqual.c         | 379 +++++++++++++++++-
 src/backend/utils/misc/guc_tables.c           |  13 +
 src/backend/utils/misc/postgresql.conf.sample |   1 +
 src/include/nodes/queryjumble.h               |   1 +
 src/include/optimizer/optimizer.h             |   2 +
 src/test/regress/expected/create_index.out    | 172 +++++++-
 src/test/regress/expected/join.out            |  60 ++-
 src/test/regress/expected/partition_prune.out | 211 +++++++++-
 src/test/regress/expected/stats_ext.out       |  12 +-
 src/test/regress/expected/tidscan.out         |  21 +-
 src/test/regress/sql/create_index.sql         |  44 ++
 src/test/regress/sql/join.sql                 |   8 +
 src/test/regress/sql/partition_prune.sql      |  18 +
 src/test/regress/sql/tidscan.sql              |   4 +
 src/tools/pgindent/typedefs.list              |   2 +
 18 files changed, 950 insertions(+), 51 deletions(-)

index b7af86d351..277ef3f385 100644
@@ -8853,18 +8853,18 @@ insert into utrtest values (2, 'qux');
 
 -- The new values are concatenated with ' triggered !'
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5468637e2e..f384edde05 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5472,6 +5472,24 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
       </listitem>
      </varlistentry>
 
+     <varlistentry id="guc-enable-or-transformation" xreflabel="or_transformation_limit">
+      <term><varname>or_transformation_limit</varname> (<type>boolean</type>)
+       <indexterm>
+        <primary><varname>or_transformation_limit</varname> configuration parameter</primary>
+       </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Sets the minimum length of arguments in an OR expression exceeding which
+        planner will try to lookup and group multiple similar OR expressions to
+        ANY (<xref linkend="functions-comparisons-any-some"/>) expressions.
+        The grouping technique of this transformation is based on the equivalence of variable sides.
+        One side of such an expression must be a constant clause, and the other must contain a variable clause.
+        The default is <literal>on</literal>.
+        </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry id="guc-enable-parallel-append" xreflabel="enable_parallel_append">
       <term><varname>enable_parallel_append</varname> (<type>boolean</type>)
       <indexterm>
diff --git a/src/backend/nodes/queryjumblefuncs.c b/src/backend/nodes/queryjumblefuncs.c
index be823a7f8f..edcb00bd31 100644
--- a/src/backend/nodes/queryjumblefuncs.c
+++ b/src/backend/nodes/queryjumblefuncs.c
@@ -141,6 +141,33 @@ JumbleQuery(Query *query)
 	return jstate;
 }
 
+JumbleState *
index cbcf83f847..e27b07e6a9 100644
 
 #include "postgres.h"

+int		or_transformation_limit = 0;

 /*
 
+	Assert(or_transformation_limit >= 0 && len_ors > or_transformation_limit);
 
 /*
  * canonicalize_qual
@@ -604,7 +964,19 @@ process_duplicate_ors(List *orlist)
 	 * If no winners, we can't transform the OR
 	 */
 	if (winners == NIL)
-		return make_orclause(orlist);
+	{
+		/*
+		 * Make an attempt to group similar OR clauses into SAOP if the list is
+		 * lengthy enough.
+		 */
+		if (or_transformation_limit >= 0 &&
+			list_length(orlist) > or_transformation_limit)
+			orlist = or_transformation(orlist);
+
+		/* Transformation could group all OR clauses to a single SAOP */
+		return (list_length(orlist) == 1) ?
+							(Expr *) linitial(orlist) : make_orclause(orlist);
+	}
 
 	/*
 	 * Generate new OR list consisting of the remaining sub-clauses.
@@ -651,6 +1023,11 @@ process_duplicate_ors(List *orlist)
 		}
 	}
 
+	/* Make an attempt to group similar OR clauses into ANY operation */
+	if (or_transformation_limit >= 0 &&
+		list_length(neworlist) > or_transformation_limit)
+		neworlist = or_transformation(neworlist);
+
 	/*
 	 * Append reduced OR to the winners list, if it's not degenerate, handling
 	 * the special case of one element correctly (can that really happen?).
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index abd9029451..a8dd701e7c 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3626,6 +3626,19 @@ struct config_int ConfigureNamesInt[] =
 		NULL, NULL, NULL
 	},
 
+	{
+		{"or_transformation_limit", PGC_USERSET, QUERY_TUNING_OTHER,
+			gettext_noop("Set the minimum length of the list of OR clauses to "
+						 "attempt the transformation."),
+			gettext_noop("The planner will try to replace expression like "
+						 "'x=c1 OR x=c2 ..' to the expression 'x = ANY(ARRAY[c1,c2,..])'"),
+			GUC_EXPLAIN
+		},
+		&or_transformation_limit,
+		0, -1, INT_MAX,
+		NULL, NULL, NULL
+	},
+
 	/* End-of-list marker */
 	{
 		{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2244ee52f7..03745c2630 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -391,6 +391,7 @@
 # - Planner Method Configuration -
 
+#or_transformation_limit = 0

index 7b63c5cf71..4c613401d6 100644

+extern PGDLLIMPORT int or_transformation_limit;
+
 extern Node *negate_clause(Node *node);
 extern Expr *canonicalize_qual(Expr *qual, bool is_check);
 
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 70ab47a92f..e9fb82b64c 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1836,6 +1836,72 @@ DROP TABLE onek_with_null;
 -- Check bitmap index path planning
 --
 EXPLAIN (COSTS OFF)
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+                                  QUERY PLAN                                  
+------------------------------------------------------------------------------
+ Index Scan using tenk1_thous_tenthous on tenk1
+   Index Cond: ((thousand = 42) AND (tenthous = ANY ('{1,3,42}'::integer[])))
+(2 rows)
+
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+ unique1 | unique2 | two | four | ten | twenty | hundred | thousand | twothousand | fivethous | tenthous | odd | even | stringu1 | stringu2 | string4 
+---------+---------+-----+------+-----+--------+---------+----------+-------------+-----------+----------+-----+------+----------+----------+---------
+      42 |    5530 |   0 |    2 |   2 |      2 |      42 |       42 |          42 |        42 |       42 |  84 |   85 | QBAAAA   | SEIAAA   | OOOOxx
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand = ANY ('{42,99}'::integer[])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = ANY ('{42,99}'::integer[]))
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+ count 
+-------
+    10
+(1 row)
+
+SET or_transformation_limit = 0;
+EXPLAIN (COSTS OFF)
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+                                  QUERY PLAN                                  
+------------------------------------------------------------------------------
+ Index Scan using tenk1_thous_tenthous on tenk1
+   Index Cond: ((thousand = 42) AND (tenthous = ANY ('{1,3,42}'::integer[])))
+(2 rows)
+
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+ unique1 | unique2 | two | four | ten | twenty | hundred | thousand | twothousand | fivethous | tenthous | odd | even | stringu1 | stringu2 | string4 
+---------+---------+-----+------+-----+--------+---------+----------+-------------+-----------+----------+-----+------+----------+----------+---------
+      42 |    5530 |   0 |    2 |   2 |      2 |      42 |       42 |          42 |        42 |       42 |  84 |   85 | QBAAAA   | SEIAAA   | OOOOxx
+(1 row)
+
+SET or_transformation_limit = 2;
+EXPLAIN (COSTS OFF) -- or_transformation still works
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+                                  QUERY PLAN                                  
+------------------------------------------------------------------------------
+ Index Scan using tenk1_thous_tenthous on tenk1
+   Index Cond: ((thousand = 42) AND (tenthous = ANY ('{1,3,42}'::integer[])))
+(2 rows)
+
+SET or_transformation_limit = 3;
+EXPLAIN (COSTS OFF) -- or_transformation must be disabled
 SELECT * FROM tenk1
   WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
                                                                QUERY PLAN                                                                
@@ -1851,38 +1917,120 @@ SELECT * FROM tenk1
                Index Cond: ((thousand = 42) AND (tenthous = 42))
 (9 rows)
 
-SELECT * FROM tenk1
-  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
- unique1 | unique2 | two | four | ten | twenty | hundred | thousand | twothousand | fivethous | tenthous | odd | even | stringu1 | stringu2 | string4 
----------+---------+-----+------+-----+--------+---------+----------+-------------+-----------+----------+-----+------+----------+----------+---------
-      42 |    5530 |   0 |    2 |   2 |      2 |      42 |       42 |          42 |        42 |       42 |  84 |   85 | QBAAAA   | SEIAAA   | OOOOxx
+RESET or_transformation_limit;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+                                     QUERY PLAN                                     
+------------------------------------------------------------------------------------
+ Aggregate
+   ->  Bitmap Heap Scan on tenk1
+         Recheck Cond: ((hundred = 42) AND (thousand = ANY ('{42,99}'::integer[])))
+         ->  BitmapAnd
+               ->  Bitmap Index Scan on tenk1_hundred
+                     Index Cond: (hundred = 42)
+               ->  Bitmap Index Scan on tenk1_thous_tenthous
+                     Index Cond: (thousand = ANY ('{42,99}'::integer[]))
+(8 rows)
+
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+ count 
+-------
+    10
 (1 row)
 
 EXPLAIN (COSTS OFF)

+RESET or_transformation_limit;
index 63cddac0d6..41e2a5ed98 100644
@@ -4223,15 +4223,61 @@ select * from tenk1 a join tenk1 b on
 
 --
 -- test placement of movable quals in a parameterized join tree
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 46b78ba3c4..388b7f838b 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -82,23 +82,43 @@ explain (costs off) select * from lp where a is null;
 (2 rows)

 
 explain (costs off) select * from lp where a <> 'g';
@@ -515,10 +535,10 @@ explain (costs off) select * from rlp where a <= 31;
 (27 rows)

 explain (costs off) select * from rlp where a = 1 or b = 'ab';
@@ -596,13 +616,13 @@ explain (costs off) select * from rlp where a < 1 or (a > 20 and a < 25);

 explain (costs off) select * from rlp3 where a = 20;   /* empty */
@@ -671,6 +691,161 @@ explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a =
          Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
 (11 rows)
 
@@ -2072,10 +2247,10 @@ explain (costs off) select * from hp where a = 1 and b = 'abcde';

 drop table hp2;

 

 
index f133b5a4ac..f112e64a87 100644
@@ -43,10 +43,25 @@ SELECT ctid, * FROM tidscan WHERE '(0,1)' = ctid;
 
 SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index d49ce9f300..2f691efcc5 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -737,6 +737,50 @@ SELECT count(*) FROM tenk1
 SELECT count(*) FROM tenk1
   WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
 
+SET or_transformation_limit = 0;
+EXPLAIN (COSTS OFF)
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+SET or_transformation_limit = 2;
+EXPLAIN (COSTS OFF) -- or_transformation still works
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+SET or_transformation_limit = 3;
+EXPLAIN (COSTS OFF) -- or_transformation must be disabled
+SELECT * FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3 OR tenthous = 42);
+RESET or_transformation_limit;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99);
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand < 42 OR thousand < 99 OR 43 > thousand OR 42 > thousand);
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+SELECT count(*) FROM tenk1
+  WHERE thousand = 42 AND (tenthous = 1 OR tenthous = 3) OR thousand = 41;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 99 OR tenthous < 2) OR thousand = 41;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
+SELECT count(*) FROM tenk1
+  WHERE hundred = 42 AND (thousand = 42 OR thousand = 41 OR thousand = 99 AND tenthous = 2);
+RESET or_transformation_limit;
+
 --
 -- Check behavior with duplicate index column contents
 --
diff --git a/src/test/regress/sql/join.sql b/src/test/regress/sql/join.sql
index c4c6c7b8ba..8158a7dd24 100644
--- a/src/test/regress/sql/join.sql
+++ b/src/test/regress/sql/join.sql
@@ -1408,6 +1408,14 @@ explain (costs off)
 select * from tenk1 a join tenk1 b on
   (a.unique1 = 1 and b.unique1 = 2) or
   ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+explain (costs off)
+select * from tenk1 a join tenk1 b on
+  (a.unique1 = 1 and b.unique1 = 2) or
+  ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
+explain (costs off)
+select * from tenk1 a join tenk1 b on
+  (a.unique1 < 20 or a.unique1 = 3 or a.unique1 = 1 and b.unique1 = 2) or
+  ((a.unique2 = 3 or a.unique2 = 7) and b.hundred = 4);
 
index dc71693861..e677ce7250 100644
@@ -21,6 +21,10 @@ explain (costs off) select * from lp where a is not null;
@@ -99,6 +103,20 @@ explain (costs off) select * from rlp where a > 1 and a >=15;	/* rlp3 onwards, i
 
+
index 313e0fb9b6..283e026751 100644
@@ -22,6 +22,10 @@ EXPLAIN (COSTS OFF)
 
+EXPLAIN (COSTS OFF)
+SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
+SELECT ctid, * FROM tidscan WHERE ctid = '(0,2)' OR '(0,1)' = ctid;
+
 -- ctid = ScalarArrayOp - implemented as tidscan
 EXPLAIN (COSTS OFF)
 SELECT ctid, * FROM tidscan WHERE ctid = ANY(ARRAY['(0,1)', '(0,2)']::tid[]);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfa9d5aaea..4f7925d78c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1672,6 +1672,8 @@ NumericVar
 OM_uint32
 OP
 OSAPerGroupState
+OrClauseGroupEntry
+OrClauseGroupKey
 OSAPerQueryState
 OSInfo
 OSSLCipher
-- 
2.44.0

From 37831eb47fe890d41e5f66bedc5d616c0f1ad570 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email-masked]>
Date: Wed, 13 Mar 2024 12:26:02 +0700
Subject: [PATCH 2/2] Teach generate_bitmap_or_paths to build BitmapOr paths
 over SAOP clauses.

 src/backend/optimizer/path/indxpath.c      |  82 +++++-
 9 files changed, 508 insertions(+), 17 deletions(-)

index f384edde05..5772574634 100644
@@ -5486,6 +5486,9 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
index 32c6a8bbdc..f78dae25ff 100644
 
@@ -1220,11 +1221,73 @@ build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel,
 	return result;
 }
 
+/*
+	 * Don't spend cycles here if the transformation is disabled.
+	 * We don't behave symmetrically here with the OR -> ANY transformation, see
+	 * the process_duplicate_ors routine, because origin list of ORs and
+	 * resulting SAOP args list lengths can differ significantly. Moreover,
+	 * Here we already limited by the MAX_SAOP_ARRAY_SIZE value.
+	if (or_transformation_limit < 0)
@@ -1247,20 +1310,25 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel,
 
-		/* Ignore RestrictInfos that aren't ORs */
-		if (!restriction_is_or_clause(rinfo))
+		orlist = extract_saop_ors(root, rinfo);
+		if (orlist == NIL)
+			/* Ignore RestrictInfo that doesn't provide proper OR list */
 			continue;
 
+		/* SAOP have splitted, remove already redundant clause */
index 6e3b376f3d..fe8862b0a0 100644
 
+/*
 
 /*
  * predicate_implied_by
 
+bool
index 4c613401d6..4ca26f65d1 100644

 /*

+
index e9fb82b64c..0d9c9334f9 100644
@@ -1980,23 +1980,25 @@ SELECT count(*) FROM tenk1
 
@@ -2008,20 +2010,22 @@ SELECT count(*) FROM tenk1
 
index 33a6dceb0e..b5fdde47ac 100644
 
+SET or_transformation_limit = -1;
+RESET or_transformation_limit;
index 019f1e7673..10ed6f0a9c 100644
 
+SET or_transformation_limit = -1;
+RESET or_transformation_limit;

---

From: Alexander Korotkov <[email-masked]>
Date: Mon, 8 Apr 2024 01:34:37 +0300

Hi!

> with a minimal number of partial indexes (likewise, previous version).

I've revised the patch.  Did some beautification, improvements for
documentation, commit messages etc.

I've pushed the 0001 patch without 0002.  I think 0001 is good by
itself given that there is the or_to_any_transform_limit GUC option.
The more similar OR clauses are here the more likely grouping them
into SOAP will be a win.  But I've changed the default value to 5.
This will make it less invasive and affect only queries with obvious
repeating patterns.  That also reduced the changes in the regression
tests expected outputs.

Regarding 0002, it seems questionable since it could cause a planning
slowdown for SAOP's with large arrays.  Also, it might reduce the win
of transformation made by 0001.  So, I think we should skip it for
now.

------
Regards,
Alexander Korotkov

---

From: Justin Pryzby <[email-masked]>
Date: Sun, 7 Apr 2024 19:38:36 -0500

> into SOAP will be a win.  But I've changed the default value to 5.

The sample config file has the wrong default

+#or_to_any_transform_limit = 0

We had a patch to catch this kind of error, but it was closed (which IMO
was itself an error).

-- 
Justin

---

From: Alexander Korotkov <[email-masked]>
Date: Fri, 14 Jun 2024 15:00:33 +0300

> of transformation made by 0001.  So, I think we should skip it for

The patch has been reverted from pg17.  Let me propose a new version
for pg18 based on the valuable feedback from Tom Lane [1][2].

 * The transformation is moved to the stage of adding restrictinfos to
the base relation (in particular add_base_clause_to_rel()).  This
leads to interesting consequences.  While this allows IndexScans to
use transformed clauses, BitmapScans and SeqScans seem unaffected.
Therefore, I wasn't able to find a planning regression.
 * As soon as there is no planning regression anymore, I've removed
or_to_any_transform_limit GUC, which was a source of critics.
 * Now, not only Consts allowed in the SAOP's list, but also Params.
 * The criticized hash based on expression jumbling has been removed.
Now, the plain list is used instead.
 * OrClauseGroup now gets a legal node tag.  That allows to mix it in
the list with other nodes without hacks.

I think this patch shouldn't be as good as before for optimizing
performance of large OR lists, given that BitmapScans and SeqScans
still deal with ORs.  However, it allows IndexScans to handle more,
doesn't seem to cause planning regression and therefore introduce no
extra GUC.  Overall, this seems like a good compromise.

This patch could use some polishing, but I'd like to first hear some
feedback on general design.

Links
1. https://www.postgresql.org/message-id/3604469.1712628736%40sss.pgh.pa.us
2. https://www.postgresql.org/message-id/3649287.1712642139%40sss.pgh.pa.us

------
Regards,
Alexander Korotkov
Supabase

---

From: Alena Rybakina <[email-masked]>
Date: Mon, 17 Jun 2024 13:33:49 +0300

Hi, thank you for your work with this subject!

On 14.06.2024 15:00, Alexander Korotkov wrote:
> On Mon, Apr 8, 2024 at 1:34 AM Alexander Korotkov<[email-masked]>  wrote:
> 1.https://www.postgresql.org/message-id/3604469.1712628736%40sss.pgh.pa.us
> 2.https://www.postgresql.org/message-id/3649287.1712642139%40sss.pgh.pa.us

Inoticedthat7librarieshave 
beenaddedtosrc/backend/optimizer/plan/initsplan.c,andas faras 
Iremember,TomLanehas alreadyexpresseddoubtsaboutthe 
approachthatrequiresaddinga largenumberof libraries[0], but I'm afraid 
I'm out of ideas about alternative approach.

In addition,Icheckedthe fixinthe 
previouscasesthatyouwroteearlier[1]andnoticedthatSeqScancontinuesto 
generate,unfortunately,withoutconvertingexpressions:

with patch:

create table test as (select (random()*10)::int x, (random()*1000) y 
from generate_series(1,1000000) i); create index test_x_1_y on test (y) 
where x = 1; create index test_x_2_y on test (y) where x = 2; vacuum 
analyze test; SELECT 1000000 CREATE INDEX CREATE INDEX VACUUM 
alena@postgres=# explain select * from test where (x = 1 or x = 2) and y 
= 100; QUERY PLAN 
-------------------------------------------------------------------------- 
Gather (cost=1000.00..12690.10 rows=1 width=12) Workers Planned: 2 -> 
Parallel Seq Scan on test (cost=0.00..11690.00 rows=1 width=12) Filter: 
(((x = 1) OR (x = 2)) AND (y = '100'::double precision)) (4 rows) 
alena@postgres=# set enable_seqscan =off; SET alena@postgres=# explain 
select * from test where (x = 1 or x = 2) and y = 100; QUERY PLAN 
------------------------------------------------------------------------- 
Seq Scan on test (cost=10000000000.00..10000020440.00 rows=1 width=12) 
Filter: (((x = 1) OR (x = 2)) AND (y = '100'::double precision)) (2 rows)

without patch:

-------------------------------------------------------------------------------------------------------------- 
Bitmap Heap Scan on test (cost=8.60..12.62 rows=1 width=12) Recheck 
Cond: (((y = '100'::double precision) AND (x = 1)) OR ((y = 
'100'::double precision) AND (x = 2))) -> BitmapOr (cost=8.60..8.60 
rows=1 width=0) -> Bitmap Index Scan on test_x_1_y (cost=0.00..4.30 
rows=1 width=0) Index Cond: (y = '100'::double precision) -> Bitmap 
Index Scan on test_x_2_y (cost=0.00..4.30 rows=1 width=0) Index Cond: (y 
= '100'::double precision) (7 rows)

[0] https://www.postgresql.org/message-id/3604469.1712628736%40sss.pgh.pa.us

[1] 
https://www.postgresql.org/message-id/CAPpHfduJtO0s9E%3DSHUTzrCD88BH0eik0UNog1_q3XBF2wLmH6g%40mail.gmail.com