public inbox for [email protected]
help / color / mirror / Atom feedSupport for CREATE MODULE?
67+ messages / 17 participants
[nested] [flat]
* Support for CREATE MODULE?
@ 2021-06-02 13:38 Jim Mlodgenski <[email protected]>
0 siblings, 3 replies; 67+ messages in thread
From: Jim Mlodgenski @ 2021-06-02 13:38 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Working with users over the years, some have large libraries of server
side code sometimes consisting of 100k+ lines of code over 1000+ functions
and procedures. This usually comes from a migration of a commercial
database like Oracle where it was best practice to put all of your
business logic into stored procedures. In these types of apps, just
managing the code is a challenge. To help classify objects, schemas
are used, but you are at the mercy of a naming convention to show
association. For example, a frequent naming convention would be having
related schemas with the names of foo_bar and foo_baz. For devs, that's
akin to keeping a file like xlog.c in a directory structure like
backend_access_transam instead of backend/access/transam. IMHO, having
a hierarchy makes it simpler to reason about related code bits.
The SQL spec does have a concept of modules that help address this. It's
defined as a persistent object within a schema that contains one or more
routines. It also defines other things like local temporary tables and
path specifications. There are other databases like DB2 that have
implemented module support each with their own way of defining the
routines within the module. The spec doesn't really give guidance on
how to manipulate the objects within the module.
Attached is a POC patch for modules. I modeled it as a sub-schema because
that is more what it seems like to me. It adds additional columns to
pg_namespace and allows for 3-part (or 4 with the database name) naming
of objects within the module. This simple example works with the patch.
CREATE SCHEMA foo;
CREATE MODULE foo.bar
CREATE FUNCTION hello() RETURNS text
LANGUAGE sql
RETURN 'hello'
CREATE FUNCTION world() RETURNS text
LANGUAGE sql
RETURN 'world';
SELECT foo.bar.hello();
Questions
- Do we want to add module support?
- If we do, should it be implemented as a type of namespace or should it
be its own object type that lives in something like pg_module?
- How should users interact with objects within a module? They could be
mostly independent like the current POC or we can introduce a path like
ALTER MODULE foo ADD FUNCTION blah
--Jim
Attachments:
[application/octet-stream] modules_poc_1.patch (29.1K, ../../CAB_5SRebSCjO12=nLsaLCBw2vnkiNH7jcNchirPc0yQ2KmiknQ@mail.gmail.com/2-modules_poc_1.patch)
download | inline diff:
unchanged:
--- a/src/backend/catalog/aclchk.c
+++ b/src/backend/catalog/aclchk.c
@@ -706,7 +706,7 @@ objectNamesToOids(ObjectType objtype, List *objnames)
char *nspname = strVal(lfirst(cell));
Oid oid;
- oid = get_namespace_oid(nspname, false);
+ oid = get_namespace_oid(nspname, InvalidOid, false);
objects = lappend_oid(objects, oid);
}
break;
@@ -1113,7 +1113,7 @@ SetDefaultACLsInSchemas(InternalDefaultACL *iacls, List *nspnames)
{
char *nspname = strVal(lfirst(nspcell));
- iacls->nspid = get_namespace_oid(nspname, false);
+ iacls->nspid = get_namespace_oid(nspname, InvalidOid, false);
/*
* We used to insist that the target role have CREATE privileges
@@ -3372,6 +3372,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype,
case OBJECT_MATVIEW:
msg = gettext_noop("permission denied for materialized view %s");
break;
+ case OBJECT_MODULE:
+ msg = gettext_noop("permission denied for module %s");
+ break;
case OBJECT_OPCLASS:
msg = gettext_noop("permission denied for operator class %s");
break;
@@ -3500,6 +3503,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype,
case OBJECT_MATVIEW:
msg = gettext_noop("must be owner of materialized view %s");
break;
+ case OBJECT_MODULE:
+ msg = gettext_noop("must be owner of module %s");
+ break;
case OBJECT_OPCLASS:
msg = gettext_noop("must be owner of operator class %s");
break;
unchanged:
--- a/src/backend/catalog/namespace.c
+++ b/src/backend/catalog/namespace.c
@@ -473,7 +473,7 @@ RangeVarGetCreationNamespace(const RangeVar *newRelation)
return myTempNamespace;
}
/* use exact schema given */
- namespaceId = get_namespace_oid(newRelation->schemaname, false);
+ namespaceId = get_namespace_oid(newRelation->schemaname, InvalidOid, false);
/* we do not check for USAGE rights here! */
}
else if (newRelation->relpersistence == RELPERSISTENCE_TEMP)
@@ -945,9 +945,11 @@ FuncnameGetCandidates(List *names, int nargs, List *argnames,
{
FuncCandidateList resultList = NULL;
bool any_special = false;
- char *schemaname;
+ char *nspname;
+ char *modulename;
char *funcname;
Oid namespaceId;
+ Oid moduleId;
CatCList *catlist;
int i;
@@ -955,19 +957,31 @@ FuncnameGetCandidates(List *names, int nargs, List *argnames,
Assert(nargs >= 0 || !(expand_variadic | expand_defaults));
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &funcname);
+ DeconstructQualifiedName(names, &nspname, &modulename, &funcname, true);
- if (schemaname)
+ if (nspname && modulename)
{
/* use exact schema given */
- namespaceId = LookupExplicitNamespace(schemaname, missing_ok);
+ namespaceId = LookupExplicitNamespace(nspname, missing_ok);
if (!OidIsValid(namespaceId))
return NULL;
+
+ moduleId = get_namespace_oid(modulename, namespaceId, true);
+ if (!OidIsValid(moduleId))
+ return NULL;
+ }
+ else if (nspname)
+ {
+ moduleId = InvalidOid;
+ namespaceId = LookupExplicitNamespace(nspname, true);
+ if (!OidIsValid(namespaceId))
+ recomputeNamespacePath();
}
else
{
/* flag to indicate we need namespace search */
namespaceId = InvalidOid;
+ moduleId = InvalidOid;
recomputeNamespacePath();
}
@@ -987,9 +1001,15 @@ FuncnameGetCandidates(List *names, int nargs, List *argnames,
int *argnumbers = NULL;
FuncCandidateList newResult;
- if (OidIsValid(namespaceId))
+ if (OidIsValid(moduleId))
{
- /* Consider only procs in specified namespace */
+ /* Consider only procs in specified module */
+ if (procform->pronamespace != moduleId)
+ continue;
+ }
+ else if (OidIsValid(namespaceId))
+ {
+ /* Consider only procs in specified schema */
if (procform->pronamespace != namespaceId)
continue;
}
@@ -1003,7 +1023,16 @@ FuncnameGetCandidates(List *names, int nargs, List *argnames,
foreach(nsp, activeSearchPath)
{
- if (procform->pronamespace == lfirst_oid(nsp) &&
+ /* check if there is a module in the schema */
+ if (nspname)
+ {
+ moduleId = get_namespace_oid(nspname, lfirst_oid(nsp), true);
+ if (OidIsValid(moduleId))
+ if (procform->pronamespace == moduleId &&
+ procform->pronamespace != myTempNamespace)
+ break;
+ }
+ else if (procform->pronamespace == lfirst_oid(nsp) &&
procform->pronamespace != myTempNamespace)
break;
pathpos++;
@@ -1449,7 +1478,6 @@ FunctionIsVisible(Oid funcid)
clist = FuncnameGetCandidates(list_make1(makeString(proname)),
nargs, NIL, false, false, false);
-
for (; clist; clist = clist->next)
{
if (memcmp(clist->args, procform->proargtypes.values,
@@ -1488,7 +1516,7 @@ OpernameGetOprid(List *names, Oid oprleft, Oid oprright)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &opername);
+ DeconstructQualifiedName(names, &schemaname, NULL, &opername, false);
if (schemaname)
{
@@ -1595,7 +1623,7 @@ OpernameGetCandidates(List *names, char oprkind, bool missing_schema_ok)
int i;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &opername);
+ DeconstructQualifiedName(names, &schemaname, NULL, &opername, false);
if (schemaname)
{
@@ -2183,7 +2211,7 @@ get_statistics_object_oid(List *names, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &stats_name);
+ DeconstructQualifiedName(names, &schemaname, NULL, &stats_name, false);
if (schemaname)
{
@@ -2224,6 +2252,60 @@ get_statistics_object_oid(List *names, bool missing_ok)
return stats_oid;
}
+/*
+ * get_module_oid - find a module by possibly qualified name
+ *
+ * If not found, returns InvalidOid if missing_ok, else throws error
+ */
+Oid
+get_module_oid(List *names, bool missing_ok)
+{
+ char *schemaname;
+ char *modulename;
+ Oid namespaceId;
+ Oid moduleId = InvalidOid;
+ ListCell *l;
+
+ /* deconstruct the name list */
+ DeconstructQualifiedName(names, &schemaname, NULL, &modulename, false);
+
+ if (schemaname)
+ {
+ /* use exact schema given */
+ namespaceId = LookupExplicitNamespace(schemaname, missing_ok);
+ if (missing_ok && !OidIsValid(namespaceId))
+ moduleId = InvalidOid;
+ else
+ moduleId = GetSysCacheOid2(NAMESPACENAME, Anum_pg_namespace_oid,
+ CStringGetDatum(modulename), ObjectIdGetDatum(namespaceId));
+ }
+ else
+ {
+ /* search for it in search path */
+ recomputeNamespacePath();
+
+ foreach(l, activeSearchPath)
+ {
+ namespaceId = lfirst_oid(l);
+
+ if (namespaceId == myTempNamespace)
+ continue; /* do not look in temp namespace */
+ moduleId = GetSysCacheOid2(NAMESPACENAME, Anum_pg_namespace_oid,
+ CStringGetDatum(modulename), ObjectIdGetDatum(namespaceId));
+ if (OidIsValid(moduleId))
+ break;
+ }
+ }
+
+ if (!OidIsValid(moduleId) && !missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("module \"%s\" does not exist",
+ NameListToString(names))));
+
+ return moduleId;
+}
+
/*
* StatisticsObjIsVisible
* Determine whether a statistics object (identified by OID) is visible in
@@ -2305,7 +2387,7 @@ get_ts_parser_oid(List *names, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &parser_name);
+ DeconstructQualifiedName(names, &schemaname, NULL, &parser_name, false);
if (schemaname)
{
@@ -2431,7 +2513,7 @@ get_ts_dict_oid(List *names, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &dict_name);
+ DeconstructQualifiedName(names, &schemaname, NULL, &dict_name, false);
if (schemaname)
{
@@ -2558,7 +2640,7 @@ get_ts_template_oid(List *names, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &template_name);
+ DeconstructQualifiedName(names, &schemaname, NULL, &template_name, false);
if (schemaname)
{
@@ -2684,7 +2766,7 @@ get_ts_config_oid(List *names, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, &config_name);
+ DeconstructQualifiedName(names, &schemaname, NULL, &config_name, false);
if (schemaname)
{
@@ -2800,17 +2882,21 @@ TSConfigIsVisible(Oid cfgid)
/*
* DeconstructQualifiedName
* Given a possibly-qualified name expressed as a list of String nodes,
- * extract the schema name and object name.
+ * extract the schema name, module name and object name.
*
* *nspname_p is set to NULL if there is no explicit schema name.
+ * *modname_p is set to NULL if there is no explicit module name.
*/
void
DeconstructQualifiedName(List *names,
char **nspname_p,
- char **objname_p)
+ char **modname_p,
+ char **objname_p,
+ bool check_module)
{
char *catalogname;
- char *schemaname = NULL;
+ char *nspname = NULL;
+ char *modulename = NULL;
char *objname = NULL;
switch (list_length(names))
@@ -2819,22 +2905,70 @@ DeconstructQualifiedName(List *names,
objname = strVal(linitial(names));
break;
case 2:
- schemaname = strVal(linitial(names));
+ nspname = strVal(linitial(names));
objname = strVal(lsecond(names));
break;
case 3:
- catalogname = strVal(linitial(names));
- schemaname = strVal(lsecond(names));
- objname = strVal(lthird(names));
+ if (check_module)
+ {
+ /*
+ * Since we don't allow cross-database references, check if the
+ * first element is the current catalog and if is different assume
+ * the first element is a schema
+ */
+ if (strcmp(strVal(linitial(names)), get_database_name(MyDatabaseId)) != 0)
+ {
+ nspname = strVal(linitial(names));
+ modulename = strVal(lsecond(names));
+ }
+ else
+ {
+ catalogname = strVal(linitial(names));
+ nspname = strVal(lsecond(names));
+ }
- /*
- * We check the catalog name and then ignore it.
- */
- if (strcmp(catalogname, get_database_name(MyDatabaseId)) != 0)
+ objname = strVal(lthird(names));
+ }
+ else
+ {
+ catalogname = strVal(linitial(names));
+ nspname = strVal(lsecond(names));
+ objname = strVal(lthird(names));
+
+ /*
+ * We check the catalog name and then ignore it.
+ */
+ if (strcmp(catalogname, get_database_name(MyDatabaseId)) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cross-database references are not implemented: %s",
+ NameListToString(names))));
+ }
+ break;
+ case 4:
+ if (check_module)
+ {
+ catalogname = strVal(linitial(names));
+ nspname = strVal(lsecond(names));
+ modulename = strVal(lthird(names));
+ objname = strVal(lfourth(names));
+
+ /*
+ * We check the catalog name and then ignore it.
+ */
+ if (strcmp(catalogname, get_database_name(MyDatabaseId)) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cross-database references are not implemented: %s",
+ NameListToString(names))));
+ }
+ else
+ {
ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("cross-database references are not implemented: %s",
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("improper qualified name (too many dotted names): %s",
NameListToString(names))));
+ }
break;
default:
ereport(ERROR,
@@ -2844,7 +2978,9 @@ DeconstructQualifiedName(List *names,
break;
}
- *nspname_p = schemaname;
+ *nspname_p = nspname;
+ if (modname_p)
+ *modname_p = modulename;
*objname_p = objname;
}
@@ -2878,7 +3014,7 @@ LookupNamespaceNoError(const char *nspname)
return InvalidOid;
}
- return get_namespace_oid(nspname, true);
+ return get_namespace_oid(nspname, InvalidOid, true);
}
/*
@@ -2907,7 +3043,7 @@ LookupExplicitNamespace(const char *nspname, bool missing_ok)
*/
}
- namespaceId = get_namespace_oid(nspname, missing_ok);
+ namespaceId = get_namespace_oid(nspname, InvalidOid, missing_ok);
if (missing_ok && !OidIsValid(namespaceId))
return InvalidOid;
@@ -2945,7 +3081,7 @@ LookupCreationNamespace(const char *nspname)
return myTempNamespace;
}
- namespaceId = get_namespace_oid(nspname, false);
+ namespaceId = get_namespace_oid(nspname, InvalidOid, false);
aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(), ACL_CREATE);
if (aclresult != ACLCHECK_OK)
@@ -2991,25 +3127,34 @@ CheckSetNamespace(Oid oldNspOid, Oid nspOid)
* if we have to create or clean out the temp namespace.
*/
Oid
-QualifiedNameGetCreationNamespace(List *names, char **objname_p)
+QualifiedNameGetCreationNamespace(List *names, char **objname_p, bool check_module)
{
- char *schemaname;
+ char *nspname = NULL;
+ char *modulename = NULL;
+ Oid nspparent;
Oid namespaceId;
/* deconstruct the name list */
- DeconstructQualifiedName(names, &schemaname, objname_p);
+ DeconstructQualifiedName(names, &nspname, &modulename, objname_p, check_module);
- if (schemaname)
+ if (nspname && modulename)
+ {
+ /* use exact schema given */
+ nspparent = LookupExplicitNamespace(nspname, false);
+
+ namespaceId = get_namespace_oid(modulename, nspparent, false);
+ }
+ else if (nspname)
{
/* check for pg_temp alias */
- if (strcmp(schemaname, "pg_temp") == 0)
+ if (strcmp(nspname, "pg_temp") == 0)
{
/* Initialize temp namespace */
AccessTempTableNamespace(false);
return myTempNamespace;
}
/* use exact schema given */
- namespaceId = get_namespace_oid(schemaname, false);
+ namespaceId = get_namespace_oid(nspname, InvalidOid, false);
/* we do not check for USAGE rights here! */
}
else
@@ -3039,12 +3184,12 @@ QualifiedNameGetCreationNamespace(List *names, char **objname_p)
* true, just return InvalidOid.
*/
Oid
-get_namespace_oid(const char *nspname, bool missing_ok)
+get_namespace_oid(const char *nspname, Oid nspnamespace, bool missing_ok)
{
Oid oid;
- oid = GetSysCacheOid1(NAMESPACENAME, Anum_pg_namespace_oid,
- CStringGetDatum(nspname));
+ oid = GetSysCacheOid2(NAMESPACENAME, Anum_pg_namespace_oid,
+ CStringGetDatum(nspname), ObjectIdGetDatum(nspnamespace));
if (!OidIsValid(oid) && !missing_ok)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_SCHEMA),
@@ -3607,7 +3752,7 @@ get_collation_oid(List *name, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(name, &schemaname, &collation_name);
+ DeconstructQualifiedName(name, &schemaname, NULL, &collation_name, false);
if (schemaname)
{
@@ -3660,7 +3805,7 @@ get_conversion_oid(List *name, bool missing_ok)
ListCell *l;
/* deconstruct the name list */
- DeconstructQualifiedName(name, &schemaname, &conversion_name);
+ DeconstructQualifiedName(name, &schemaname, NULL, &conversion_name, false);
if (schemaname)
{
@@ -3789,7 +3934,7 @@ recomputeNamespacePath(void)
char *rname;
rname = NameStr(((Form_pg_authid) GETSTRUCT(tuple))->rolname);
- namespaceId = get_namespace_oid(rname, true);
+ namespaceId = get_namespace_oid(rname, InvalidOid, true);
ReleaseSysCache(tuple);
if (OidIsValid(namespaceId) &&
!list_member_oid(oidlist, namespaceId) &&
@@ -3818,7 +3963,7 @@ recomputeNamespacePath(void)
else
{
/* normal namespace reference */
- namespaceId = get_namespace_oid(curname, true);
+ namespaceId = get_namespace_oid(curname, InvalidOid, true);
if (OidIsValid(namespaceId) &&
!list_member_oid(oidlist, namespaceId) &&
pg_namespace_aclcheck(namespaceId, roleid,
@@ -3987,7 +4132,7 @@ InitTempTableNamespace(void)
snprintf(namespaceName, sizeof(namespaceName), "pg_temp_%d", MyBackendId);
- namespaceId = get_namespace_oid(namespaceName, true);
+ namespaceId = get_namespace_oid(namespaceName, InvalidOid, true);
if (!OidIsValid(namespaceId))
{
/*
@@ -3998,8 +4143,8 @@ InitTempTableNamespace(void)
* temp tables. This works because the places that access the temp
* namespace for my own backend skip permissions checks on it.
*/
- namespaceId = NamespaceCreate(namespaceName, BOOTSTRAP_SUPERUSERID,
- true);
+ namespaceId = NamespaceCreate(namespaceName, InvalidOid, NSPKIND_SCHEMA,
+ BOOTSTRAP_SUPERUSERID, true);
/* Advance command counter to make namespace visible */
CommandCounterIncrement();
}
@@ -4020,11 +4165,11 @@ InitTempTableNamespace(void)
snprintf(namespaceName, sizeof(namespaceName), "pg_toast_temp_%d",
MyBackendId);
- toastspaceId = get_namespace_oid(namespaceName, true);
+ toastspaceId = get_namespace_oid(namespaceName, InvalidOid, true);
if (!OidIsValid(toastspaceId))
{
- toastspaceId = NamespaceCreate(namespaceName, BOOTSTRAP_SUPERUSERID,
- true);
+ toastspaceId = NamespaceCreate(namespaceName, InvalidOid, NSPKIND_SCHEMA,
+ BOOTSTRAP_SUPERUSERID, true);
/* Advance command counter to make namespace visible */
CommandCounterIncrement();
}
unchanged:
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -406,12 +406,26 @@ static const ObjectPropertyType ObjectProperty[] =
NAMESPACENAME,
Anum_pg_namespace_oid,
Anum_pg_namespace_nspname,
- InvalidAttrNumber,
+ Anum_pg_namespace_nspnamespace,
Anum_pg_namespace_nspowner,
Anum_pg_namespace_nspacl,
OBJECT_SCHEMA,
true
},
+ {
+ "module",
+ NamespaceRelationId,
+ NamespaceOidIndexId,
+ NAMESPACEOID,
+ NAMESPACENAME,
+ Anum_pg_namespace_oid,
+ Anum_pg_namespace_nspname,
+ Anum_pg_namespace_nspnamespace,
+ Anum_pg_namespace_nspowner,
+ Anum_pg_namespace_nspacl,
+ OBJECT_MODULE,
+ true
+ },
{
"relation",
RelationRelationId,
@@ -765,6 +779,10 @@ static const struct object_type_map
{
"schema", OBJECT_SCHEMA
},
+ /* OCLASS_MODULE */
+ {
+ "module", OBJECT_MODULE
+ },
/* OCLASS_TSPARSER */
{
"text search parser", OBJECT_TSPARSER
@@ -1128,6 +1146,12 @@ get_object_address(ObjectType objtype, Node *object,
missing_ok);
address.objectSubId = 0;
break;
+ case OBJECT_MODULE:
+ address.classId = NamespaceRelationId;
+ address.objectId = get_module_oid(castNode(List, object),
+ missing_ok);
+ address.objectSubId = 0;
+ break;
default:
elog(ERROR, "unrecognized objtype: %d", (int) objtype);
/* placate compiler, in case it thinks elog might return */
@@ -1281,7 +1305,7 @@ get_object_address_unqualified(ObjectType objtype,
break;
case OBJECT_SCHEMA:
address.classId = NamespaceRelationId;
- address.objectId = get_namespace_oid(name, missing_ok);
+ address.objectId = get_namespace_oid(name, InvalidOid, missing_ok);
address.objectSubId = 0;
break;
case OBJECT_LANGUAGE:
@@ -2013,7 +2037,7 @@ get_object_address_defacl(List *object, bool missing_ok)
*/
if (schema)
{
- schemaid = get_namespace_oid(schema, true);
+ schemaid = get_namespace_oid(schema, InvalidOid, true);
if (schemaid == InvalidOid)
goto not_found;
}
@@ -2267,6 +2291,7 @@ pg_get_object_address(PG_FUNCTION_ARGS)
case OBJECT_TABCONSTRAINT:
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
+ case OBJECT_MODULE:
objnode = (Node *) name;
break;
case OBJECT_ACCESS_METHOD:
@@ -2431,6 +2456,7 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address,
NameListToString((castNode(ObjectWithArgs, object))->objname));
break;
case OBJECT_SCHEMA:
+ case OBJECT_MODULE:
if (!pg_namespace_ownercheck(address.objectId, roleid))
aclcheck_error(ACLCHECK_NOT_OWNER, objtype,
strVal((Value *) object));
unchanged:
--- a/src/backend/catalog/pg_namespace.c
+++ b/src/backend/catalog/pg_namespace.c
@@ -40,7 +40,8 @@
* ---------------
*/
Oid
-NamespaceCreate(const char *nspName, Oid ownerId, bool isTemp)
+NamespaceCreate(const char *nspName, Oid nspnamespace, char nspkind,
+ Oid ownerId, bool isTemp)
{
Relation nspdesc;
HeapTuple tup;
@@ -58,7 +59,7 @@ NamespaceCreate(const char *nspName, Oid ownerId, bool isTemp)
elog(ERROR, "no namespace name supplied");
/* make sure there is no existing namespace of same name */
- if (SearchSysCacheExists1(NAMESPACENAME, PointerGetDatum(nspName)))
+ if (SearchSysCacheExists2(NAMESPACENAME, PointerGetDatum(nspName), nspnamespace))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_SCHEMA),
errmsg("schema \"%s\" already exists", nspName)));
@@ -85,6 +86,8 @@ NamespaceCreate(const char *nspName, Oid ownerId, bool isTemp)
namestrcpy(&nname, nspName);
values[Anum_pg_namespace_nspname - 1] = NameGetDatum(&nname);
values[Anum_pg_namespace_nspowner - 1] = ObjectIdGetDatum(ownerId);
+ values[Anum_pg_namespace_nspnamespace - 1] = ObjectIdGetDatum(nspnamespace);
+ values[Anum_pg_namespace_nspkind - 1] = CharGetDatum(nspkind);
if (nspacl != NULL)
values[Anum_pg_namespace_nspacl - 1] = PointerGetDatum(nspacl);
else
unchanged:
--- a/src/backend/commands/extension.c
+++ b/src/backend/commands/extension.c
@@ -1460,7 +1460,7 @@ CreateExtensionInternal(char *extensionName,
if (schemaName)
{
/* If the user is giving us the schema name, it must exist already. */
- schemaOid = get_namespace_oid(schemaName, false);
+ schemaOid = get_namespace_oid(schemaName, InvalidOid, false);
}
if (control->schema != NULL)
@@ -1484,7 +1484,7 @@ CreateExtensionInternal(char *extensionName,
schemaName = control->schema;
/* Find or create the schema in case it does not exist. */
- schemaOid = get_namespace_oid(schemaName, true);
+ schemaOid = get_namespace_oid(schemaName, InvalidOid, true);
if (!OidIsValid(schemaOid))
{
@@ -1501,7 +1501,7 @@ CreateExtensionInternal(char *extensionName,
* CreateSchemaCommand includes CommandCounterIncrement, so new
* schema is now visible.
*/
- schemaOid = get_namespace_oid(schemaName, false);
+ schemaOid = get_namespace_oid(schemaName, InvalidOid, false);
}
}
else if (!OidIsValid(schemaOid))
unchanged:
--- a/src/backend/commands/schemacmds.c
+++ b/src/backend/commands/schemacmds.c
@@ -113,7 +113,7 @@ CreateSchemaCommand(CreateSchemaStmt *stmt, const char *queryString,
* creation-permission check first, we do likewise.
*/
if (stmt->if_not_exists &&
- SearchSysCacheExists1(NAMESPACENAME, PointerGetDatum(schemaName)))
+ SearchSysCacheExists2(NAMESPACENAME, PointerGetDatum(schemaName), InvalidOid))
{
ereport(NOTICE,
(errcode(ERRCODE_DUPLICATE_SCHEMA),
@@ -135,7 +135,8 @@ CreateSchemaCommand(CreateSchemaStmt *stmt, const char *queryString,
save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
/* Create the schema's namespace */
- namespaceId = NamespaceCreate(schemaName, owner_uid, false);
+ namespaceId = NamespaceCreate(schemaName, InvalidOid, NSPKIND_SCHEMA,
+ owner_uid, false);
/* Advance cmd counter to make the namespace visible */
CommandCounterIncrement();
@@ -226,7 +227,7 @@ RenameSchema(const char *oldname, const char *newname)
rel = table_open(NamespaceRelationId, RowExclusiveLock);
- tup = SearchSysCacheCopy1(NAMESPACENAME, CStringGetDatum(oldname));
+ tup = SearchSysCacheCopy2(NAMESPACENAME, CStringGetDatum(oldname), InvalidOid);
if (!HeapTupleIsValid(tup))
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_SCHEMA),
@@ -236,7 +237,7 @@ RenameSchema(const char *oldname, const char *newname)
nspOid = nspform->oid;
/* make sure the new name doesn't exist */
- if (OidIsValid(get_namespace_oid(newname, true)))
+ if (OidIsValid(get_namespace_oid(newname, InvalidOid, true)))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_SCHEMA),
errmsg("schema \"%s\" already exists", newname)));
@@ -306,7 +307,7 @@ AlterSchemaOwner(const char *name, Oid newOwnerId)
rel = table_open(NamespaceRelationId, RowExclusiveLock);
- tup = SearchSysCache1(NAMESPACENAME, CStringGetDatum(name));
+ tup = SearchSysCache2(NAMESPACENAME, CStringGetDatum(name), InvalidOid);
if (!HeapTupleIsValid(tup))
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_SCHEMA),
unchanged:
--- a/src/backend/utils/adt/acl.c
+++ b/src/backend/utils/adt/acl.c
@@ -3827,7 +3827,7 @@ convert_schema_name(text *schemaname)
{
char *nspname = text_to_cstring(schemaname);
- return get_namespace_oid(nspname, false);
+ return get_namespace_oid(nspname, InvalidOid, false);
}
/*
unchanged:
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -184,7 +184,7 @@ binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
InsertExtensionTuple(text_to_cstring(extName),
GetUserId(),
- get_namespace_oid(text_to_cstring(schemaName), false),
+ get_namespace_oid(text_to_cstring(schemaName), InvalidOid, false),
relocatable,
text_to_cstring(extVersion),
extConfig,
unchanged:
--- a/src/backend/utils/adt/regproc.c
+++ b/src/backend/utils/adt/regproc.c
@@ -1762,7 +1762,7 @@ regnamespacein(PG_FUNCTION_ARGS)
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid name syntax")));
- result = get_namespace_oid(strVal(linitial(names)), false);
+ result = get_namespace_oid(strVal(linitial(names)), InvalidOid, false);
PG_RETURN_OID(result);
}
@@ -1786,7 +1786,7 @@ to_regnamespace(PG_FUNCTION_ARGS)
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid name syntax")));
- result = get_namespace_oid(strVal(linitial(names)), true);
+ result = get_namespace_oid(strVal(linitial(names)), InvalidOid, true);
if (OidIsValid(result))
PG_RETURN_OID(result);
unchanged:
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -509,10 +509,10 @@ static const struct cachedesc cacheinfo[] = {
},
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
- 1,
+ 2,
{
Anum_pg_namespace_nspname,
- 0,
+ Anum_pg_namespace_nspnamespace,
0,
0
},
only in patch2:
unchanged:
--- a/src/include/catalog/namespace.h
+++ b/src/include/catalog/namespace.h
@@ -135,15 +135,18 @@ extern Oid get_ts_config_oid(List *names, bool missing_ok);
extern bool TSConfigIsVisible(Oid cfgid);
extern void DeconstructQualifiedName(List *names,
- char **nspname_p,
- char **objname_p);
+ char **nspname_p,
+ char **modname_p,
+ char **objname_p,
+ bool check_module);
extern Oid LookupNamespaceNoError(const char *nspname);
extern Oid LookupExplicitNamespace(const char *nspname, bool missing_ok);
-extern Oid get_namespace_oid(const char *nspname, bool missing_ok);
+extern Oid get_namespace_oid(const char *nspname, Oid nspnamespace, bool missing_ok);
+extern Oid get_module_oid(List *names, bool missing_ok);
extern Oid LookupCreationNamespace(const char *nspname);
extern void CheckSetNamespace(Oid oldNspOid, Oid nspOid);
-extern Oid QualifiedNameGetCreationNamespace(List *names, char **objname_p);
+extern Oid QualifiedNameGetCreationNamespace(List *names, char **objname_p, bool check_module);
extern RangeVar *makeRangeVarFromNameList(List *names);
extern char *NameListToString(List *names);
extern char *NameListToQuotedString(List *names);
only in patch2:
unchanged:
--- a/src/include/catalog/pg_namespace.dat
+++ b/src/include/catalog/pg_namespace.dat
@@ -14,12 +14,15 @@
{ oid => '11', oid_symbol => 'PG_CATALOG_NAMESPACE',
descr => 'system catalog schema',
- nspname => 'pg_catalog', nspacl => '_null_' },
+ nspname => 'pg_catalog', nspnamespace => '0',
+ nspkind => 's', nspacl => '_null_' },
{ oid => '99', oid_symbol => 'PG_TOAST_NAMESPACE',
descr => 'reserved schema for TOAST tables',
- nspname => 'pg_toast', nspacl => '_null_' },
+ nspname => 'pg_toast', nspnamespace => '0',
+ nspkind => 's', nspacl => '_null_' },
{ oid => '2200', oid_symbol => 'PG_PUBLIC_NAMESPACE',
descr => 'standard public schema',
- nspname => 'public', nspacl => '_null_' },
+ nspname => 'public', nspnamespace => '0',
+ nspkind => 's', nspacl => '_null_' },
]
only in patch2:
unchanged:
--- a/src/include/catalog/pg_namespace.h
+++ b/src/include/catalog/pg_namespace.h
@@ -29,6 +29,8 @@
*
* nspname name of the namespace
* nspowner owner (creator) of the namespace
+ * nspnamespace parent of the namespace
+ * nspkind the type namespace
* nspacl access privilege list
* ----------------------------------------------------------------
*/
@@ -38,6 +40,9 @@ CATALOG(pg_namespace,2615,NamespaceRelationId)
NameData nspname;
Oid nspowner BKI_DEFAULT(POSTGRES) BKI_LOOKUP(pg_authid);
+ Oid nspnamespace BKI_DEFAULT(0);
+ char nspkind BKI_DEFAULT(s);
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
aclitem nspacl[1];
@@ -53,14 +58,18 @@ typedef FormData_pg_namespace *Form_pg_namespace;
DECLARE_TOAST(pg_namespace, 4163, 4164);
-DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
+DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops, nspnamespace oid_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX_PKEY(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
#define NamespaceOidIndexId 2685
+#define NSPKIND_SCHEMA 's' /* schema */
+#define NSPKIND_MODULE 'm' /* module */
+
/*
* prototypes for functions in pg_namespace.c
*/
-extern Oid NamespaceCreate(const char *nspName, Oid ownerId, bool isTemp);
+extern Oid NamespaceCreate(const char *nspName, Oid nspnamespace, char nspkind,
+ Oid ownerId, bool isTemp);
#endif /* PG_NAMESPACE_H */
[application/octet-stream] modules_poc_2.patch (7.5K, ../../CAB_5SRebSCjO12=nLsaLCBw2vnkiNH7jcNchirPc0yQ2KmiknQ@mail.gmail.com/3-modules_poc_2.patch)
download | inline diff:
unchanged:
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -40,6 +40,7 @@ OBJS = \
indexcmds.o \
lockcmds.o \
matview.o \
+ modulecmds.o \
opclasscmds.o \
operatorcmds.o \
policy.o \
unchanged:
--- /dev/null
+++ b/src/backend/commands/modulecmds.c
@@ -0,0 +1,202 @@
+/*-------------------------------------------------------------------------
+ *
+ * modulecmds.c
+ * module creation/manipulation commands
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/modulecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/table.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/namespace.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_authid.h"
+#include "catalog/pg_namespace.h"
+#include "commands/dbcommands.h"
+#include "commands/event_trigger.h"
+#include "commands/modulecmds.h"
+#include "miscadmin.h"
+#include "parser/parse_utilcmd.h"
+#include "tcop/utility.h"
+#include "utils/acl.h"
+#include "utils/builtins.h"
+#include "utils/rel.h"
+#include "utils/lsyscache.h"
+#include "utils/syscache.h"
+
+/*
+ * CREATE MODULE
+ *
+ * Note: caller should pass in location information for the whole
+ * CREATE MODULE statement, which in turn we pass down as the location
+ * of the component commands. This comports with our general plan of
+ * reporting location/len for the whole command even when executing
+ * a subquery.
+ */
+ObjectAddress
+CreateModuleCommand(ParseState *pstate, CreateModuleStmt *stmt, const char *queryString,
+ int stmt_location, int stmt_len)
+{
+ char *modulename;
+ Oid namespaceId;
+ Oid moduleId;
+ OverrideSearchPath *overridePath;
+ List *parsetree_list;
+ ListCell *parsetree_item;
+ Oid owner_uid;
+ Oid saved_uid;
+ int save_sec_context;
+ AclResult aclresult;
+ ObjectAddress myself,
+ referenced;
+ ObjectAddresses *addrs;
+
+ GetUserIdAndSecContext(&saved_uid, &save_sec_context);
+
+ /*
+ * Who is supposed to own the new module?
+ */
+ if (stmt->authrole)
+ owner_uid = get_rolespec_oid(stmt->authrole, false);
+ else
+ owner_uid = saved_uid;
+
+ /* Convert list of names to a name and namespace */
+ namespaceId = QualifiedNameGetCreationNamespace(stmt->modulename,
+ &modulename, false);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(namespaceId));
+
+ /*
+ * If if_not_exists was given and the module already exists, bail out.
+ * (Note: we needn't check this when not if_not_exists, because
+ * NamespaceCreate will complain anyway.) We could do this before making
+ * the permissions checks, but since CREATE TABLE IF NOT EXISTS makes its
+ * creation-permission check first, we do likewise.
+ */
+
+ if (stmt->if_not_exists &&
+ SearchSysCacheExists2(NAMESPACENAME, PointerGetDatum(modulename),
+ ObjectIdGetDatum(namespaceId)))
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_DUPLICATE_SCHEMA),
+ errmsg("module \"%s\" already exists, skipping",
+ modulename)));
+ return InvalidObjectAddress;
+ }
+
+ /*
+ * If the requested authorization is different from the current user,
+ * temporarily set the current user so that the object(s) will be created
+ * with the correct ownership.
+ *
+ * (The setting will be restored at the end of this routine, or in case of
+ * error, transaction abort will clean things up.)
+ */
+
+ if (saved_uid != owner_uid)
+ SetUserIdAndSecContext(owner_uid,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+ /* Create the module's namespace */
+ moduleId = NamespaceCreate(modulename, namespaceId, NSPKIND_MODULE,
+ owner_uid, false);
+
+ /* Advance cmd counter to make the namespace visible */
+ CommandCounterIncrement();
+
+ /*
+ * Temporarily make the new namespace be the front of the search path, as
+ * well as the default creation target namespace. This will be undone at
+ * the end of this routine, or upon error.
+ */
+ overridePath = GetOverrideSearchPath(CurrentMemoryContext);
+ overridePath->schemas = lcons_oid(moduleId, overridePath->schemas);
+ /* XXX should we clear overridePath->useTemp? */
+ PushOverrideSearchPath(overridePath);
+
+ /*
+ * Report the new module to possibly interested event triggers. Note we
+ * must do this here and not in ProcessUtilitySlow because otherwise the
+ * objects created below are reported before the module, which would be
+ * wrong.
+ */
+ ObjectAddressSet(myself, NamespaceRelationId, moduleId);
+ EventTriggerCollectSimpleCommand(myself, InvalidObjectAddress,
+ (Node *) stmt);
+
+ /*
+ * Examine the list of commands embedded in the CREATE MODULE command, and
+ * reorganize them into a sequentially executable order with no forward
+ * references. Note that the result is still a list of raw parsetrees ---
+ * we cannot, in general, run parse analysis on one statement until we
+ * have actually executed the prior ones.
+ */
+ parsetree_list = transformCreateModuleStmt(stmt);
+
+ /*
+ * Execute each command contained in the CREATE MODULE. Since the grammar
+ * allows only utility commands in CREATE MODULE, there is no need to pass
+ * them through parse_analyze() or the rewriter; we can just hand them
+ * straight to ProcessUtility.
+ */
+ foreach(parsetree_item, parsetree_list)
+ {
+ Node *stmt = (Node *) lfirst(parsetree_item);
+ PlannedStmt *wrapper;
+
+ wrapper = makeNode(PlannedStmt);
+ wrapper->commandType = CMD_UTILITY;
+ wrapper->canSetTag = false;
+ wrapper->utilityStmt = stmt;
+ wrapper->stmt_location = stmt_location;
+ wrapper->stmt_len = stmt_len;
+
+ ProcessUtility(wrapper,
+ queryString,
+ PROCESS_UTILITY_SUBCOMMAND,
+ NULL,
+ NULL,
+ None_Receiver,
+ NULL);
+
+ CommandCounterIncrement();
+ }
+
+ /* Reset search path to normal state */
+ PopOverrideSearchPath();
+
+ /* Reset current user and security context */
+ SetUserIdAndSecContext(saved_uid, save_sec_context);
+
+ addrs = new_object_addresses();
+
+
+ /* dependency on namespace */
+ ObjectAddressSet(referenced, NamespaceRelationId, namespaceId);
+ add_exact_object_address(&referenced, addrs);
+
+ record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL);
+ free_object_addresses(addrs);
+
+ recordDependencyOnOwner(NamespaceRelationId, moduleId, owner_uid);
+
+ return myself;
+}
only in patch2:
unchanged:
--- /dev/null
+++ b/src/include/commands/modulecmds.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * modulecmds.h
+ * prototypes for modulecmds.c.
+ *
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/commands/modulecmds.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef MODULECMDS_H
+#define MODULECMDS_H
+
+#include "catalog/objectaddress.h"
+#include "nodes/parsenodes.h"
+
+extern ObjectAddress CreateModuleCommand(ParseState *pstate,
+ CreateModuleStmt *parsetree,
+ const char *queryString,
+ int stmt_location, int stmt_len);
+
+#endif /* MODULECMDS_H */
[application/octet-stream] modules_poc_3.patch (28.8K, ../../CAB_5SRebSCjO12=nLsaLCBw2vnkiNH7jcNchirPc0yQ2KmiknQ@mail.gmail.com/4-modules_poc_3.patch)
download | inline diff:
unchanged:
--- a/src/backend/catalog/pg_operator.c
+++ b/src/backend/catalog/pg_operator.c
@@ -604,7 +604,8 @@ get_other_operator(List *otherOp, Oid otherLeftTypeId, Oid otherRightTypeId,
}
otherNamespace = QualifiedNameGetCreationNamespace(otherOp,
- &otherName);
+ &otherName,
+ false);
if (strcmp(otherName, operatorName) == 0 &&
otherNamespace == operatorNamespace &&
unchanged:
--- a/src/backend/commands/aggregatecmds.c
+++ b/src/backend/commands/aggregatecmds.c
@@ -101,7 +101,7 @@ DefineAggregate(ParseState *pstate,
ListCell *pl;
/* Convert list of names to a name and namespace */
- aggNamespace = QualifiedNameGetCreationNamespace(name, &aggName);
+ aggNamespace = QualifiedNameGetCreationNamespace(name, &aggName, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(aggNamespace, GetUserId(), ACL_CREATE);
unchanged:
--- a/src/backend/commands/alter.c
+++ b/src/backend/commands/alter.c
@@ -384,6 +384,7 @@ ExecRenameStmt(RenameStmt *stmt)
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
case OBJECT_LANGUAGE:
+ case OBJECT_MODULE:
case OBJECT_PROCEDURE:
case OBJECT_ROUTINE:
case OBJECT_STATISTIC_EXT:
unchanged:
--- a/src/backend/commands/collationcmds.c
+++ b/src/backend/commands/collationcmds.c
@@ -73,7 +73,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
Oid newoid;
ObjectAddress address;
- collNamespace = QualifiedNameGetCreationNamespace(names, &collName);
+ collNamespace = QualifiedNameGetCreationNamespace(names, &collName, false);
aclresult = pg_namespace_aclcheck(collNamespace, GetUserId(), ACL_CREATE);
if (aclresult != ACLCHECK_OK)
unchanged:
--- a/src/backend/commands/conversioncmds.c
+++ b/src/backend/commands/conversioncmds.c
@@ -51,7 +51,7 @@ CreateConversionCommand(CreateConversionStmt *stmt)
/* Convert list of names to a name and namespace */
namespaceId = QualifiedNameGetCreationNamespace(stmt->conversion_name,
- &conversion_name);
+ &conversion_name, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(), ACL_CREATE);
unchanged:
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -967,6 +967,7 @@ EventTriggerSupportsObjectType(ObjectType obtype)
case OBJECT_LANGUAGE:
case OBJECT_LARGEOBJECT:
case OBJECT_MATVIEW:
+ case OBJECT_MODULE:
case OBJECT_OPCLASS:
case OBJECT_OPERATOR:
case OBJECT_OPFAMILY:
@@ -2087,6 +2088,8 @@ stringify_grant_objtype(ObjectType objtype)
return "LANGUAGE";
case OBJECT_LARGEOBJECT:
return "LARGE OBJECT";
+ case OBJECT_MODULE:
+ return "MODULE";
case OBJECT_SCHEMA:
return "SCHEMA";
case OBJECT_PROCEDURE:
@@ -2169,6 +2172,8 @@ stringify_adefprivs_objtype(ObjectType objtype)
return "LANGUAGES";
case OBJECT_LARGEOBJECT:
return "LARGE OBJECTS";
+ case OBJECT_MODULE:
+ return "MODULES";
case OBJECT_SCHEMA:
return "SCHEMAS";
case OBJECT_PROCEDURE:
unchanged:
--- a/src/backend/commands/functioncmds.c
+++ b/src/backend/commands/functioncmds.c
@@ -148,7 +148,7 @@ compute_return_type(TypeName *returnType, Oid languageOid,
errmsg("type \"%s\" is not yet defined", typnam),
errdetail("Creating a shell type definition.")));
namespaceId = QualifiedNameGetCreationNamespace(returnType->names,
- &typname);
+ &typname, false);
aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(),
ACL_CREATE);
if (aclresult != ACLCHECK_OK)
@@ -1050,7 +1050,7 @@ CreateFunction(ParseState *pstate, CreateFunctionStmt *stmt)
/* Convert list of names to a name and namespace */
namespaceId = QualifiedNameGetCreationNamespace(stmt->funcname,
- &funcname);
+ &funcname, true);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(), ACL_CREATE);
unchanged:
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2039,7 +2039,7 @@ ResolveOpClass(List *opclass, Oid attrType,
*/
/* deconstruct the name list */
- DeconstructQualifiedName(opclass, &schemaname, &opcname);
+ DeconstructQualifiedName(opclass, &schemaname, NULL, &opcname, false);
if (schemaname)
{
@@ -2787,7 +2787,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind,
*/
if (objectKind == REINDEX_OBJECT_SCHEMA)
{
- objectOid = get_namespace_oid(objectName, false);
+ objectOid = get_namespace_oid(objectName, InvalidOid, false);
if (!pg_namespace_ownercheck(objectOid, GetUserId()))
aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_SCHEMA,
unchanged:
--- a/src/backend/commands/opclasscmds.c
+++ b/src/backend/commands/opclasscmds.c
@@ -85,7 +85,7 @@ OpFamilyCacheLookup(Oid amID, List *opfamilyname, bool missing_ok)
HeapTuple htup;
/* deconstruct the name list */
- DeconstructQualifiedName(opfamilyname, &schemaname, &opfname);
+ DeconstructQualifiedName(opfamilyname, &schemaname, NULL, &opfname, false);
if (schemaname)
{
@@ -166,7 +166,7 @@ OpClassCacheLookup(Oid amID, List *opclassname, bool missing_ok)
HeapTuple htup;
/* deconstruct the name list */
- DeconstructQualifiedName(opclassname, &schemaname, &opcname);
+ DeconstructQualifiedName(opclassname, &schemaname, NULL, &opcname, false);
if (schemaname)
{
@@ -354,7 +354,7 @@ DefineOpClass(CreateOpClassStmt *stmt)
/* Convert list of names to a name and namespace */
namespaceoid = QualifiedNameGetCreationNamespace(stmt->opclassname,
- &opcname);
+ &opcname, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(namespaceoid, GetUserId(), ACL_CREATE);
@@ -769,7 +769,7 @@ DefineOpFamily(CreateOpFamilyStmt *stmt)
/* Convert list of names to a name and namespace */
namespaceoid = QualifiedNameGetCreationNamespace(stmt->opfamilyname,
- &opfname);
+ &opfname, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(namespaceoid, GetUserId(), ACL_CREATE);
unchanged:
--- a/src/backend/commands/operatorcmds.c
+++ b/src/backend/commands/operatorcmds.c
@@ -87,7 +87,7 @@ DefineOperator(List *names, List *parameters)
ListCell *pl;
/* Convert list of names to a name and namespace */
- oprNamespace = QualifiedNameGetCreationNamespace(names, &oprName);
+ oprNamespace = QualifiedNameGetCreationNamespace(names, &oprName, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(oprNamespace, GetUserId(), ACL_CREATE);
unchanged:
--- a/src/backend/commands/seclabel.c
+++ b/src/backend/commands/seclabel.c
@@ -48,6 +48,7 @@ SecLabelSupportsObjectType(ObjectType objtype)
case OBJECT_LANGUAGE:
case OBJECT_LARGEOBJECT:
case OBJECT_MATVIEW:
+ case OBJECT_MODULE:
case OBJECT_PROCEDURE:
case OBJECT_PUBLICATION:
case OBJECT_ROLE:
unchanged:
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -163,7 +163,7 @@ CreateStatistics(CreateStatsStmt *stmt)
*/
if (stmt->defnames)
namespaceId = QualifiedNameGetCreationNamespace(stmt->defnames,
- &namestr);
+ &namestr, false);
else
{
namespaceId = RelationGetNamespace(rel);
@@ -630,7 +630,7 @@ AlterStatistics(AlterStatsStmt *stmt)
Assert(stmt->missing_ok);
- DeconstructQualifiedName(stmt->defnames, &schemaname, &statname);
+ DeconstructQualifiedName(stmt->defnames, &schemaname, NULL, &statname, false);
if (schemaname)
ereport(NOTICE,
unchanged:
--- a/src/backend/commands/tsearchcmds.c
+++ b/src/backend/commands/tsearchcmds.c
@@ -196,7 +196,7 @@ DefineTSParser(List *names, List *parameters)
prsRel = table_open(TSParserRelationId, RowExclusiveLock);
/* Convert list of names to a name and namespace */
- namespaceoid = QualifiedNameGetCreationNamespace(names, &prsname);
+ namespaceoid = QualifiedNameGetCreationNamespace(names, &prsname, false);
/* initialize tuple fields with name/namespace */
memset(values, 0, sizeof(values));
@@ -405,7 +405,7 @@ DefineTSDictionary(List *names, List *parameters)
ObjectAddress address;
/* Convert list of names to a name and namespace */
- namespaceoid = QualifiedNameGetCreationNamespace(names, &dictname);
+ namespaceoid = QualifiedNameGetCreationNamespace(names, &dictname, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(namespaceoid, GetUserId(), ACL_CREATE);
@@ -701,7 +701,7 @@ DefineTSTemplate(List *names, List *parameters)
errmsg("must be superuser to create text search templates")));
/* Convert list of names to a name and namespace */
- namespaceoid = QualifiedNameGetCreationNamespace(names, &tmplname);
+ namespaceoid = QualifiedNameGetCreationNamespace(names, &tmplname, false);
tmplRel = table_open(TSTemplateRelationId, RowExclusiveLock);
@@ -908,7 +908,7 @@ DefineTSConfiguration(List *names, List *parameters, ObjectAddress *copied)
ObjectAddress address;
/* Convert list of names to a name and namespace */
- namespaceoid = QualifiedNameGetCreationNamespace(names, &cfgname);
+ namespaceoid = QualifiedNameGetCreationNamespace(names, &cfgname, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(namespaceoid, GetUserId(), ACL_CREATE);
unchanged:
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -217,7 +217,7 @@ DefineType(ParseState *pstate, List *names, List *parameters)
errmsg("must be superuser to create a base type")));
/* Convert list of names to a name and namespace */
- typeNamespace = QualifiedNameGetCreationNamespace(names, &typeName);
+ typeNamespace = QualifiedNameGetCreationNamespace(names, &typeName, false);
#ifdef NOT_USED
/* XXX this is unnecessary given the superuser check above */
@@ -733,7 +733,7 @@ DefineDomain(CreateDomainStmt *stmt)
/* Convert list of names to a name and namespace */
domainNamespace = QualifiedNameGetCreationNamespace(stmt->domainname,
- &domainName);
+ &domainName, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(domainNamespace, GetUserId(),
@@ -1149,7 +1149,7 @@ DefineEnum(CreateEnumStmt *stmt)
/* Convert list of names to a name and namespace */
enumNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
- &enumName);
+ &enumName, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(enumNamespace, GetUserId(), ACL_CREATE);
@@ -1369,7 +1369,7 @@ DefineRange(CreateRangeStmt *stmt)
/* Convert list of names to a name and namespace */
typeNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
- &typeName);
+ &typeName, false);
/* Check we have creation rights in target namespace */
aclresult = pg_namespace_aclcheck(typeNamespace, GetUserId(), ACL_CREATE);
@@ -1457,7 +1457,7 @@ DefineRange(CreateRangeStmt *stmt)
errmsg("conflicting or redundant options")));
/* we can look up the subtype name immediately */
multirangeNamespace = QualifiedNameGetCreationNamespace(defGetQualifiedName(defel),
- &multirangeTypeName);
+ &multirangeTypeName, false);
}
else
ereport(ERROR,
unchanged:
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -262,7 +262,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
struct GroupClause *groupclause;
}
-%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt
+%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt module_stmt
AlterEventTrigStmt AlterCollationStmt
AlterDatabaseStmt AlterDatabaseSetStmt AlterDomainStmt AlterEnumStmt
AlterFdwStmt AlterForeignServerStmt AlterGroupStmt
@@ -301,6 +301,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
CreateMatViewStmt RefreshMatViewStmt CreateAmStmt
CreatePublicationStmt AlterPublicationStmt
CreateSubscriptionStmt AlterSubscriptionStmt DropSubscriptionStmt
+ CreateModuleStmt
%type <node> select_no_parens select_with_parens select_clause
simple_select values_clause
@@ -344,7 +345,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <str> opt_in_database
%type <str> OptSchemaName
-%type <list> OptSchemaEltList
+%type <list> OptSchemaEltList OptModuleEltList
%type <chr> am_type
@@ -368,7 +369,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> func_name handler_name qual_Op qual_all_Op subquery_Op
opt_class opt_inline_handler opt_validator validator_clause
- opt_collate
+ opt_collate module_name
%type <range> qualified_name insert_target OptConstrFromTable
@@ -681,7 +682,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED
- MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE
+ MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MODULE
+ MONTH_P MOVE
NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NFC NFD NFKC NFKD NO NONE
NORMALIZE NORMALIZED
@@ -954,6 +956,7 @@ stmt:
| CreateFunctionStmt
| CreateGroupStmt
| CreateMatViewStmt
+ | CreateModuleStmt
| CreateOpClassStmt
| CreateOpFamilyStmt
| CreatePublicationStmt
@@ -1466,6 +1469,84 @@ schema_stmt:
| ViewStmt
;
+/*****************************************************************************
+ *
+ * Manipulate a module
+ *
+ *****************************************************************************/
+
+CreateModuleStmt:
+ CREATE MODULE module_name OptModuleEltList
+ {
+ CreateModuleStmt *n = makeNode(CreateModuleStmt);
+ n->modulename = $3;
+ n->authrole = NULL;
+ n->moduleElts = $4;
+ n->if_not_exists = false;
+ $$ = (Node *)n;
+ }
+ | CREATE MODULE module_name OWNER RoleSpec OptModuleEltList
+ {
+ CreateModuleStmt *n = makeNode(CreateModuleStmt);
+ n->modulename = $3;
+ n->authrole = $5;
+ n->moduleElts = $6;
+ n->if_not_exists = false;
+ $$ = (Node *)n;
+ }
+ | CREATE MODULE IF_P NOT EXISTS module_name OptModuleEltList
+ {
+ CreateModuleStmt *n = makeNode(CreateModuleStmt);
+ n->modulename = $6;
+ n->authrole = NULL;
+ if ($7 != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("CREATE MODULE IF NOT EXISTS cannot include module elements"),
+ parser_errposition(@7)));
+ n->moduleElts = $7;
+ n->if_not_exists = true;
+ $$ = (Node *)n;
+ }
+ | CREATE MODULE IF_P NOT EXISTS module_name OWNER RoleSpec OptModuleEltList
+ {
+ CreateModuleStmt *n = makeNode(CreateModuleStmt);
+ n->modulename = $6;
+ n->authrole = $8;
+ if ($9 != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("CREATE MODULE IF NOT EXISTS cannot include module elements"),
+ parser_errposition(@9)));
+ n->moduleElts = $9;
+ n->if_not_exists = true;
+ $$ = (Node *)n;
+ }
+ ;
+
+module_name:
+ name { $$ = list_make1(makeString($1)); }
+ | name attrs { $$ = lcons(makeString($1), $2); }
+ ;
+
+OptModuleEltList:
+ OptModuleEltList module_stmt
+ {
+ if (@$ < 0) /* see comments for YYLLOC_DEFAULT */
+ @$ = @2;
+ $$ = lappend($1, $2);
+ }
+ | /* EMPTY */
+ { $$ = NIL; }
+ ;
+
+/*
+ * module_stmt are the ones that can show up inside a CREATE MODULE statement.
+ */
+module_stmt:
+ CreateFunctionStmt
+ ;
+
/*****************************************************************************
*
@@ -6308,6 +6389,26 @@ DropStmt: DROP object_type_any_name IF_P EXISTS any_name_list opt_drop_behavior
n->concurrent = true;
$$ = (Node *)n;
}
+ | DROP MODULE any_name_list opt_drop_behavior
+ {
+ DropStmt *n = makeNode(DropStmt);
+ n->removeType = OBJECT_MODULE;
+ n->objects = $3;
+ n->behavior = $4;
+ n->missing_ok = false;
+ n->concurrent = false;
+ $$ = (Node *)n;
+ }
+ | DROP MODULE IF_P EXISTS any_name_list opt_drop_behavior
+ {
+ DropStmt *n = makeNode(DropStmt);
+ n->removeType = OBJECT_MODULE;
+ n->objects = $5;
+ n->behavior = $6;
+ n->missing_ok = true;
+ n->concurrent = false;
+ $$ = (Node *)n;
+ }
;
/* object types taking any_name/any_name_list */
@@ -8569,6 +8670,15 @@ RenameStmt: ALTER AGGREGATE aggregate_with_argtypes RENAME TO name
n->missing_ok = false;
$$ = (Node *)n;
}
+ | ALTER MODULE module_name RENAME TO name
+ {
+ RenameStmt *n = makeNode(RenameStmt);
+ n->renameType = OBJECT_MODULE;
+ n->object = (Node *) $3;
+ n->newname = $6;
+ n->missing_ok = false;
+ $$ = (Node *)n;
+ }
| ALTER opt_procedural LANGUAGE name RENAME TO name
{
RenameStmt *n = makeNode(RenameStmt);
@@ -15639,6 +15749,7 @@ unreserved_keyword:
| MINUTE_P
| MINVALUE
| MODE
+ | MODULE
| MONTH_P
| MOVE
| NAME_P
@@ -16205,6 +16316,7 @@ bare_label_keyword:
| METHOD
| MINVALUE
| MODE
+ | MODULE
| MOVE
| NAME_P
| NAMES
unchanged:
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -18,8 +18,10 @@
#include "catalog/pg_aggregate.h"
#include "catalog/pg_proc.h"
#include "catalog/pg_type.h"
+#include "commands/dbcommands.h"
#include "funcapi.h"
#include "lib/stringinfo.h"
+#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "parser/parse_agg.h"
@@ -1421,7 +1423,6 @@ func_get_detail(List *funcname,
raw_candidates = FuncnameGetCandidates(funcname, nargs, fargnames,
expand_variadic, expand_defaults,
false);
-
/*
* Quickly check if there is an exact match to the input datatypes (there
* can be only one)
@@ -1549,7 +1550,6 @@ func_get_detail(List *funcname,
/* one match only? then run with it... */
if (ncandidates == 1)
best_candidate = current_candidates;
-
/*
* multiple candidates? then better decide or throw an error...
*/
@@ -1882,6 +1882,16 @@ FuncNameAsType(List *funcname)
Oid result;
Type typtup;
+ /*
+ * check if this may be in a module. If it could be, don't check if it
+ * may be a type since they can not be in a module
+ */
+ if (list_length(funcname) == 3)
+ {
+ if (strcmp(strVal(linitial(funcname)), get_database_name(MyDatabaseId)) != 0)
+ return InvalidOid;
+ }
+
/*
* temp_ok=false protects the <refsect1 id="sql-createfunction-security">
* contract for writing SECURITY DEFINER functions safely.
@@ -2058,7 +2068,6 @@ LookupFuncNameInternal(List *funcname, int nargs, const Oid *argtypes,
clist = FuncnameGetCandidates(funcname, nargs, NIL, false, false,
missing_ok);
-
/*
* If no arguments were specified, the name must yield a unique candidate.
*/
unchanged:
--- a/src/backend/parser/parse_oper.c
+++ b/src/backend/parser/parse_oper.c
@@ -954,7 +954,7 @@ make_oper_cache_key(ParseState *pstate, OprCacheKey *key, List *opname,
char *opername;
/* deconstruct the name list */
- DeconstructQualifiedName(opname, &schemaname, &opername);
+ DeconstructQualifiedName(opname, &schemaname, NULL, &opername, false);
/* ensure zero-fill for stable hashing */
MemSet(key, 0, sizeof(OprCacheKey));
unchanged:
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -166,7 +166,7 @@ LookupTypeNameExtended(ParseState *pstate,
char *typname;
/* deconstruct the name list */
- DeconstructQualifiedName(typeName->names, &schemaname, &typname);
+ DeconstructQualifiedName(typeName->names, &schemaname, NULL, &typname, false);
if (schemaname)
{
unchanged:
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -113,6 +113,14 @@ typedef struct
List *grants; /* GRANT items */
} CreateSchemaStmtContext;
+/* State shared by transformCreateModuleStmt and its subroutines */
+typedef struct
+{
+ const char *stmtType; /* "CREATE MODULE" or "ALTER MODULE" */
+ List *modulename; /* name of module */
+ RoleSpec *authrole; /* owner of module */
+ List *functions; /* CREATE FUNCTION items */
+} CreateModuleStmtContext;
static void transformColumnDefinition(CreateStmtContext *cxt,
ColumnDef *column);
@@ -3944,6 +3952,70 @@ transformCreateSchemaStmt(CreateSchemaStmt *stmt)
return result;
}
+/*
+ * transformCreateModuleStmt -
+ * analyzes the CREATE MODULE statement
+ *
+ * Split the module element list into individual commands and place
+ * them in the result list in an order such that there are no forward
+ * references. Currently there are only functions allowed in modules
+ * but the spec allows variables and temp tables so this provides a
+ * way to have them created before the functions that could use them.
+ *
+ * The functions are also checked to make sure there is not an explicit
+ * namespace attempted to be used.
+ */
+List *
+transformCreateModuleStmt(CreateModuleStmt *stmt)
+{
+ CreateModuleStmtContext cxt;
+ List *result;
+ ListCell *elements;
+
+
+ cxt.stmtType = "CREATE MODULE";
+ cxt.modulename = stmt->modulename;
+ cxt.authrole = (RoleSpec *) stmt->authrole;
+ cxt.functions = NIL;
+
+ /*
+ * Run through each module element in the module element list. Separate
+ * statements by type, and do preliminary analysis.
+ */
+ foreach(elements, stmt->moduleElts)
+ {
+ Node *element = lfirst(elements);
+
+ switch (nodeTag(element))
+ {
+ case T_CreateFunctionStmt:
+ {
+ CreateFunctionStmt *elp = (CreateFunctionStmt *) element;
+
+ if (list_length(elp->funcname) > 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_SCHEMA_DEFINITION),
+ errmsg("CREATE FUNCTION (%s) specifies a "
+ "namespace inside of CREATE MODULE (%s)",
+ NameListToString(elp->funcname),
+ NameListToString(cxt.modulename))));
+
+ cxt.functions = lappend(cxt.functions, element);
+ }
+ break;
+
+ default:
+ elog(ERROR, "unrecognized node type: %d",
+ (int) nodeTag(element));
+ }
+ }
+
+ result = NIL;
+ result = list_concat(result, cxt.functions);
+
+ return result;
+}
+
/*
* setSchemaName
* Set or check schema name in an element of a CREATE SCHEMA command
unchanged:
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -42,6 +42,7 @@
#include "commands/extension.h"
#include "commands/lockcmds.h"
#include "commands/matview.h"
+#include "commands/modulecmds.h"
#include "commands/policy.h"
#include "commands/portalcmds.h"
#include "commands/prepare.h"
@@ -178,6 +179,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateForeignServerStmt:
case T_CreateForeignTableStmt:
case T_CreateFunctionStmt:
+ case T_CreateModuleStmt:
case T_CreateOpClassStmt:
case T_CreateOpFamilyStmt:
case T_CreatePLangStmt:
@@ -1097,6 +1099,19 @@ ProcessUtilitySlow(ParseState *pstate,
commandCollected = true;
break;
+ case T_CreateModuleStmt: /* CREATE Module */
+ CreateModuleCommand(pstate, (CreateModuleStmt *) parsetree,
+ queryString,
+ pstmt->stmt_location,
+ pstmt->stmt_len);
+ /*
+ * EventTriggerCollectSimpleCommand called by
+ * CreateModuleCommand
+ */
+ commandCollected = true;
+ break;
+
+
case T_CreateStmt:
case T_CreateForeignTableStmt:
{
@@ -2280,6 +2295,9 @@ AlterObjectTypeCommandTag(ObjectType objtype)
case OBJECT_STATISTIC_EXT:
tag = CMDTAG_ALTER_STATISTICS;
break;
+ case OBJECT_MODULE:
+ tag = CMDTAG_ALTER_MODULE;
+ break;
default:
tag = CMDTAG_UNKNOWN;
break;
@@ -2584,6 +2602,9 @@ CreateCommandTag(Node *parsetree)
case OBJECT_STATISTIC_EXT:
tag = CMDTAG_DROP_STATISTICS;
break;
+ case OBJECT_MODULE:
+ tag = CMDTAG_DROP_MODULE;
+ break;
default:
tag = CMDTAG_UNKNOWN;
}
@@ -2739,6 +2760,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_FUNCTION;
break;
+ case T_CreateModuleStmt:
+ tag = CMDTAG_CREATE_MODULE;
+ break;
+
case T_IndexStmt:
tag = CMDTAG_CREATE_INDEX;
break;
@@ -3382,6 +3407,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateModuleStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_IndexStmt:
lev = LOGSTMT_DDL;
break;
only in patch2:
unchanged:
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -432,6 +432,7 @@ typedef enum NodeTag
T_AlterCollationStmt,
T_CallStmt,
T_AlterStatsStmt,
+ T_CreateModuleStmt,
/*
* TAGS FOR PARSE TREE NODES (parsenodes.h)
only in patch2:
unchanged:
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1798,6 +1798,7 @@ typedef enum ObjectType
OBJECT_LANGUAGE,
OBJECT_LARGEOBJECT,
OBJECT_MATVIEW,
+ OBJECT_MODULE,
OBJECT_OPCLASS,
OBJECT_OPERATOR,
OBJECT_OPFAMILY,
@@ -2975,6 +2976,24 @@ typedef struct AlterFunctionStmt
List *actions; /* list of DefElem */
} AlterFunctionStmt;
+/* ----------------------
+ * Create Module Statement
+ *
+ * NOTE: the moduleElts list contains raw parsetrees for component statements
+ * of the schema, such as CREATE FUNCTION, CREATE PROCEDURE, etc. These are
+ * analyzed and executed after the module itself is created.
+ * ----------------------
+ */
+typedef struct CreateModuleStmt
+{
+ NodeTag type;
+ List *modulename; /* the name of the schema to create */
+ RoleSpec *authrole; /* the owner of the created schema */
+ List *moduleElts; /* module components (list of parsenodes) */
+ bool if_not_exists; /* just do nothing if module already exists? */
+} CreateModuleStmt;
+
+
/* ----------------------
* DO Statement
*
only in patch2:
unchanged:
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -257,6 +257,7 @@ PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("minvalue", MINVALUE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("mode", MODE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("module", MODULE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("month", MONTH_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("move", MOVE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("name", NAME_P, UNRESERVED_KEYWORD, BARE_LABEL)
only in patch2:
unchanged:
--- a/src/include/parser/parse_utilcmd.h
+++ b/src/include/parser/parse_utilcmd.h
@@ -31,6 +31,7 @@ extern CreateStatsStmt *transformStatsStmt(Oid relid, CreateStatsStmt *stmt,
extern void transformRuleStmt(RuleStmt *stmt, const char *queryString,
List **actions, Node **whereClause);
extern List *transformCreateSchemaStmt(CreateSchemaStmt *stmt);
+extern List *transformCreateModuleStmt(CreateModuleStmt *stmt);
extern PartitionBoundSpec *transformPartitionBound(ParseState *pstate, Relation parent,
PartitionBoundSpec *spec);
extern List *expandTableLikeClause(RangeVar *heapRel,
only in patch2:
unchanged:
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -43,6 +43,7 @@ PG_CMDTAG(CMDTAG_ALTER_INDEX, "ALTER INDEX", true, false, false)
PG_CMDTAG(CMDTAG_ALTER_LANGUAGE, "ALTER LANGUAGE", true, false, false)
PG_CMDTAG(CMDTAG_ALTER_LARGE_OBJECT, "ALTER LARGE OBJECT", true, false, false)
PG_CMDTAG(CMDTAG_ALTER_MATERIALIZED_VIEW, "ALTER MATERIALIZED VIEW", true, false, false)
+PG_CMDTAG(CMDTAG_ALTER_MODULE, "ALTER MODULE", true, false, false)
PG_CMDTAG(CMDTAG_ALTER_OPERATOR, "ALTER OPERATOR", true, false, false)
PG_CMDTAG(CMDTAG_ALTER_OPERATOR_CLASS, "ALTER OPERATOR CLASS", true, false, false)
PG_CMDTAG(CMDTAG_ALTER_OPERATOR_FAMILY, "ALTER OPERATOR FAMILY", true, false, false)
@@ -98,6 +99,7 @@ PG_CMDTAG(CMDTAG_CREATE_FUNCTION, "CREATE FUNCTION", true, false, false)
PG_CMDTAG(CMDTAG_CREATE_INDEX, "CREATE INDEX", true, false, false)
PG_CMDTAG(CMDTAG_CREATE_LANGUAGE, "CREATE LANGUAGE", true, false, false)
PG_CMDTAG(CMDTAG_CREATE_MATERIALIZED_VIEW, "CREATE MATERIALIZED VIEW", true, false, false)
+PG_CMDTAG(CMDTAG_CREATE_MODULE, "CREATE MODULE", true, false, false)
PG_CMDTAG(CMDTAG_CREATE_OPERATOR, "CREATE OPERATOR", true, false, false)
PG_CMDTAG(CMDTAG_CREATE_OPERATOR_CLASS, "CREATE OPERATOR CLASS", true, false, false)
PG_CMDTAG(CMDTAG_CREATE_OPERATOR_FAMILY, "CREATE OPERATOR FAMILY", true, false, false)
@@ -150,6 +152,7 @@ PG_CMDTAG(CMDTAG_DROP_FUNCTION, "DROP FUNCTION", true, false, false)
PG_CMDTAG(CMDTAG_DROP_INDEX, "DROP INDEX", true, false, false)
PG_CMDTAG(CMDTAG_DROP_LANGUAGE, "DROP LANGUAGE", true, false, false)
PG_CMDTAG(CMDTAG_DROP_MATERIALIZED_VIEW, "DROP MATERIALIZED VIEW", true, false, false)
+PG_CMDTAG(CMDTAG_DROP_MODULE, "DROP MODULE", true, false, false)
PG_CMDTAG(CMDTAG_DROP_OPERATOR, "DROP OPERATOR", true, false, false)
PG_CMDTAG(CMDTAG_DROP_OPERATOR_CLASS, "DROP OPERATOR CLASS", true, false, false)
PG_CMDTAG(CMDTAG_DROP_OPERATOR_FAMILY, "DROP OPERATOR FAMILY", true, false, false)
[application/octet-stream] modules_poc_4.patch (4.5K, ../../CAB_5SRebSCjO12=nLsaLCBw2vnkiNH7jcNchirPc0yQ2KmiknQ@mail.gmail.com/5-modules_poc_4.patch)
download | inline diff:
diff --git a/src/test/regress/expected/create_module.out b/src/test/regress/expected/create_module.out
new file mode 100644
index 0000000000..634df03e4c
--- /dev/null
+++ b/src/test/regress/expected/create_module.out
@@ -0,0 +1,83 @@
+CREATE TABLE cm_test (a int, b text);
+CREATE MODULE mtest1
+ CREATE FUNCTION m1testa() RETURNS text
+ LANGUAGE sql
+ RETURN '1x'
+ CREATE FUNCTION m1testb() RETURNS text
+ LANGUAGE sql
+ RETURN '1y';
+CREATE SCHEMA temp_mod_test;
+GRANT ALL ON SCHEMA temp_mod_test TO public;
+CREATE MODULE temp_mod_test.mtest2
+ CREATE PROCEDURE m2testa(x text)
+ LANGUAGE SQL
+ AS $$
+ INSERT INTO cm_test VALUES (1, x);
+ $$
+ CREATE FUNCTION m2testb() RETURNS text
+ LANGUAGE sql
+ RETURN '2y';
+CREATE MODULE mtest3
+ CREATE FUNCTION mtest3.m3testa() RETURNS text
+ LANGUAGE sql
+ RETURN '3x';
+ERROR: CREATE FUNCTION (mtest3.m3testa) specifies a namespace inside of CREATE MODULE (mtest3)
+SELECT mtest1.m1testa();
+ m1testa
+---------
+ 1x
+(1 row)
+
+SELECT mtest1.m1testb();
+ m1testb
+---------
+ 1y
+(1 row)
+
+SELECT public.mtest1.m1testa();
+ m1testa
+---------
+ 1x
+(1 row)
+
+SELECT public.mtest1.m1testb();
+ m1testb
+---------
+ 1y
+(1 row)
+
+SELECT temp_mod_test.mtest2.m2testb();
+ m2testb
+---------
+ 2y
+(1 row)
+
+SELECT temp_mod_test.mtest2.m2testa('x'); -- error
+ERROR: temp_mod_test.mtest2.m2testa(unknown) is a procedure
+LINE 1: SELECT temp_mod_test.mtest2.m2testa('x');
+ ^
+HINT: To call a procedure, use CALL.
+CALL temp_mod_test.mtest2.m2testa('a'); -- ok
+CALL temp_mod_test.mtest2.m2testa('xy' || 'zzy'); -- ok, constant-folded arg
+CREATE PROCEDURE mtest1.m1testc(x text)
+ LANGUAGE SQL
+ AS $$
+ INSERT INTO cm_test VALUES (2, x);
+ $$;
+ERROR: schema "mtest1" does not exist
+CALL mtest1.m1testc('a'); -- ok
+ERROR: schema "mtest1" does not exist
+LINE 1: CALL mtest1.m1testc('a');
+ ^
+DROP PROCEDURE mtest1.m1testc(text);
+ERROR: procedure mtest1.m1testc(text) does not exist
+DROP FUNCTION temp_mod_test.mtest2.m2testb();
+-- cleanup
+DROP MODULE mtest1 CASCADE;
+NOTICE: drop cascades to 2 other objects
+DETAIL: drop cascades to function mtest1.m1testa()
+drop cascades to function mtest1.m1testb()
+DROP MODULE temp_mod_test.mtest2 CASCADE;
+NOTICE: drop cascades to function mtest2.m2testa(text)
+DROP SCHEMA temp_mod_test;
+DROP TABLE cm_test;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 22b0d3584d..e572bf212d 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -51,7 +51,7 @@ test: copy copyselect copydml insert insert_conflict
# ----------
# More groups of parallel tests
# ----------
-test: create_misc create_operator create_procedure
+test: create_misc create_operator create_procedure create_module
# These depend on create_misc and create_operator
test: create_index create_index_spgist create_view index_including index_including_gist
diff --git a/src/test/regress/sql/create_module.sql b/src/test/regress/sql/create_module.sql
new file mode 100644
index 0000000000..8c1db1e636
--- /dev/null
+++ b/src/test/regress/sql/create_module.sql
@@ -0,0 +1,58 @@
+CREATE TABLE cm_test (a int, b text);
+
+CREATE MODULE mtest1
+ CREATE FUNCTION m1testa() RETURNS text
+ LANGUAGE sql
+ RETURN '1x'
+ CREATE FUNCTION m1testb() RETURNS text
+ LANGUAGE sql
+ RETURN '1y';
+
+CREATE SCHEMA temp_mod_test;
+GRANT ALL ON SCHEMA temp_mod_test TO public;
+
+CREATE MODULE temp_mod_test.mtest2
+ CREATE PROCEDURE m2testa(x text)
+ LANGUAGE SQL
+ AS $$
+ INSERT INTO cm_test VALUES (1, x);
+ $$
+ CREATE FUNCTION m2testb() RETURNS text
+ LANGUAGE sql
+ RETURN '2y';
+
+CREATE MODULE mtest3
+ CREATE FUNCTION mtest3.m3testa() RETURNS text
+ LANGUAGE sql
+ RETURN '3x';
+
+SELECT mtest1.m1testa();
+SELECT mtest1.m1testb();
+
+SELECT public.mtest1.m1testa();
+SELECT public.mtest1.m1testb();
+
+SELECT temp_mod_test.mtest2.m2testb();
+
+SELECT temp_mod_test.mtest2.m2testa('x'); -- error
+CALL temp_mod_test.mtest2.m2testa('a'); -- ok
+CALL temp_mod_test.mtest2.m2testa('xy' || 'zzy'); -- ok, constant-folded arg
+
+CREATE PROCEDURE mtest1.m1testc(x text)
+ LANGUAGE SQL
+ AS $$
+ INSERT INTO cm_test VALUES (2, x);
+ $$;
+
+CALL mtest1.m1testc('a'); -- ok
+
+DROP PROCEDURE mtest1.m1testc(text);
+DROP FUNCTION temp_mod_test.mtest2.m2testb();
+
+-- cleanup
+
+DROP MODULE mtest1 CASCADE;
+DROP MODULE temp_mod_test.mtest2 CASCADE;
+
+DROP SCHEMA temp_mod_test;
+DROP TABLE cm_test;
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 13:58 Tom Lane <[email protected]>
parent: Jim Mlodgenski <[email protected]>
2 siblings, 1 reply; 67+ messages in thread
From: Tom Lane @ 2021-06-02 13:58 UTC (permalink / raw)
To: Jim Mlodgenski <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Jim Mlodgenski <[email protected]> writes:
> Questions
> - Do we want to add module support?
Certainly many people have asked for that, or things like that.
> - If we do, should it be implemented as a type of namespace or should it
> be its own object type that lives in something like pg_module?
While I didn't read the actual patch, your sketch just above this makes
me want to run away screaming. In the first place, what do you think
the primary key of pg_namespace is now? But the bigger problem is that
sub-namespaces just do not work in SQL syntax. Back when we first added
schema support, I had some ambitions towards allowing nested schemas,
which is a big part of the reason why pg_namespace is named that and not
pg_schema. But the idea fell apart after I understood the syntactic
ambiguities it'd introduce. It's already quite hard to tell which part
of a multiply.qualified.name is which, given that SQL says that you can
optionally put a "catalog" (database) name in front of the others.
I really doubt there is a way to shoehorn sub-schemas in there without
creating terrible ambiguities. Is "a.b.c" a reference to object c in
schema b in database a, or is it a reference to object c in sub-schema b
in schema a? This is why we've ended up with bastard syntax like
(table.column).subcolumn.
> - How should users interact with objects within a module? They could be
> mostly independent like the current POC or we can introduce a path like
> ALTER MODULE foo ADD FUNCTION blah
I wonder whether it'd be better to consider modules as a kind of
extension, or at least things with the same sort of ownership relations
as extensions have.
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 14:43 Jim Mlodgenski <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 4 replies; 67+ messages in thread
From: Jim Mlodgenski @ 2021-06-02 14:43 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Wed, Jun 2, 2021 at 9:58 AM Tom Lane <[email protected]> wrote:
>
> In the first place, what do you think the primary key of pg_namespace is now?
In the patch the unique constraint is (nspname, nspnamespace) which is
certainly awkward. I initially went down the pg_module route to avoid
adding another catalog, but in retrospect, that may be a cleaner way.
> It's already quite hard to tell which part
> of a multiply.qualified.name is which, given that SQL says that you can
> optionally put a "catalog" (database) name in front of the others.
> I really doubt there is a way to shoehorn sub-schemas in there without
> creating terrible ambiguities. Is "a.b.c" a reference to object c in
> schema b in database a, or is it a reference to object c in sub-schema b
> in schema a?
That was the area I had the most difficult part to reason about. I tried to make
some simplifying assumptions by checking if "a" was the current database.
Since we don't support cross database access, if it was not, I assumed "a"
was a schema. I not sure if that would be valid, but it did scope things
to a more manageable problem.
> I wonder whether it'd be better to consider modules as a kind of
> extension, or at least things with the same sort of ownership relations
> as extensions have.
That would solve the problem of associating objects which is the larger
problem for users today. The objects can all live in their respective
schemas with the module tying them all together.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 15:07 Bruce Momjian <[email protected]>
parent: Jim Mlodgenski <[email protected]>
3 siblings, 1 reply; 67+ messages in thread
From: Bruce Momjian @ 2021-06-02 15:07 UTC (permalink / raw)
To: Jim Mlodgenski <[email protected]>; +Cc: Tom Lane <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jun 2, 2021 at 10:43:10AM -0400, Jim Mlodgenski wrote:
> On Wed, Jun 2, 2021 at 9:58 AM Tom Lane <[email protected]> wrote:
> >
> > In the first place, what do you think the primary key of pg_namespace is now?
>
> In the patch the unique constraint is (nspname, nspnamespace) which is
> certainly awkward. I initially went down the pg_module route to avoid
> adding another catalog, but in retrospect, that may be a cleaner way.
>
>
> > It's already quite hard to tell which part
> > of a multiply.qualified.name is which, given that SQL says that you can
> > optionally put a "catalog" (database) name in front of the others.
> > I really doubt there is a way to shoehorn sub-schemas in there without
> > creating terrible ambiguities. Is "a.b.c" a reference to object c in
> > schema b in database a, or is it a reference to object c in sub-schema b
> > in schema a?
>
> That was the area I had the most difficult part to reason about. I tried to make
> some simplifying assumptions by checking if "a" was the current database.
> Since we don't support cross database access, if it was not, I assumed "a"
> was a schema. I not sure if that would be valid, but it did scope things
> to a more manageable problem.
If we go in this direction, I assume we would just disallow a schema
name matching the database name. CREATE DATABASE with TEMPLATE would
have to check that. Also the common case where you create a database
name to match the user name, and also a schema inside to match the
username, would have to be disallowed, e.g. creating a 'postgres' schema
to match the 'postgres' user in the 'postgres' database.
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 15:11 Joe Conway <[email protected]>
parent: Jim Mlodgenski <[email protected]>
3 siblings, 0 replies; 67+ messages in thread
From: Joe Conway @ 2021-06-02 15:11 UTC (permalink / raw)
To: Jim Mlodgenski <[email protected]>; Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 6/2/21 10:43 AM, Jim Mlodgenski wrote:
> On Wed, Jun 2, 2021 at 9:58 AM Tom Lane <[email protected]> wrote:
>> I wonder whether it'd be better to consider modules as a kind of
>> extension, or at least things with the same sort of ownership relations
>> as extensions have.
>
> That would solve the problem of associating objects which is the larger
> problem for users today. The objects can all live in their respective
> schemas with the module tying them all together.
Maybe something similar to "CREATE EXTENSION ... FROM unpackaged"?
Something like:
CREATE EXTENSION myfoo; /* shell extension */
ALTER EXTENSION myfoo ADD type ...;
ALTER EXTENSION myfoo ADD function ...;
...
Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 15:14 Tom Lane <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Tom Lane @ 2021-06-02 15:14 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Jim Mlodgenski <[email protected]>; PostgreSQL Hackers <[email protected]>
Bruce Momjian <[email protected]> writes:
> If we go in this direction, I assume we would just disallow a schema
> name matching the database name.
That seems quite impossible to enforce.
regression=# create database d1;
CREATE DATABASE
regression=# alter database d1 rename to d2;
ALTER DATABASE
The system had no way to know that d1 doesn't contain a schema named d2.
And you can't fix that by restricting the ALTER to be done on the
current database:
regression=# \c d2
You are now connected to database "d2" as user "postgres".
d2=# alter database d2 rename to d3;
ERROR: current database cannot be renamed
Between that and the point that this restriction would certainly break
existing installations, this is a non-starter.
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 15:22 Joel Jacobson <[email protected]>
parent: Jim Mlodgenski <[email protected]>
3 siblings, 0 replies; 67+ messages in thread
From: Joel Jacobson @ 2021-06-02 15:22 UTC (permalink / raw)
To: [email protected]
On Wed, Jun 2, 2021, at 16:43, Jim Mlodgenski wrote:
> On Wed, Jun 2, 2021 at 9:58 AM Tom Lane <[email protected] <mailto:tgl%40sss.pgh.pa.us>> wrote:
> > I wonder whether it'd be better to consider modules as a kind of
> > extension, or at least things with the same sort of ownership relations
> > as extensions have.
>
> That would solve the problem of associating objects which is the larger
> problem for users today. The objects can all live in their respective
> schemas with the module tying them all together.
I like the idea of somehow using extensions.
Right now, extensions can only be added from the command-line, via `make install`.
But maybe a new extension could be packaged from the SQL prompt, out of existing database objects that are not already part of an extension?
Maybe the interface could be:
init_new_extension(extension_name text) function, to register a new empty extension.
add_object_to_extension(extension_name text, type text, object_names text[], object_args text[])
Then, if dropping the extension, all objects would be dropped, and if creating the extension, all objects would be restored.
I don't have an idea on how to handle update scripts, but since it's not mandatory to provide extension update scripts, maybe that's not a problem.
/Joel
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-02 19:08 Alvaro Herrera <[email protected]>
parent: Jim Mlodgenski <[email protected]>
2 siblings, 0 replies; 67+ messages in thread
From: Alvaro Herrera @ 2021-06-02 19:08 UTC (permalink / raw)
To: Jim Mlodgenski <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 2021-Jun-02, Jim Mlodgenski wrote:
> Attached is a POC patch for modules. I modeled it as a sub-schema because
> that is more what it seems like to me. It adds additional columns to
> pg_namespace and allows for 3-part (or 4 with the database name) naming
> of objects within the module. This simple example works with the patch.
Given the downthread discussion, this idea doesn't seem workable.
People are now discussing "what if the module is some kind of
extension". But to me that seems to go against the grain; you'd have to
implement a ton of stuff in order to let "extension-modules" be
installed without on-disk foo.control files.
But what if the module is just a particular kind of *namespace*? I
mean, what if CREATE MODULE is implemented by creating a row in
pg_namespace with nspkind='m'? So a pg_namespace row can refer to
either a regular schema (nspkind='s') or a module. In a schema you can
create objects of any kind just like today, but in a module you're
restricted to having only functions (and maybe also operators? other
types of objects?).
Then, a qualified object name foo.bar() can refer to either the routine
bar() in schema foo, or routine bar in module foo. To the low-level
code it's pretty much the same thing (look the namespace in pg_namespace
just as today).
What other properties do you want modules to have? Are there "private"
functions? (What *is* a private function in this context? I mean, how
does "being in a module" interact with object lookup rules? Does
plpgsql have to be aware that a routine is in a module?)
Are there module-scoped variables? (If so, you probably want Pavel
Stehule's variable patch pushed ahead of time).
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-03 12:49 Peter Eisentraut <[email protected]>
parent: Jim Mlodgenski <[email protected]>
3 siblings, 1 reply; 67+ messages in thread
From: Peter Eisentraut @ 2021-06-03 12:49 UTC (permalink / raw)
To: Jim Mlodgenski <[email protected]>; Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 02.06.21 16:43, Jim Mlodgenski wrote:
>> It's already quite hard to tell which part
>> of a multiply.qualified.name is which, given that SQL says that you can
>> optionally put a "catalog" (database) name in front of the others.
>> I really doubt there is a way to shoehorn sub-schemas in there without
>> creating terrible ambiguities. Is "a.b.c" a reference to object c in
>> schema b in database a, or is it a reference to object c in sub-schema b
>> in schema a?
> That was the area I had the most difficult part to reason about. I tried to make
> some simplifying assumptions by checking if "a" was the current database.
> Since we don't support cross database access, if it was not, I assumed "a"
> was a schema. I not sure if that would be valid, but it did scope things
> to a more manageable problem.
Given that, as you said, the concept of modules is in the SQL standard,
there is surely some guidance in there about how this is supposed to
affect name resolution. So let's start with that. Maybe we won't like
it in the end or whatever, but we should surely look there first.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-04 19:31 Jim Mlodgenski <[email protected]>
parent: Peter Eisentraut <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Jim Mlodgenski @ 2021-06-04 19:31 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Tom Lane <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Jun 3, 2021 at 8:49 AM Peter Eisentraut
<[email protected]> wrote:
> Given that, as you said, the concept of modules is in the SQL standard,
> there is surely some guidance in there about how this is supposed to
> affect name resolution. So let's start with that. Maybe we won't like
> it in the end or whatever, but we should surely look there first.
Studying the spec further, catalog/schema/module are all used to
identify a module-level routine. I don't see it spelled out that
is needs to be in that format of catalog.schema.module.routine to
fully qualify the routine, but it would likely be awkward for users
to come up with an alternative syntax like
(catalog.schema.module).routine or catalog.scheme.module->routine
The way the spec is worded, I read it as that schemas take precedence
over modules regarding path resolution. So for example with 2-level
naming if there is a schema 'foo' and a module 'public.foo' both with
functions 'bar' 'foo.bar' would refer to the schema-level function not
the module-level function. I've not found guidance on throwing catalog
into the mix and 3-level naming. Say we had a catalog 'postgres' with a
schema 'foo' with a function 'bar' and a schema 'postgres' with a module
'foo' with a function 'bar'. What would 'postgres.foo.bar' refer to? If
the SQL was executed from a catalog other than 'postgres', we'd have no
way of knowing if 'foo.bar' existed there. So if it's implementation
dependent, saying schemas take precedence over catalogs may make sense
and 'postgres.foo.bar' refers to the module-level function in the
'postgres' schema.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Support for CREATE MODULE?
@ 2021-06-04 20:09 Pavel Stehule <[email protected]>
parent: Jim Mlodgenski <[email protected]>
2 siblings, 0 replies; 67+ messages in thread
From: Pavel Stehule @ 2021-06-04 20:09 UTC (permalink / raw)
To: Jim Mlodgenski <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi
st 2. 6. 2021 v 15:39 odesílatel Jim Mlodgenski <[email protected]> napsal:
> Working with users over the years, some have large libraries of server
> side code sometimes consisting of 100k+ lines of code over 1000+ functions
> and procedures. This usually comes from a migration of a commercial
> database like Oracle where it was best practice to put all of your
> business logic into stored procedures. In these types of apps, just
> managing the code is a challenge. To help classify objects, schemas
> are used, but you are at the mercy of a naming convention to show
> association. For example, a frequent naming convention would be having
> related schemas with the names of foo_bar and foo_baz. For devs, that's
> akin to keeping a file like xlog.c in a directory structure like
> backend_access_transam instead of backend/access/transam. IMHO, having
> a hierarchy makes it simpler to reason about related code bits.
>
> The SQL spec does have a concept of modules that help address this. It's
> defined as a persistent object within a schema that contains one or more
> routines. It also defines other things like local temporary tables and
> path specifications. There are other databases like DB2 that have
> implemented module support each with their own way of defining the
> routines within the module. The spec doesn't really give guidance on
> how to manipulate the objects within the module.
>
> Attached is a POC patch for modules. I modeled it as a sub-schema because
> that is more what it seems like to me. It adds additional columns to
> pg_namespace and allows for 3-part (or 4 with the database name) naming
> of objects within the module. This simple example works with the patch.
>
> CREATE SCHEMA foo;
> CREATE MODULE foo.bar
> CREATE FUNCTION hello() RETURNS text
> LANGUAGE sql
> RETURN 'hello'
> CREATE FUNCTION world() RETURNS text
> LANGUAGE sql
> RETURN 'world';
> SELECT foo.bar.hello();
>
> Questions
> - Do we want to add module support?
>
> - If we do, should it be implemented as a type of namespace or should it
> be its own object type that lives in something like pg_module?
>
> - How should users interact with objects within a module? They could be
> mostly independent like the current POC or we can introduce a path like
> ALTER MODULE foo ADD FUNCTION blah
>
I never liked the SQL/PSM concept of modules. The possibility to assign
database objects to schema or to modules looks like schizophrenia.
There are only two advantages of modules - a) possibility to define private
objects, b) local scope - the objects from modules shadows external objects
without dependency of search_path.
But both these features are pretty hard to implement in PL/pgSQL - where
expression executor is SQL executor.
Without these features I don't see strong benefits for modules.
Regards
Pavel
>
> --Jim
>
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH v2 2/3] Calculate MaxBackends earlier in PostmasterMain().
@ 2021-08-02 17:42 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Nathan Bossart @ 2021-08-02 17:42 UTC (permalink / raw)
Presently, InitializeMaxBackends() is called after processing
shared_preload_libraries because it used to tally up the number of
registered background workers requested by the libraries. Since
6bc8ef0b, InitializeMaxBackends() has simply used the
max_worker_processes GUC instead, so all the comments about needing
to register background workers before initializing MaxBackends are
no longer correct.
In addition to revising the comments, this patch reorders
InitializeMaxBackends() to before shared_preload_libraries is
processed so that modules can make use of MaxBackends in their
_PG_init() functions.
---
src/backend/postmaster/postmaster.c | 19 +++++++++----------
src/backend/utils/init/postinit.c | 4 +---
2 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 3dcaf8a4a5..3d3b2e376c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1005,10 +1005,15 @@ PostmasterMain(int argc, char *argv[])
LocalProcessControlFile(false);
/*
- * Register the apply launcher. Since it registers a background worker,
- * it needs to be called before InitializeMaxBackends(), and it's probably
- * a good idea to call it before any modules had chance to take the
- * background worker slots.
+ * Calculate MaxBackends. This is done before processing
+ * shared_preload_libraries so that such libraries can make use of it in
+ * _PG_init().
+ */
+ InitializeMaxBackends();
+
+ /*
+ * Register the apply launcher. It's probably a good idea to call it before
+ * any modules had chance to take the background worker slots.
*/
ApplyLauncherRegister();
@@ -1028,12 +1033,6 @@ PostmasterMain(int argc, char *argv[])
}
#endif
- /*
- * Now that loadable modules have had their chance to register background
- * workers, calculate MaxBackends.
- */
- InitializeMaxBackends();
-
/*
* Now that loadable modules have had their chance to request additional
* shared memory, determine the value of any runtime-computed GUCs that
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 9139fe895c..0be1eb0f44 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -538,9 +538,7 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
/*
* Initialize MaxBackends value from config options.
*
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called before shared memory size is determined.
*
* Note that in EXEC_BACKEND environment, the value is passed down from
* postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
--
2.25.1
--+HP7ph2BbKc20aGI
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0003-Block-attempts-to-set-GUCs-while-loading-shared_p.patch"
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH v1 2/3] Calculate MaxBackends earlier in PostmasterMain().
@ 2021-08-02 17:42 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Nathan Bossart @ 2021-08-02 17:42 UTC (permalink / raw)
Presently, InitializeMaxBackends() is called after processing
shared_preload_libraries because it used to tally up the number of
registered background workers requested by the libraries. Since
6bc8ef0b, InitializeMaxBackends() has simply used the
max_worker_processes GUC instead, so all the comments about needing
to register background workers before initializing MaxBackends are
no longer correct.
In addition to revising the comments, this patch reorders
InitializeMaxBackends() to before shared_preload_libraries is
processed so that modules can make use of MaxBackends in their
_PG_init() functions.
---
src/backend/postmaster/postmaster.c | 19 +++++++++----------
src/backend/utils/init/postinit.c | 4 +---
2 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 3dcaf8a4a5..3d3b2e376c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1005,10 +1005,15 @@ PostmasterMain(int argc, char *argv[])
LocalProcessControlFile(false);
/*
- * Register the apply launcher. Since it registers a background worker,
- * it needs to be called before InitializeMaxBackends(), and it's probably
- * a good idea to call it before any modules had chance to take the
- * background worker slots.
+ * Calculate MaxBackends. This is done before processing
+ * shared_preload_libraries so that such libraries can make use of it in
+ * _PG_init().
+ */
+ InitializeMaxBackends();
+
+ /*
+ * Register the apply launcher. It's probably a good idea to call it before
+ * any modules had chance to take the background worker slots.
*/
ApplyLauncherRegister();
@@ -1028,12 +1033,6 @@ PostmasterMain(int argc, char *argv[])
}
#endif
- /*
- * Now that loadable modules have had their chance to register background
- * workers, calculate MaxBackends.
- */
- InitializeMaxBackends();
-
/*
* Now that loadable modules have had their chance to request additional
* shared memory, determine the value of any runtime-computed GUCs that
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 9139fe895c..0be1eb0f44 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -538,9 +538,7 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
/*
* Initialize MaxBackends value from config options.
*
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called before shared memory size is determined.
*
* Note that in EXEC_BACKEND environment, the value is passed down from
* postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
--
2.25.1
--bg08WKrSYDhXBjb5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v1-0003-Block-attempts-to-set-GUCs-while-loading-shared_p.patch"
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: support for CREATE MODULE
@ 2022-07-18 18:06 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Nathan Bossart @ 2022-07-18 18:06 UTC (permalink / raw)
To: Swaha Miller <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 17, 2022 at 04:30:43PM -0700, Nathan Bossart wrote:
> On Thu, Mar 17, 2022 at 04:26:31PM -0700, Swaha Miller wrote:
>> On Thu, Mar 17, 2022 at 4:16 PM Nathan Bossart <[email protected]>
>> wrote:
>>> It seems unlikely that this will be committed for v15. Swaha, should the
>>> commitfest entry be adjusted to v16 and moved to the next commitfest?
>>>
>>>
>> Yes please, thank you Nathan
>
> Done!
I spoke with Swaha off-list, and we agreed that this commitfest entry can
be closed for now. I'm going to mark it as returned-with-feedback.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-13 15:00 Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-01-13 15:00 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Sat, Jan 13, 2024 at 11:09 AM Andrei Lepikhov
<[email protected]> wrote:
> On 11/1/2024 18:30, Alexander Korotkov wrote:
> > On Tue, Jan 9, 2024 at 1:14 PM Pavel Borisov <[email protected]> wrote:
> >>> Hmm, I don't see this old code in these patches. Resend 0002-* because
> >>> of trailing spaces.
> >>
> >>
> >> AFAIK, cfbot does not seek old versions of patchset parts in previous messages. So for it to run correctly, a new version of the whole patchset should be sent even if most patches are unchanged.
> >
> > Please, find the revised patchset with some refactoring and comments
> > improvement from me. I'll continue looking into this.
> The patch looks better, thanks to your refactoring.
> I propose additional comments and tests to make the code more
> understandable (see attachment).
> I intended to look into this part of the code more, but the tests show a
> difference between PostgreSQL v.15 and v.16, which causes changes in the
> code of this feature.
Makes sense. I've incorporated your changes into the patchset.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Generalize-common-code-of-adding-sort-befor-20240113.patch (9.1K, ../../CAPpHfdscoK4u4PhR354Swso3Tx0cFOjJKcbG02mi98jTwRxW9A@mail.gmail.com/2-0001-Generalize-common-code-of-adding-sort-befor-20240113.patch)
download | inline diff:
From cc7d0123c4923b2cf10a4ac6805f362da1f6e7e6 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:27:58 +0200
Subject: [PATCH 1/2] Generalize common code of adding sort before generation
of grouping paths
Extract the repetitive code pattern into a new function make_ordered_path().
Discussion: https://postgr.es/m/CAPpHfdtzaVa7S4onKy3YvttF2rrH5hQNHx9HtcSTLbpjx%2BMJ%2Bw%40mail.gmail.com
Author: Andrei Lepikhov
---
src/backend/optimizer/plan/planner.c | 228 ++++++++++-----------------
1 file changed, 80 insertions(+), 148 deletions(-)
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 667723b6753..014b179c3f0 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6809,6 +6809,58 @@ done:
return parallel_workers;
}
+/*
+ * make_ordered_path
+ * Return a path ordered by 'pathkeys' based on the given 'path'. May
+ * return NULL if it doesn't make sense to generate an ordered path in
+ * this case.
+ */
+static Path *
+make_ordered_path(PlannerInfo *root, RelOptInfo *rel, Path *path,
+ Path *cheapest_path, List *pathkeys)
+{
+ bool is_sorted;
+ int presorted_keys;
+
+ is_sorted = pathkeys_count_contained_in(pathkeys,
+ path->pathkeys,
+ &presorted_keys);
+
+ if (!is_sorted)
+ {
+ /*
+ * Try at least sorting the cheapest path and also try incrementally
+ * sorting any path which is partially sorted already (no need to deal
+ * with paths which have presorted keys when incremental sort is
+ * disabled unless it's the cheapest input path).
+ */
+ if (path != cheapest_path &&
+ (presorted_keys == 0 || !enable_incremental_sort))
+ return NULL;
+
+ /*
+ * We've no need to consider both a sort and incremental sort. We'll
+ * just do a sort if there are no presorted keys and an incremental
+ * sort when there are presorted keys.
+ */
+ if (presorted_keys == 0 || !enable_incremental_sort)
+ path = (Path *) create_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ -1.0);
+ else
+ path = (Path *) create_incremental_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ presorted_keys,
+ -1.0);
+ }
+
+ return path;
+}
+
/*
* add_paths_to_grouping_rel
*
@@ -6840,45 +6892,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ cheapest_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6935,46 +6957,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, partially_grouped_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially
- * sorted already (no need to deal with paths which have
- * presorted keys when incremental sort is disabled unless
- * it's the cheapest input path).
- */
- if (path != partially_grouped_rel->cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental
- * sort. We'll just do a sort if there are no pre-sorted
- * keys and an incremental sort when there are presorted
- * keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(grouped_rel, (Path *)
@@ -7200,44 +7191,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(partially_grouped_rel, (Path *)
@@ -7268,45 +7230,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->partial_pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
-
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_partial_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
--
2.39.3 (Apple Git-145)
[application/octet-stream] 0002-Explore-alternative-orderings-of-group-by-p-20240113.patch (33.8K, ../../CAPpHfdscoK4u4PhR354Swso3Tx0cFOjJKcbG02mi98jTwRxW9A@mail.gmail.com/3-0002-Explore-alternative-orderings-of-group-by-p-20240113.patch)
download | inline diff:
From 159936c405e51b721e14033af35d886f9948b959 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:26:49 +0200
Subject: [PATCH 2/2] Explore alternative orderings of group-by pathkeys during
optimization.
When evaluating a query with a multi-column GROUP BY clause, we can minimize
sort operations or avoid them if we synchronize the order of GROUP BY clauses
with the ORDER BY sort clause or sort order, which comes from the underlying
query tree. Grouping does not imply any ordering, so we can compare
the keys in arbitrary order, and a Hash Agg leverages this. But for Group Agg,
we simply compared keys in the order specified in the query. This commit
explores alternative ordering of the keys, trying to find a cheaper one.
The ordering of group keys may interact with other parts of the query, some of
which may not be known while planning the grouping. For example, there may be
an explicit ORDER BY clause or some other ordering-dependent operation higher up
in the query, and using the same ordering may allow using either incremental
sort or even eliminating the sort entirely.
The patch always keeps the ordering specified in the query, assuming the user
might have additional insights.
This introduces a new GUC enable_group_by_reordering so that the optimization
may be disabled if needed.
Discussion: https://postgr.es/m/7c79e6a5-8597-74e8-0671-1c39d124c9d6%40sigaev.ru
Author: Andrey V. Lepikhov, Teodor Sigaev
Reviewed-by: Tomas Vondra, Claudio Freire, Gavin Flower, Dmitry Dolgov
Reviewed-by: Robert Haas, Pavel Borisov, David Rowley, Zhihong Yu
Reviewed-by: Tom Lane, Alexander Korotkov
---
src/backend/optimizer/path/equivclass.c | 13 +-
src/backend/optimizer/path/pathkeys.c | 220 ++++++++++++
src/backend/optimizer/plan/planner.c | 316 +++++++++++-------
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/pathnodes.h | 10 +
src/include/optimizer/paths.h | 2 +
src/test/regress/expected/aggregates.out | 154 +++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/aggregates.sql | 56 ++++
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 661 insertions(+), 125 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index e86dfeaecd4..7dd14d0a43b 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,7 +652,18 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- return cur_ec; /* Match! */
+ {
+ /*
+ * Match!
+ *
+ * Copy the sortref if it wasn't set yet. That may happen if
+ * the ec was constructed from WHERE clause, i.e. it doesn't
+ * have a target reference at all.
+ */
+ if (cur_ec->ec_sortref == 0 && sortref > 0)
+ cur_ec->ec_sortref = sortref;
+ return cur_ec;
+ }
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index ca94a31f71e..5aac6d66776 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -22,12 +22,15 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
+#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
+/* Consider reordering of GROUP BY keys? */
+bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -350,6 +353,170 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
+/*
+ * group_keys_reorder_by_pathkeys
+ * Reorder GROUP BY keys to match the input pathkeys.
+ *
+ * Function returns new lists (pathkeys and clauses), original GROUP BY lists
+ * stay untouched.
+ *
+ * Returns the number of GROUP BY keys with a matching pathkey.
+ */
+static int
+group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
+ List **group_clauses,
+ int num_groupby_pathkeys)
+{
+ List *new_group_pathkeys = NIL,
+ *new_group_clauses = NIL;
+ ListCell *lc;
+ int n;
+
+ if (pathkeys == NIL || *group_pathkeys == NIL)
+ return 0;
+
+ /*
+ * Walk the pathkeys (determining ordering of the input path) and see if
+ * there's a matching GROUP BY key. If we find one, we append it to the
+ * list, and do the same for the clauses.
+ *
+ * Once we find the first pathkey without a matching GROUP BY key, the
+ * rest of the pathkeys are useless and can't be used to evaluate the
+ * grouping, so we abort the loop and ignore the remaining pathkeys.
+ */
+ foreach(lc, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(lc);
+ SortGroupClause *sgc;
+
+ /*
+ * Pathkeys are built in a way that allows simply comparing pointers.
+ * Give up if we can't find the matching pointer.
+ */
+ if (foreach_current_index(lc) >= num_groupby_pathkeys ||
+ !list_member_ptr(*group_pathkeys, pathkey))
+ break;
+
+ /*
+ * Since 1349d27 pathkey coming from underlying node can be in the
+ * root->group_pathkeys but not in the processed_groupClause. So, we
+ * should be careful here.
+ */
+ sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
+ *group_clauses);
+ if (!sgc)
+ /* The grouping clause does not cover this pathkey */
+ break;
+
+ new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ }
+
+ /* remember the number of pathkeys with a matching GROUP BY key */
+ n = list_length(new_group_pathkeys);
+
+ /* append the remaining group pathkeys (will be treated as not sorted) */
+ *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
+ *group_pathkeys);
+ *group_clauses = list_concat_unique_ptr(new_group_clauses,
+ *group_clauses);
+
+ return n;
+}
+
+/*
+ * get_useful_group_keys_orderings
+ * Determine which orderings of GROUP BY keys are potentially interesting.
+ *
+ * Returns a list of PathKeyInfo items, each representing an interesting
+ * ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
+ * matching order.
+ *
+ * The function considers (and keeps) multiple GROUP BY orderings:
+ *
+ * - the original ordering, as specified by the GROUP BY clause,
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
+ * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ */
+List *
+get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
+{
+ Query *parse = root->parse;
+ List *infos = NIL;
+ PathKeyInfo *info;
+
+ List *pathkeys = root->group_pathkeys;
+ List *clauses = root->processed_groupClause;
+
+ /* always return at least the original pathkeys/clauses */
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+ infos = lappend(infos, info);
+
+ /*
+ * Should we try generating alternative orderings of the group keys? If
+ * not, we produce only the order specified in the query, i.e. the
+ * optimization is effectively disabled.
+ */
+ if (!enable_group_by_reordering)
+ return infos;
+
+ /*
+ * Grouping sets have own and more complex logic to decide the ordering.
+ */
+ if (parse->groupingSets)
+ return infos;
+
+ /*
+ * If the path is sorted in some way, try reordering the group keys to
+ * match the path as much of the ordering as possible. Then thanks to
+ * incremental sort we would get this sort as cheap as possible.
+ */
+ if (path->pathkeys)
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(path->pathkeys, &pathkeys, &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(path->pathkeys)))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ /*
+ * Try reordering pathkeys to minimize the sort cost (this time consider
+ * the ORDER BY clause).
+ */
+ if (root->sort_pathkeys)
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
+ &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(path->pathkeys)))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ return infos;
+}
+
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -1939,6 +2106,54 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
+/*
+ * pathkeys_useful_for_grouping
+ * Count the number of pathkeys that are useful for grouping (instead of
+ * explicit sort)
+ *
+ * Group pathkeys could be reordered to benefit from the ordering. The
+ * ordering may not be "complete" and may require incremental sort, but that's
+ * fine. So we simply count prefix pathkeys with a matching group key, and
+ * stop once we find the first pathkey without a match.
+ *
+ * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
+ * pathkeys are useful for grouping, and we might do incremental sort to get
+ * path ordered by (a,b,e).
+ *
+ * This logic is necessary to retain paths with ordering not matching grouping
+ * keys directly, without the reordering.
+ *
+ * Returns the length of pathkey prefix with matching group keys.
+ */
+static int
+pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
+{
+ ListCell *key;
+ int n = 0;
+
+ /* no special ordering requested for grouping */
+ if (root->group_pathkeys == NIL)
+ return 0;
+
+ /* unordered path */
+ if (pathkeys == NIL)
+ return 0;
+
+ /* walk the pathkeys and search for matching group key */
+ foreach(key, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(key);
+
+ /* no matching group key, we're done */
+ if (!list_member_ptr(root->group_pathkeys, pathkey))
+ break;
+
+ n++;
+ }
+
+ return n;
+}
+
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -1953,6 +2168,9 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
+ if (nuseful2 > nuseful)
+ nuseful = nuseful2;
+ nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -1988,6 +2206,8 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
+ if (root->group_pathkeys != NIL)
+ return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 014b179c3f0..239b799a6b2 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2805,8 +2805,9 @@ remove_useless_groupby_columns(PlannerInfo *root)
*
* In principle it might be interesting to consider other orderings of the
* GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. We don't
- * bother with that, though. Hashed grouping will frequently win anyway.
+ * possible plans (eg an indexscan) and thereby reduce cost. However, we
+ * don't yet have sufficient information to do that here, so that's left until
+ * later in planning. See get_useful_group_keys_orderings().
*
* Note: we need no comparable processing of the distinctClause because
* the parser already enforced that that matches ORDER BY.
@@ -6891,103 +6892,135 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- grouped_rel,
- path,
- cheapest_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY. Make an
- * AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- root->processed_groupClause,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (parse->groupClause)
+ foreach(lc2, pathkey_orderings)
{
- /*
- * We have GROUP BY without aggregation or grouping sets. Make
- * a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- root->processed_groupClause,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
- /*
- * Instead of operating directly on the input relation, we can
- * consider finalizing a partially aggregated path.
- */
- if (partially_grouped_rel != NULL)
- {
- foreach(lc, partially_grouped_rel->pathlist)
- {
- Path *path = (Path *) lfirst(lc);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
path = make_ordered_path(root,
grouped_rel,
path,
- partially_grouped_rel->cheapest_total_path,
- root->group_pathkeys);
-
+ cheapest_path,
+ info->pathkeys);
if (path == NULL)
continue;
- if (parse->hasAggs)
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make
+ * an AggPath.
+ */
add_path(grouped_rel, (Path *)
create_agg_path(root,
grouped_rel,
path,
grouped_rel->reltarget,
parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- root->processed_groupClause,
+ AGGSPLIT_SIMPLE,
+ info->clauses,
havingQual,
- agg_final_costs,
+ agg_costs,
dNumGroups));
- else
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets.
+ * Make a GroupPath.
+ */
add_path(grouped_rel, (Path *)
create_group_path(root,
grouped_rel,
path,
- root->processed_groupClause,
+ info->clauses,
havingQual,
dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
+ }
+ }
+
+ /*
+ * Instead of operating directly on the input relation, we can
+ * consider finalizing a partially aggregated path.
+ */
+ if (partially_grouped_rel != NULL)
+ {
+ foreach(lc, partially_grouped_rel->pathlist)
+ {
+ ListCell *lc2;
+ Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
+
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
+
+ Assert(list_length(pathkey_orderings) > 0);
+
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ info->clauses,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ info->clauses,
+ havingQual,
+ dNumGroups));
+
+ }
}
}
}
@@ -7190,37 +7223,54 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_total_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
partially_grouped_rel,
path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialGroups));
+ cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialGroups));
+ }
}
}
@@ -7229,37 +7279,55 @@ create_partial_grouping_paths(PlannerInfo *root,
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_partial_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialPartialGroups));
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialPartialGroups));
+ }
}
}
@@ -7373,6 +7441,8 @@ gather_grouping_paths(PlannerInfo *root, RelOptInfo *rel)
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted prefix
* of the list without having the list be fully sorted.
+ *
+ * XXX Shouldn't this also consider the group-key-reordering?
*/
if (!enable_incremental_sort || list_length(root->group_pathkeys) == 1)
return;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e53ebc6dc2b..7fe58518d7d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1050,6 +1050,16 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables reordering of GROUP BY keys."),
+ NULL,
+ GUC_EXPLAIN
+ },
+ &enable_group_by_reordering,
+ true,
+ NULL, NULL, NULL
+ },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 835b0e9ba89..da10b43dac3 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -399,6 +399,7 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
+#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index b9713ec9aa6..137da178dc7 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1456,6 +1456,16 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
+/*
+ * Combines the information about pathkeys and the associated clauses.
+ */
+typedef struct PathKeyInfo
+{
+ NodeTag type;
+ List *pathkeys;
+ List *clauses;
+} PathKeyInfo;
+
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index efd4abc28f3..0e8a9c94ba6 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,6 +24,7 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
+extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -204,6 +205,7 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
+extern List *get_useful_group_keys_orderings(PlannerInfo *root, Path *path);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index f635c5a1afb..0d46e096e5a 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,6 +2728,160 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+-- GROUP BY optimization by reorder columns by frequency
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Limit
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(7 rows)
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+ QUERY PLAN
+------------------------------
+ Group
+ Group Key: y, x, z, w
+ -> Sort
+ Sort Key: y, x, z, w
+ -> Seq Scan on btg
+(5 rows)
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+ QUERY PLAN
+-----------------------------------------------
+ Sort
+ Sort Key: ((x * x)), z
+ -> Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(8 rows)
+
+DROP TABLE btg;
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+ QUERY PLAN
+--------------------------------------------------------
+ Sort
+ Sort Key: c2
+ -> GroupAggregate
+ Group Key: c1
+ -> Sort
+ Sort Key: c1, c2
+ -> Bitmap Heap Scan on t1
+ Recheck Cond: (c2 < 100)
+ -> Bitmap Index Scan on t1_c2_idx
+ Index Cond: (c2 < 100)
+(10 rows)
+
+DROP TABLE t1 CASCADE;
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 271313ebf86..9be7aca2b8a 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,6 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
+ enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -133,7 +134,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(22 rows)
+(23 rows)
-- There are always wait event descriptions for various types.
select type, count(*) > 0 as ok FROM pg_wait_events
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index cc8f0efad55..f99167ac9ea 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,6 +1181,62 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+
+-- GROUP BY optimization by reorder columns by frequency
+
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+
+DROP TABLE btg;
+
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+DROP TABLE t1 CASCADE;
+
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
+
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..e575fa13c63 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+PathKeyInfo
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-14 12:14 Andrei Lepikhov <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-01-14 12:14 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On 13/1/2024 22:00, Alexander Korotkov wrote:
> On Sat, Jan 13, 2024 at 11:09 AM Andrei Lepikhov
> <[email protected]> wrote:
>> On 11/1/2024 18:30, Alexander Korotkov wrote:
>>> On Tue, Jan 9, 2024 at 1:14 PM Pavel Borisov <[email protected]> wrote:
>>>>> Hmm, I don't see this old code in these patches. Resend 0002-* because
>>>>> of trailing spaces.
>>>>
>>>>
>>>> AFAIK, cfbot does not seek old versions of patchset parts in previous messages. So for it to run correctly, a new version of the whole patchset should be sent even if most patches are unchanged.
>>>
>>> Please, find the revised patchset with some refactoring and comments
>>> improvement from me. I'll continue looking into this.
>> The patch looks better, thanks to your refactoring.
>> I propose additional comments and tests to make the code more
>> understandable (see attachment).
>> I intended to look into this part of the code more, but the tests show a
>> difference between PostgreSQL v.15 and v.16, which causes changes in the
>> code of this feature.
>
> Makes sense. I've incorporated your changes into the patchset.
One more improvement. To underpin code change:
- return cur_ec; /* Match! */
+ {
+ /*
+ * Match!
+ *
+ * Copy the sortref if it wasn't set yet. That may happen if
+ * the ec was constructed from WHERE clause, i.e. it doesn't
+ * have a target reference at all.
+ */
+ if (cur_ec->ec_sortref == 0 && sortref > 0)
+ cur_ec->ec_sortref = sortref;
+ return cur_ec;
+ }
I propose the test (see attachment). It shows why we introduce this
change: GROUP-BY should juggle not only pathkeys generated by explicit
sort operators but also planner-made, likewise in this example, by
MergeJoin.
--
regards,
Andrei Lepikhov
Postgres Professional
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 0d46e096e5..ca38e78f21 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2879,6 +2879,37 @@ FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
(10 rows)
DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------
+ Incremental Sort
+ Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
+ Presorted Key: c1.relpages, c1.relname
+ -> Group
+ Group Key: c1.relpages, c1.relname, c1.reltuples
+ -> Incremental Sort
+ Sort Key: c1.relpages, c1.relname, c1.reltuples
+ Presorted Key: c1.relpages, c1.relname
+ -> Merge Join
+ Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ -> Sort
+ Sort Key: c1.relpages, c1.relname
+ -> Seq Scan on pg_class c1
+ -> Sort
+ Sort Key: c2.relpages, c2.relname
+ -> Seq Scan on pg_class c2
+(16 rows)
+
+RESET enable_hashjoin;
+RESET enable_nestloop;
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index f99167ac9e..cf87b5d5dd 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1233,6 +1233,18 @@ SELECT array_agg(c1 ORDER BY c2),c2
FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+RESET enable_hashjoin;
+RESET enable_nestloop;
+
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
Attachments:
[text/plain] details-2.txt (2.7K, ../../[email protected]/2-details-2.txt)
download | inline diff:
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 0d46e096e5..ca38e78f21 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2879,6 +2879,37 @@ FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
(10 rows)
DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------
+ Incremental Sort
+ Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
+ Presorted Key: c1.relpages, c1.relname
+ -> Group
+ Group Key: c1.relpages, c1.relname, c1.reltuples
+ -> Incremental Sort
+ Sort Key: c1.relpages, c1.relname, c1.reltuples
+ Presorted Key: c1.relpages, c1.relname
+ -> Merge Join
+ Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ -> Sort
+ Sort Key: c1.relpages, c1.relname
+ -> Seq Scan on pg_class c1
+ -> Sort
+ Sort Key: c2.relpages, c2.relname
+ -> Seq Scan on pg_class c2
+(16 rows)
+
+RESET enable_hashjoin;
+RESET enable_nestloop;
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index f99167ac9e..cf87b5d5dd 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1233,6 +1233,18 @@ SELECT array_agg(c1 ORDER BY c2),c2
FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+RESET enable_hashjoin;
+RESET enable_nestloop;
+
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-15 00:19 Alexander Korotkov <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-01-15 00:19 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Sun, Jan 14, 2024 at 2:14 PM Andrei Lepikhov
<[email protected]> wrote:
> On 13/1/2024 22:00, Alexander Korotkov wrote:
> > On Sat, Jan 13, 2024 at 11:09 AM Andrei Lepikhov
> > <[email protected]> wrote:
> >> On 11/1/2024 18:30, Alexander Korotkov wrote:
> >>> On Tue, Jan 9, 2024 at 1:14 PM Pavel Borisov <[email protected]> wrote:
> >>>>> Hmm, I don't see this old code in these patches. Resend 0002-* because
> >>>>> of trailing spaces.
> >>>>
> >>>>
> >>>> AFAIK, cfbot does not seek old versions of patchset parts in previous messages. So for it to run correctly, a new version of the whole patchset should be sent even if most patches are unchanged.
> >>>
> >>> Please, find the revised patchset with some refactoring and comments
> >>> improvement from me. I'll continue looking into this.
> >> The patch looks better, thanks to your refactoring.
> >> I propose additional comments and tests to make the code more
> >> understandable (see attachment).
> >> I intended to look into this part of the code more, but the tests show a
> >> difference between PostgreSQL v.15 and v.16, which causes changes in the
> >> code of this feature.
> >
> > Makes sense. I've incorporated your changes into the patchset.
> One more improvement. To underpin code change:
>
> - return cur_ec; /* Match! */
> + {
> + /*
> + * Match!
> + *
> + * Copy the sortref if it wasn't set yet. That may happen if
> + * the ec was constructed from WHERE clause, i.e. it doesn't
> + * have a target reference at all.
> + */
> + if (cur_ec->ec_sortref == 0 && sortref > 0)
> + cur_ec->ec_sortref = sortref;
> + return cur_ec;
> + }
>
> I propose the test (see attachment). It shows why we introduce this
> change: GROUP-BY should juggle not only pathkeys generated by explicit
> sort operators but also planner-made, likewise in this example, by
> MergeJoin.
Thank you for providing the test case relevant for this code change.
The revised patch incorporating this change is attached. Now the
patchset looks good to me. I'm going to push it if there are no
objections.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0002-Explore-alternative-orderings-of-group-by-p-20240115.patch (35.7K, ../../CAPpHfdtWiQk587H3F-SHm4VUrb26rU7kBF1GtzV1CKOzz+GcOw@mail.gmail.com/2-0002-Explore-alternative-orderings-of-group-by-p-20240115.patch)
download | inline diff:
From 3ee8c4b602285882af116df6e35f76d4b1972433 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:26:49 +0200
Subject: [PATCH 2/2] Explore alternative orderings of group-by pathkeys during
optimization.
When evaluating a query with a multi-column GROUP BY clause, we can minimize
sort operations or avoid them if we synchronize the order of GROUP BY clauses
with the ORDER BY sort clause or sort order, which comes from the underlying
query tree. Grouping does not imply any ordering, so we can compare
the keys in arbitrary order, and a Hash Agg leverages this. But for Group Agg,
we simply compared keys in the order specified in the query. This commit
explores alternative ordering of the keys, trying to find a cheaper one.
The ordering of group keys may interact with other parts of the query, some of
which may not be known while planning the grouping. For example, there may be
an explicit ORDER BY clause or some other ordering-dependent operation higher up
in the query, and using the same ordering may allow using either incremental
sort or even eliminating the sort entirely.
The patch always keeps the ordering specified in the query, assuming the user
might have additional insights.
This introduces a new GUC enable_group_by_reordering so that the optimization
may be disabled if needed.
Discussion: https://postgr.es/m/7c79e6a5-8597-74e8-0671-1c39d124c9d6%40sigaev.ru
Author: Andrey V. Lepikhov, Teodor Sigaev
Reviewed-by: Tomas Vondra, Claudio Freire, Gavin Flower, Dmitry Dolgov
Reviewed-by: Robert Haas, Pavel Borisov, David Rowley, Zhihong Yu
Reviewed-by: Tom Lane, Alexander Korotkov
---
src/backend/optimizer/path/equivclass.c | 13 +-
src/backend/optimizer/path/pathkeys.c | 220 ++++++++++++
src/backend/optimizer/plan/planner.c | 316 +++++++++++-------
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/pathnodes.h | 10 +
src/include/optimizer/paths.h | 2 +
src/test/regress/expected/aggregates.out | 185 ++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/aggregates.sql | 68 ++++
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 704 insertions(+), 125 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index e86dfeaecd4..4bd60a09c69 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,7 +652,18 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- return cur_ec; /* Match! */
+ {
+ /*
+ * Match!
+ *
+ * Copy the sortref if it wasn't set yet. That may happen if
+ * the ec was constructed from a WHERE clause, i.e. it doesn't
+ * have a target reference at all.
+ */
+ if (cur_ec->ec_sortref == 0 && sortref > 0)
+ cur_ec->ec_sortref = sortref;
+ return cur_ec;
+ }
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index ca94a31f71e..5aac6d66776 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -22,12 +22,15 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
+#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
+/* Consider reordering of GROUP BY keys? */
+bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -350,6 +353,170 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
+/*
+ * group_keys_reorder_by_pathkeys
+ * Reorder GROUP BY keys to match the input pathkeys.
+ *
+ * Function returns new lists (pathkeys and clauses), original GROUP BY lists
+ * stay untouched.
+ *
+ * Returns the number of GROUP BY keys with a matching pathkey.
+ */
+static int
+group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
+ List **group_clauses,
+ int num_groupby_pathkeys)
+{
+ List *new_group_pathkeys = NIL,
+ *new_group_clauses = NIL;
+ ListCell *lc;
+ int n;
+
+ if (pathkeys == NIL || *group_pathkeys == NIL)
+ return 0;
+
+ /*
+ * Walk the pathkeys (determining ordering of the input path) and see if
+ * there's a matching GROUP BY key. If we find one, we append it to the
+ * list, and do the same for the clauses.
+ *
+ * Once we find the first pathkey without a matching GROUP BY key, the
+ * rest of the pathkeys are useless and can't be used to evaluate the
+ * grouping, so we abort the loop and ignore the remaining pathkeys.
+ */
+ foreach(lc, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(lc);
+ SortGroupClause *sgc;
+
+ /*
+ * Pathkeys are built in a way that allows simply comparing pointers.
+ * Give up if we can't find the matching pointer.
+ */
+ if (foreach_current_index(lc) >= num_groupby_pathkeys ||
+ !list_member_ptr(*group_pathkeys, pathkey))
+ break;
+
+ /*
+ * Since 1349d27 pathkey coming from underlying node can be in the
+ * root->group_pathkeys but not in the processed_groupClause. So, we
+ * should be careful here.
+ */
+ sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
+ *group_clauses);
+ if (!sgc)
+ /* The grouping clause does not cover this pathkey */
+ break;
+
+ new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ }
+
+ /* remember the number of pathkeys with a matching GROUP BY key */
+ n = list_length(new_group_pathkeys);
+
+ /* append the remaining group pathkeys (will be treated as not sorted) */
+ *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
+ *group_pathkeys);
+ *group_clauses = list_concat_unique_ptr(new_group_clauses,
+ *group_clauses);
+
+ return n;
+}
+
+/*
+ * get_useful_group_keys_orderings
+ * Determine which orderings of GROUP BY keys are potentially interesting.
+ *
+ * Returns a list of PathKeyInfo items, each representing an interesting
+ * ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
+ * matching order.
+ *
+ * The function considers (and keeps) multiple GROUP BY orderings:
+ *
+ * - the original ordering, as specified by the GROUP BY clause,
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
+ * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ */
+List *
+get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
+{
+ Query *parse = root->parse;
+ List *infos = NIL;
+ PathKeyInfo *info;
+
+ List *pathkeys = root->group_pathkeys;
+ List *clauses = root->processed_groupClause;
+
+ /* always return at least the original pathkeys/clauses */
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+ infos = lappend(infos, info);
+
+ /*
+ * Should we try generating alternative orderings of the group keys? If
+ * not, we produce only the order specified in the query, i.e. the
+ * optimization is effectively disabled.
+ */
+ if (!enable_group_by_reordering)
+ return infos;
+
+ /*
+ * Grouping sets have own and more complex logic to decide the ordering.
+ */
+ if (parse->groupingSets)
+ return infos;
+
+ /*
+ * If the path is sorted in some way, try reordering the group keys to
+ * match the path as much of the ordering as possible. Then thanks to
+ * incremental sort we would get this sort as cheap as possible.
+ */
+ if (path->pathkeys)
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(path->pathkeys, &pathkeys, &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(path->pathkeys)))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ /*
+ * Try reordering pathkeys to minimize the sort cost (this time consider
+ * the ORDER BY clause).
+ */
+ if (root->sort_pathkeys)
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
+ &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(path->pathkeys)))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ return infos;
+}
+
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -1939,6 +2106,54 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
+/*
+ * pathkeys_useful_for_grouping
+ * Count the number of pathkeys that are useful for grouping (instead of
+ * explicit sort)
+ *
+ * Group pathkeys could be reordered to benefit from the ordering. The
+ * ordering may not be "complete" and may require incremental sort, but that's
+ * fine. So we simply count prefix pathkeys with a matching group key, and
+ * stop once we find the first pathkey without a match.
+ *
+ * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
+ * pathkeys are useful for grouping, and we might do incremental sort to get
+ * path ordered by (a,b,e).
+ *
+ * This logic is necessary to retain paths with ordering not matching grouping
+ * keys directly, without the reordering.
+ *
+ * Returns the length of pathkey prefix with matching group keys.
+ */
+static int
+pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
+{
+ ListCell *key;
+ int n = 0;
+
+ /* no special ordering requested for grouping */
+ if (root->group_pathkeys == NIL)
+ return 0;
+
+ /* unordered path */
+ if (pathkeys == NIL)
+ return 0;
+
+ /* walk the pathkeys and search for matching group key */
+ foreach(key, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(key);
+
+ /* no matching group key, we're done */
+ if (!list_member_ptr(root->group_pathkeys, pathkey))
+ break;
+
+ n++;
+ }
+
+ return n;
+}
+
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -1953,6 +2168,9 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
+ if (nuseful2 > nuseful)
+ nuseful = nuseful2;
+ nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -1988,6 +2206,8 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
+ if (root->group_pathkeys != NIL)
+ return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 014b179c3f0..239b799a6b2 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2805,8 +2805,9 @@ remove_useless_groupby_columns(PlannerInfo *root)
*
* In principle it might be interesting to consider other orderings of the
* GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. We don't
- * bother with that, though. Hashed grouping will frequently win anyway.
+ * possible plans (eg an indexscan) and thereby reduce cost. However, we
+ * don't yet have sufficient information to do that here, so that's left until
+ * later in planning. See get_useful_group_keys_orderings().
*
* Note: we need no comparable processing of the distinctClause because
* the parser already enforced that that matches ORDER BY.
@@ -6891,103 +6892,135 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- grouped_rel,
- path,
- cheapest_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY. Make an
- * AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- root->processed_groupClause,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (parse->groupClause)
+ foreach(lc2, pathkey_orderings)
{
- /*
- * We have GROUP BY without aggregation or grouping sets. Make
- * a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- root->processed_groupClause,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
- /*
- * Instead of operating directly on the input relation, we can
- * consider finalizing a partially aggregated path.
- */
- if (partially_grouped_rel != NULL)
- {
- foreach(lc, partially_grouped_rel->pathlist)
- {
- Path *path = (Path *) lfirst(lc);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
path = make_ordered_path(root,
grouped_rel,
path,
- partially_grouped_rel->cheapest_total_path,
- root->group_pathkeys);
-
+ cheapest_path,
+ info->pathkeys);
if (path == NULL)
continue;
- if (parse->hasAggs)
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make
+ * an AggPath.
+ */
add_path(grouped_rel, (Path *)
create_agg_path(root,
grouped_rel,
path,
grouped_rel->reltarget,
parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- root->processed_groupClause,
+ AGGSPLIT_SIMPLE,
+ info->clauses,
havingQual,
- agg_final_costs,
+ agg_costs,
dNumGroups));
- else
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets.
+ * Make a GroupPath.
+ */
add_path(grouped_rel, (Path *)
create_group_path(root,
grouped_rel,
path,
- root->processed_groupClause,
+ info->clauses,
havingQual,
dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
+ }
+ }
+
+ /*
+ * Instead of operating directly on the input relation, we can
+ * consider finalizing a partially aggregated path.
+ */
+ if (partially_grouped_rel != NULL)
+ {
+ foreach(lc, partially_grouped_rel->pathlist)
+ {
+ ListCell *lc2;
+ Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
+
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
+
+ Assert(list_length(pathkey_orderings) > 0);
+
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ info->clauses,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ info->clauses,
+ havingQual,
+ dNumGroups));
+
+ }
}
}
}
@@ -7190,37 +7223,54 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_total_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
partially_grouped_rel,
path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialGroups));
+ cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialGroups));
+ }
}
}
@@ -7229,37 +7279,55 @@ create_partial_grouping_paths(PlannerInfo *root,
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_partial_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialPartialGroups));
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialPartialGroups));
+ }
}
}
@@ -7373,6 +7441,8 @@ gather_grouping_paths(PlannerInfo *root, RelOptInfo *rel)
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted prefix
* of the list without having the list be fully sorted.
+ *
+ * XXX Shouldn't this also consider the group-key-reordering?
*/
if (!enable_incremental_sort || list_length(root->group_pathkeys) == 1)
return;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e53ebc6dc2b..7fe58518d7d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1050,6 +1050,16 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables reordering of GROUP BY keys."),
+ NULL,
+ GUC_EXPLAIN
+ },
+ &enable_group_by_reordering,
+ true,
+ NULL, NULL, NULL
+ },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 835b0e9ba89..da10b43dac3 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -399,6 +399,7 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
+#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index b9713ec9aa6..137da178dc7 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1456,6 +1456,16 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
+/*
+ * Combines the information about pathkeys and the associated clauses.
+ */
+typedef struct PathKeyInfo
+{
+ NodeTag type;
+ List *pathkeys;
+ List *clauses;
+} PathKeyInfo;
+
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index efd4abc28f3..0e8a9c94ba6 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,6 +24,7 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
+extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -204,6 +205,7 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
+extern List *get_useful_group_keys_orderings(PlannerInfo *root, Path *path);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index f635c5a1afb..ca38e78f211 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,6 +2728,191 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+-- GROUP BY optimization by reorder columns by frequency
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Limit
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(7 rows)
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+ QUERY PLAN
+------------------------------
+ Group
+ Group Key: y, x, z, w
+ -> Sort
+ Sort Key: y, x, z, w
+ -> Seq Scan on btg
+(5 rows)
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+ QUERY PLAN
+-----------------------------------------------
+ Sort
+ Sort Key: ((x * x)), z
+ -> Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(8 rows)
+
+DROP TABLE btg;
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+ QUERY PLAN
+--------------------------------------------------------
+ Sort
+ Sort Key: c2
+ -> GroupAggregate
+ Group Key: c1
+ -> Sort
+ Sort Key: c1, c2
+ -> Bitmap Heap Scan on t1
+ Recheck Cond: (c2 < 100)
+ -> Bitmap Index Scan on t1_c2_idx
+ Index Cond: (c2 < 100)
+(10 rows)
+
+DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------
+ Incremental Sort
+ Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
+ Presorted Key: c1.relpages, c1.relname
+ -> Group
+ Group Key: c1.relpages, c1.relname, c1.reltuples
+ -> Incremental Sort
+ Sort Key: c1.relpages, c1.relname, c1.reltuples
+ Presorted Key: c1.relpages, c1.relname
+ -> Merge Join
+ Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ -> Sort
+ Sort Key: c1.relpages, c1.relname
+ -> Seq Scan on pg_class c1
+ -> Sort
+ Sort Key: c2.relpages, c2.relname
+ -> Seq Scan on pg_class c2
+(16 rows)
+
+RESET enable_hashjoin;
+RESET enable_nestloop;
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 271313ebf86..9be7aca2b8a 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,6 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
+ enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -133,7 +134,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(22 rows)
+(23 rows)
-- There are always wait event descriptions for various types.
select type, count(*) > 0 as ok FROM pg_wait_events
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index cc8f0efad55..cf87b5d5ddd 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,6 +1181,74 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+
+-- GROUP BY optimization by reorder columns by frequency
+
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+
+DROP TABLE btg;
+
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+DROP TABLE t1 CASCADE;
+
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+RESET enable_hashjoin;
+RESET enable_nestloop;
+
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
+
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..e575fa13c63 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+PathKeyInfo
--
2.39.3 (Apple Git-145)
[application/octet-stream] 0001-Generalize-common-code-of-adding-sort-befor-20240115.patch (9.1K, ../../CAPpHfdtWiQk587H3F-SHm4VUrb26rU7kBF1GtzV1CKOzz+GcOw@mail.gmail.com/3-0001-Generalize-common-code-of-adding-sort-befor-20240115.patch)
download | inline diff:
From 2167ada5fc9e2d07b1f20cd840bea409e0e594c9 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:27:58 +0200
Subject: [PATCH 1/2] Generalize common code of adding sort before generation
of grouping paths
Extract the repetitive code pattern into a new function make_ordered_path().
Discussion: https://postgr.es/m/CAPpHfdtzaVa7S4onKy3YvttF2rrH5hQNHx9HtcSTLbpjx%2BMJ%2Bw%40mail.gmail.com
Author: Andrei Lepikhov
---
src/backend/optimizer/plan/planner.c | 228 ++++++++++-----------------
1 file changed, 80 insertions(+), 148 deletions(-)
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 667723b6753..014b179c3f0 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6809,6 +6809,58 @@ done:
return parallel_workers;
}
+/*
+ * make_ordered_path
+ * Return a path ordered by 'pathkeys' based on the given 'path'. May
+ * return NULL if it doesn't make sense to generate an ordered path in
+ * this case.
+ */
+static Path *
+make_ordered_path(PlannerInfo *root, RelOptInfo *rel, Path *path,
+ Path *cheapest_path, List *pathkeys)
+{
+ bool is_sorted;
+ int presorted_keys;
+
+ is_sorted = pathkeys_count_contained_in(pathkeys,
+ path->pathkeys,
+ &presorted_keys);
+
+ if (!is_sorted)
+ {
+ /*
+ * Try at least sorting the cheapest path and also try incrementally
+ * sorting any path which is partially sorted already (no need to deal
+ * with paths which have presorted keys when incremental sort is
+ * disabled unless it's the cheapest input path).
+ */
+ if (path != cheapest_path &&
+ (presorted_keys == 0 || !enable_incremental_sort))
+ return NULL;
+
+ /*
+ * We've no need to consider both a sort and incremental sort. We'll
+ * just do a sort if there are no presorted keys and an incremental
+ * sort when there are presorted keys.
+ */
+ if (presorted_keys == 0 || !enable_incremental_sort)
+ path = (Path *) create_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ -1.0);
+ else
+ path = (Path *) create_incremental_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ presorted_keys,
+ -1.0);
+ }
+
+ return path;
+}
+
/*
* add_paths_to_grouping_rel
*
@@ -6840,45 +6892,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ cheapest_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6935,46 +6957,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, partially_grouped_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially
- * sorted already (no need to deal with paths which have
- * presorted keys when incremental sort is disabled unless
- * it's the cheapest input path).
- */
- if (path != partially_grouped_rel->cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental
- * sort. We'll just do a sort if there are no pre-sorted
- * keys and an incremental sort when there are presorted
- * keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(grouped_rel, (Path *)
@@ -7200,44 +7191,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(partially_grouped_rel, (Path *)
@@ -7268,45 +7230,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->partial_pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
-
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_partial_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-15 06:42 Richard Guo <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: Richard Guo @ 2024-01-15 06:42 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Mon, Jan 15, 2024 at 8:20 AM Alexander Korotkov <[email protected]>
wrote:
> Thank you for providing the test case relevant for this code change.
> The revised patch incorporating this change is attached. Now the
> patchset looks good to me. I'm going to push it if there are no
> objections.
Seems I'm late for the party. Can we hold for several more days? I'd
like to have a review on this patch.
Thanks
Richard
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-15 07:56 Alexander Korotkov <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-01-15 07:56 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Mon, Jan 15, 2024 at 8:42 AM Richard Guo <[email protected]> wrote:
> On Mon, Jan 15, 2024 at 8:20 AM Alexander Korotkov <[email protected]> wrote:
>>
>> Thank you for providing the test case relevant for this code change.
>> The revised patch incorporating this change is attached. Now the
>> patchset looks good to me. I'm going to push it if there are no
>> objections.
>
> Seems I'm late for the party. Can we hold for several more days? I'd
> like to have a review on this patch.
Sure, NP. I'll hold it till your review.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-15 09:46 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-01-15 09:46 UTC (permalink / raw)
To: Richard Guo <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On 15/1/2024 13:42, Richard Guo wrote:
>
> On Mon, Jan 15, 2024 at 8:20 AM Alexander Korotkov <[email protected]
> <mailto:[email protected]>> wrote:
>
> Thank you for providing the test case relevant for this code change.
> The revised patch incorporating this change is attached. Now the
> patchset looks good to me. I'm going to push it if there are no
> objections.
>
>
> Seems I'm late for the party. Can we hold for several more days? I'd
> like to have a review on this patch.
Get on board! It looks like this feature needs as much review as
possible (likewise SJE).
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-16 02:48 Richard Guo <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Richard Guo @ 2024-01-16 02:48 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Mon, Jan 15, 2024 at 3:56 PM Alexander Korotkov <[email protected]>
wrote:
> On Mon, Jan 15, 2024 at 8:42 AM Richard Guo <[email protected]>
> wrote:
> > On Mon, Jan 15, 2024 at 8:20 AM Alexander Korotkov <[email protected]>
> wrote:
> >>
> >> Thank you for providing the test case relevant for this code change.
> >> The revised patch incorporating this change is attached. Now the
> >> patchset looks good to me. I'm going to push it if there are no
> >> objections.
> >
> > Seems I'm late for the party. Can we hold for several more days? I'd
> > like to have a review on this patch.
>
> Sure, NP. I'll hold it till your review.
Thanks. Appreciate that.
I have briefly reviewed this patch and here are some comments.
* When trying to match the ordering of GROUP BY to that of ORDER BY in
get_useful_group_keys_orderings, this patch checks against the length of
path->pathkeys. This does not make sense. I guess this is a typo and
the real intention is to check against root->sort_pathkeys.
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -504,7 +504,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path
*path)
root->num_groupby_pathkeys);
if (n > 0 &&
- (enable_incremental_sort || n == list_length(path->pathkeys)))
+ (enable_incremental_sort || n ==
list_length(root->sort_pathkeys)))
However, I think this is also incorrect. When incremental sort is
disabled, it is reasonable to consider reordering the GROUP BY keys only
if the number of matching pathkeys is equal to the length of
root->group_pathkeys i.e. if 'n == list_length(root->group_pathkeys)'.
* When the original ordering of GROUP BY keys matches the path's
pathkeys or ORDER BY keys, get_useful_group_keys_orderings would
generate duplicate PathKeyInfos. Consequently, this duplication would
lead to the creation and addition of identical paths multiple times.
This is not great. Consider the query below:
create table t (a int, b int);
create index on t (a, b);
set enable_hashagg to off;
explain select count(*) from t group by a, b;
QUERY PLAN
----------------------------------------------------------------------------------
GroupAggregate (cost=0.15..97.27 rows=226 width=16)
Group Key: a, b
-> Index Only Scan using t_a_b_idx on t (cost=0.15..78.06 rows=2260
width=8)
(3 rows)
The same path with group keys {a, b} is created and added twice.
* Part of the work performed in this patch overlaps with that of
preprocess_groupclause. They are both trying to adjust the ordering of
the GROUP BY keys to match ORDER BY. I wonder if it would be better to
perform this work only once.
* When reordering the GROUP BY keys, I think the following checks need
some improvements.
+ /*
+ * Since 1349d27 pathkey coming from underlying node can be in the
+ * root->group_pathkeys but not in the processed_groupClause. So, we
+ * should be careful here.
+ */
+ sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
+ *group_clauses);
+ if (!sgc)
+ /* The grouping clause does not cover this pathkey */
+ break;
I think we need to check or assert 'pathkey->pk_eclass->ec_sortref' is
valid before calling get_sortgroupref_clause_noerr with it. Also, how
can we guarantee that the returned GROUP BY item is sortable? Should we
check that with OidIsValid(sgc->sortop)?
Thanks
Richard
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-16 15:05 Alexander Korotkov <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-01-16 15:05 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
Hi!
Thank you for your review. The revised patchset is attached.
On Tue, Jan 16, 2024 at 4:48 AM Richard Guo <[email protected]> wrote:
> On Mon, Jan 15, 2024 at 3:56 PM Alexander Korotkov <[email protected]> wrote:
>>
>> On Mon, Jan 15, 2024 at 8:42 AM Richard Guo <[email protected]> wrote:
>> > On Mon, Jan 15, 2024 at 8:20 AM Alexander Korotkov <[email protected]> wrote:
>> >>
>> >> Thank you for providing the test case relevant for this code change.
>> >> The revised patch incorporating this change is attached. Now the
>> >> patchset looks good to me. I'm going to push it if there are no
>> >> objections.
>> >
>> > Seems I'm late for the party. Can we hold for several more days? I'd
>> > like to have a review on this patch.
>>
>> Sure, NP. I'll hold it till your review.
>
>
> Thanks. Appreciate that.
>
> I have briefly reviewed this patch and here are some comments.
>
> * When trying to match the ordering of GROUP BY to that of ORDER BY in
> get_useful_group_keys_orderings, this patch checks against the length of
> path->pathkeys. This does not make sense. I guess this is a typo and
> the real intention is to check against root->sort_pathkeys.
>
> --- a/src/backend/optimizer/path/pathkeys.c
> +++ b/src/backend/optimizer/path/pathkeys.c
> @@ -504,7 +504,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
> root->num_groupby_pathkeys);
>
> if (n > 0 &&
> - (enable_incremental_sort || n == list_length(path->pathkeys)))
> + (enable_incremental_sort || n == list_length(root->sort_pathkeys)))
>
> However, I think this is also incorrect. When incremental sort is
> disabled, it is reasonable to consider reordering the GROUP BY keys only
> if the number of matching pathkeys is equal to the length of
> root->group_pathkeys i.e. if 'n == list_length(root->group_pathkeys)'.
Hmm... Why should this be list_length(root->group_pathkeys) while
we're targeting root->sort_pathkeys. I yet changed that to
list_length(root->sort_pathkeys).
> * When the original ordering of GROUP BY keys matches the path's
> pathkeys or ORDER BY keys, get_useful_group_keys_orderings would
> generate duplicate PathKeyInfos. Consequently, this duplication would
> lead to the creation and addition of identical paths multiple times.
> This is not great. Consider the query below:
>
> create table t (a int, b int);
> create index on t (a, b);
> set enable_hashagg to off;
>
> explain select count(*) from t group by a, b;
> QUERY PLAN
> ----------------------------------------------------------------------------------
> GroupAggregate (cost=0.15..97.27 rows=226 width=16)
> Group Key: a, b
> -> Index Only Scan using t_a_b_idx on t (cost=0.15..78.06 rows=2260 width=8)
> (3 rows)
>
> The same path with group keys {a, b} is created and added twice.
I tried to address that.
> * Part of the work performed in this patch overlaps with that of
> preprocess_groupclause. They are both trying to adjust the ordering of
> the GROUP BY keys to match ORDER BY. I wonder if it would be better to
> perform this work only once.
Andrei, could you take a look.
> * When reordering the GROUP BY keys, I think the following checks need
> some improvements.
>
> + /*
> + * Since 1349d27 pathkey coming from underlying node can be in the
> + * root->group_pathkeys but not in the processed_groupClause. So, we
> + * should be careful here.
> + */
> + sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
> + *group_clauses);
> + if (!sgc)
> + /* The grouping clause does not cover this pathkey */
> + break;
>
> I think we need to check or assert 'pathkey->pk_eclass->ec_sortref' is
> valid before calling get_sortgroupref_clause_noerr with it. Also, how
> can we guarantee that the returned GROUP BY item is sortable? Should we
> check that with OidIsValid(sgc->sortop)?
Added.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Generalize-common-code-of-adding-sort-befor-20240116.patch (9.1K, ../../CAPpHfdsfHn89cAFEa6-6Xm1sB_xf8Rf2WNab6DQzb8Rvs0yD8g@mail.gmail.com/2-0001-Generalize-common-code-of-adding-sort-befor-20240116.patch)
download | inline diff:
From 2167ada5fc9e2d07b1f20cd840bea409e0e594c9 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:27:58 +0200
Subject: [PATCH 1/2] Generalize common code of adding sort before generation
of grouping paths
Extract the repetitive code pattern into a new function make_ordered_path().
Discussion: https://postgr.es/m/CAPpHfdtzaVa7S4onKy3YvttF2rrH5hQNHx9HtcSTLbpjx%2BMJ%2Bw%40mail.gmail.com
Author: Andrei Lepikhov
---
src/backend/optimizer/plan/planner.c | 228 ++++++++++-----------------
1 file changed, 80 insertions(+), 148 deletions(-)
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 667723b6753..014b179c3f0 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6809,6 +6809,58 @@ done:
return parallel_workers;
}
+/*
+ * make_ordered_path
+ * Return a path ordered by 'pathkeys' based on the given 'path'. May
+ * return NULL if it doesn't make sense to generate an ordered path in
+ * this case.
+ */
+static Path *
+make_ordered_path(PlannerInfo *root, RelOptInfo *rel, Path *path,
+ Path *cheapest_path, List *pathkeys)
+{
+ bool is_sorted;
+ int presorted_keys;
+
+ is_sorted = pathkeys_count_contained_in(pathkeys,
+ path->pathkeys,
+ &presorted_keys);
+
+ if (!is_sorted)
+ {
+ /*
+ * Try at least sorting the cheapest path and also try incrementally
+ * sorting any path which is partially sorted already (no need to deal
+ * with paths which have presorted keys when incremental sort is
+ * disabled unless it's the cheapest input path).
+ */
+ if (path != cheapest_path &&
+ (presorted_keys == 0 || !enable_incremental_sort))
+ return NULL;
+
+ /*
+ * We've no need to consider both a sort and incremental sort. We'll
+ * just do a sort if there are no presorted keys and an incremental
+ * sort when there are presorted keys.
+ */
+ if (presorted_keys == 0 || !enable_incremental_sort)
+ path = (Path *) create_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ -1.0);
+ else
+ path = (Path *) create_incremental_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ presorted_keys,
+ -1.0);
+ }
+
+ return path;
+}
+
/*
* add_paths_to_grouping_rel
*
@@ -6840,45 +6892,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ cheapest_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6935,46 +6957,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, partially_grouped_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially
- * sorted already (no need to deal with paths which have
- * presorted keys when incremental sort is disabled unless
- * it's the cheapest input path).
- */
- if (path != partially_grouped_rel->cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental
- * sort. We'll just do a sort if there are no pre-sorted
- * keys and an incremental sort when there are presorted
- * keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(grouped_rel, (Path *)
@@ -7200,44 +7191,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(partially_grouped_rel, (Path *)
@@ -7268,45 +7230,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->partial_pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
-
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_partial_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
--
2.39.3 (Apple Git-145)
[application/octet-stream] 0002-Explore-alternative-orderings-of-group-by-p-20240116.patch (36.2K, ../../CAPpHfdsfHn89cAFEa6-6Xm1sB_xf8Rf2WNab6DQzb8Rvs0yD8g@mail.gmail.com/3-0002-Explore-alternative-orderings-of-group-by-p-20240116.patch)
download | inline diff:
From f0c0340555a50415b03d0d684c4cb8422ccdb926 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:26:49 +0200
Subject: [PATCH 2/2] Explore alternative orderings of group-by pathkeys during
optimization.
When evaluating a query with a multi-column GROUP BY clause, we can minimize
sort operations or avoid them if we synchronize the order of GROUP BY clauses
with the ORDER BY sort clause or sort order, which comes from the underlying
query tree. Grouping does not imply any ordering, so we can compare
the keys in arbitrary order, and a Hash Agg leverages this. But for Group Agg,
we simply compared keys in the order specified in the query. This commit
explores alternative ordering of the keys, trying to find a cheaper one.
The ordering of group keys may interact with other parts of the query, some of
which may not be known while planning the grouping. For example, there may be
an explicit ORDER BY clause or some other ordering-dependent operation higher up
in the query, and using the same ordering may allow using either incremental
sort or even eliminating the sort entirely.
The patch always keeps the ordering specified in the query, assuming the user
might have additional insights.
This introduces a new GUC enable_group_by_reordering so that the optimization
may be disabled if needed.
Discussion: https://postgr.es/m/7c79e6a5-8597-74e8-0671-1c39d124c9d6%40sigaev.ru
Author: Andrey V. Lepikhov, Teodor Sigaev
Reviewed-by: Tomas Vondra, Claudio Freire, Gavin Flower, Dmitry Dolgov
Reviewed-by: Robert Haas, Pavel Borisov, David Rowley, Zhihong Yu
Reviewed-by: Tom Lane, Alexander Korotkov
---
src/backend/optimizer/path/equivclass.c | 13 +-
src/backend/optimizer/path/pathkeys.c | 230 +++++++++++++
src/backend/optimizer/plan/planner.c | 316 +++++++++++-------
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/pathnodes.h | 10 +
src/include/optimizer/paths.h | 2 +
src/test/regress/expected/aggregates.out | 185 ++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/aggregates.sql | 68 ++++
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 714 insertions(+), 125 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index e86dfeaecd4..4bd60a09c69 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,7 +652,18 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- return cur_ec; /* Match! */
+ {
+ /*
+ * Match!
+ *
+ * Copy the sortref if it wasn't set yet. That may happen if
+ * the ec was constructed from a WHERE clause, i.e. it doesn't
+ * have a target reference at all.
+ */
+ if (cur_ec->ec_sortref == 0 && sortref > 0)
+ cur_ec->ec_sortref = sortref;
+ return cur_ec;
+ }
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index ca94a31f71e..919d54dd790 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -22,12 +22,15 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
+#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
+/* Consider reordering of GROUP BY keys? */
+bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -350,6 +353,180 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
+/*
+ * group_keys_reorder_by_pathkeys
+ * Reorder GROUP BY keys to match the input pathkeys.
+ *
+ * Function returns new lists (pathkeys and clauses), original GROUP BY lists
+ * stay untouched.
+ *
+ * Returns the number of GROUP BY keys with a matching pathkey.
+ */
+static int
+group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
+ List **group_clauses,
+ int num_groupby_pathkeys)
+{
+ List *new_group_pathkeys = NIL,
+ *new_group_clauses = NIL;
+ ListCell *lc;
+ int n;
+
+ if (pathkeys == NIL || *group_pathkeys == NIL)
+ return 0;
+
+ /*
+ * Walk the pathkeys (determining ordering of the input path) and see if
+ * there's a matching GROUP BY key. If we find one, we append it to the
+ * list, and do the same for the clauses.
+ *
+ * Once we find the first pathkey without a matching GROUP BY key, the
+ * rest of the pathkeys are useless and can't be used to evaluate the
+ * grouping, so we abort the loop and ignore the remaining pathkeys.
+ */
+ foreach(lc, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(lc);
+ SortGroupClause *sgc;
+
+ /*
+ * Pathkeys are built in a way that allows simply comparing pointers.
+ * Give up if we can't find the matching pointer. Also give up if
+ * there is no sortclause reference for some reason.
+ */
+ if (foreach_current_index(lc) >= num_groupby_pathkeys ||
+ !list_member_ptr(*group_pathkeys, pathkey) ||
+ pathkey->pk_eclass->ec_sortref == 0)
+ break;
+
+ /*
+ * Since 1349d27 pathkey coming from underlying node can be in the
+ * root->group_pathkeys but not in the processed_groupClause. So, we
+ * should be careful here.
+ */
+ sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
+ *group_clauses);
+ if (!sgc)
+ /* The grouping clause does not cover this pathkey */
+ break;
+
+ /*
+ * Sort group clause should have an ordering operator as long as there
+ * is an associated pathkey.
+ */
+ Assert(OidIsValid(sgc->sortop));
+
+ new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ }
+
+ /* remember the number of pathkeys with a matching GROUP BY key */
+ n = list_length(new_group_pathkeys);
+
+ /* append the remaining group pathkeys (will be treated as not sorted) */
+ *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
+ *group_pathkeys);
+ *group_clauses = list_concat_unique_ptr(new_group_clauses,
+ *group_clauses);
+
+ return n;
+}
+
+/*
+ * get_useful_group_keys_orderings
+ * Determine which orderings of GROUP BY keys are potentially interesting.
+ *
+ * Returns a list of PathKeyInfo items, each representing an interesting
+ * ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
+ * matching order.
+ *
+ * The function considers (and keeps) multiple GROUP BY orderings:
+ *
+ * - the original ordering, as specified by the GROUP BY clause,
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
+ * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ */
+List *
+get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
+{
+ Query *parse = root->parse;
+ List *infos = NIL;
+ PathKeyInfo *info;
+
+ List *pathkeys = root->group_pathkeys;
+ List *clauses = root->processed_groupClause;
+
+ /* always return at least the original pathkeys/clauses */
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+ infos = lappend(infos, info);
+
+ /*
+ * Should we try generating alternative orderings of the group keys? If
+ * not, we produce only the order specified in the query, i.e. the
+ * optimization is effectively disabled.
+ */
+ if (!enable_group_by_reordering)
+ return infos;
+
+ /*
+ * Grouping sets have own and more complex logic to decide the ordering.
+ */
+ if (parse->groupingSets)
+ return infos;
+
+ /*
+ * If the path is sorted in some way, try reordering the group keys to
+ * match the path as much of the ordering as possible. Then thanks to
+ * incremental sort we would get this sort as cheap as possible.
+ */
+ if (path->pathkeys &&
+ !pathkeys_contained_in(path->pathkeys, root->group_pathkeys))
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(path->pathkeys, &pathkeys, &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 && compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL &&
+ (enable_incremental_sort || n == list_length(path->pathkeys)))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ /*
+ * Try reordering pathkeys to minimize the sort cost (this time consider
+ * the ORDER BY clause).
+ */
+ if (root->sort_pathkeys &&
+ !pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys))
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
+ &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 && compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL &&
+ (enable_incremental_sort || n == list_length(root->sort_pathkeys)))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ return infos;
+}
+
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -1939,6 +2116,54 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
+/*
+ * pathkeys_useful_for_grouping
+ * Count the number of pathkeys that are useful for grouping (instead of
+ * explicit sort)
+ *
+ * Group pathkeys could be reordered to benefit from the ordering. The
+ * ordering may not be "complete" and may require incremental sort, but that's
+ * fine. So we simply count prefix pathkeys with a matching group key, and
+ * stop once we find the first pathkey without a match.
+ *
+ * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
+ * pathkeys are useful for grouping, and we might do incremental sort to get
+ * path ordered by (a,b,e).
+ *
+ * This logic is necessary to retain paths with ordering not matching grouping
+ * keys directly, without the reordering.
+ *
+ * Returns the length of pathkey prefix with matching group keys.
+ */
+static int
+pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
+{
+ ListCell *key;
+ int n = 0;
+
+ /* no special ordering requested for grouping */
+ if (root->group_pathkeys == NIL)
+ return 0;
+
+ /* unordered path */
+ if (pathkeys == NIL)
+ return 0;
+
+ /* walk the pathkeys and search for matching group key */
+ foreach(key, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(key);
+
+ /* no matching group key, we're done */
+ if (!list_member_ptr(root->group_pathkeys, pathkey))
+ break;
+
+ n++;
+ }
+
+ return n;
+}
+
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -1953,6 +2178,9 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
+ if (nuseful2 > nuseful)
+ nuseful = nuseful2;
+ nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -1988,6 +2216,8 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
+ if (root->group_pathkeys != NIL)
+ return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 014b179c3f0..239b799a6b2 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2805,8 +2805,9 @@ remove_useless_groupby_columns(PlannerInfo *root)
*
* In principle it might be interesting to consider other orderings of the
* GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. We don't
- * bother with that, though. Hashed grouping will frequently win anyway.
+ * possible plans (eg an indexscan) and thereby reduce cost. However, we
+ * don't yet have sufficient information to do that here, so that's left until
+ * later in planning. See get_useful_group_keys_orderings().
*
* Note: we need no comparable processing of the distinctClause because
* the parser already enforced that that matches ORDER BY.
@@ -6891,103 +6892,135 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- grouped_rel,
- path,
- cheapest_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY. Make an
- * AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- root->processed_groupClause,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (parse->groupClause)
+ foreach(lc2, pathkey_orderings)
{
- /*
- * We have GROUP BY without aggregation or grouping sets. Make
- * a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- root->processed_groupClause,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
- /*
- * Instead of operating directly on the input relation, we can
- * consider finalizing a partially aggregated path.
- */
- if (partially_grouped_rel != NULL)
- {
- foreach(lc, partially_grouped_rel->pathlist)
- {
- Path *path = (Path *) lfirst(lc);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
path = make_ordered_path(root,
grouped_rel,
path,
- partially_grouped_rel->cheapest_total_path,
- root->group_pathkeys);
-
+ cheapest_path,
+ info->pathkeys);
if (path == NULL)
continue;
- if (parse->hasAggs)
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make
+ * an AggPath.
+ */
add_path(grouped_rel, (Path *)
create_agg_path(root,
grouped_rel,
path,
grouped_rel->reltarget,
parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- root->processed_groupClause,
+ AGGSPLIT_SIMPLE,
+ info->clauses,
havingQual,
- agg_final_costs,
+ agg_costs,
dNumGroups));
- else
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets.
+ * Make a GroupPath.
+ */
add_path(grouped_rel, (Path *)
create_group_path(root,
grouped_rel,
path,
- root->processed_groupClause,
+ info->clauses,
havingQual,
dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
+ }
+ }
+
+ /*
+ * Instead of operating directly on the input relation, we can
+ * consider finalizing a partially aggregated path.
+ */
+ if (partially_grouped_rel != NULL)
+ {
+ foreach(lc, partially_grouped_rel->pathlist)
+ {
+ ListCell *lc2;
+ Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
+
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
+
+ Assert(list_length(pathkey_orderings) > 0);
+
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ info->clauses,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ info->clauses,
+ havingQual,
+ dNumGroups));
+
+ }
}
}
}
@@ -7190,37 +7223,54 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_total_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
partially_grouped_rel,
path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialGroups));
+ cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialGroups));
+ }
}
}
@@ -7229,37 +7279,55 @@ create_partial_grouping_paths(PlannerInfo *root,
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_partial_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialPartialGroups));
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialPartialGroups));
+ }
}
}
@@ -7373,6 +7441,8 @@ gather_grouping_paths(PlannerInfo *root, RelOptInfo *rel)
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted prefix
* of the list without having the list be fully sorted.
+ *
+ * XXX Shouldn't this also consider the group-key-reordering?
*/
if (!enable_incremental_sort || list_length(root->group_pathkeys) == 1)
return;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e53ebc6dc2b..7fe58518d7d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1050,6 +1050,16 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables reordering of GROUP BY keys."),
+ NULL,
+ GUC_EXPLAIN
+ },
+ &enable_group_by_reordering,
+ true,
+ NULL, NULL, NULL
+ },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 835b0e9ba89..da10b43dac3 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -399,6 +399,7 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
+#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index b9713ec9aa6..137da178dc7 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1456,6 +1456,16 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
+/*
+ * Combines the information about pathkeys and the associated clauses.
+ */
+typedef struct PathKeyInfo
+{
+ NodeTag type;
+ List *pathkeys;
+ List *clauses;
+} PathKeyInfo;
+
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index efd4abc28f3..0e8a9c94ba6 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,6 +24,7 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
+extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -204,6 +205,7 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
+extern List *get_useful_group_keys_orderings(PlannerInfo *root, Path *path);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index f635c5a1afb..ca38e78f211 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,6 +2728,191 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+-- GROUP BY optimization by reorder columns by frequency
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Limit
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(7 rows)
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+ QUERY PLAN
+------------------------------
+ Group
+ Group Key: y, x, z, w
+ -> Sort
+ Sort Key: y, x, z, w
+ -> Seq Scan on btg
+(5 rows)
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+ QUERY PLAN
+-----------------------------------------------
+ Sort
+ Sort Key: ((x * x)), z
+ -> Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(8 rows)
+
+DROP TABLE btg;
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+ QUERY PLAN
+--------------------------------------------------------
+ Sort
+ Sort Key: c2
+ -> GroupAggregate
+ Group Key: c1
+ -> Sort
+ Sort Key: c1, c2
+ -> Bitmap Heap Scan on t1
+ Recheck Cond: (c2 < 100)
+ -> Bitmap Index Scan on t1_c2_idx
+ Index Cond: (c2 < 100)
+(10 rows)
+
+DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------
+ Incremental Sort
+ Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
+ Presorted Key: c1.relpages, c1.relname
+ -> Group
+ Group Key: c1.relpages, c1.relname, c1.reltuples
+ -> Incremental Sort
+ Sort Key: c1.relpages, c1.relname, c1.reltuples
+ Presorted Key: c1.relpages, c1.relname
+ -> Merge Join
+ Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ -> Sort
+ Sort Key: c1.relpages, c1.relname
+ -> Seq Scan on pg_class c1
+ -> Sort
+ Sort Key: c2.relpages, c2.relname
+ -> Seq Scan on pg_class c2
+(16 rows)
+
+RESET enable_hashjoin;
+RESET enable_nestloop;
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 271313ebf86..9be7aca2b8a 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,6 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
+ enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -133,7 +134,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(22 rows)
+(23 rows)
-- There are always wait event descriptions for various types.
select type, count(*) > 0 as ok FROM pg_wait_events
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index cc8f0efad55..cf87b5d5ddd 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,6 +1181,74 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+
+-- GROUP BY optimization by reorder columns by frequency
+
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+
+DROP TABLE btg;
+
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+DROP TABLE t1 CASCADE;
+
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+RESET enable_hashjoin;
+RESET enable_nestloop;
+
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
+
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..e575fa13c63 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+PathKeyInfo
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-18 09:48 Andrei Lepikhov <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-01-18 09:48 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; Richard Guo <[email protected]>; +Cc: Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On 16/1/2024 22:05, Alexander Korotkov wrote:
> On Tue, Jan 16, 2024 at 4:48 AM Richard Guo <[email protected]> wrote:
>> * When trying to match the ordering of GROUP BY to that of ORDER BY in
>> get_useful_group_keys_orderings, this patch checks against the length of
>> path->pathkeys. This does not make sense. I guess this is a typo and
>> the real intention is to check against root->sort_pathkeys.
Thanks! It is really my blunder - fresh look works.
>>
>> --- a/src/backend/optimizer/path/pathkeys.c
>> +++ b/src/backend/optimizer/path/pathkeys.c
>> @@ -504,7 +504,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
>> root->num_groupby_pathkeys);
>>
>> if (n > 0 &&
>> - (enable_incremental_sort || n == list_length(path->pathkeys)))
>> + (enable_incremental_sort || n == list_length(root->sort_pathkeys)))
>>
>> However, I think this is also incorrect. When incremental sort is
>> disabled, it is reasonable to consider reordering the GROUP BY keys only
>> if the number of matching pathkeys is equal to the length of
>> root->group_pathkeys i.e. if 'n == list_length(root->group_pathkeys)'.
>
> Hmm... Why should this be list_length(root->group_pathkeys) while
> we're targeting root->sort_pathkeys. I yet changed that to
> list_length(root->sort_pathkeys).
I think, in the first case, when we are trying to arrange group-by keys
according to underlying pathkeys with incremental sort = off, it makes
sense to do if we fetch all group-by keys regardless of a more or equal
number of path keys the incoming path contains. The code and test case
are in step1.txt.
>
>> * When the original ordering of GROUP BY keys matches the path's
>> pathkeys or ORDER BY keys, get_useful_group_keys_orderings would
>> generate duplicate PathKeyInfos. Consequently, this duplication would
>> lead to the creation and addition of identical paths multiple times.
>> This is not great. Consider the query below:
>>
>> create table t (a int, b int);
>> create index on t (a, b);
>> set enable_hashagg to off;
>>
>> explain select count(*) from t group by a, b;
>> QUERY PLAN
>> ----------------------------------------------------------------------------------
>> GroupAggregate (cost=0.15..97.27 rows=226 width=16)
>> Group Key: a, b
>> -> Index Only Scan using t_a_b_idx on t (cost=0.15..78.06 rows=2260 width=8)
>> (3 rows)
>>
>> The same path with group keys {a, b} is created and added twice.
>
> I tried to address that.
>
>> * Part of the work performed in this patch overlaps with that of
>> preprocess_groupclause. They are both trying to adjust the ordering of
>> the GROUP BY keys to match ORDER BY. I wonder if it would be better to
>> perform this work only once.
>
> Andrei, could you take a look.
As I see, the PathKeyInfo list often contains duplicated pathkeys,
coming from either sort_pathkeys or path->pathkeys orderings. So, I can
propose to check duplicates each time (see step2.txt in attachment).
>
>> * When reordering the GROUP BY keys, I think the following checks need
>> some improvements.
>>
>> + /*
>> + * Since 1349d27 pathkey coming from underlying node can be in the
>> + * root->group_pathkeys but not in the processed_groupClause. So, we
>> + * should be careful here.
>> + */
>> + sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
>> + *group_clauses);
>> + if (!sgc)
>> + /* The grouping clause does not cover this pathkey */
>> + break;
>>
>> I think we need to check or assert 'pathkey->pk_eclass->ec_sortref' is
>> valid before calling get_sortgroupref_clause_noerr with it. Also, how
>> can we guarantee that the returned GROUP BY item is sortable? Should we
>> check that with OidIsValid(sgc->sortop)?
>
> Added.
Reviewed it, looks good.
--
regards,
Andrei Lepikhov
Postgres Professional
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 919d54dd79..1095b73dac 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -489,8 +489,9 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
n = group_keys_reorder_by_pathkeys(path->pathkeys, &pathkeys, &clauses,
root->num_groupby_pathkeys);
- if (n > 0 && compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL &&
- (enable_incremental_sort || n == list_length(path->pathkeys)))
+ if (n > 0 &&
+ (enable_incremental_sort || n == root->num_groupby_pathkeys) &&
+ compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
{
info = makeNode(PathKeyInfo);
info->pathkeys = pathkeys;
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index ca38e78f21..67dd20f375 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2856,6 +2856,23 @@ explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
-> Index Scan using abc on btg
(8 rows)
+SET enable_incremental_sort = off;
+-- The case when the number of incoming subtree path keys is more than
+-- the number of grouping keys.
+CREATE INDEX idx_y_x_z ON btg(y,x,w);
+EXPLAIN (VERBOSE, COSTS OFF)
+SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
+ QUERY PLAN
+-----------------------------------------------------
+ GroupAggregate
+ Output: y, x, array_agg(DISTINCT w)
+ Group Key: btg.y, btg.x
+ -> Index Only Scan using idx_y_x_z on public.btg
+ Output: y, x, w
+ Index Cond: (btg.y < 0)
+(6 rows)
+
+RESET enable_incremental_sort;
DROP TABLE btg;
-- The case, when scanning sort order correspond to aggregate sort order but
-- can not be found in the group-by list
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 9880a188d3..524bdfa67d 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1182,7 +1182,6 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
-- GROUP BY optimization by reorder columns
-
CREATE TABLE btg AS SELECT
i % 100 AS x,
i % 100 AS y,
@@ -1223,8 +1222,11 @@ explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
SET enable_incremental_sort = off;
-EXPLAIN (VERBOSE, COSTS ON)
-SELECT x,y, array_agg(z) FROM btg WHERE x < 2 GROUP BY y,x;
+-- The case when the number of incoming subtree path keys is more than
+-- the number of grouping keys.
+CREATE INDEX idx_y_x_z ON btg(y,x,w);
+EXPLAIN (VERBOSE, COSTS OFF)
+SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
RESET enable_incremental_sort;
DROP TABLE btg;
Attachments:
[text/plain] step1.txt (2.9K, ../../[email protected]/2-step1.txt)
download | inline diff:
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 919d54dd79..1095b73dac 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -489,8 +489,9 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
n = group_keys_reorder_by_pathkeys(path->pathkeys, &pathkeys, &clauses,
root->num_groupby_pathkeys);
- if (n > 0 && compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL &&
- (enable_incremental_sort || n == list_length(path->pathkeys)))
+ if (n > 0 &&
+ (enable_incremental_sort || n == root->num_groupby_pathkeys) &&
+ compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
{
info = makeNode(PathKeyInfo);
info->pathkeys = pathkeys;
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index ca38e78f21..67dd20f375 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2856,6 +2856,23 @@ explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
-> Index Scan using abc on btg
(8 rows)
+SET enable_incremental_sort = off;
+-- The case when the number of incoming subtree path keys is more than
+-- the number of grouping keys.
+CREATE INDEX idx_y_x_z ON btg(y,x,w);
+EXPLAIN (VERBOSE, COSTS OFF)
+SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
+ QUERY PLAN
+-----------------------------------------------------
+ GroupAggregate
+ Output: y, x, array_agg(DISTINCT w)
+ Group Key: btg.y, btg.x
+ -> Index Only Scan using idx_y_x_z on public.btg
+ Output: y, x, w
+ Index Cond: (btg.y < 0)
+(6 rows)
+
+RESET enable_incremental_sort;
DROP TABLE btg;
-- The case, when scanning sort order correspond to aggregate sort order but
-- can not be found in the group-by list
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 9880a188d3..524bdfa67d 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1182,7 +1182,6 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
-- GROUP BY optimization by reorder columns
-
CREATE TABLE btg AS SELECT
i % 100 AS x,
i % 100 AS y,
@@ -1223,8 +1222,11 @@ explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
SET enable_incremental_sort = off;
-EXPLAIN (VERBOSE, COSTS ON)
-SELECT x,y, array_agg(z) FROM btg WHERE x < 2 GROUP BY y,x;
+-- The case when the number of incoming subtree path keys is more than
+-- the number of grouping keys.
+CREATE INDEX idx_y_x_z ON btg(y,x,w);
+EXPLAIN (VERBOSE, COSTS OFF)
+SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
RESET enable_incremental_sort;
DROP TABLE btg;
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-18 14:18 Andrei Lepikhov <[email protected]>
parent: Andrei Lepikhov <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-01-18 14:18 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; Richard Guo <[email protected]>; +Cc: Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
Just forgotten second attachment.
--
regards,
Andrei Lepikhov
Postgres Professional
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1095b73dac..b612420547 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -432,6 +432,21 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
return n;
}
+static bool
+duplicated_pathkey_combination(List *infos, List *pathkeys)
+{
+ ListCell *lc;
+
+ foreach (lc, infos)
+ {
+ PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+
+ if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
+ return true;
+ }
+ return false;
+}
+
/*
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
@@ -491,7 +506,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
if (n > 0 &&
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
- compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
+ !duplicated_pathkey_combination(infos, pathkeys))
{
info = makeNode(PathKeyInfo);
info->pathkeys = pathkeys;
@@ -514,8 +529,9 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
&clauses,
root->num_groupby_pathkeys);
- if (n > 0 && compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL &&
- (enable_incremental_sort || n == list_length(root->sort_pathkeys)))
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
+ !duplicated_pathkey_combination(infos, pathkeys))
{
info = makeNode(PathKeyInfo);
info->pathkeys = pathkeys;
Attachments:
[text/plain] step2.txt (1.6K, ../../[email protected]/2-step2.txt)
download | inline diff:
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1095b73dac..b612420547 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -432,6 +432,21 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
return n;
}
+static bool
+duplicated_pathkey_combination(List *infos, List *pathkeys)
+{
+ ListCell *lc;
+
+ foreach (lc, infos)
+ {
+ PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+
+ if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
+ return true;
+ }
+ return false;
+}
+
/*
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
@@ -491,7 +506,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
if (n > 0 &&
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
- compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
+ !duplicated_pathkey_combination(infos, pathkeys))
{
info = makeNode(PathKeyInfo);
info->pathkeys = pathkeys;
@@ -514,8 +529,9 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
&clauses,
root->num_groupby_pathkeys);
- if (n > 0 && compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL &&
- (enable_incremental_sort || n == list_length(root->sort_pathkeys)))
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
+ !duplicated_pathkey_combination(infos, pathkeys))
{
info = makeNode(PathKeyInfo);
info->pathkeys = pathkeys;
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-19 14:21 Alexander Korotkov <[email protected]>
parent: Andrei Lepikhov <[email protected]>
1 sibling, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2024-01-19 14:21 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
Hi!
I've applied your changes with minor editing, thank you.
On Thu, Jan 18, 2024 at 11:49 AM Andrei Lepikhov
<[email protected]> wrote:
> >> * Part of the work performed in this patch overlaps with that of
> >> preprocess_groupclause. They are both trying to adjust the ordering of
> >> the GROUP BY keys to match ORDER BY. I wonder if it would be better to
> >> perform this work only once.
> >
> > Andrei, could you take a look.
> As I see, the PathKeyInfo list often contains duplicated pathkeys,
> coming from either sort_pathkeys or path->pathkeys orderings. So, I can
> propose to check duplicates each time (see step2.txt in attachment).
Actually I asked to recheck if we can cut some part of
preprocess_groupclause() given that we're reordering the pathkeys
later. It seems that we can remove everything except the work with a
grouping set. I've done this in the revised patchset.
I'm going to push this if there are no objections.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0002-Explore-alternative-orderings-of-group-by-p-20240119.patch (43.0K, ../../CAPpHfdtNnX9PYmgEsh9DXYws2Px6UJhH5kYUxsg3hu-zJTYuWw@mail.gmail.com/2-0002-Explore-alternative-orderings-of-group-by-p-20240119.patch)
download | inline diff:
From e9a6e72777c2e776b480f2d5e0cf8dcf49f273b8 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Fri, 19 Jan 2024 16:05:08 +0200
Subject: [PATCH 2/2] Explore alternative orderings of group-by pathkeys during
optimization.
When evaluating a query with a multi-column GROUP BY clause, we can minimize
sort operations or avoid them if we synchronize the order of GROUP BY clauses
with the ORDER BY sort clause or sort order, which comes from the underlying
query tree. Grouping does not imply any ordering, so we can compare
the keys in arbitrary order, and a Hash Agg leverages this. But for Group Agg,
we simply compared keys in the order specified in the query. This commit
explores alternative ordering of the keys, trying to find a cheaper one.
The ordering of group keys may interact with other parts of the query, some of
which may not be known while planning the grouping. For example, there may be
an explicit ORDER BY clause or some other ordering-dependent operation higher up
in the query, and using the same ordering may allow using either incremental
sort or even eliminating the sort entirely.
The patch always keeps the ordering specified in the query, assuming the user
might have additional insights.
This introduces a new GUC enable_group_by_reordering so that the optimization
may be disabled if needed.
Discussion: https://postgr.es/m/7c79e6a5-8597-74e8-0671-1c39d124c9d6%40sigaev.ru
Author: Andrey V. Lepikhov, Teodor Sigaev
Reviewed-by: Tomas Vondra, Claudio Freire, Gavin Flower, Dmitry Dolgov
Reviewed-by: Robert Haas, Pavel Borisov, David Rowley, Zhihong Yu
Reviewed-by: Tom Lane, Alexander Korotkov
---
src/backend/optimizer/path/equivclass.c | 13 +-
src/backend/optimizer/path/pathkeys.c | 252 +++++++++++
src/backend/optimizer/plan/planner.c | 424 +++++++++---------
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/pathnodes.h | 10 +
src/include/optimizer/paths.h | 2 +
src/test/regress/expected/aggregates.out | 202 +++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/aggregates.sql | 75 ++++
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 770 insertions(+), 223 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index e86dfeaecd4..4bd60a09c69 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,7 +652,18 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- return cur_ec; /* Match! */
+ {
+ /*
+ * Match!
+ *
+ * Copy the sortref if it wasn't set yet. That may happen if
+ * the ec was constructed from a WHERE clause, i.e. it doesn't
+ * have a target reference at all.
+ */
+ if (cur_ec->ec_sortref == 0 && sortref > 0)
+ cur_ec->ec_sortref = sortref;
+ return cur_ec;
+ }
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index ca94a31f71e..82ff31273b9 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -22,12 +22,15 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
+#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
+/* Consider reordering of GROUP BY keys? */
+bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -350,6 +353,202 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
+/*
+ * group_keys_reorder_by_pathkeys
+ * Reorder GROUP BY keys to match the input pathkeys.
+ *
+ * Function returns new lists (pathkeys and clauses), original GROUP BY lists
+ * stay untouched.
+ *
+ * Returns the number of GROUP BY keys with a matching pathkey.
+ */
+static int
+group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
+ List **group_clauses,
+ int num_groupby_pathkeys)
+{
+ List *new_group_pathkeys = NIL,
+ *new_group_clauses = NIL;
+ ListCell *lc;
+ int n;
+
+ if (pathkeys == NIL || *group_pathkeys == NIL)
+ return 0;
+
+ /*
+ * Walk the pathkeys (determining ordering of the input path) and see if
+ * there's a matching GROUP BY key. If we find one, we append it to the
+ * list, and do the same for the clauses.
+ *
+ * Once we find the first pathkey without a matching GROUP BY key, the
+ * rest of the pathkeys are useless and can't be used to evaluate the
+ * grouping, so we abort the loop and ignore the remaining pathkeys.
+ */
+ foreach(lc, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(lc);
+ SortGroupClause *sgc;
+
+ /*
+ * Pathkeys are built in a way that allows simply comparing pointers.
+ * Give up if we can't find the matching pointer. Also give up if
+ * there is no sortclause reference for some reason.
+ */
+ if (foreach_current_index(lc) >= num_groupby_pathkeys ||
+ !list_member_ptr(*group_pathkeys, pathkey) ||
+ pathkey->pk_eclass->ec_sortref == 0)
+ break;
+
+ /*
+ * Since 1349d27 pathkey coming from underlying node can be in the
+ * root->group_pathkeys but not in the processed_groupClause. So, we
+ * should be careful here.
+ */
+ sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
+ *group_clauses);
+ if (!sgc)
+ /* The grouping clause does not cover this pathkey */
+ break;
+
+ /*
+ * Sort group clause should have an ordering operator as long as there
+ * is an associated pathkey.
+ */
+ Assert(OidIsValid(sgc->sortop));
+
+ new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ }
+
+ /* remember the number of pathkeys with a matching GROUP BY key */
+ n = list_length(new_group_pathkeys);
+
+ /* append the remaining group pathkeys (will be treated as not sorted) */
+ *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
+ *group_pathkeys);
+ *group_clauses = list_concat_unique_ptr(new_group_clauses,
+ *group_clauses);
+
+ return n;
+}
+
+/*
+ * pathkeys_are_duplicate
+ * Check if give pathkeys are already contained the list of
+ * PathKeyInfo's.
+ */
+static bool
+pathkeys_are_duplicate(List *infos, List *pathkeys)
+{
+ ListCell *lc;
+
+ foreach(lc, infos)
+ {
+ PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+
+ if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
+ return true;
+ }
+ return false;
+}
+
+/*
+ * get_useful_group_keys_orderings
+ * Determine which orderings of GROUP BY keys are potentially interesting.
+ *
+ * Returns a list of PathKeyInfo items, each representing an interesting
+ * ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
+ * matching order.
+ *
+ * The function considers (and keeps) multiple GROUP BY orderings:
+ *
+ * - the original ordering, as specified by the GROUP BY clause,
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
+ * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ */
+List *
+get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
+{
+ Query *parse = root->parse;
+ List *infos = NIL;
+ PathKeyInfo *info;
+
+ List *pathkeys = root->group_pathkeys;
+ List *clauses = root->processed_groupClause;
+
+ /* always return at least the original pathkeys/clauses */
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+ infos = lappend(infos, info);
+
+ /*
+ * Should we try generating alternative orderings of the group keys? If
+ * not, we produce only the order specified in the query, i.e. the
+ * optimization is effectively disabled.
+ */
+ if (!enable_group_by_reordering)
+ return infos;
+
+ /*
+ * Grouping sets have own and more complex logic to decide the ordering.
+ */
+ if (parse->groupingSets)
+ return infos;
+
+ /*
+ * If the path is sorted in some way, try reordering the group keys to
+ * match the path as much of the ordering as possible. Then thanks to
+ * incremental sort we would get this sort as cheap as possible.
+ */
+ if (path->pathkeys &&
+ !pathkeys_contained_in(path->pathkeys, root->group_pathkeys))
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(path->pathkeys, &pathkeys, &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 &&
+ (enable_incremental_sort || n == root->num_groupby_pathkeys) &&
+ !pathkeys_are_duplicate(infos, pathkeys))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ /*
+ * Try reordering pathkeys to minimize the sort cost (this time consider
+ * the ORDER BY clause).
+ */
+ if (root->sort_pathkeys &&
+ !pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys))
+ {
+ int n;
+
+ n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
+ &clauses,
+ root->num_groupby_pathkeys);
+
+ if (n > 0 &&
+ (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
+ !pathkeys_are_duplicate(infos, pathkeys))
+ {
+ info = makeNode(PathKeyInfo);
+ info->pathkeys = pathkeys;
+ info->clauses = clauses;
+
+ infos = lappend(infos, info);
+ }
+ }
+
+ return infos;
+}
+
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -1939,6 +2138,54 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
+/*
+ * pathkeys_useful_for_grouping
+ * Count the number of pathkeys that are useful for grouping (instead of
+ * explicit sort)
+ *
+ * Group pathkeys could be reordered to benefit from the ordering. The
+ * ordering may not be "complete" and may require incremental sort, but that's
+ * fine. So we simply count prefix pathkeys with a matching group key, and
+ * stop once we find the first pathkey without a match.
+ *
+ * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
+ * pathkeys are useful for grouping, and we might do incremental sort to get
+ * path ordered by (a,b,e).
+ *
+ * This logic is necessary to retain paths with ordering not matching grouping
+ * keys directly, without the reordering.
+ *
+ * Returns the length of pathkey prefix with matching group keys.
+ */
+static int
+pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
+{
+ ListCell *key;
+ int n = 0;
+
+ /* no special ordering requested for grouping */
+ if (root->group_pathkeys == NIL)
+ return 0;
+
+ /* unordered path */
+ if (pathkeys == NIL)
+ return 0;
+
+ /* walk the pathkeys and search for matching group key */
+ foreach(key, pathkeys)
+ {
+ PathKey *pathkey = (PathKey *) lfirst(key);
+
+ /* no matching group key, we're done */
+ if (!list_member_ptr(root->group_pathkeys, pathkey))
+ break;
+
+ n++;
+ }
+
+ return n;
+}
+
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -1953,6 +2200,9 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
+ if (nuseful2 > nuseful)
+ nuseful = nuseful2;
+ nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -1988,6 +2238,8 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
+ if (root->group_pathkeys != NIL)
+ return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 014b179c3f0..2e2458b1284 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -140,7 +140,7 @@ static double preprocess_limit(PlannerInfo *root,
double tuple_fraction,
int64 *offset_est, int64 *count_est);
static void remove_useless_groupby_columns(PlannerInfo *root);
-static List *preprocess_groupclause(PlannerInfo *root, List *force);
+static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
static List *extract_rollup_sets(List *groupingSets);
static List *reorder_grouping_sets(List *groupingSets, List *sortclause);
static void standard_qp_callback(PlannerInfo *root, void *extra);
@@ -1423,7 +1423,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction)
else if (parse->groupClause)
{
/* Preprocess regular GROUP BY clause, if any */
- root->processed_groupClause = preprocess_groupclause(root, NIL);
+ root->processed_groupClause = list_copy(parse->groupClause);;
/* Remove any redundant GROUP BY columns */
remove_useless_groupby_columns(root);
}
@@ -2144,7 +2144,7 @@ preprocess_grouping_sets(PlannerInfo *root)
* The groupClauses for hashed grouping sets are built later on.)
*/
if (gs->set)
- rollup->groupClause = preprocess_groupclause(root, gs->set);
+ rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
else
rollup->groupClause = NIL;
@@ -2796,111 +2796,24 @@ remove_useless_groupby_columns(PlannerInfo *root)
}
/*
- * preprocess_groupclause - do preparatory work on GROUP BY clause
- *
- * The idea here is to adjust the ordering of the GROUP BY elements
- * (which in itself is semantically insignificant) to match ORDER BY,
- * thereby allowing a single sort operation to both implement the ORDER BY
- * requirement and set up for a Unique step that implements GROUP BY.
- *
- * In principle it might be interesting to consider other orderings of the
- * GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. We don't
- * bother with that, though. Hashed grouping will frequently win anyway.
- *
- * Note: we need no comparable processing of the distinctClause because
- * the parser already enforced that that matches ORDER BY.
- *
- * Note: we return a fresh List, but its elements are the same
- * SortGroupClauses appearing in parse->groupClause. This is important
- * because later processing may modify the processed_groupClause list.
- *
- * For grouping sets, the order of items is instead forced to agree with that
- * of the grouping set (and items not in the grouping set are skipped). The
- * work of sorting the order of grouping set elements to match the ORDER BY if
- * possible is done elsewhere.
+ * groupclause_apply_groupingset
+ * Apply the order of GROUP BY clauses defined by grouping sets. Items
+ * not in the grouping set are skipped.
*/
static List *
-preprocess_groupclause(PlannerInfo *root, List *force)
+groupclause_apply_groupingset(PlannerInfo *root, List *gset)
{
Query *parse = root->parse;
List *new_groupclause = NIL;
- bool partial_match;
ListCell *sl;
- ListCell *gl;
- /* For grouping sets, we need to force the ordering */
- if (force)
+ foreach(sl, gset)
{
- foreach(sl, force)
- {
- Index ref = lfirst_int(sl);
- SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+ Index ref = lfirst_int(sl);
+ SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
- new_groupclause = lappend(new_groupclause, cl);
- }
-
- return new_groupclause;
+ new_groupclause = lappend(new_groupclause, cl);
}
-
- /* If no ORDER BY, nothing useful to do here */
- if (parse->sortClause == NIL)
- return list_copy(parse->groupClause);
-
- /*
- * Scan the ORDER BY clause and construct a list of matching GROUP BY
- * items, but only as far as we can make a matching prefix.
- *
- * This code assumes that the sortClause contains no duplicate items.
- */
- foreach(sl, parse->sortClause)
- {
- SortGroupClause *sc = lfirst_node(SortGroupClause, sl);
-
- foreach(gl, parse->groupClause)
- {
- SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
-
- if (equal(gc, sc))
- {
- new_groupclause = lappend(new_groupclause, gc);
- break;
- }
- }
- if (gl == NULL)
- break; /* no match, so stop scanning */
- }
-
- /* Did we match all of the ORDER BY list, or just some of it? */
- partial_match = (sl != NULL);
-
- /* If no match at all, no point in reordering GROUP BY */
- if (new_groupclause == NIL)
- return list_copy(parse->groupClause);
-
- /*
- * Add any remaining GROUP BY items to the new list, but only if we were
- * able to make a complete match. In other words, we only rearrange the
- * GROUP BY list if the result is that one list is a prefix of the other
- * --- otherwise there's no possibility of a common sort. Also, give up
- * if there are any non-sortable GROUP BY items, since then there's no
- * hope anyway.
- */
- foreach(gl, parse->groupClause)
- {
- SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
-
- if (list_member_ptr(new_groupclause, gc))
- continue; /* it matched an ORDER BY item */
- if (partial_match) /* give up, no common sort possible */
- return list_copy(parse->groupClause);
- if (!OidIsValid(gc->sortop)) /* give up, GROUP BY can't be sorted */
- return list_copy(parse->groupClause);
- new_groupclause = lappend(new_groupclause, gc);
- }
-
- /* Success --- install the rearranged GROUP BY list */
- Assert(list_length(parse->groupClause) == list_length(new_groupclause));
return new_groupclause;
}
@@ -4200,7 +4113,7 @@ consider_groupingsets_paths(PlannerInfo *root,
{
rollup = makeNode(RollupData);
- rollup->groupClause = preprocess_groupclause(root, gset);
+ rollup->groupClause = groupclause_apply_groupingset(root, gset);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
@@ -4389,7 +4302,7 @@ consider_groupingsets_paths(PlannerInfo *root,
Assert(gs->set != NIL);
- rollup->groupClause = preprocess_groupclause(root, gs->set);
+ rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
@@ -6891,103 +6804,135 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- grouped_rel,
- path,
- cheapest_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY. Make an
- * AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- root->processed_groupClause,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (parse->groupClause)
+ foreach(lc2, pathkey_orderings)
{
- /*
- * We have GROUP BY without aggregation or grouping sets. Make
- * a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- root->processed_groupClause,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
- /*
- * Instead of operating directly on the input relation, we can
- * consider finalizing a partially aggregated path.
- */
- if (partially_grouped_rel != NULL)
- {
- foreach(lc, partially_grouped_rel->pathlist)
- {
- Path *path = (Path *) lfirst(lc);
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
path = make_ordered_path(root,
grouped_rel,
path,
- partially_grouped_rel->cheapest_total_path,
- root->group_pathkeys);
-
+ cheapest_path,
+ info->pathkeys);
if (path == NULL)
continue;
- if (parse->hasAggs)
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make
+ * an AggPath.
+ */
add_path(grouped_rel, (Path *)
create_agg_path(root,
grouped_rel,
path,
grouped_rel->reltarget,
parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- root->processed_groupClause,
+ AGGSPLIT_SIMPLE,
+ info->clauses,
havingQual,
- agg_final_costs,
+ agg_costs,
dNumGroups));
- else
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets.
+ * Make a GroupPath.
+ */
add_path(grouped_rel, (Path *)
create_group_path(root,
grouped_rel,
path,
- root->processed_groupClause,
+ info->clauses,
havingQual,
dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
+ }
+ }
+ /*
+ * Instead of operating directly on the input relation, we can
+ * consider finalizing a partially aggregated path.
+ */
+ if (partially_grouped_rel != NULL)
+ {
+ foreach(lc, partially_grouped_rel->pathlist)
+ {
+ ListCell *lc2;
+ Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
+
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
+
+ Assert(list_length(pathkey_orderings) > 0);
+
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ info->clauses,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ info->clauses,
+ havingQual,
+ dNumGroups));
+
+ }
}
}
}
@@ -7190,37 +7135,54 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_total_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
partially_grouped_rel,
path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialGroups));
+ cheapest_total_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialGroups));
+ }
}
}
@@ -7229,37 +7191,55 @@ create_partial_grouping_paths(PlannerInfo *root,
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
+ ListCell *lc2;
Path *path = (Path *) lfirst(lc);
+ Path *path_save = path;
+ List *pathkey_orderings = NIL;
- path = make_ordered_path(root,
- partially_grouped_rel,
- path,
- cheapest_partial_path,
- root->group_pathkeys);
+ /* generate alternative group orderings that might be useful */
+ pathkey_orderings = get_useful_group_keys_orderings(root, path);
- if (path == NULL)
- continue;
+ Assert(list_length(pathkey_orderings) > 0);
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- parse->groupClause ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- root->processed_groupClause,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- root->processed_groupClause,
- NIL,
- dNumPartialPartialGroups));
+ /* process all potentially interesting grouping reorderings */
+ foreach(lc2, pathkey_orderings)
+ {
+ PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+
+
+ /* restore the path (we replace it in the loop) */
+ path = path_save;
+
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ info->pathkeys);
+
+ if (path == NULL)
+ continue;
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ info->clauses,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ info->clauses,
+ NIL,
+ dNumPartialPartialGroups));
+ }
}
}
@@ -7373,6 +7353,8 @@ gather_grouping_paths(PlannerInfo *root, RelOptInfo *rel)
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted prefix
* of the list without having the list be fully sorted.
+ *
+ * XXX Shouldn't this also consider the group-key-reordering?
*/
if (!enable_incremental_sort || list_length(root->group_pathkeys) == 1)
return;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e53ebc6dc2b..7fe58518d7d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1050,6 +1050,16 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables reordering of GROUP BY keys."),
+ NULL,
+ GUC_EXPLAIN
+ },
+ &enable_group_by_reordering,
+ true,
+ NULL, NULL, NULL
+ },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 835b0e9ba89..da10b43dac3 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -399,6 +399,7 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
+#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index b9713ec9aa6..137da178dc7 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1456,6 +1456,16 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
+/*
+ * Combines the information about pathkeys and the associated clauses.
+ */
+typedef struct PathKeyInfo
+{
+ NodeTag type;
+ List *pathkeys;
+ List *clauses;
+} PathKeyInfo;
+
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index efd4abc28f3..0e8a9c94ba6 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,6 +24,7 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
+extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -204,6 +205,7 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
+extern List *get_useful_group_keys_orderings(PlannerInfo *root, Path *path);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index f635c5a1afb..67dd20f3751 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,6 +2728,208 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+-- GROUP BY optimization by reorder columns by frequency
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+ QUERY PLAN
+----------------------------------------
+ GroupAggregate
+ Group Key: x, y
+ -> Index Only Scan using abc on btg
+(3 rows)
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, z, w
+ -> Incremental Sort
+ Sort Key: x, y, z, w
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+ QUERY PLAN
+-----------------------------------------
+ Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(6 rows)
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(6 rows)
+
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+ QUERY PLAN
+----------------------------------------------------
+ Group
+ Group Key: btg.x, btg.y, btg.w, btg.z
+ -> Limit
+ -> Incremental Sort
+ Sort Key: btg.x, btg.y, btg.w, btg.z
+ Presorted Key: btg.x, btg.y
+ -> Index Scan using abc on btg
+(7 rows)
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+ QUERY PLAN
+------------------------------
+ Group
+ Group Key: y, x, z, w
+ -> Sort
+ Sort Key: y, x, z, w
+ -> Seq Scan on btg
+(5 rows)
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+ QUERY PLAN
+-----------------------------------------------
+ Sort
+ Sort Key: ((x * x)), z
+ -> Group
+ Group Key: x, y, w, z
+ -> Incremental Sort
+ Sort Key: x, y, w, z
+ Presorted Key: x, y
+ -> Index Scan using abc on btg
+(8 rows)
+
+SET enable_incremental_sort = off;
+-- The case when the number of incoming subtree path keys is more than
+-- the number of grouping keys.
+CREATE INDEX idx_y_x_z ON btg(y,x,w);
+EXPLAIN (VERBOSE, COSTS OFF)
+SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
+ QUERY PLAN
+-----------------------------------------------------
+ GroupAggregate
+ Output: y, x, array_agg(DISTINCT w)
+ Group Key: btg.y, btg.x
+ -> Index Only Scan using idx_y_x_z on public.btg
+ Output: y, x, w
+ Index Cond: (btg.y < 0)
+(6 rows)
+
+RESET enable_incremental_sort;
+DROP TABLE btg;
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+ QUERY PLAN
+--------------------------------------------------------
+ Sort
+ Sort Key: c2
+ -> GroupAggregate
+ Group Key: c1
+ -> Sort
+ Sort Key: c1, c2
+ -> Bitmap Heap Scan on t1
+ Recheck Cond: (c2 < 100)
+ -> Bitmap Index Scan on t1_c2_idx
+ Index Cond: (c2 < 100)
+(10 rows)
+
+DROP TABLE t1 CASCADE;
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------
+ Incremental Sort
+ Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
+ Presorted Key: c1.relpages, c1.relname
+ -> Group
+ Group Key: c1.relpages, c1.relname, c1.reltuples
+ -> Incremental Sort
+ Sort Key: c1.relpages, c1.relname, c1.reltuples
+ Presorted Key: c1.relpages, c1.relname
+ -> Merge Join
+ Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ -> Sort
+ Sort Key: c1.relpages, c1.relname
+ -> Seq Scan on pg_class c1
+ -> Sort
+ Sort Key: c2.relpages, c2.relname
+ -> Seq Scan on pg_class c2
+(16 rows)
+
+RESET enable_hashjoin;
+RESET enable_nestloop;
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 271313ebf86..9be7aca2b8a 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,6 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
+ enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -133,7 +134,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(22 rows)
+(23 rows)
-- There are always wait event descriptions for various types.
select type, count(*) > 0 as ok FROM pg_wait_events
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index cc8f0efad55..524bdfa67d6 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,6 +1181,81 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+
+-- GROUP BY optimization by reorder columns by frequency
+
+SET enable_hashagg=off;
+SET max_parallel_workers= 0;
+SET max_parallel_workers_per_gather = 0;
+
+-- Utilize index scan ordering to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
+EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+
+-- Engage incremental sort
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+
+-- Subqueries
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
+GROUP BY (w,x,z,y);
+explain (COSTS OFF) SELECT x,y
+FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
+GROUP BY (w,x,z,y);
+
+-- Should work with and without GROUP-BY optimization
+explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+
+-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
+explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+
+SET enable_incremental_sort = off;
+-- The case when the number of incoming subtree path keys is more than
+-- the number of grouping keys.
+CREATE INDEX idx_y_x_z ON btg(y,x,w);
+EXPLAIN (VERBOSE, COSTS OFF)
+SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
+RESET enable_incremental_sort;
+
+DROP TABLE btg;
+
+-- The case, when scanning sort order correspond to aggregate sort order but
+-- can not be found in the group-by list
+CREATE TABLE t1 (c1 int PRIMARY KEY, c2 int);
+CREATE UNIQUE INDEX ON t1(c2);
+explain (costs off)
+SELECT array_agg(c1 ORDER BY c2),c2
+FROM t1 WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+DROP TABLE t1 CASCADE;
+
+-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
+-- by planner itself. For example, by MergeJoin.
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+explain (COSTS OFF)
+SELECT c1.relname,c1.relpages
+FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
+GROUP BY c1.reltuples,c1.relpages,c1.relname
+ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+RESET enable_hashjoin;
+RESET enable_nestloop;
+
+RESET enable_hashagg;
+RESET max_parallel_workers;
+RESET max_parallel_workers_per_gather;
+
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 16421f034cb..97750ddcf8f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4042,3 +4042,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+PathKeyInfo
--
2.39.3 (Apple Git-145)
[application/octet-stream] 0001-Generalize-common-code-of-adding-sort-befor-20240119.patch (9.1K, ../../CAPpHfdtNnX9PYmgEsh9DXYws2Px6UJhH5kYUxsg3hu-zJTYuWw@mail.gmail.com/3-0001-Generalize-common-code-of-adding-sort-befor-20240119.patch)
download | inline diff:
From c76a5828ff4925d03ce071658cc4e413e9a3aaf3 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 11 Jan 2024 13:27:58 +0200
Subject: [PATCH 1/2] Generalize common code of adding sort before generation
of grouping paths
Extract the repetitive code pattern into a new function make_ordered_path().
Discussion: https://postgr.es/m/CAPpHfdtzaVa7S4onKy3YvttF2rrH5hQNHx9HtcSTLbpjx%2BMJ%2Bw%40mail.gmail.com
Author: Andrei Lepikhov
---
src/backend/optimizer/plan/planner.c | 228 ++++++++++-----------------
1 file changed, 80 insertions(+), 148 deletions(-)
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 667723b6753..014b179c3f0 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6809,6 +6809,58 @@ done:
return parallel_workers;
}
+/*
+ * make_ordered_path
+ * Return a path ordered by 'pathkeys' based on the given 'path'. May
+ * return NULL if it doesn't make sense to generate an ordered path in
+ * this case.
+ */
+static Path *
+make_ordered_path(PlannerInfo *root, RelOptInfo *rel, Path *path,
+ Path *cheapest_path, List *pathkeys)
+{
+ bool is_sorted;
+ int presorted_keys;
+
+ is_sorted = pathkeys_count_contained_in(pathkeys,
+ path->pathkeys,
+ &presorted_keys);
+
+ if (!is_sorted)
+ {
+ /*
+ * Try at least sorting the cheapest path and also try incrementally
+ * sorting any path which is partially sorted already (no need to deal
+ * with paths which have presorted keys when incremental sort is
+ * disabled unless it's the cheapest input path).
+ */
+ if (path != cheapest_path &&
+ (presorted_keys == 0 || !enable_incremental_sort))
+ return NULL;
+
+ /*
+ * We've no need to consider both a sort and incremental sort. We'll
+ * just do a sort if there are no presorted keys and an incremental
+ * sort when there are presorted keys.
+ */
+ if (presorted_keys == 0 || !enable_incremental_sort)
+ path = (Path *) create_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ -1.0);
+ else
+ path = (Path *) create_incremental_sort_path(root,
+ rel,
+ path,
+ pathkeys,
+ presorted_keys,
+ -1.0);
+ }
+
+ return path;
+}
+
/*
* add_paths_to_grouping_rel
*
@@ -6840,45 +6892,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ cheapest_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6935,46 +6957,15 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc, partially_grouped_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially
- * sorted already (no need to deal with paths which have
- * presorted keys when incremental sort is disabled unless
- * it's the cheapest input path).
- */
- if (path != partially_grouped_rel->cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ grouped_rel,
+ path,
+ partially_grouped_rel->cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental
- * sort. We'll just do a sort if there are no pre-sorted
- * keys and an incremental sort when there are presorted
- * keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(grouped_rel, (Path *)
@@ -7200,44 +7191,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_total_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_total_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_path(partially_grouped_rel, (Path *)
@@ -7268,45 +7230,15 @@ create_partial_grouping_paths(PlannerInfo *root,
foreach(lc, input_rel->partial_pathlist)
{
Path *path = (Path *) lfirst(lc);
- bool is_sorted;
- int presorted_keys;
-
- is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
- path->pathkeys,
- &presorted_keys);
- if (!is_sorted)
- {
- /*
- * Try at least sorting the cheapest path and also try
- * incrementally sorting any path which is partially sorted
- * already (no need to deal with paths which have presorted
- * keys when incremental sort is disabled unless it's the
- * cheapest input path).
- */
- if (path != cheapest_partial_path &&
- (presorted_keys == 0 || !enable_incremental_sort))
- continue;
+ path = make_ordered_path(root,
+ partially_grouped_rel,
+ path,
+ cheapest_partial_path,
+ root->group_pathkeys);
- /*
- * We've no need to consider both a sort and incremental sort.
- * We'll just do a sort if there are no presorted keys and an
- * incremental sort when there are presorted keys.
- */
- if (presorted_keys == 0 || !enable_incremental_sort)
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- -1.0);
- else
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- root->group_pathkeys,
- presorted_keys,
- -1.0);
- }
+ if (path == NULL)
+ continue;
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-24 17:37 Nathan Bossart <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 67+ messages in thread
From: Nathan Bossart @ 2024-01-24 17:37 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
A recent buildfarm failure [0] seems to indicate a name collision with the
"abc" index in the aggregates.sql test and the "abc" table in
namespace.sql.
[0] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=piculet&dt=2024-01-24%2014%3A05%3A14
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-24 19:44 Alexander Korotkov <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-01-24 19:44 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Wed, Jan 24, 2024 at 7:38 PM Nathan Bossart <[email protected]> wrote:
> A recent buildfarm failure [0] seems to indicate a name collision with the
> "abc" index in the aggregates.sql test and the "abc" table in
> namespace.sql.
>
> [0] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=piculet&dt=2024-01-24%2014%3A05%3A14
Thank you for catching this. Fixed.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-01-26 14:37 vignesh C <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: vignesh C @ 2024-01-26 14:37 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andrei Lepikhov <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>; Tom Lane <[email protected]>
On Thu, 25 Jan 2024 at 01:15, Alexander Korotkov <[email protected]> wrote:
>
> On Wed, Jan 24, 2024 at 7:38 PM Nathan Bossart <[email protected]> wrote:
> > A recent buildfarm failure [0] seems to indicate a name collision with the
> > "abc" index in the aggregates.sql test and the "abc" table in
> > namespace.sql.
> >
> > [0] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=piculet&dt=2024-01-24%2014%3A05%3A14
>
> Thank you for catching this. Fixed.
Since the patch has been committed, I have marked this entry as committed.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-02 02:02 Tom Lane <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 2 replies; 67+ messages in thread
From: Tom Lane @ 2024-02-02 02:02 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Alexander Korotkov <[email protected]> writes:
> I'm going to push this if there are no objections.
One of the test cases added by this commit has not been very
stable in the buildfarm. Latest example is here:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04
and I've seen similar failures intermittently on other machines.
I'd suggest building this test atop a table that is more stable
than pg_class. You're just waving a red flag in front of a bull
if you expect stable statistics from that during a regression run.
Nor do I see any particular reason for pg_class to be especially
suited to the test.
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-02 03:32 Richard Guo <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 67+ messages in thread
From: Richard Guo @ 2024-02-02 03:32 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Fri, Feb 2, 2024 at 10:02 AM Tom Lane <[email protected]> wrote:
> Alexander Korotkov <[email protected]> writes:
> > I'm going to push this if there are no objections.
>
> One of the test cases added by this commit has not been very
> stable in the buildfarm. Latest example is here:
>
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04
>
> and I've seen similar failures intermittently on other machines.
>
> I'd suggest building this test atop a table that is more stable
> than pg_class. You're just waving a red flag in front of a bull
> if you expect stable statistics from that during a regression run.
> Nor do I see any particular reason for pg_class to be especially
> suited to the test.
Yeah, it's not a good practice to use pg_class in this place. While
looking through the test cases added by this commit, I noticed some
other minor issues that are not great. Such as
* The table 'btg' is inserted with 10000 tuples, which seems a bit
expensive for a test. I don't think we need such a big table to test
what we want.
* I don't see why we need to manipulate GUC max_parallel_workers and
max_parallel_workers_per_gather.
* I think we'd better write the tests with the keywords being all upper
or all lower. A mixed use of upper and lower is not great. Such as in
explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
* Some comments for the test queries are not easy to read.
* For this statement
CREATE INDEX idx_y_x_z ON btg(y,x,w);
I think the index name would cause confusion. It creates an index on
columns y, x and w, but the name indicates an index on y, x and z.
I'd like to write a draft patch for the fixes.
Thanks
Richard
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-02 03:42 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-02-02 03:42 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 2/2/2024 09:02, Tom Lane wrote:
> Alexander Korotkov <[email protected]> writes:
>> I'm going to push this if there are no objections.
>
> One of the test cases added by this commit has not been very
> stable in the buildfarm. Latest example is here:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04
>
> and I've seen similar failures intermittently on other machines.
>
> I'd suggest building this test atop a table that is more stable
> than pg_class. You're just waving a red flag in front of a bull
> if you expect stable statistics from that during a regression run.
> Nor do I see any particular reason for pg_class to be especially
> suited to the test.
Yeah, It is my fault. Please, see in the attachment the patch fixing that.
--
regards,
Andrei Lepikhov
Postgres Professional
From 11a049d95ee48e38ad569aab7663d8de91f946ad Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email protected]>
Date: Fri, 2 Feb 2024 10:39:55 +0700
Subject: [PATCH] Replace the GROUP-BY optimization test with the same based on
something less volatile when the pg_class relation.
---
src/test/regress/expected/aggregates.out | 32 +++++++++++-------------
src/test/regress/sql/aggregates.sql | 9 +++----
2 files changed, 18 insertions(+), 23 deletions(-)
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 7a73c19314..c2e1b8c9ed 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2873,7 +2873,6 @@ SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
(6 rows)
RESET enable_incremental_sort;
-DROP TABLE btg;
-- The case, when scanning sort order correspond to aggregate sort order but
-- can not be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
@@ -2901,32 +2900,31 @@ DROP TABLE agg_sort_order CASCADE;
SET enable_hashjoin = off;
SET enable_nestloop = off;
explain (COSTS OFF)
-SELECT c1.relname,c1.relpages
-FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
-GROUP BY c1.reltuples,c1.relpages,c1.relname
-ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
- QUERY PLAN
----------------------------------------------------------------------------------------------
+SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
+GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
+ QUERY PLAN
+-------------------------------------------------------------------
Incremental Sort
- Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
- Presorted Key: c1.relpages, c1.relname
+ Sort Key: b1.z, b1.w, ((b1.x * b1.x))
+ Presorted Key: b1.z, b1.w
-> Group
- Group Key: c1.relpages, c1.relname, c1.reltuples
+ Group Key: b1.z, b1.w, b1.x
-> Incremental Sort
- Sort Key: c1.relpages, c1.relname, c1.reltuples
- Presorted Key: c1.relpages, c1.relname
+ Sort Key: b1.z, b1.w, b1.x
+ Presorted Key: b1.z, b1.w
-> Merge Join
- Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ Merge Cond: ((b1.z = b2.z) AND (b1.w = b2.w))
-> Sort
- Sort Key: c1.relpages, c1.relname
- -> Seq Scan on pg_class c1
+ Sort Key: b1.z, b1.w
+ -> Seq Scan on btg b1
-> Sort
- Sort Key: c2.relpages, c2.relname
- -> Seq Scan on pg_class c2
+ Sort Key: b2.z, b2.w
+ -> Seq Scan on btg b2
(16 rows)
RESET enable_hashjoin;
RESET enable_nestloop;
+DROP TABLE btg;
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 916dbf908f..3548fbb8db 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1229,8 +1229,6 @@ EXPLAIN (VERBOSE, COSTS OFF)
SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
RESET enable_incremental_sort;
-DROP TABLE btg;
-
-- The case, when scanning sort order correspond to aggregate sort order but
-- can not be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
@@ -1245,13 +1243,12 @@ DROP TABLE agg_sort_order CASCADE;
SET enable_hashjoin = off;
SET enable_nestloop = off;
explain (COSTS OFF)
-SELECT c1.relname,c1.relpages
-FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
-GROUP BY c1.reltuples,c1.relpages,c1.relname
-ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
+GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
RESET enable_hashjoin;
RESET enable_nestloop;
+DROP TABLE btg;
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
--
2.43.0
Attachments:
[text/plain] 0001-Replace-the-GROUP-BY-optimization-test-with-the-same.patch (4.4K, ../../[email protected]/2-0001-Replace-the-GROUP-BY-optimization-test-with-the-same.patch)
download | inline diff:
From 11a049d95ee48e38ad569aab7663d8de91f946ad Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <[email protected]>
Date: Fri, 2 Feb 2024 10:39:55 +0700
Subject: [PATCH] Replace the GROUP-BY optimization test with the same based on
something less volatile when the pg_class relation.
---
src/test/regress/expected/aggregates.out | 32 +++++++++++-------------
src/test/regress/sql/aggregates.sql | 9 +++----
2 files changed, 18 insertions(+), 23 deletions(-)
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 7a73c19314..c2e1b8c9ed 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2873,7 +2873,6 @@ SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
(6 rows)
RESET enable_incremental_sort;
-DROP TABLE btg;
-- The case, when scanning sort order correspond to aggregate sort order but
-- can not be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
@@ -2901,32 +2900,31 @@ DROP TABLE agg_sort_order CASCADE;
SET enable_hashjoin = off;
SET enable_nestloop = off;
explain (COSTS OFF)
-SELECT c1.relname,c1.relpages
-FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
-GROUP BY c1.reltuples,c1.relpages,c1.relname
-ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
- QUERY PLAN
----------------------------------------------------------------------------------------------
+SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
+GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
+ QUERY PLAN
+-------------------------------------------------------------------
Incremental Sort
- Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
- Presorted Key: c1.relpages, c1.relname
+ Sort Key: b1.z, b1.w, ((b1.x * b1.x))
+ Presorted Key: b1.z, b1.w
-> Group
- Group Key: c1.relpages, c1.relname, c1.reltuples
+ Group Key: b1.z, b1.w, b1.x
-> Incremental Sort
- Sort Key: c1.relpages, c1.relname, c1.reltuples
- Presorted Key: c1.relpages, c1.relname
+ Sort Key: b1.z, b1.w, b1.x
+ Presorted Key: b1.z, b1.w
-> Merge Join
- Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
+ Merge Cond: ((b1.z = b2.z) AND (b1.w = b2.w))
-> Sort
- Sort Key: c1.relpages, c1.relname
- -> Seq Scan on pg_class c1
+ Sort Key: b1.z, b1.w
+ -> Seq Scan on btg b1
-> Sort
- Sort Key: c2.relpages, c2.relname
- -> Seq Scan on pg_class c2
+ Sort Key: b2.z, b2.w
+ -> Seq Scan on btg b2
(16 rows)
RESET enable_hashjoin;
RESET enable_nestloop;
+DROP TABLE btg;
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 916dbf908f..3548fbb8db 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1229,8 +1229,6 @@ EXPLAIN (VERBOSE, COSTS OFF)
SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
RESET enable_incremental_sort;
-DROP TABLE btg;
-
-- The case, when scanning sort order correspond to aggregate sort order but
-- can not be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
@@ -1245,13 +1243,12 @@ DROP TABLE agg_sort_order CASCADE;
SET enable_hashjoin = off;
SET enable_nestloop = off;
explain (COSTS OFF)
-SELECT c1.relname,c1.relpages
-FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
-GROUP BY c1.reltuples,c1.relpages,c1.relname
-ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
+SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
+GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
RESET enable_hashjoin;
RESET enable_nestloop;
+DROP TABLE btg;
RESET enable_hashagg;
RESET max_parallel_workers;
RESET max_parallel_workers_per_gather;
--
2.43.0
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-02 04:06 Richard Guo <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Richard Guo @ 2024-02-02 04:06 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Fri, Feb 2, 2024 at 11:32 AM Richard Guo <[email protected]> wrote:
> On Fri, Feb 2, 2024 at 10:02 AM Tom Lane <[email protected]> wrote:
>
>> One of the test cases added by this commit has not been very
>> stable in the buildfarm. Latest example is here:
>>
>>
>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04
>>
>> and I've seen similar failures intermittently on other machines.
>>
>> I'd suggest building this test atop a table that is more stable
>> than pg_class. You're just waving a red flag in front of a bull
>> if you expect stable statistics from that during a regression run.
>> Nor do I see any particular reason for pg_class to be especially
>> suited to the test.
>
>
> Yeah, it's not a good practice to use pg_class in this place. While
> looking through the test cases added by this commit, I noticed some
> other minor issues that are not great. Such as
>
> * The table 'btg' is inserted with 10000 tuples, which seems a bit
> expensive for a test. I don't think we need such a big table to test
> what we want.
>
> * I don't see why we need to manipulate GUC max_parallel_workers and
> max_parallel_workers_per_gather.
>
> * I think we'd better write the tests with the keywords being all upper
> or all lower. A mixed use of upper and lower is not great. Such as in
>
> explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
>
> * Some comments for the test queries are not easy to read.
>
> * For this statement
>
> CREATE INDEX idx_y_x_z ON btg(y,x,w);
>
> I think the index name would cause confusion. It creates an index on
> columns y, x and w, but the name indicates an index on y, x and z.
>
> I'd like to write a draft patch for the fixes.
>
Here is the draft patch that fixes the issues I complained about in
upthread.
Thanks
Richard
Attachments:
[application/octet-stream] v1-0001-Multiple-revises-for-the-GROUP-BY-reordering-tests.patch (15.6K, ../../CAMbWs4-NKLa+Ss+X=WR6h0x=T07YBJoAz70ZGHzc-2zcHUHb0A@mail.gmail.com/3-v1-0001-Multiple-revises-for-the-GROUP-BY-reordering-tests.patch)
download | inline diff:
From f14c3ae49042608aa43f01f7b95d738e793b4418 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Fri, 2 Feb 2024 12:01:10 +0800
Subject: [PATCH v1] Multiple revises for the GROUP BY reordering tests
---
src/test/regress/expected/aggregates.out | 200 ++++++++---------------
src/test/regress/sql/aggregates.sql | 92 +++++------
2 files changed, 113 insertions(+), 179 deletions(-)
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 7a73c19314..9fb68fc280 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,29 +2728,20 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
--- GROUP BY optimization by reorder columns
+-- GROUP BY optimization by reordering GROUP BY clauses
CREATE TABLE btg AS SELECT
- i % 100 AS x,
- i % 100 AS y,
+ i % 10 AS x,
+ i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
-CREATE INDEX btg_x_y_idx ON btg(x,y);
+FROM generate_series(1,1000) AS i;
+CREATE INDEX btg_x_y_idx ON btg(x, y);
ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
--- Utilize index scan ordering to avoid a Sort operation
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: x, y
- -> Index Only Scan using btg_x_y_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+SET enable_seqscan=off;
+-- Utilize the ordering of index scan to avoid a Sort operation
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY y, x;
QUERY PLAN
------------------------------------------------
GroupAggregate
@@ -2759,21 +2750,11 @@ EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
(3 rows)
-- Engage incremental sort
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, z, w
- -> Incremental Sort
- Sort Key: x, y, z, w
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY z, y, w, x;
QUERY PLAN
-------------------------------------------------
- Group
+ GroupAggregate
Group Key: x, y, z, w
-> Incremental Sort
Sort Key: x, y, z, w
@@ -2781,35 +2762,13 @@ explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
-> Index Scan using btg_x_y_idx on btg
(6 rows)
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, w, z
- -> Incremental Sort
- Sort Key: x, y, w, z
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, w, z
- -> Incremental Sort
- Sort Key: x, y, w, z
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
--- Subqueries
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
-GROUP BY (w,x,z,y);
+-- Utilize the ordering of subquery scan to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*)
+FROM (SELECT * FROM btg ORDER BY x, y, w, z) AS q1
+GROUP BY w, x, z, y;
QUERY PLAN
-------------------------------------------------
- Group
+ GroupAggregate
Group Key: btg.x, btg.y, btg.w, btg.z
-> Incremental Sort
Sort Key: btg.x, btg.y, btg.w, btg.z
@@ -2817,38 +2776,52 @@ GROUP BY (w,x,z,y);
-> Index Scan using btg_x_y_idx on btg
(6 rows)
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
-GROUP BY (w,x,z,y);
- QUERY PLAN
--------------------------------------------------------
- Group
- Group Key: btg.x, btg.y, btg.w, btg.z
- -> Limit
- -> Incremental Sort
- Sort Key: btg.x, btg.y, btg.w, btg.z
- Presorted Key: btg.x, btg.y
- -> Index Scan using btg_x_y_idx on btg
-(7 rows)
+-- Utilize the ordering of merge join to avoid a full-Sort operation
+SET enable_hashjoin=off;
+SET enable_nestloop=off;
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM btg t1 JOIN btg t2 ON t1.z = t2.z AND t1.w = t2.w AND t1.x = t2.x
+ GROUP BY t1.x, t1.y, t1.z, t1.w;
+ QUERY PLAN
+-------------------------------------------------------------------------------
+ GroupAggregate
+ Group Key: t1.z, t1.w, t1.x, t1.y
+ -> Incremental Sort
+ Sort Key: t1.z, t1.w, t1.x, t1.y
+ Presorted Key: t1.z, t1.w, t1.x
+ -> Merge Join
+ Merge Cond: ((t1.z = t2.z) AND (t1.w = t2.w) AND (t1.x = t2.x))
+ -> Sort
+ Sort Key: t1.z, t1.w, t1.x
+ -> Index Scan using btg_x_y_idx on btg t1
+ -> Sort
+ Sort Key: t2.z, t2.w, t2.x
+ -> Index Scan using btg_x_y_idx on btg t2
+(13 rows)
+RESET enable_nestloop;
+RESET enable_hashjoin;
-- Should work with and without GROUP-BY optimization
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
- QUERY PLAN
-------------------------------
- Group
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER BY y, x, z, w;
+ QUERY PLAN
+-------------------------------------------------
+ GroupAggregate
Group Key: y, x, z, w
-> Sort
Sort Key: y, x, z, w
- -> Seq Scan on btg
+ -> Index Scan using btg_x_y_idx on btg
(5 rows)
-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
-explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER BY x*x, z;
QUERY PLAN
-------------------------------------------------------
Sort
Sort Key: ((x * x)), z
- -> Group
+ -> GroupAggregate
Group Key: x, y, w, z
-> Incremental Sort
Sort Key: x, y, w, z
@@ -2856,80 +2829,47 @@ explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
-> Index Scan using btg_x_y_idx on btg
(8 rows)
-SET enable_incremental_sort = off;
--- The case when the number of incoming subtree path keys is more than
+-- Test the case where the number of incoming subtree path keys is more than
-- the number of grouping keys.
-CREATE INDEX idx_y_x_z ON btg(y,x,w);
+CREATE INDEX btg_y_x_w_idx ON btg(y, x, w);
EXPLAIN (VERBOSE, COSTS OFF)
-SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
- QUERY PLAN
------------------------------------------------------
+SELECT y, x, array_agg(distinct w)
+ FROM btg WHERE y < 0 GROUP BY x, y;
+ QUERY PLAN
+---------------------------------------------------------
GroupAggregate
Output: y, x, array_agg(DISTINCT w)
Group Key: btg.y, btg.x
- -> Index Only Scan using idx_y_x_z on public.btg
+ -> Index Only Scan using btg_y_x_w_idx on public.btg
Output: y, x, w
Index Cond: (btg.y < 0)
(6 rows)
-RESET enable_incremental_sort;
DROP TABLE btg;
--- The case, when scanning sort order correspond to aggregate sort order but
--- can not be found in the group-by list
+-- Test the case where the the ordering of scan matches the ordering within the
+-- aggregate but cannot be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
-CREATE UNIQUE INDEX ON agg_sort_order(c2);
-explain (costs off)
+CREATE UNIQUE INDEX agg_sort_order_c2_idx ON agg_sort_order(c2);
+INSERT INTO agg_sort_order SELECT i, i FROM generate_series(1,100)i;
+ANALYZE agg_sort_order;
+EXPLAIN (COSTS OFF)
SELECT array_agg(c1 ORDER BY c2),c2
FROM agg_sort_order WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
- QUERY PLAN
---------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------------------------------------
Sort
Sort Key: c2
-> GroupAggregate
Group Key: c1
-> Sort
Sort Key: c1, c2
- -> Bitmap Heap Scan on agg_sort_order
- Recheck Cond: (c2 < 100)
- -> Bitmap Index Scan on agg_sort_order_c2_idx
- Index Cond: (c2 < 100)
-(10 rows)
+ -> Index Scan using agg_sort_order_c2_idx on agg_sort_order
+ Index Cond: (c2 < 100)
+(8 rows)
DROP TABLE agg_sort_order CASCADE;
--- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
--- by planner itself. For example, by MergeJoin.
-SET enable_hashjoin = off;
-SET enable_nestloop = off;
-explain (COSTS OFF)
-SELECT c1.relname,c1.relpages
-FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
-GROUP BY c1.reltuples,c1.relpages,c1.relname
-ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
- QUERY PLAN
----------------------------------------------------------------------------------------------
- Incremental Sort
- Sort Key: c1.relpages, c1.relname, ((c1.relpages * c1.relpages))
- Presorted Key: c1.relpages, c1.relname
- -> Group
- Group Key: c1.relpages, c1.relname, c1.reltuples
- -> Incremental Sort
- Sort Key: c1.relpages, c1.relname, c1.reltuples
- Presorted Key: c1.relpages, c1.relname
- -> Merge Join
- Merge Cond: ((c1.relpages = c2.relpages) AND (c1.relname = c2.relname))
- -> Sort
- Sort Key: c1.relpages, c1.relname
- -> Seq Scan on pg_class c1
- -> Sort
- Sort Key: c2.relpages, c2.relname
- -> Seq Scan on pg_class c2
-(16 rows)
-
-RESET enable_hashjoin;
-RESET enable_nestloop;
RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
+RESET enable_seqscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 916dbf908f..77cddad4fc 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,80 +1181,74 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
--- GROUP BY optimization by reorder columns
+-- GROUP BY optimization by reordering GROUP BY clauses
CREATE TABLE btg AS SELECT
- i % 100 AS x,
- i % 100 AS y,
+ i % 10 AS x,
+ i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
-CREATE INDEX btg_x_y_idx ON btg(x,y);
+FROM generate_series(1,1000) AS i;
+CREATE INDEX btg_x_y_idx ON btg(x, y);
ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-
SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
+SET enable_seqscan=off;
--- Utilize index scan ordering to avoid a Sort operation
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+-- Utilize the ordering of index scan to avoid a Sort operation
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY y, x;
-- Engage incremental sort
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
-
--- Subqueries
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
-GROUP BY (w,x,z,y);
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
-GROUP BY (w,x,z,y);
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY z, y, w, x;
+
+-- Utilize the ordering of subquery scan to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*)
+FROM (SELECT * FROM btg ORDER BY x, y, w, z) AS q1
+GROUP BY w, x, z, y;
+
+-- Utilize the ordering of merge join to avoid a full-Sort operation
+SET enable_hashjoin=off;
+SET enable_nestloop=off;
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM btg t1 JOIN btg t2 ON t1.z = t2.z AND t1.w = t2.w AND t1.x = t2.x
+ GROUP BY t1.x, t1.y, t1.z, t1.w;
+RESET enable_nestloop;
+RESET enable_hashjoin;
-- Should work with and without GROUP-BY optimization
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER BY y, x, z, w;
-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
-explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER BY x*x, z;
-SET enable_incremental_sort = off;
--- The case when the number of incoming subtree path keys is more than
+-- Test the case where the number of incoming subtree path keys is more than
-- the number of grouping keys.
-CREATE INDEX idx_y_x_z ON btg(y,x,w);
+CREATE INDEX btg_y_x_w_idx ON btg(y, x, w);
EXPLAIN (VERBOSE, COSTS OFF)
-SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
-RESET enable_incremental_sort;
+SELECT y, x, array_agg(distinct w)
+ FROM btg WHERE y < 0 GROUP BY x, y;
DROP TABLE btg;
--- The case, when scanning sort order correspond to aggregate sort order but
--- can not be found in the group-by list
+-- Test the case where the the ordering of scan matches the ordering within the
+-- aggregate but cannot be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
-CREATE UNIQUE INDEX ON agg_sort_order(c2);
-explain (costs off)
+CREATE UNIQUE INDEX agg_sort_order_c2_idx ON agg_sort_order(c2);
+INSERT INTO agg_sort_order SELECT i, i FROM generate_series(1,100)i;
+ANALYZE agg_sort_order;
+
+EXPLAIN (COSTS OFF)
SELECT array_agg(c1 ORDER BY c2),c2
FROM agg_sort_order WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
-DROP TABLE agg_sort_order CASCADE;
--- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
--- by planner itself. For example, by MergeJoin.
-SET enable_hashjoin = off;
-SET enable_nestloop = off;
-explain (COSTS OFF)
-SELECT c1.relname,c1.relpages
-FROM pg_class c1 JOIN pg_class c2 ON (c1.relname=c2.relname AND c1.relpages=c2.relpages)
-GROUP BY c1.reltuples,c1.relpages,c1.relname
-ORDER BY c1.relpages, c1.relname, c1.relpages*c1.relpages;
-RESET enable_hashjoin;
-RESET enable_nestloop;
+DROP TABLE agg_sort_order CASCADE;
RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
+RESET enable_seqscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
--
2.31.0
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-02 04:40 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-02-02 04:40 UTC (permalink / raw)
To: Richard Guo <[email protected]>; Tom Lane <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 2/2/2024 11:06, Richard Guo wrote:
>
> On Fri, Feb 2, 2024 at 11:32 AM Richard Guo <[email protected]
> <mailto:[email protected]>> wrote:
>
> On Fri, Feb 2, 2024 at 10:02 AM Tom Lane <[email protected]
> <mailto:[email protected]>> wrote:
>
> One of the test cases added by this commit has not been very
> stable in the buildfarm. Latest example is here:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04 <https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04;
>
> and I've seen similar failures intermittently on other machines.
>
> I'd suggest building this test atop a table that is more stable
> than pg_class. You're just waving a red flag in front of a bull
> if you expect stable statistics from that during a regression run.
> Nor do I see any particular reason for pg_class to be especially
> suited to the test.
>
>
> Yeah, it's not a good practice to use pg_class in this place. While
> looking through the test cases added by this commit, I noticed some
> other minor issues that are not great. Such as
>
> * The table 'btg' is inserted with 10000 tuples, which seems a bit
> expensive for a test. I don't think we need such a big table to test
> what we want.
>
> * I don't see why we need to manipulate GUC max_parallel_workers and
> max_parallel_workers_per_gather.
>
> * I think we'd better write the tests with the keywords being all upper
> or all lower. A mixed use of upper and lower is not great. Such as in
>
> explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
>
> * Some comments for the test queries are not easy to read.
>
> * For this statement
>
> CREATE INDEX idx_y_x_z ON btg(y,x,w);
>
> I think the index name would cause confusion. It creates an index on
> columns y, x and w, but the name indicates an index on y, x and z.
>
> I'd like to write a draft patch for the fixes.
>
>
> Here is the draft patch that fixes the issues I complained about in
> upthread.
I passed through the patch. Looks like it doesn't break anything. Why do
you prefer to use count(*) in EXPLAIN instead of plain targetlist, like
"SELECT x,y,..."?
Also, according to the test mentioned by Tom:
1. I see, PG uses IndexScan on (x,y). So, column x will be already
sorted before the MergeJoin. Why not use Incremental Sort on (x,z,w)
instead of full sort?
2. For memo, IMO, this test shows us the future near perspective of this
feature: It is cheaper to use grouping order (w,z) instead of (z,w).
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-21 08:08 Richard Guo <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Richard Guo @ 2024-02-21 08:08 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Fri, Feb 2, 2024 at 12:40 PM Andrei Lepikhov <[email protected]>
wrote:
> On 2/2/2024 11:06, Richard Guo wrote:
> >
> > On Fri, Feb 2, 2024 at 11:32 AM Richard Guo <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > On Fri, Feb 2, 2024 at 10:02 AM Tom Lane <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > One of the test cases added by this commit has not been very
> > stable in the buildfarm. Latest example is here:
> >
> >
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04
> <
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2024-02-01%2021%3A28%3A04
> >
> >
> > and I've seen similar failures intermittently on other machines.
> >
> > I'd suggest building this test atop a table that is more stable
> > than pg_class. You're just waving a red flag in front of a bull
> > if you expect stable statistics from that during a regression
> run.
> > Nor do I see any particular reason for pg_class to be especially
> > suited to the test.
> >
> >
> > Yeah, it's not a good practice to use pg_class in this place. While
> > looking through the test cases added by this commit, I noticed some
> > other minor issues that are not great. Such as
> >
> > * The table 'btg' is inserted with 10000 tuples, which seems a bit
> > expensive for a test. I don't think we need such a big table to test
> > what we want.
> >
> > * I don't see why we need to manipulate GUC max_parallel_workers and
> > max_parallel_workers_per_gather.
> >
> > * I think we'd better write the tests with the keywords being all
> upper
> > or all lower. A mixed use of upper and lower is not great. Such as
> in
> >
> > explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
> >
> > * Some comments for the test queries are not easy to read.
> >
> > * For this statement
> >
> > CREATE INDEX idx_y_x_z ON btg(y,x,w);
> >
> > I think the index name would cause confusion. It creates an index on
> > columns y, x and w, but the name indicates an index on y, x and z.
> >
> > I'd like to write a draft patch for the fixes.
> >
> >
> > Here is the draft patch that fixes the issues I complained about in
> > upthread.
>
> I passed through the patch. Looks like it doesn't break anything. Why do
> you prefer to use count(*) in EXPLAIN instead of plain targetlist, like
> "SELECT x,y,..."?
Nothing special. Just making the test cases consistent as much as
possible.
> Also, according to the test mentioned by Tom:
> 1. I see, PG uses IndexScan on (x,y). So, column x will be already
> sorted before the MergeJoin. Why not use Incremental Sort on (x,z,w)
> instead of full sort?
I think that's because the planner chooses to use (z, w, x) to perform
the mergejoin. I did not delve into the details, but I guess the cost
estimation decides this is cheaper.
Hi Alexander,
What do you think about the revisions for the test cases?
Thanks
Richard
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-21 10:19 Alexander Korotkov <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2024-02-21 10:19 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi, Richard!
> What do you think about the revisions for the test cases?
I've rebased your patch upthread. Did some minor beautifications.
> * The table 'btg' is inserted with 10000 tuples, which seems a bit
> expensive for a test. I don't think we need such a big table to test
> what we want.
Your patch reduces the number of rows to 1000 tuples. I found it
possible to further reduce it to 100 tuples. That also allowed me to
save the plan in the test case introduced by e1b7fde418.
Please check if you're OK with the patch attached.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Multiple-revises-for-the-GROUP-BY-reordering-test-v2.patch (16.3K, ../../CAPpHfdtMD=RhmB80DMRejs8s72Nr_y0OprUD9YiJFGgPVMdwXw@mail.gmail.com/2-0001-Multiple-revises-for-the-GROUP-BY-reordering-test-v2.patch)
download | inline diff:
From 3da12defe67efcb10ca719bd0bd64e2acf6b37fa Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Wed, 21 Feb 2024 11:43:00 +0200
Subject: [PATCH] Multiple revises for the GROUP BY reordering tests
Reported-by: Richard Guo
Discussion: https://postgr.es/m/CAMbWs4_NF0stupM8guasC_yCJxdTvRqm3a%2BJLarjvAxcH314TQ%40mail.gmail.com
---
src/test/regress/expected/aggregates.out | 185 ++++++++++-------------
src/test/regress/sql/aggregates.sql | 90 ++++++-----
2 files changed, 126 insertions(+), 149 deletions(-)
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index dba6f230014..12460bcb7f5 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,29 +2728,20 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
--- GROUP BY optimization by reorder columns
+-- GROUP BY optimization by reordering GROUP BY clauses
CREATE TABLE btg AS SELECT
- i % 100 AS x,
- i % 100 AS y,
+ i % 10 AS x,
+ i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
-CREATE INDEX btg_x_y_idx ON btg(x,y);
+FROM generate_series(1, 100) AS i;
+CREATE INDEX btg_x_y_idx ON btg(x, y);
ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
--- Utilize index scan ordering to avoid a Sort operation
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: x, y
- -> Index Only Scan using btg_x_y_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+SET enable_hashagg = off;
+SET enable_seqscan = off;
+-- Utilize the ordering of index scan to avoid a Sort operation
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY y, x;
QUERY PLAN
------------------------------------------------
GroupAggregate
@@ -2759,21 +2750,11 @@ EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
(3 rows)
-- Engage incremental sort
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, z, w
- -> Incremental Sort
- Sort Key: x, y, z, w
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY z, y, w, x;
QUERY PLAN
-------------------------------------------------
- Group
+ GroupAggregate
Group Key: x, y, z, w
-> Incremental Sort
Sort Key: x, y, z, w
@@ -2781,35 +2762,13 @@ explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
-> Index Scan using btg_x_y_idx on btg
(6 rows)
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, w, z
- -> Incremental Sort
- Sort Key: x, y, w, z
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, w, z
- -> Incremental Sort
- Sort Key: x, y, w, z
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
--- Subqueries
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
-GROUP BY (w,x,z,y);
+-- Utilize the ordering of subquery scan to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*)
+FROM (SELECT * FROM btg ORDER BY x, y, w, z) AS q1
+GROUP BY w, x, z, y;
QUERY PLAN
-------------------------------------------------
- Group
+ GroupAggregate
Group Key: btg.x, btg.y, btg.w, btg.z
-> Incremental Sort
Sort Key: btg.x, btg.y, btg.w, btg.z
@@ -2817,38 +2776,52 @@ GROUP BY (w,x,z,y);
-> Index Scan using btg_x_y_idx on btg
(6 rows)
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
-GROUP BY (w,x,z,y);
- QUERY PLAN
--------------------------------------------------------
- Group
- Group Key: btg.x, btg.y, btg.w, btg.z
- -> Limit
- -> Incremental Sort
- Sort Key: btg.x, btg.y, btg.w, btg.z
- Presorted Key: btg.x, btg.y
- -> Index Scan using btg_x_y_idx on btg
-(7 rows)
+-- Utilize the ordering of merge join to avoid a full Sort operation
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM btg t1 JOIN btg t2 ON t1.z = t2.z AND t1.w = t2.w AND t1.x = t2.x
+ GROUP BY t1.x, t1.y, t1.z, t1.w;
+ QUERY PLAN
+-------------------------------------------------------------------------------
+ GroupAggregate
+ Group Key: t1.z, t1.w, t1.x, t1.y
+ -> Incremental Sort
+ Sort Key: t1.z, t1.w, t1.x, t1.y
+ Presorted Key: t1.z, t1.w, t1.x
+ -> Merge Join
+ Merge Cond: ((t1.z = t2.z) AND (t1.w = t2.w) AND (t1.x = t2.x))
+ -> Sort
+ Sort Key: t1.z, t1.w, t1.x
+ -> Index Scan using btg_x_y_idx on btg t1
+ -> Sort
+ Sort Key: t2.z, t2.w, t2.x
+ -> Index Scan using btg_x_y_idx on btg t2
+(13 rows)
+RESET enable_nestloop;
+RESET enable_hashjoin;
-- Should work with and without GROUP-BY optimization
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
- QUERY PLAN
-------------------------------
- Group
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER BY y, x, z, w;
+ QUERY PLAN
+-------------------------------------------------
+ GroupAggregate
Group Key: y, x, z, w
-> Sort
Sort Key: y, x, z, w
- -> Seq Scan on btg
+ -> Index Scan using btg_x_y_idx on btg
(5 rows)
-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
-explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER BY x*x, z;
QUERY PLAN
-------------------------------------------------------
Sort
Sort Key: ((x * x)), z
- -> Group
+ -> GroupAggregate
Group Key: x, y, w, z
-> Incremental Sort
Sort Key: x, y, w, z
@@ -2856,23 +2829,22 @@ explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
-> Index Scan using btg_x_y_idx on btg
(8 rows)
-SET enable_incremental_sort = off;
--- The case when the number of incoming subtree path keys is more than
+-- Test the case where the number of incoming subtree path keys is more than
-- the number of grouping keys.
-CREATE INDEX idx_y_x_z ON btg(y,x,w);
+CREATE INDEX btg_y_x_w_idx ON btg(y, x, w);
EXPLAIN (VERBOSE, COSTS OFF)
-SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
- QUERY PLAN
------------------------------------------------------
+SELECT y, x, array_agg(distinct w)
+ FROM btg WHERE y < 0 GROUP BY x, y;
+ QUERY PLAN
+---------------------------------------------------------
GroupAggregate
Output: y, x, array_agg(DISTINCT w)
Group Key: btg.y, btg.x
- -> Index Only Scan using idx_y_x_z on public.btg
+ -> Index Only Scan using btg_y_x_w_idx on public.btg
Output: y, x, w
Index Cond: (btg.y < 0)
(6 rows)
-RESET enable_incremental_sort;
-- Check we don't pick aggregate path key instead of grouping path key
CREATE TABLE group_agg_pk AS SELECT
i % 10 AS x,
@@ -2897,37 +2869,37 @@ GROUP BY c1.w, c1.z;
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--- The case, when scanning sort order correspond to aggregate sort order but
--- can not be found in the group-by list
+-- Test the case where the the ordering of scan matches the ordering within the
+-- aggregate but cannot be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
-CREATE UNIQUE INDEX ON agg_sort_order(c2);
-explain (costs off)
+CREATE UNIQUE INDEX agg_sort_order_c2_idx ON agg_sort_order(c2);
+INSERT INTO agg_sort_order SELECT i, i FROM generate_series(1,100)i;
+ANALYZE agg_sort_order;
+EXPLAIN (COSTS OFF)
SELECT array_agg(c1 ORDER BY c2),c2
FROM agg_sort_order WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
- QUERY PLAN
---------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------------------------------------
Sort
Sort Key: c2
-> GroupAggregate
Group Key: c1
-> Sort
Sort Key: c1, c2
- -> Bitmap Heap Scan on agg_sort_order
- Recheck Cond: (c2 < 100)
- -> Bitmap Index Scan on agg_sort_order_c2_idx
- Index Cond: (c2 < 100)
-(10 rows)
+ -> Index Scan using agg_sort_order_c2_idx on agg_sort_order
+ Index Cond: (c2 < 100)
+(8 rows)
DROP TABLE agg_sort_order CASCADE;
-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
--- by planner itself. For example, by MergeJoin.
+-- by planner itself. For example, by MergeJoin.
SET enable_hashjoin = off;
SET enable_nestloop = off;
-explain (COSTS OFF)
-SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
-GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
- QUERY PLAN
--------------------------------------------------------------------
+EXPLAIN (COSTS OFF)
+SELECT b1.x, b1.w FROM btg b1 JOIN btg b2 ON (b1.z = b2.z AND b1.w = b2.w)
+GROUP BY b1.x, b1.z, b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
+ QUERY PLAN
+----------------------------------------------------------------------
Incremental Sort
Sort Key: b1.z, b1.w, ((b1.x * b1.x))
Presorted Key: b1.z, b1.w
@@ -2940,7 +2912,7 @@ GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
Merge Cond: ((b1.z = b2.z) AND (b1.w = b2.w))
-> Sort
Sort Key: b1.z, b1.w
- -> Seq Scan on btg b1
+ -> Index Scan using btg_x_y_idx on btg b1
-> Sort
Sort Key: b2.z, b2.w
-> Seq Scan on btg b2
@@ -2950,8 +2922,7 @@ RESET enable_hashjoin;
RESET enable_nestloop;
DROP TABLE btg;
RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
+RESET enable_seqscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index d6ed5d0effa..3c75c2aaa65 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,53 +1181,56 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
--- GROUP BY optimization by reorder columns
+-- GROUP BY optimization by reordering GROUP BY clauses
CREATE TABLE btg AS SELECT
- i % 100 AS x,
- i % 100 AS y,
+ i % 10 AS x,
+ i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
-CREATE INDEX btg_x_y_idx ON btg(x,y);
+FROM generate_series(1, 100) AS i;
+CREATE INDEX btg_x_y_idx ON btg(x, y);
ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
+SET enable_hashagg = off;
+SET enable_seqscan = off;
--- Utilize index scan ordering to avoid a Sort operation
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+-- Utilize the ordering of index scan to avoid a Sort operation
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY y, x;
-- Engage incremental sort
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
-
--- Subqueries
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
-GROUP BY (w,x,z,y);
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
-GROUP BY (w,x,z,y);
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY z, y, w, x;
+
+-- Utilize the ordering of subquery scan to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*)
+FROM (SELECT * FROM btg ORDER BY x, y, w, z) AS q1
+GROUP BY w, x, z, y;
+
+-- Utilize the ordering of merge join to avoid a full Sort operation
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM btg t1 JOIN btg t2 ON t1.z = t2.z AND t1.w = t2.w AND t1.x = t2.x
+ GROUP BY t1.x, t1.y, t1.z, t1.w;
+RESET enable_nestloop;
+RESET enable_hashjoin;
-- Should work with and without GROUP-BY optimization
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER BY y, x, z, w;
-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
-explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER BY x*x, z;
-SET enable_incremental_sort = off;
--- The case when the number of incoming subtree path keys is more than
+-- Test the case where the number of incoming subtree path keys is more than
-- the number of grouping keys.
-CREATE INDEX idx_y_x_z ON btg(y,x,w);
+CREATE INDEX btg_y_x_w_idx ON btg(y, x, w);
EXPLAIN (VERBOSE, COSTS OFF)
-SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
-RESET enable_incremental_sort;
+SELECT y, x, array_agg(distinct w)
+ FROM btg WHERE y < 0 GROUP BY x, y;
-- Check we don't pick aggregate path key instead of grouping path key
CREATE TABLE group_agg_pk AS SELECT
@@ -1248,29 +1251,32 @@ RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--- The case, when scanning sort order correspond to aggregate sort order but
--- can not be found in the group-by list
+-- Test the case where the the ordering of scan matches the ordering within the
+-- aggregate but cannot be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
-CREATE UNIQUE INDEX ON agg_sort_order(c2);
-explain (costs off)
+CREATE UNIQUE INDEX agg_sort_order_c2_idx ON agg_sort_order(c2);
+INSERT INTO agg_sort_order SELECT i, i FROM generate_series(1,100)i;
+ANALYZE agg_sort_order;
+
+EXPLAIN (COSTS OFF)
SELECT array_agg(c1 ORDER BY c2),c2
FROM agg_sort_order WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
+
DROP TABLE agg_sort_order CASCADE;
-- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
--- by planner itself. For example, by MergeJoin.
+-- by planner itself. For example, by MergeJoin.
SET enable_hashjoin = off;
SET enable_nestloop = off;
-explain (COSTS OFF)
-SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
-GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
+EXPLAIN (COSTS OFF)
+SELECT b1.x, b1.w FROM btg b1 JOIN btg b2 ON (b1.z = b2.z AND b1.w = b2.w)
+GROUP BY b1.x, b1.z, b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
RESET enable_hashjoin;
RESET enable_nestloop;
DROP TABLE btg;
RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
+RESET enable_seqscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-21 15:08 Maxim Orlov <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Maxim Orlov @ 2024-02-21 15:08 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi!
Another issue on test introduced in 0452b461bc405. I think it may be
unstable in some circumstances.
For example, if we'll try to use different BLCKSZ. See, I've made a little
change in the number of tuples to be inserted:
$ git diff
diff --git a/src/test/regress/sql/aggregates.sql
b/src/test/regress/sql/aggregates.sql
index d6ed5d0eff..414078d4ec 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1187,7 +1187,7 @@ CREATE TABLE btg AS SELECT
i % 100 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
+FROM generate_series(1,11900) AS i;
CREATE INDEX btg_x_y_idx ON btg(x,y);
ANALYZE btg;
And the bulk extension is kicked, so we got zeroed pages in the relation.
The plane is also changed,
switched to seq scan from index scan:
@@ -2734,7 +2734,7 @@
i % 100 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
+FROM generate_series(1,11900) AS i;
CREATE INDEX btg_x_y_idx ON btg(x,y);
ANALYZE btg;
-- GROUP BY optimization by reorder columns by frequency
@@ -2760,62 +2760,57 @@
-- Engage incremental sort
explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+------------------------------
Group
Group Key: x, y, z, w
- -> Incremental Sort
+ -> Sort
Sort Key: x, y, z, w
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
+ -> Seq Scan on btg
+(5 rows)
... and so on.
So, my proposal is simple. I think we need not just "ANALYZE btg", but
"VACUUM ANALYZE btg", to get rid of zeroed pages in this particular
case. PFA corresponding patch.
--
Best regards,
Maxim Orlov.
Attachments:
[application/octet-stream] 0001-Force-VACUUM-to-avoid-zeroed-pages-from-bulk-extensi.patch (1.3K, ../../CACG=ezaYM1tr6Lmp8PRH1aeZq=rBKXEoTwgzMcLaD5MPhfW0Lg@mail.gmail.com/3-0001-Force-VACUUM-to-avoid-zeroed-pages-from-bulk-extensi.patch)
download | inline diff:
From 4352e08c22ffc19d764e037967f164e8e58dba5d Mon Sep 17 00:00:00 2001
From: Maxim Orlov <[email protected]>
Date: Wed, 21 Feb 2024 17:58:26 +0300
Subject: [PATCH] Force VACUUM to avoid zeroed pages from bulk extension mess
with planning.
---
src/test/regress/expected/aggregates.out | 2 +-
src/test/regress/sql/aggregates.sql | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index dba6f23001..7f387e3023 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2736,7 +2736,7 @@ CREATE TABLE btg AS SELECT
i AS w
FROM generate_series(1,10000) AS i;
CREATE INDEX btg_x_y_idx ON btg(x,y);
-ANALYZE btg;
+VACUUM ANALYZE btg;
-- GROUP BY optimization by reorder columns by frequency
SET enable_hashagg=off;
SET max_parallel_workers= 0;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index d6ed5d0eff..31e73ec1f1 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1189,7 +1189,7 @@ CREATE TABLE btg AS SELECT
i AS w
FROM generate_series(1,10000) AS i;
CREATE INDEX btg_x_y_idx ON btg(x,y);
-ANALYZE btg;
+VACUUM ANALYZE btg;
-- GROUP BY optimization by reorder columns by frequency
--
2.43.0
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-22 02:09 Richard Guo <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 67+ messages in thread
From: Richard Guo @ 2024-02-22 02:09 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Wed, Feb 21, 2024 at 6:20 PM Alexander Korotkov <[email protected]>
wrote:
> Hi, Richard!
>
> > What do you think about the revisions for the test cases?
>
> I've rebased your patch upthread. Did some minor beautifications.
>
> > * The table 'btg' is inserted with 10000 tuples, which seems a bit
> > expensive for a test. I don't think we need such a big table to test
> > what we want.
>
> Your patch reduces the number of rows to 1000 tuples. I found it
> possible to further reduce it to 100 tuples. That also allowed me to
> save the plan in the test case introduced by e1b7fde418.
>
> Please check if you're OK with the patch attached.
I looked through the v2 patch and have two comments.
* The test case under "Check we don't pick aggregate path key instead of
grouping path key" does not have EXPLAIN to show the plan. So how can
we ensure we do not mistakenly select the aggregate pathkeys instead of
the grouping pathkeys?
* I don't think the test case introduced by e1b7fde418 is still needed,
because we already have one under "Utilize the ordering of merge join to
avoid a full Sort operation". This kind of test case is just to ensure
that we are able to utilize the ordering of the subplans underneath. So
it should be parallel to the test cases for utilize the ordering of
index scan and subquery scan.
See the attached v3 patch. I also made cosmetic tweaks to the comments,
and simplified a test case query.
Thanks
Richard
Attachments:
[application/octet-stream] v3-0001-Multiple-revisions-to-the-GROUP-BY-reordering-tests.patch (17.6K, ../../CAMbWs48MSejYST6pX8q-rEJMmRwo+fycDpc5vkkDusc=KF5y7Q@mail.gmail.com/3-v3-0001-Multiple-revisions-to-the-GROUP-BY-reordering-tests.patch)
download | inline diff:
From 13a383f16f9057ea8a96efe57221bb2458a456f0 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Thu, 22 Feb 2024 09:20:16 +0800
Subject: [PATCH v3] Multiple revisions to the GROUP BY reordering tests
Author: Richard Guo
---
src/test/regress/expected/aggregates.out | 236 ++++++++++-------------
src/test/regress/sql/aggregates.sql | 107 +++++-----
2 files changed, 154 insertions(+), 189 deletions(-)
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index dba6f23001..f86cf8d258 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2728,29 +2728,20 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
--- GROUP BY optimization by reorder columns
+-- GROUP BY optimization by reordering GROUP BY clauses
CREATE TABLE btg AS SELECT
- i % 100 AS x,
- i % 100 AS y,
+ i % 10 AS x,
+ i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
-CREATE INDEX btg_x_y_idx ON btg(x,y);
+FROM generate_series(1, 100) AS i;
+CREATE INDEX btg_x_y_idx ON btg(x, y);
ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
--- Utilize index scan ordering to avoid a Sort operation
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: x, y
- -> Index Only Scan using btg_x_y_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+SET enable_hashagg = off;
+SET enable_seqscan = off;
+-- Utilize the ordering of index scan to avoid a Sort operation
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY y, x;
QUERY PLAN
------------------------------------------------
GroupAggregate
@@ -2759,21 +2750,11 @@ EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
(3 rows)
-- Engage incremental sort
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY z, y, w, x;
QUERY PLAN
-------------------------------------------------
- Group
- Group Key: x, y, z, w
- -> Incremental Sort
- Sort Key: x, y, z, w
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
- QUERY PLAN
--------------------------------------------------
- Group
+ GroupAggregate
Group Key: x, y, z, w
-> Incremental Sort
Sort Key: x, y, z, w
@@ -2781,35 +2762,13 @@ explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
-> Index Scan using btg_x_y_idx on btg
(6 rows)
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
- QUERY PLAN
--------------------------------------------------
- Group
- Group Key: x, y, w, z
- -> Incremental Sort
- Sort Key: x, y, w, z
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
+-- Utilize the ordering of subquery scan to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*)
+FROM (SELECT * FROM btg ORDER BY x, y, w, z) AS q1
+GROUP BY w, x, z, y;
QUERY PLAN
-------------------------------------------------
- Group
- Group Key: x, y, w, z
- -> Incremental Sort
- Sort Key: x, y, w, z
- Presorted Key: x, y
- -> Index Scan using btg_x_y_idx on btg
-(6 rows)
-
--- Subqueries
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
-GROUP BY (w,x,z,y);
- QUERY PLAN
--------------------------------------------------
- Group
+ GroupAggregate
Group Key: btg.x, btg.y, btg.w, btg.z
-> Incremental Sort
Sort Key: btg.x, btg.y, btg.w, btg.z
@@ -2817,38 +2776,52 @@ GROUP BY (w,x,z,y);
-> Index Scan using btg_x_y_idx on btg
(6 rows)
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
-GROUP BY (w,x,z,y);
- QUERY PLAN
--------------------------------------------------------
- Group
- Group Key: btg.x, btg.y, btg.w, btg.z
- -> Limit
- -> Incremental Sort
- Sort Key: btg.x, btg.y, btg.w, btg.z
- Presorted Key: btg.x, btg.y
- -> Index Scan using btg_x_y_idx on btg
-(7 rows)
+-- Utilize the ordering of merge join to avoid a full Sort operation
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM btg t1 JOIN btg t2 ON t1.z = t2.z AND t1.w = t2.w AND t1.x = t2.x
+ GROUP BY t1.x, t1.y, t1.z, t1.w;
+ QUERY PLAN
+-------------------------------------------------------------------------------
+ GroupAggregate
+ Group Key: t1.z, t1.w, t1.x, t1.y
+ -> Incremental Sort
+ Sort Key: t1.z, t1.w, t1.x, t1.y
+ Presorted Key: t1.z, t1.w, t1.x
+ -> Merge Join
+ Merge Cond: ((t1.z = t2.z) AND (t1.w = t2.w) AND (t1.x = t2.x))
+ -> Sort
+ Sort Key: t1.z, t1.w, t1.x
+ -> Index Scan using btg_x_y_idx on btg t1
+ -> Sort
+ Sort Key: t2.z, t2.w, t2.x
+ -> Index Scan using btg_x_y_idx on btg t2
+(13 rows)
+RESET enable_nestloop;
+RESET enable_hashjoin;
-- Should work with and without GROUP-BY optimization
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
- QUERY PLAN
-------------------------------
- Group
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER BY y, x, z, w;
+ QUERY PLAN
+-------------------------------------------------
+ GroupAggregate
Group Key: y, x, z, w
-> Sort
Sort Key: y, x, z, w
- -> Seq Scan on btg
+ -> Index Scan using btg_x_y_idx on btg
(5 rows)
-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
-explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER BY x*x, z;
QUERY PLAN
-------------------------------------------------------
Sort
Sort Key: ((x * x)), z
- -> Group
+ -> GroupAggregate
Group Key: x, y, w, z
-> Incremental Sort
Sort Key: x, y, w, z
@@ -2856,24 +2829,24 @@ explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
-> Index Scan using btg_x_y_idx on btg
(8 rows)
-SET enable_incremental_sort = off;
--- The case when the number of incoming subtree path keys is more than
+-- Test the case where the number of incoming subtree path keys is more than
-- the number of grouping keys.
-CREATE INDEX idx_y_x_z ON btg(y,x,w);
+CREATE INDEX btg_y_x_w_idx ON btg(y, x, w);
EXPLAIN (VERBOSE, COSTS OFF)
-SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
- QUERY PLAN
------------------------------------------------------
+SELECT y, x, array_agg(distinct w)
+ FROM btg WHERE y < 0 GROUP BY x, y;
+ QUERY PLAN
+---------------------------------------------------------
GroupAggregate
Output: y, x, array_agg(DISTINCT w)
Group Key: btg.y, btg.x
- -> Index Only Scan using idx_y_x_z on public.btg
+ -> Index Only Scan using btg_y_x_w_idx on public.btg
Output: y, x, w
Index Cond: (btg.y < 0)
(6 rows)
-RESET enable_incremental_sort;
--- Check we don't pick aggregate path key instead of grouping path key
+-- Ensure that we do not select the aggregate pathkeys instead of the grouping
+-- pathkeys
CREATE TABLE group_agg_pk AS SELECT
i % 10 AS x,
i % 2 AS y,
@@ -2884,74 +2857,63 @@ FROM generate_series(1,100) AS i;
ANALYZE group_agg_pk;
SET enable_nestloop = off;
SET enable_hashjoin = off;
-SELECT
- c1.z, c1.w, string_agg(''::text, repeat(''::text, c1.f) ORDER BY c1.x,c1.y)
-FROM group_agg_pk c1 JOIN group_agg_pk c2 ON (c1.x = c2.f)
+EXPLAIN (COSTS OFF)
+SELECT avg(c1.f ORDER BY c1.x, c1.y)
+FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
- z | w | string_agg
----+---+------------
- 0 | 2 |
- 1 | 2 |
+ QUERY PLAN
+-----------------------------------------------------
+ GroupAggregate
+ Group Key: c1.w, c1.z
+ -> Sort
+ Sort Key: c1.w, c1.z, c1.x, c1.y
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(12 rows)
+
+SELECT avg(c1.f ORDER BY c1.x, c1.y)
+FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
+GROUP BY c1.w, c1.z;
+ avg
+--------------------
+ 4.0000000000000000
+ 5.0000000000000000
(2 rows)
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--- The case, when scanning sort order correspond to aggregate sort order but
--- can not be found in the group-by list
+-- Test the case where the the ordering of scan matches the ordering within the
+-- aggregate but cannot be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
-CREATE UNIQUE INDEX ON agg_sort_order(c2);
-explain (costs off)
+CREATE UNIQUE INDEX agg_sort_order_c2_idx ON agg_sort_order(c2);
+INSERT INTO agg_sort_order SELECT i, i FROM generate_series(1,100)i;
+ANALYZE agg_sort_order;
+EXPLAIN (COSTS OFF)
SELECT array_agg(c1 ORDER BY c2),c2
FROM agg_sort_order WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
- QUERY PLAN
---------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------------------------------------
Sort
Sort Key: c2
-> GroupAggregate
Group Key: c1
-> Sort
Sort Key: c1, c2
- -> Bitmap Heap Scan on agg_sort_order
- Recheck Cond: (c2 < 100)
- -> Bitmap Index Scan on agg_sort_order_c2_idx
- Index Cond: (c2 < 100)
-(10 rows)
+ -> Index Scan using agg_sort_order_c2_idx on agg_sort_order
+ Index Cond: (c2 < 100)
+(8 rows)
DROP TABLE agg_sort_order CASCADE;
--- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
--- by planner itself. For example, by MergeJoin.
-SET enable_hashjoin = off;
-SET enable_nestloop = off;
-explain (COSTS OFF)
-SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
-GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
- QUERY PLAN
--------------------------------------------------------------------
- Incremental Sort
- Sort Key: b1.z, b1.w, ((b1.x * b1.x))
- Presorted Key: b1.z, b1.w
- -> Group
- Group Key: b1.z, b1.w, b1.x
- -> Incremental Sort
- Sort Key: b1.z, b1.w, b1.x
- Presorted Key: b1.z, b1.w
- -> Merge Join
- Merge Cond: ((b1.z = b2.z) AND (b1.w = b2.w))
- -> Sort
- Sort Key: b1.z, b1.w
- -> Seq Scan on btg b1
- -> Sort
- Sort Key: b2.z, b2.w
- -> Seq Scan on btg b2
-(16 rows)
-
-RESET enable_hashjoin;
-RESET enable_nestloop;
DROP TABLE btg;
RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
+RESET enable_seqscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index d6ed5d0eff..441e01d150 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1181,55 +1181,59 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
--- GROUP BY optimization by reorder columns
+-- GROUP BY optimization by reordering GROUP BY clauses
CREATE TABLE btg AS SELECT
- i % 100 AS x,
- i % 100 AS y,
+ i % 10 AS x,
+ i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
-FROM generate_series(1,10000) AS i;
-CREATE INDEX btg_x_y_idx ON btg(x,y);
+FROM generate_series(1, 100) AS i;
+CREATE INDEX btg_x_y_idx ON btg(x, y);
ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
+SET enable_hashagg = off;
+SET enable_seqscan = off;
--- Utilize index scan ordering to avoid a Sort operation
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY x,y;
-EXPLAIN (COSTS OFF) SELECT count(*) FROM btg GROUP BY y,x;
+-- Utilize the ordering of index scan to avoid a Sort operation
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY y, x;
-- Engage incremental sort
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY x,y,z,w;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY z,y,w,x;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,z,x,y;
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y;
-
--- Subqueries
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z) AS q1
-GROUP BY (w,x,z,y);
-explain (COSTS OFF) SELECT x,y
-FROM (SELECT * FROM btg ORDER BY x,y,w,z LIMIT 100) AS q1
-GROUP BY (w,x,z,y);
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY z, y, w, x;
+
+-- Utilize the ordering of subquery scan to avoid a Sort operation
+EXPLAIN (COSTS OFF) SELECT count(*)
+FROM (SELECT * FROM btg ORDER BY x, y, w, z) AS q1
+GROUP BY w, x, z, y;
+
+-- Utilize the ordering of merge join to avoid a full Sort operation
+SET enable_hashjoin = off;
+SET enable_nestloop = off;
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM btg t1 JOIN btg t2 ON t1.z = t2.z AND t1.w = t2.w AND t1.x = t2.x
+ GROUP BY t1.x, t1.y, t1.z, t1.w;
+RESET enable_nestloop;
+RESET enable_hashjoin;
-- Should work with and without GROUP-BY optimization
-explain (COSTS OFF) SELECT x,y FROM btg GROUP BY w,x,z,y ORDER BY y,x,z,w;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER BY y, x, z, w;
-- Utilize incremental sort to make the ORDER BY rule a bit cheaper
-explain (COSTS OFF) SELECT x,w FROM btg GROUP BY w,x,y,z ORDER BY x*x,z;
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER BY x*x, z;
-SET enable_incremental_sort = off;
--- The case when the number of incoming subtree path keys is more than
+-- Test the case where the number of incoming subtree path keys is more than
-- the number of grouping keys.
-CREATE INDEX idx_y_x_z ON btg(y,x,w);
+CREATE INDEX btg_y_x_w_idx ON btg(y, x, w);
EXPLAIN (VERBOSE, COSTS OFF)
-SELECT y,x,array_agg(distinct w) FROM btg WHERE y < 0 GROUP BY x,y;
-RESET enable_incremental_sort;
+SELECT y, x, array_agg(distinct w)
+ FROM btg WHERE y < 0 GROUP BY x, y;
--- Check we don't pick aggregate path key instead of grouping path key
+-- Ensure that we do not select the aggregate pathkeys instead of the grouping
+-- pathkeys
CREATE TABLE group_agg_pk AS SELECT
i % 10 AS x,
i % 2 AS y,
@@ -1240,37 +1244,36 @@ FROM generate_series(1,100) AS i;
ANALYZE group_agg_pk;
SET enable_nestloop = off;
SET enable_hashjoin = off;
-SELECT
- c1.z, c1.w, string_agg(''::text, repeat(''::text, c1.f) ORDER BY c1.x,c1.y)
-FROM group_agg_pk c1 JOIN group_agg_pk c2 ON (c1.x = c2.f)
+
+EXPLAIN (COSTS OFF)
+SELECT avg(c1.f ORDER BY c1.x, c1.y)
+FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
+GROUP BY c1.w, c1.z;
+SELECT avg(c1.f ORDER BY c1.x, c1.y)
+FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--- The case, when scanning sort order correspond to aggregate sort order but
--- can not be found in the group-by list
+-- Test the case where the the ordering of scan matches the ordering within the
+-- aggregate but cannot be found in the group-by list
CREATE TABLE agg_sort_order (c1 int PRIMARY KEY, c2 int);
-CREATE UNIQUE INDEX ON agg_sort_order(c2);
-explain (costs off)
+CREATE UNIQUE INDEX agg_sort_order_c2_idx ON agg_sort_order(c2);
+INSERT INTO agg_sort_order SELECT i, i FROM generate_series(1,100)i;
+ANALYZE agg_sort_order;
+
+EXPLAIN (COSTS OFF)
SELECT array_agg(c1 ORDER BY c2),c2
FROM agg_sort_order WHERE c2 < 100 GROUP BY c1 ORDER BY 2;
-DROP TABLE agg_sort_order CASCADE;
--- Check, that GROUP-BY reordering optimization can operate with pathkeys, built
--- by planner itself. For example, by MergeJoin.
-SET enable_hashjoin = off;
-SET enable_nestloop = off;
-explain (COSTS OFF)
-SELECT b1.x,b1.w FROM btg b1 JOIN btg b2 ON (b1.z=b2.z AND b1.w=b2.w)
-GROUP BY b1.x,b1.z,b1.w ORDER BY b1.z, b1.w, b1.x*b1.x;
-RESET enable_hashjoin;
-RESET enable_nestloop;
+DROP TABLE agg_sort_order CASCADE;
DROP TABLE btg;
+
RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
+RESET enable_seqscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
--
2.31.0
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-22 04:18 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-02-22 04:18 UTC (permalink / raw)
To: Richard Guo <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Tom Lane <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 22/2/2024 09:09, Richard Guo wrote:
>
> On Wed, Feb 21, 2024 at 6:20 PM Alexander Korotkov <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi, Richard!
>
> > What do you think about the revisions for the test cases?
>
> I've rebased your patch upthread. Did some minor beautifications.
>
> > * The table 'btg' is inserted with 10000 tuples, which seems a bit
> > expensive for a test. I don't think we need such a big table to test
> > what we want.
>
> Your patch reduces the number of rows to 1000 tuples. I found it
> possible to further reduce it to 100 tuples. That also allowed me to
> save the plan in the test case introduced by e1b7fde418.
>
> Please check if you're OK with the patch attached.
>
>
> I looked through the v2 patch and have two comments.
>
> * The test case under "Check we don't pick aggregate path key instead of
> grouping path key" does not have EXPLAIN to show the plan. So how can
> we ensure we do not mistakenly select the aggregate pathkeys instead of
> the grouping pathkeys?
I confirm it works correctly. I am not sure about the stability of the
zeros number in the output on different platforms here:
avg
--------------------
4.0000000000000000
5.0000000000000000
It was why I'd used the format() function before. So, may we elaborate
on the test and restrict the output?
>
> * I don't think the test case introduced by e1b7fde418 is still needed,
> because we already have one under "Utilize the ordering of merge join to
> avoid a full Sort operation". This kind of test case is just to ensure
> that we are able to utilize the ordering of the subplans underneath. So
> it should be parallel to the test cases for utilize the ordering of
> index scan and subquery scan.
I confirm, this version also checks ec_sortref initialization in the
case when ec are contructed from WHERE clause. Generally, I like more
one test for one issue instead of one test for all at once. But it works
and I don't have any reason to dispute it.
Also, I'm unsure about removing the disabling of the
max_parallel_workers_per_gather parameter. Have you discovered the
domination of the current plan over the partial one? Do the cost
fluctuations across platforms not trigger a parallel plan?
What's more, I suggest to address here the complaint from [1]. As I see,
cost difference between Sort and IncrementalSort strategies in that case
is around 0.5. To make the test more stable I propose to change it a bit
and add a limit:
SELECT count(*) FROM btg GROUP BY z, y, w, x LIMIT 10;
It makes efficacy of IncrementalSort more obvious difference around 10
cost points.
[1]
https://www.postgresql.org/message-id/[email protected]...
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-22 06:35 Richard Guo <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: Richard Guo @ 2024-02-22 06:35 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Tom Lane <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Thu, Feb 22, 2024 at 12:18 PM Andrei Lepikhov <[email protected]>
wrote:
> On 22/2/2024 09:09, Richard Guo wrote:
> > I looked through the v2 patch and have two comments.
> >
> > * The test case under "Check we don't pick aggregate path key instead of
> > grouping path key" does not have EXPLAIN to show the plan. So how can
> > we ensure we do not mistakenly select the aggregate pathkeys instead of
> > the grouping pathkeys?
>
> I confirm it works correctly. I am not sure about the stability of the
> zeros number in the output on different platforms here:
> avg
> --------------------
> 4.0000000000000000
> 5.0000000000000000
> It was why I'd used the format() function before. So, may we elaborate
> on the test and restrict the output?
The avg() function on integer argument is commonly used in
aggregates.sql. I don't think this is an issue. See the first test
query in aggregates.sql.
SELECT avg(four) AS avg_1 FROM onek;
avg_1
--------------------
1.5000000000000000
(1 row)
> > * I don't think the test case introduced by e1b7fde418 is still needed,
> > because we already have one under "Utilize the ordering of merge join to
> > avoid a full Sort operation". This kind of test case is just to ensure
> > that we are able to utilize the ordering of the subplans underneath. So
> > it should be parallel to the test cases for utilize the ordering of
> > index scan and subquery scan.
>
> Also, I'm unsure about removing the disabling of the
> max_parallel_workers_per_gather parameter. Have you discovered the
> domination of the current plan over the partial one? Do the cost
> fluctuations across platforms not trigger a parallel plan?
The table used for testing contains only 100 tuples, which is the size
of only one page. I don't believe it would trigger any parallel plans,
unless we manually change min_parallel_table_scan_size.
> What's more, I suggest to address here the complaint from [1]. As I see,
> cost difference between Sort and IncrementalSort strategies in that case
> is around 0.5. To make the test more stable I propose to change it a bit
> and add a limit:
> SELECT count(*) FROM btg GROUP BY z, y, w, x LIMIT 10;
> It makes efficacy of IncrementalSort more obvious difference around 10
> cost points.
I don't think that's necessary. With Incremental Sort the final cost
is:
GroupAggregate (cost=1.66..19.00 rows=100 width=25)
while with full Sort it is:
GroupAggregate (cost=16.96..19.46 rows=100 width=25)
With the STD_FUZZ_FACTOR (1.01), there is no doubt that the first path
is cheaper on total cost. Not to say that even if somehow we decide the
two paths are fuzzily the same on total cost, the first path still
dominates because its startup cost is much cheaper.
Thanks
Richard
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-02-22 07:04 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-02-22 07:04 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Tom Lane <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 22/2/2024 13:35, Richard Guo wrote:
> The avg() function on integer argument is commonly used in
> aggregates.sql. I don't think this is an issue. See the first test
> query in aggregates.sql.
Make sense
> > it should be parallel to the test cases for utilize the ordering of
> > index scan and subquery scan.
>
> Also, I'm unsure about removing the disabling of the
> max_parallel_workers_per_gather parameter. Have you discovered the
> domination of the current plan over the partial one? Do the cost
> fluctuations across platforms not trigger a parallel plan?
>
>
> The table used for testing contains only 100 tuples, which is the size
> of only one page. I don't believe it would trigger any parallel plans,
> unless we manually change min_parallel_table_scan_size.
I don't intend to argue it, but just for the information, I frequently
reduce it to zero, allowing PostgreSQL to make a decision based on
costs. It sometimes works much better, because one small table in multi
join can disallow an effective parallel plan.
>
> What's more, I suggest to address here the complaint from [1]. As I
> see,
> cost difference between Sort and IncrementalSort strategies in that
> case
> is around 0.5. To make the test more stable I propose to change it a
> bit
> and add a limit:
> SELECT count(*) FROM btg GROUP BY z, y, w, x LIMIT 10;
> It makes efficacy of IncrementalSort more obvious difference around 10
> cost points.
>
>
> I don't think that's necessary. With Incremental Sort the final cost
> is:
>
> GroupAggregate (cost=1.66..19.00 rows=100 width=25)
>
> while with full Sort it is:
>
> GroupAggregate (cost=16.96..19.46 rows=100 width=25)
>
> With the STD_FUZZ_FACTOR (1.01), there is no doubt that the first path
> is cheaper on total cost. Not to say that even if somehow we decide the
> two paths are fuzzily the same on total cost, the first path still
> dominates because its startup cost is much cheaper.
As before, I won't protest here - it needs some computations about how
much cost can be added by bulk extension of the relation blocks. If
Maxim will answer that it's enough to resolve his issue, why not?
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-11 23:44 Tom Lane <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 6 replies; 67+ messages in thread
From: Tom Lane @ 2024-04-11 23:44 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Andrei Lepikhov <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
I'm late to the party, and I apologize for not having paid attention
to this thread for months ... but I am wondering why 0452b461b got
committed. It seems to add a substantial amount of complexity and
planner cycles in return for not much. The reason why I'm dubious
about it can be found in this comment that the patch deleted:
- * In principle it might be interesting to consider other orderings of the
- * GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. We don't
- * bother with that, though. Hashed grouping will frequently win anyway.
Indeed, if you remove the
SET enable_hashagg = off;
SET enable_seqscan = off;
lines added to aggregates.sql, you find that every single one of the
test cases added there changes plan, except for the one case that is
there to prove that the planner *doesn't* pick this type of plan.
That shows that the planner doesn't believe any of the plans developed
here are cheaper than other alternatives it would normally consider
(usually HashAgg). So it sure seems like we're spending a lot of
effort on uninteresting plans. Maybe this is an artifact of testing
against toy tables and a useful win could be shown on bigger tables,
but I don't see any indication in the thread that anyone actually
demonstrated that.
I might hold still for this patch anyway if the patch were simpler
and more obviously correct, but there are scary bits in it:
* The very first hunk causes get_eclass_for_sort_expr to have
side-effects on existing EquivalenceClass data structures.
What is the argument that that's not going to cause problems?
At the very least it introduces asymmetry, in that the first
caller wins the chance to change the EC's ec_sortref and later
callers won't be able to.
* I'm pretty unconvinced by group_keys_reorder_by_pathkeys (which
I notice has already had one band-aid added to it since commit).
In particular, it seems to believe that the pathkeys and clauses
lists match one-for-one, but I seriously doubt that that invariant
remains guaranteed after the cleanup steps
/* append the remaining group pathkeys (will be treated as not sorted) */
*group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
*group_pathkeys);
*group_clauses = list_concat_unique_ptr(new_group_clauses,
*group_clauses);
For that to be reliable, the SortGroupClauses added to
new_group_clauses in the main loop have to be exactly those
that are associated with the same pathkeys in the old lists.
I doubt that that's necessarily true in the presence of redundant
grouping clauses. (Maybe we can't get here with any redundant
grouping clauses, but still, we don't really guarantee uniqueness of
SortGroupClauses, much less that they are never copied which is what
you need if you want to believe that pointer equality is sufficient
for de-duping here. PathKeys are explicitly made to be safe to compare
pointer-wise, but I know of no such guarantee for SortGroupClauses.)
* It seems like root->processed_groupClause no longer has much to do
with the grouping behavior, which is scary given how much code still
believes that it does. I suspect there are bugs lurking there, and
am not comforted by the fact that the patch changed exactly nothing
in the pathnodes.h documentation of that field. This comment looks
pretty silly now too:
/* Preprocess regular GROUP BY clause, if any */
root->processed_groupClause = list_copy(parse->groupClause);
What "preprocessing" is going on there? This comment was adequate
before, when the code line invoked preprocess_groupclause which had
a bunch of commentary; but now it has to stand on its own and it's
not doing a great job of that.
* Speaking of pathnodes.h, PathKeyInfo is a pretty uninformative node
type name, and the comments provided for it are not going to educate
anybody. What is the "association" between the pathkeys and clauses?
I guess the clauses are supposed to be SortGroupClauses, but not all
pathkeys match up to SortGroupClauses, so what then? This is very
underspecified, and fuzzy thinking about data structures tends to lead
to bugs.
So I'm quite afraid that there are still bugs lurking here.
What's more, given that the plans this patch makes apparently
seldom win when you don't put a thumb on the scales, it might
take a very long time to isolate those bugs. If the patch
produces a faulty plan (e.g. not sorted properly) we'd never
notice if that plan isn't chosen, and even if it is chosen
it probably wouldn't produce anything as obviously wrong as
a crash.
If this patch were producing better results I'd be more excited
about putting more work into it. But on the basis of what I'm
seeing right now, I think maybe we ought to give up on it.
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-12 05:05 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
5 siblings, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-04-12 05:05 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 4/12/24 06:44, Tom Lane wrote:
> If this patch were producing better results I'd be more excited
> about putting more work into it. But on the basis of what I'm
> seeing right now, I think maybe we ought to give up on it.
First, thanks for the deep review - sometimes, only a commit gives us a
chance to get such observation :))).
On a broader note, introducing automatic group-by-order choosing is a
step towards training the optimiser to handle poorly tuned incoming
queries. While it's true that this may initially impact performance,
it's crucial to weigh the potential benefits. So, beforehand, we should
agree: Is it worth it?
If yes, I would say I see how often hashing doesn't work in grouping.
Sometimes because of estimation errors, sometimes because grouping
already has sorted input, sometimes in analytical queries when planner
doesn't have enough memory for hashing. In analytical cases, the only
way to speed up queries sometimes is to be smart with features like
IncrementalSort and this one.
About low efficiency. Remember the previous version of the GROUP-BY
optimisation - we disagreed on operator costs and the cost model in
general. In the current version, we went the opposite - adding small
features step-by-step. The current commit contains an integral part of
the feature and is designed for safely testing the approach and adding
more profitable parts like choosing group-by-order according to distinct
values or unique indexes on grouping columns.
I have passed through the code being steered by the issues explained in
detail. I see seven issues. Two of them definitely should be scrutinised
right now, and I'm ready to do that.
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-16 05:59 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
5 siblings, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-04-16 05:59 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 4/12/24 06:44, Tom Lane wrote:
> * The very first hunk causes get_eclass_for_sort_expr to have
> side-effects on existing EquivalenceClass data structures.
> What is the argument that that's not going to cause problems?
> At the very least it introduces asymmetry, in that the first
> caller wins the chance to change the EC's ec_sortref and later
> callers won't be able to.
Let me resolve issues bit by bit.
Addressing your first note, 'asymmetry,' I discovered this part of the
code. Before the 8d83a5d, it could cause some artefacts, but since then,
a kind of asymmetry has been introduced into the GROUP-BY clause list.
As you can see in the attached patch, GROUP-BY reordering works even
when the asymmetry of the 8d83a5d chooses different keys.
At the same time, I agree that implicitly setting anything in an
exporting function, which should just look up for EC, is a questionable
coding style. To avoid possible usage issues (in extensions, for
example), I propose setting it up afterwards, explicitly forcing this
action by input parameter - see my attempt in the attachment.
--
regards,
Andrei Lepikhov
Postgres Professional
Attachments:
[text/x-patch] group_by_asymmetry.diff (7.8K, ../../[email protected]/2-group_by_asymmetry.diff)
download | inline diff:
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index a619ff9177..bc08dfadaf 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,18 +652,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from a WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1d61881a6b..5d9597adcd 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -1355,7 +1355,8 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
&sortclauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
/* It's caller error if not all clauses were sortable */
Assert(sortable);
return result;
@@ -1379,13 +1380,16 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
* to remove any clauses that can be proven redundant via the eclass logic.
* Even though we'll have to hash in that case, we might as well not hash
* redundant columns.)
+ * *set_ec_sortref is true if caller wants to set up ec_sortref field in
+ * the pathkey Equivalence Class, if it have not initialized beforehand.
*/
List *
make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable)
+ bool *sortable,
+ bool set_ec_sortref)
{
List *pathkeys = NIL;
ListCell *l;
@@ -1409,6 +1413,13 @@ make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
sortcl->nulls_first,
sortcl->tleSortGroupRef,
true);
+ if (pathkey->pk_eclass->ec_sortref == 0 && set_ec_sortref)
+ /*
+ * Copy the sortref if it wasn't set yet. That may happen if
+ * the ec was constructed from a WHERE clause, i.e. it doesn't
+ * have a target reference at all.
+ */
+ pathkey->pk_eclass->ec_sortref = sortcl->tleSortGroupRef;
/* Canonical form eliminates redundant ordering keys */
if (!pathkey_is_redundant(pathkey, pathkeys))
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 5320da51a0..dee98c9a90 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -3391,7 +3391,7 @@ standard_qp_callback(PlannerInfo *root, void *extra)
/*
* With a plain GROUP BY list, we can remove any grouping items that
* are proven redundant by EquivalenceClass processing. For example,
- * we can remove y given "WHERE x = y GROUP BY x, y". These aren't
+ * we can remove y given "WHERE x get_eclass_for_sort_expr= y GROUP BY x, y". These aren't
* especially common cases, but they're nearly free to detect. Note
* that we remove redundant items from processed_groupClause but not
* the original parse->groupClause.
@@ -3403,7 +3403,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&root->processed_groupClause,
tlist,
true,
- &sortable);
+ &sortable,
+ true);
if (!sortable)
{
/* Can't sort; no point in considering aggregate ordering either */
@@ -3453,7 +3454,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&root->processed_distinctClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->distinct_pathkeys = NIL;
}
@@ -3479,7 +3481,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&groupClauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->setop_pathkeys = NIL;
}
@@ -6027,7 +6030,8 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc,
&wc->partitionClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
Assert(sortable);
}
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 39ba461548..fff4b69dfd 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -242,7 +242,8 @@ extern List *make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable);
+ bool *sortable,
+ bool set_ec_sortref);
extern void initialize_mergeclause_eclasses(PlannerInfo *root,
RestrictInfo *restrictinfo);
extern void update_mergeclause_eclasses(PlannerInfo *root,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 50695d83a2..1b06f9ca97 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2886,6 +2886,53 @@ GROUP BY c1.w, c1.z;
5.0000000000000000
(2 rows)
+-- Pathkeys, built in subtree, can be used to optimize GROUP-BY clause ordering.
+-- Also, here we check that it doesn't depend on the initial clause order in the
+-- GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c1.x, c1.y
+ -> Incremental Sort
+ Sort Key: c1.x, c1.y
+ Presorted Key: c1.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c2.x, c1.y
+ -> Incremental Sort
+ Sort Key: c2.x, c1.y
+ Presorted Key: c2.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 2905848ead..0be9c48483 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1253,6 +1253,20 @@ SELECT avg(c1.f ORDER BY c1.x, c1.y)
FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
+-- Pathkeys, built in subtree, can be used to optimize GROUP-BY clause ordering.
+-- Also, here we check that it doesn't depend on the initial clause order in the
+-- GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-17 03:11 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
5 siblings, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-04-17 03:11 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 4/12/24 06:44, Tom Lane wrote:
> * I'm pretty unconvinced by group_keys_reorder_by_pathkeys (which
> I notice has already had one band-aid added to it since commit).
> In particular, it seems to believe that the pathkeys and clauses
> lists match one-for-one, but I seriously doubt that that invariant
> remains guaranteed after the cleanup steps
>
> /* append the remaining group pathkeys (will be treated as not sorted) */
> *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
> *group_pathkeys);
> *group_clauses = list_concat_unique_ptr(new_group_clauses,
> *group_clauses);
>
> For that to be reliable, the SortGroupClauses added to
> new_group_clauses in the main loop have to be exactly those
> that are associated with the same pathkeys in the old lists.
> I doubt that that's necessarily true in the presence of redundant
> grouping clauses. (Maybe we can't get here with any redundant
> grouping clauses, but still, we don't really guarantee uniqueness of
> SortGroupClauses, much less that they are never copied which is what
> you need if you want to believe that pointer equality is sufficient
> for de-duping here. PathKeys are explicitly made to be safe to compare
> pointer-wise, but I know of no such guarantee for SortGroupClauses.)
I spent a lot of time inventing situations with SortGroupClause
duplicates. Unfortunately, it looks impossible so far. But because we
really don't guarantee uniqueness, I changed the code to survive in this
case. Also, I added assertion checking to be sure we don't have logical
mistakes here - see attachment.
About the band-aid mentioned above - as I see, 4169850 introduces the
same trick in planner.c. So, it looks like result of design of the
current code.
--
regards,
Andrei Lepikhov
Postgres Professional
Attachments:
[text/x-patch] get_useful_group_keys_orderings.patch (3.8K, ../../[email protected]/2-get_useful_group_keys_orderings.patch)
download | inline diff:
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 5d9597adcd..aa80f6486c 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -380,15 +380,18 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* We're going to search within just the first num_groupby_pathkeys of
- * *group_pathkeys. The thing is that root->group_pathkeys is passed as
+ * *group_pathkeys. The thing is that root->group_pathkeys is passed as
* *group_pathkeys containing grouping pathkeys altogether with aggregate
- * pathkeys. If we process aggregate pathkeys we could get an invalid
+ * pathkeys. If we process aggregate pathkeys we could get an invalid
* result of get_sortgroupref_clause_noerr(), because their
- * pathkey->pk_eclass->ec_sortref doesn't referece query targetlist. So,
+ * pathkey->pk_eclass->ec_sortref doesn't reference query targetlist. So,
* we allocate a separate list of pathkeys for lookups.
*/
grouping_pathkeys = list_copy_head(*group_pathkeys, num_groupby_pathkeys);
+ /* Make a new copy before reordering clauses */
+ *group_clauses = list_copy(*group_clauses);
+
/*
* Walk the pathkeys (determining ordering of the input path) and see if
* there's a matching GROUP BY key. If we find one, we append it to the
@@ -400,8 +403,8 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
*/
foreach(lc, pathkeys)
{
- PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
+ PathKey *pathkey = (PathKey *) lfirst(lc);
+ SortGroupClause *sgc;
/*
* Pathkeys are built in a way that allows simply comparing pointers.
@@ -431,17 +434,25 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
Assert(OidIsValid(sgc->sortop));
new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
+
+ /*
+ * Keeping in mind that the SortGroupClause list doesn't guarantee
+ * unique pointers we must explicitly transfer elements one-by-one.
+ */
new_group_clauses = lappend(new_group_clauses, sgc);
+ *group_clauses = list_delete_ptr(*group_clauses, sgc);
}
/* remember the number of pathkeys with a matching GROUP BY key */
n = list_length(new_group_pathkeys);
- /* append the remaining group pathkeys (will be treated as not sorted) */
+ /*
+ * Append the remaining group pathkeys (will be treated as not sorted) and
+ * grouping clauses.
+ */
*group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
*group_pathkeys);
- *group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
+ *group_clauses = list_concat(new_group_clauses, *group_clauses);
list_free(grouping_pathkeys);
return n;
@@ -484,9 +495,9 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
List *
get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
{
- Query *parse = root->parse;
- List *infos = NIL;
- PathKeyInfo *info;
+ Query *parse = root->parse;
+ List *infos = NIL;
+ PathKeyInfo *info;
List *pathkeys = root->group_pathkeys;
List *clauses = root->processed_groupClause;
@@ -561,6 +572,23 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
+#ifdef USE_ASSERT_CHECKING
+ {
+ PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ ListCell *lc;
+
+ /* Test consistency of info structures */
+ for_each_from(lc, infos, 1)
+ {
+ info = lfirst_node(PathKeyInfo, lc);
+
+ Assert(list_length(info->clauses) == list_length(pinfo->clauses));
+ Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
+ Assert(list_difference(info->clauses, pinfo->clauses) == NIL);
+ Assert(list_difference_ptr(info->pathkeys, pinfo->pathkeys) == NIL);
+ }
+ }
+#endif
return infos;
}
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-18 05:01 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
5 siblings, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-04-18 05:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 4/12/24 06:44, Tom Lane wrote:
> * It seems like root->processed_groupClause no longer has much to do
> with the grouping behavior, which is scary given how much code still
> believes that it does. I suspect there are bugs lurking there, and
1. Analysing the code, processed_groupClause is used in:
- adjust_foreign_grouping_path_cost - to decide, do we need to add cost
of sort to the foreign grouping.
- just for replacing relids during SJE process.
- create_groupingsets_plan(), preprocess_grouping_sets,
remove_useless_groupby_columns - we don't apply this feature in the case
of grouping sets.
- get_number_of_groups - estimation shouldn't depend on the order of
clauses.
- create_grouping_paths - to analyse hashability of grouping clause
- make_group_input_target, make_partial_grouping_target - doesn't depend
on the order of clauses
planner.c: add_paths_to_grouping_rel, create_partial_grouping_paths - in
the case of hash grouping.
So, the only case we can impact current behavior lies in the
postgres_fdw. But here we don't really know which plan will be chosen
during planning on foreign node and stay the same behavior is legal for me.
> am not comforted by the fact that the patch changed exactly nothing
> in the pathnodes.h documentation of that field. This comment looks
> pretty silly now too:
>
> /* Preprocess regular GROUP BY clause, if any */
> root->processed_groupClause = list_copy(parse->groupClause);
>
> What "preprocessing" is going on there? This comment was adequate
> before, when the code line invoked preprocess_groupclause which had
> a bunch of commentary; but now it has to stand on its own and it's
> not doing a great job of that.
Thanks for picking this place. I fixed it.
More interesting here is that we move decisions on the order of group-by
clauses to the stage, where we already have underlying subtree and
incoming path keys. In principle, we could return the preprocessing
routine and arrange GROUP-BY with the ORDER-BY clause as it was before.
But looking into the code, I found only one reason to do this:
adjust_group_pathkeys_for_groupagg. Curious about how much profit we get
here, I think we can discover it later with no hurry. A good outcome
here will be a test case that can show the importance of arranging
GROUP-BY and ORDER-BY at an early stage.
--
regards,
Andrei Lepikhov
Postgres Professional
Attachments:
[text/x-patch] minor_comment.patch (656B, ../../[email protected]/2-minor_comment.patch)
download | inline diff:
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 5ea3705796..861656a192 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -1424,7 +1424,10 @@ grouping_planner(PlannerInfo *root, double tuple_fraction,
}
else if (parse->groupClause)
{
- /* Preprocess regular GROUP BY clause, if any */
+ /*
+ * Make a copy of origin groupClause because we are going to
+ * remove redundant clauses.
+ */
root->processed_groupClause = list_copy(parse->groupClause);
/* Remove any redundant GROUP BY columns */
remove_useless_groupby_columns(root);
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-18 08:54 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
5 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-04-18 08:54 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 4/12/24 06:44, Tom Lane wrote:
> * Speaking of pathnodes.h, PathKeyInfo is a pretty uninformative node
> type name, and the comments provided for it are not going to educate
> anybody. What is the "association" between the pathkeys and clauses?
> I guess the clauses are supposed to be SortGroupClauses, but not all
> pathkeys match up to SortGroupClauses, so what then? This is very
> underspecified, and fuzzy thinking about data structures tends to lead
> to bugs.
I'm not the best in English documentation and naming style. So, feel
free to check my version.
>
> So I'm quite afraid that there are still bugs lurking here.
> What's more, given that the plans this patch makes apparently
> seldom win when you don't put a thumb on the scales, it might
> take a very long time to isolate those bugs. If the patch
> produces a faulty plan (e.g. not sorted properly) we'd never
> notice if that plan isn't chosen, and even if it is chosen
> it probably wouldn't produce anything as obviously wrong as
> a crash.
I added checkings on the proper order of pathkeys and clauses.
If you really care about that, we should spend additional time and
rewrite the code to generate an order of clauses somewhere in the plan
creation phase. For example, during the create_group_plan call, we could
use the processed_groupClause list, cycle through subpath->pathkeys, set
the order, and detect whether the pathkeys list corresponds to the
group-by or is enough to build a grouping plan.
Anyway, make this part of code more resistant to blunders is another story.
--
regards,
Andrei Lepikhov
Postgres Professional
Attachments:
[text/x-patch] final_improvements.patch (7.0K, ../../[email protected]/2-final_improvements.patch)
download | inline diff:
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index aa80f6486c..9c079270ec 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -461,7 +461,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* pathkeys_are_duplicate
* Check if give pathkeys are already contained the list of
- * PathKeyInfo's.
+ * GroupByOrdering's.
*/
static bool
pathkeys_are_duplicate(List *infos, List *pathkeys)
@@ -470,7 +470,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
foreach(lc, infos)
{
- PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+ GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
return true;
@@ -482,7 +482,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
*
- * Returns a list of PathKeyInfo items, each representing an interesting
+ * Returns a list of GroupByOrdering items, each representing an interesting
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
@@ -495,15 +495,15 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
List *
get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
{
- Query *parse = root->parse;
- List *infos = NIL;
- PathKeyInfo *info;
+ Query *parse = root->parse;
+ List *infos = NIL;
+ GroupByOrdering *info;
List *pathkeys = root->group_pathkeys;
List *clauses = root->processed_groupClause;
/* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
infos = lappend(infos, info);
@@ -539,7 +539,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -564,7 +564,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -574,18 +574,29 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
#ifdef USE_ASSERT_CHECKING
{
- PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ GroupByOrdering *pinfo = linitial_node(GroupByOrdering, infos);
ListCell *lc;
/* Test consistency of info structures */
for_each_from(lc, infos, 1)
{
- info = lfirst_node(PathKeyInfo, lc);
+ ListCell *lc1, *lc2;
+
+ info = lfirst_node(GroupByOrdering, lc);
Assert(list_length(info->clauses) == list_length(pinfo->clauses));
Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
Assert(list_difference(info->clauses, pinfo->clauses) == NIL);
Assert(list_difference_ptr(info->pathkeys, pinfo->pathkeys) == NIL);
+
+ forboth(lc1, info->clauses, lc2, info->pathkeys)
+ {
+ SortGroupClause *sgc = lfirst_node(SortGroupClause, lc1);
+ PathKey *pk = lfirst_node(PathKey, lc2);
+
+ if (pk->pk_eclass->ec_sortref != sgc->tleSortGroupRef)
+ elog(ERROR, "Order of group-by clauses doesn't correspond incoming sort order");
+ }
}
}
#endif
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 861656a192..4751a700ab 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6882,7 +6882,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -6963,7 +6963,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7214,7 +7214,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7270,7 +7270,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 6c71098f2d..b3862500b6 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -427,6 +427,10 @@ struct PlannerInfo
* containing all the query's rows. Hence, if you want to check whether
* GROUP BY was specified, test for nonempty parse->groupClause, not for
* nonempty processed_groupClause.
+ * Optimiser chooses specific order of group-by clauses during the upper
+ * paths generation process, attempting to use different strategies to
+ * minimize number of sorts or engage incremental sort.
+ * See get_useful_group_keys_orderings for details.
*
* Currently, when grouping sets are specified we do not attempt to
* optimize the groupClause, so that processed_groupClause will be
@@ -1468,14 +1472,20 @@ typedef struct PathKey
} PathKey;
/*
- * Combines the information about pathkeys and the associated clauses.
+ * Contains an order of group-by clauses and corresponding list of pathkeys.
+ *
+ * Order of SortGroupClause elements must correspond the order in the head of
+ * PathKey list:
+ * tleSortGroupRef of N-th element in the clauses must be the same as the value
+ * of ec_sortref in N-th pathkey equivalence class.
*/
-typedef struct PathKeyInfo
+typedef struct GroupByOrdering
{
NodeTag type;
+
List *pathkeys;
List *clauses;
-} PathKeyInfo;
+} GroupByOrdering;
/*
* VolatileFunctionStatus -- allows nodes to cache their
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d551ada325..33eff1e63a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4113,7 +4113,7 @@ manifest_writer
rfile
ws_options
ws_file_info
-PathKeyInfo
+GroupByOrdering
TidStore
TidStoreIter
TidStoreIterResult
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-18 10:57 Alexander Korotkov <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2024-04-18 10:57 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi, Andrei!
On Thu, Apr 18, 2024 at 11:54 AM Andrei Lepikhov
<[email protected]> wrote:
> On 4/12/24 06:44, Tom Lane wrote:
> > * Speaking of pathnodes.h, PathKeyInfo is a pretty uninformative node
> > type name, and the comments provided for it are not going to educate
> > anybody. What is the "association" between the pathkeys and clauses?
> > I guess the clauses are supposed to be SortGroupClauses, but not all
> > pathkeys match up to SortGroupClauses, so what then? This is very
> > underspecified, and fuzzy thinking about data structures tends to lead
> > to bugs.
> I'm not the best in English documentation and naming style. So, feel
> free to check my version.
> >
> > So I'm quite afraid that there are still bugs lurking here.
> > What's more, given that the plans this patch makes apparently
> > seldom win when you don't put a thumb on the scales, it might
> > take a very long time to isolate those bugs. If the patch
> > produces a faulty plan (e.g. not sorted properly) we'd never
> > notice if that plan isn't chosen, and even if it is chosen
> > it probably wouldn't produce anything as obviously wrong as
> > a crash.
> I added checkings on the proper order of pathkeys and clauses.
> If you really care about that, we should spend additional time and
> rewrite the code to generate an order of clauses somewhere in the plan
> creation phase. For example, during the create_group_plan call, we could
> use the processed_groupClause list, cycle through subpath->pathkeys, set
> the order, and detect whether the pathkeys list corresponds to the
> group-by or is enough to build a grouping plan.
> Anyway, make this part of code more resistant to blunders is another story.
Thank you for the fixes you've proposed. I didn't look much into
details yet, but I think the main concern Tom expressed in [1] is
whether the feature is reasonable at all. I think at this stage the
most important thing is to come up with convincing examples showing
how huge performance benefits it could cause. I will return to this
later today and will try to provide some convincing examples.
Links
1. https://www.postgresql.org/message-id/266850.1712879082%40sss.pgh.pa.us
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-19 10:44 jian he <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 67+ messages in thread
From: jian he @ 2024-04-19 10:44 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Thu, Apr 18, 2024 at 6:58 PM Alexander Korotkov <[email protected]> wrote:
>
> Thank you for the fixes you've proposed. I didn't look much into
> details yet, but I think the main concern Tom expressed in [1] is
> whether the feature is reasonable at all. I think at this stage the
> most important thing is to come up with convincing examples showing
> how huge performance benefits it could cause. I will return to this
> later today and will try to provide some convincing examples.
>
hi.
I found a case where it improved performance.
+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+
I change
+FROM generate_series(1,10000) AS i;
to
+ FROM generate_series(1, 1e6) AS i;
Then I found out about these 2 queries performance improved a lot.
A: explain(analyze) SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER
BY y, x \watch i=0.1 c=10
B: explain(analyze) SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER
BY y, x, z, w \watch i=0.1 c=10
set (enable_seqscan,enable_hashagg) from on to off:
queryA execution time from 1533.013 ms to 533.430 ms
queryB execution time from 1996.817 ms to 497.020 ms
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-22 03:51 Andrei Lepikhov <[email protected]>
parent: Tom Lane <[email protected]>
5 siblings, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-04-22 03:51 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 4/12/24 06:44, Tom Lane wrote:
> If this patch were producing better results I'd be more excited
> about putting more work into it. But on the basis of what I'm
> seeing right now, I think maybe we ought to give up on it.
Let me show current cases where users have a profit with this tiny
improvement (see queries and execution results in query.sql):
1. 'Not optimised query text' — when we didn't consider group-by
ordering during database development.
2. 'Accidental pathkeys' - we didn't see any explicit orderings, but
accidentally, the planner used merge join that caused some orderings and
we can utilise it.
3. 'Uncertain scan path' — We have options regarding which index to use,
and because of that, we can't predict the optimal group-by ordering
before the start of query planning.
4. 'HashAgg V/S GroupAgg' — sometimes, the GroupAgg strategy outperforms
HashAgg just because we don't need any ordering procedure at all.
And the last thing here — this code introduces the basics needed to add
more sophisticated strategies, like ordering according to uniqueness or
preferring to set constant-width columns first in grouping.
--
regards,
Andrei Lepikhov
Postgres Professional
Attachments:
[application/sql] query.sql (10.4K, ../../[email protected]/2-query.sql)
download
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-22 08:56 jian he <[email protected]>
parent: jian he <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: jian he @ 2024-04-22 08:56 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Fri, Apr 19, 2024 at 6:44 PM jian he <[email protected]> wrote:
>
> On Thu, Apr 18, 2024 at 6:58 PM Alexander Korotkov <[email protected]> wrote:
> >
> > Thank you for the fixes you've proposed. I didn't look much into
> > details yet, but I think the main concern Tom expressed in [1] is
> > whether the feature is reasonable at all. I think at this stage the
> > most important thing is to come up with convincing examples showing
> > how huge performance benefits it could cause. I will return to this
> > later today and will try to provide some convincing examples.
> >
hi.
previously preprocess_groupclause will not process cases
where no ORDER BY clause is specified.
commit 0452b461b will reorder the GROUP BY element even though no
ORDER BY clause is specified
, if there are associated indexes on it.
(hope I understand it correctly).
for example (when enable_hashagg is false)
explain(verbose) select count(*) FROM btg GROUP BY y,x;
in pg16 will not reorder, it will be as is: `GROUP BY y,x`
after commit 0452b461b, it will reorder to `GROUP BY x,y`.
because there is an index `btree (x, y)` (only one) associated with it.
if you drop the index `btree (x, y)` , it will be `GROUP BY y,x` as pg16.
This reordering GROUP BY element when no ORDER BY clause is not specified
is performant useful when the work_mem is small.
I've attached some tests comparing master with REL_16_STABLE to
demonstrate that.
all the tests attached are under the condition:
work_mem='64kB', buildtype=release, max_parallel_workers_per_gather=0.
one example:
CREATE TABLE btg5 AS
SELECT i::numeric % 10 AS x, i % 10 AS y, 'abc' || i % 10 AS z, i % 100000 AS w
FROM generate_series(1, 1e6) AS i;
CREATE INDEX btg5_x_y_idx ON btg5(x, y);
explain(analyze) SELECT count(*) FROM btg5 GROUP BY z, y, w, x;
in pg17, the execution time is: 746.574 ms
in pg16, the execution time is: 1693.483 ms
if I reorder it manually as:
`explain(analyze) SELECT count(*) FROM btg5 GROUP BY x, y, w, z;`
then in pg16, the execution time is 630.394 ms
Attachments:
[application/sql] low_mem_groupby_reorder.sql (8.1K, ../../CACJufxFZ8vLd6_M0krR8xG8vBNKGSYiLXPMpmK1c60wXzbsTJA@mail.gmail.com/2-low_mem_groupby_reorder.sql)
download
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-22 22:19 Alexander Korotkov <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2024-04-22 22:19 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi!
On Thu, Apr 18, 2024 at 1:57 PM Alexander Korotkov <[email protected]> wrote:
> Thank you for the fixes you've proposed. I didn't look much into
> details yet, but I think the main concern Tom expressed in [1] is
> whether the feature is reasonable at all. I think at this stage the
> most important thing is to come up with convincing examples showing
> how huge performance benefits it could cause. I will return to this
> later today and will try to provide some convincing examples.
I took a fresh look at 0452b461b, and have the following thoughts:
1) Previously, preprocess_groupclause() tries to reorder GROUP BY
clauses to match the required ORDER BY order. It only reorders if
GROUP BY pathkeys are the prefix of ORDER BY pathkeys or vice versa.
So, both of them need to be satisfied by one ordering. 0452b461b also
tries to match GROUP BY clauses to ORDER BY clauses, but takes into
account an incremental sort. Actually, instead of removing
preprocess_groupclause(), we could just teach it to take incremental
sort into account.
2) The get_useful_group_keys_orderings() function takes into account 3
orderings of pathkeys and clauses: original order as written in GROUP
BY, matching ORDER BY clauses as much as possible, and matching the
input path as much as possible. Given that even before 0452b461b,
preprocess_groupclause() could change the original order of GROUP BY
clauses, so we don't need to distinguish the first two. We could just
consider output of new preprocess_groupclause() taking into account an
incremental sort and the ordering matching the input path as much as
possible. This seems like significant simplification.
Let me also describe my thoughts about the justification of the
feature itself. As Tom pointed upthread, Sort + Grouping is generally
unlikely faster than Hash Aggregate. The main point of this feature
is being able to match the order of GROUP BY clauses to the order of
the input path. That input path could be Index Scan or Subquery.
Let's concentrate on Index Scan. Index Scan can give us the required
order, so we can do grouping without Sort or with significantly
cheaper Incremental Sort. That could be much faster than Hash
Aggregate. But if we scan the full table (or its significant
fraction), Index Scan is typically much more expensive than Sequential
Scan because of random page access. However, there are cases when
Index Scan could be cheaper.
1) If the heap row is wide and the index contains all the required
columns, Index Only Scan can be cheaper than Sequential Scan because
of lesser volume.
2) If the query predicate is selective enough and matches the index,
Index Scan might be significantly cheaper. One may say that if the
query predicate is selective enough then there are not that many
matching rows, so aggregating them either way isn't a big deal anyway.
However, given that data volumes are growing tremendously, it's not
hard to imagine that the index selected a small fraction of table
rows, but they are still enough to be costly for aggregating.
Therefore, I think there are significant cases where matching GROUP BY
clauses to the order of the input path could give a substantial
improvement over Hash Aggregate.
While there are some particular use-cases by Jian He, I hope that
above could give some rationale.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-24 06:25 jian he <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 3 replies; 67+ messages in thread
From: jian he @ 2024-04-24 06:25 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
hi.
I found an interesting case.
CREATE TABLE t1 AS
SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
z, i::int4 AS w
FROM generate_series(1, 100) AS i;
CREATE INDEX t1_x_y_idx ON t1 (x, y);
ANALYZE t1;
SET enable_hashagg = off;
SET enable_seqscan = off;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,y,z;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,w,y;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,z,y;
the above part will use:
-> Incremental Sort
Sort Key: x, $, $, $
Presorted Key: x
-> Index Scan using t1_x_y_idx on t1
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY z,y,w,x;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY w,y,z,x;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,z,x,w;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,w,x,z;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,x,z,w;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,x,w,z;
these will use:
-> Incremental Sort
Sort Key: x, y, $, $
Presorted Key: x, y
-> Index Scan using t1_x_y_idx on t1
I guess this is fine, but not optimal?
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-24 11:17 jian he <[email protected]>
parent: jian he <[email protected]>
2 siblings, 0 replies; 67+ messages in thread
From: jian he @ 2024-04-24 11:17 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
hi
one more question (maybe a dumb one....)
drop table if exists t1;
CREATE TABLE t1 AS
SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
z, i::int4 AS w
FROM generate_series(1, 100) AS i;
CREATE INDEX t1_x_y_idx ON t1 (x, y);
ANALYZE t1;
SET enable_hashagg = off;
SET enable_seqscan = off;
EXPLAIN (COSTS OFF, verbose) SELECT count(*) FROM t1 GROUP BY z,x,y,w
order by w;
QUERY PLAN
------------------------------------------------------
GroupAggregate
Output: count(*), w, z, x, y
Group Key: t1.w, t1.x, t1.y, t1.z
-> Sort
Output: w, z, x, y
Sort Key: t1.w, t1.x, t1.y, t1.z
-> Index Scan using t1_x_y_idx on public.t1
Output: w, z, x, y
(8 rows)
if you do
` Sort Key: t1.w, t1.x, t1.y, t1.z`
then the output is supposed to be:
Output: w, x, y, z
?
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-04-29 02:50 jian he <[email protected]>
parent: jian he <[email protected]>
2 siblings, 0 replies; 67+ messages in thread
From: jian he @ 2024-04-29 02:50 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Wed, Apr 24, 2024 at 2:25 PM jian he <[email protected]> wrote:
>
> hi.
> I found an interesting case.
>
> CREATE TABLE t1 AS
> SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
> z, i::int4 AS w
> FROM generate_series(1, 100) AS i;
> CREATE INDEX t1_x_y_idx ON t1 (x, y);
> ANALYZE t1;
> SET enable_hashagg = off;
> SET enable_seqscan = off;
>
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,y,z;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,w,y;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,z,y;
> the above part will use:
> -> Incremental Sort
> Sort Key: x, $, $, $
> Presorted Key: x
> -> Index Scan using t1_x_y_idx on t1
We can make these cases also `Presorted Key: x, y`.
in
`if (path->pathkeys && !pathkeys_contained_in(path->pathkeys,
root->group_pathkeys))` branch
we can simple do
- infos = lappend(infos, info);
+ infos = lcons(info, infos);
similar to what we did at plancat.c (search lcons).
get_useful_group_keys_orderings returns a list of PathKeyInfo,
then the caller function just iterates each element.
so for the caller, order of the returned list element from
get_useful_group_keys_orderings
does not matter.
for path Incremental Sort:
function make_ordered_path will return the same cost for different
numbers of presorted keys.
for example:
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
make_ordered_path cost is same for:
`info->pathkeys: x,y,z,w`
`info->pathkeys:x,z,y,w`
if we arrange `info->pathkeys: x,y,z,w` before `info->pathkeys:x,z,y,w`
in get_useful_group_keys_orderings.
then with the same cost, we will choose the first one
(`info->pathkeys: x,y,z,w`),
if we use IncrementalSort, then we use `Presorted Key: x, y`.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-16 07:47 Andrei Lepikhov <[email protected]>
parent: jian he <[email protected]>
2 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-05-16 07:47 UTC (permalink / raw)
To: jian he <[email protected]>; Alexander Korotkov <[email protected]>; +Cc: Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 24.04.2024 13:25, jian he wrote:
> hi.
> I found an interesting case.
>
> CREATE TABLE t1 AS
> SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
> z, i::int4 AS w
> FROM generate_series(1, 100) AS i;
> CREATE INDEX t1_x_y_idx ON t1 (x, y);
> ANALYZE t1;
> SET enable_hashagg = off;
> SET enable_seqscan = off;
>
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,y,z;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,w,y;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,z,y;
> the above part will use:
> -> Incremental Sort
> Sort Key: x, $, $, $
> Presorted Key: x
> -> Index Scan using t1_x_y_idx on t1
>
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY z,y,w,x;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY w,y,z,x;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,z,x,w;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,w,x,z;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,x,z,w;
> EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,x,w,z;
>
> these will use:
> -> Incremental Sort
> Sort Key: x, y, $, $
> Presorted Key: x, y
> -> Index Scan using t1_x_y_idx on t1
>
> I guess this is fine, but not optimal?
It looks like a bug right now - in current implementation we don't
differentiate different orders. So:
1. Applying all the patches from the thread which I proposed as an
answer to T.Lane last rebuke - does behavior still the same?.
2. Could you try to find the reason?
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-20 08:54 jian he <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: jian he @ 2024-05-20 08:54 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Thu, May 16, 2024 at 3:47 PM Andrei Lepikhov
<[email protected]> wrote:
>
> On 24.04.2024 13:25, jian he wrote:
> > hi.
> > I found an interesting case.
> >
> > CREATE TABLE t1 AS
> > SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
> > z, i::int4 AS w
> > FROM generate_series(1, 100) AS i;
> > CREATE INDEX t1_x_y_idx ON t1 (x, y);
> > ANALYZE t1;
> > SET enable_hashagg = off;
> > SET enable_seqscan = off;
> >
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,y,z;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,w,y;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,z,y;
> > the above part will use:
> > -> Incremental Sort
> > Sort Key: x, $, $, $
> > Presorted Key: x
> > -> Index Scan using t1_x_y_idx on t1
> >
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY z,y,w,x;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY w,y,z,x;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,z,x,w;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,w,x,z;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,x,z,w;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY y,x,w,z;
> >
> > these will use:
> > -> Incremental Sort
> > Sort Key: x, y, $, $
> > Presorted Key: x, y
> > -> Index Scan using t1_x_y_idx on t1
> >
> > I guess this is fine, but not optimal?
> It looks like a bug right now - in current implementation we don't
> differentiate different orders. So:
> 1. Applying all the patches from the thread which I proposed as an
> answer to T.Lane last rebuke - does behavior still the same?.
I've applied these 4 patches in this thread.
final_improvements.patch
minor_comment.patch
get_useful_group_keys_orderings.patch
group_by_asymmetry.diff
The behavior is still the same as the master.
meaning that below quoted queries are still using "Presorted Key: x".
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,y,z;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,w,y;
> > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,z,y;
> > the above part will use:
> > -> Incremental Sort
> > Sort Key: x, $, $, $
> > Presorted Key: x
> > -> Index Scan using t1_x_y_idx on t1
> 2. Could you try to find the reason?
the following are my understanding, it may be wrong...
function get_useful_group_keys_orderings, may generate alternative
pathkeys and group clauses.
for different PathKeyInfo
sub functions within add_paths_to_grouping_rel:
make_ordered_path, create_agg_path will yield the same cost.
function add_path (within add_paths_to_grouping_rel) will add these
paths (different PathKeyInfos) in a *sequential* order.
then later in the function set_cheapest.
`foreach(p, parent_rel->pathlist)` will iterate these different paths
in a sequential order.
so in set_cheapest if 2 path costs are the same, then the first path
residing in parent_rel->pathlist wins.
fo a concrete example:
CREATE TABLE t1 AS
SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
z, i::float4 AS w FROM generate_series(1, 100) AS i;
CREATE INDEX t1_x_y_idx ON t1 (x, y);
ANALYZE t1;
SET enable_hashagg = off;
SET enable_seqscan = off;
EXPLAIN (COSTS OFF, verbose) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
add_paths_to_grouping_rel will do the following in a sequential order.
A. generate and add original PathKeyInfo(groupby x,z,y,w)
B. generate and add alternative PathKeyInfo(groupby x,y,z,w). (because
of the index path)
C. can_hash is true, generate a HashAgg Path (because of
enable_hashagg is set to off, so this path the cost is very high)
As mentioned previously,
both A and B can use Incremental Sort, set_cheapest will choose the A
instead of B,
because A step generated path **first** satisfies increment sort.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-21 09:01 Andrei Lepikhov <[email protected]>
parent: jian he <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Andrei Lepikhov @ 2024-05-21 09:01 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 20/5/2024 15:54, jian he wrote:
> As mentioned previously,
> both A and B can use Incremental Sort, set_cheapest will choose the A
> instead of B,
> because A step generated path **first** satisfies increment sort.
Yeah, I agree with your analysis.
Looking into the cost_incremental_sort, I see that we have ten
group_tuples. This value doesn't depend on how many presorted keys (1 or
2) we use. This is caused by default estimation.
Given the current circumstances, it seems unlikely that we can make any
significant changes without introducing a new sort cost model that
accounts for the number of sorted columns.
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-24 00:00 jian he <[email protected]>
parent: jian he <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: jian he @ 2024-05-24 00:00 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Mon, May 20, 2024 at 4:54 PM jian he <[email protected]> wrote:
>
>
> The behavior is still the same as the master.
> meaning that below quoted queries are still using "Presorted Key: x".
>
> > > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
> > > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,y,z;
> > > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,w,y;
> > > EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,w,z,y;
> > > the above part will use:
> > > -> Incremental Sort
> > > Sort Key: x, $, $, $
> > > Presorted Key: x
> > > -> Index Scan using t1_x_y_idx on t1
>
>
> > 2. Could you try to find the reason?
> the following are my understanding, it may be wrong...
>
I think
my previous thread only explained that two paths have the same cost,
the planner chooses the one that was first added into the pathlist.
but didn't explain why Incremental Sort path, presorted by one key and
presorted by two keys
yield the same cost.
--------------------------------------------
if (rel->tuples > 0)
{
/*
* Clamp to size of rel, or size of rel / 10 if multiple Vars. The
* fudge factor is because the Vars are probably correlated but we
* don't know by how much. We should never clamp to less than the
* largest ndistinct value for any of the Vars, though, since
* there will surely be at least that many groups.
*/
double clamp = rel->tuples;
if (relvarcount > 1)
{
clamp *= 0.1;
if (clamp < relmaxndistinct)
{
clamp = relmaxndistinct;
/* for sanity in case some ndistinct is too large: */
if (clamp > rel->tuples)
clamp = rel->tuples;
}
}
if (reldistinct > clamp)
reldistinct = clamp;
...
}
i think,
the above code[0] snippet within function estimate_num_groups
makes Incremental Sort Path not pickup the optimal presorted keys.
see original problem thread: [1]
----------------------------------------------------------------------------------------
CREATE TABLE t1 AS
SELECT (i % 10)::numeric AS x,(i % 10)::int8 AS y,'abc' || i % 10 AS
z, i::int4 AS w
FROM generate_series(1, 100) AS i;
CREATE INDEX t1_x_y_idx ON t1 (x, y);
ANALYZE t1;
SET enable_hashagg = off;
SET enable_seqscan = off;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t1 GROUP BY x,z,y,w;
will generate 2 Incremental Sort path, one: "Presorted Key: x",
another one: "Presorted Key: x,y".
The first Incremental Sort path is added first.
The function estimate_num_groups returned value (input_groups) is the
main key to
calculate the cost of Incremental Sort path!
But here, the estimate_num_groups function returned the same
value: 10 for
"Presorted Key: x" and "Presorted Key: x,y".
then later cost_incremental_sort returns the same cost for "Presorted
Key: x" and "Presorted Key: x,y".
(line refers to src/backend/utils/adt/selfuncs line number).
why "Presorted Key: x,y" return 10 in estimate_num_groups:
line 3667 assign clamp to 100.0, because rel->tuples is 100.
line 3671 clamp *= 0.1; make clamp because 10.
line 3680, 3681 `if (reldistinct > clamp) branch` make reldistinct
from 100 to 10.
line 3733 make `numdistinct *= reldistinct;` makes numdistinct because 10.
If I change the total number of rows in a relation, or change the
distinct number of y values
then the planner will use "Incremental Sort Presorted Key: x,y".
for example:
CREATE TABLE t10 AS
SELECT (i % 10)::numeric AS x,(i % 11)::int8 AS y,'abc' || i % 10
AS z, i::float4 AS w FROM generate_series(1, 1E2) AS i;
CREATE INDEX t10_x_y_idx ON t10 (x, y);
ANALYZE t10;
The above setup will make the following query using "Incremental Sort
Presorted Key: x,y".
EXPLAIN (COSTS OFF) SELECT count(*) FROM t10 GROUP BY x,z,y,w;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t10 GROUP BY x,w,y,z;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t10 GROUP BY x,z,w,y;
EXPLAIN (COSTS OFF) SELECT count(*) FROM t10 GROUP BY x,w,z,y;
summary:
* the regression test setup at [2] can make some cases not pickup the
best optimal
Incremental Sort path.
we use this test in src/test/regress/sql/aggregates.sql
-- Engage incremental sort
EXPLAIN (COSTS OFF)
SELECT count(*) FROM btg GROUP BY z, y, w, x;
but if we use:
EXPLAIN (COSTS OFF)
SELECT count(*) FROM btg GROUP BY x, z, w, y;
then the plan is not the best.
* The regression test setup makes estimate_num_groups code logic
return the same value.
therefore make Incremental Sort presort by one key, two keys yield the
same cost.
the cost_incremental_sort will return the same cost.
* https://git.postgresql.org/cgit/postgresql.git/tree/src/test/regress/sql/aggregates.sql#n1194
maybe we can change:
CREATE TABLE btg AS SELECT
i % 10 AS x,
i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
FROM generate_series(1, 100) AS i;
to
CREATE TABLE btg AS SELECT
i % 10 AS x,
i % 10 AS y,
'abc' || i % 10 AS z,
i AS w
FROM generate_series(1, 1000) AS i;
[0] https://git.postgresql.org/cgit/postgresql.git/tree/src/backend/utils/adt/selfuncs.c#n3669
[1] https://www.postgresql.org/message-id/CACJufxGt99nZ%2Bnir%2BaB6pFQ%3DK8oNiHAQ3OELqSbGMqNxok8nxA%40ma...
[2] https://git.postgresql.org/cgit/postgresql.git/tree/src/test/regress/sql/aggregates.sql#n1194
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-27 12:41 Alexander Korotkov <[email protected]>
parent: Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-05-27 12:41 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi!
On Tue, Apr 23, 2024 at 1:19 AM Alexander Korotkov <[email protected]> wrote:
> On Thu, Apr 18, 2024 at 1:57 PM Alexander Korotkov <[email protected]> wrote:
> > Thank you for the fixes you've proposed. I didn't look much into
> > details yet, but I think the main concern Tom expressed in [1] is
> > whether the feature is reasonable at all. I think at this stage the
> > most important thing is to come up with convincing examples showing
> > how huge performance benefits it could cause. I will return to this
> > later today and will try to provide some convincing examples.
>
> I took a fresh look at 0452b461b, and have the following thoughts:
> 1) Previously, preprocess_groupclause() tries to reorder GROUP BY
> clauses to match the required ORDER BY order. It only reorders if
> GROUP BY pathkeys are the prefix of ORDER BY pathkeys or vice versa.
> So, both of them need to be satisfied by one ordering. 0452b461b also
> tries to match GROUP BY clauses to ORDER BY clauses, but takes into
> account an incremental sort. Actually, instead of removing
> preprocess_groupclause(), we could just teach it to take incremental
> sort into account.
> 2) The get_useful_group_keys_orderings() function takes into account 3
> orderings of pathkeys and clauses: original order as written in GROUP
> BY, matching ORDER BY clauses as much as possible, and matching the
> input path as much as possible. Given that even before 0452b461b,
> preprocess_groupclause() could change the original order of GROUP BY
> clauses, so we don't need to distinguish the first two. We could just
> consider output of new preprocess_groupclause() taking into account an
> incremental sort and the ordering matching the input path as much as
> possible. This seems like significant simplification.
>
> Let me also describe my thoughts about the justification of the
> feature itself. As Tom pointed upthread, Sort + Grouping is generally
> unlikely faster than Hash Aggregate. The main point of this feature
> is being able to match the order of GROUP BY clauses to the order of
> the input path. That input path could be Index Scan or Subquery.
> Let's concentrate on Index Scan. Index Scan can give us the required
> order, so we can do grouping without Sort or with significantly
> cheaper Incremental Sort. That could be much faster than Hash
> Aggregate. But if we scan the full table (or its significant
> fraction), Index Scan is typically much more expensive than Sequential
> Scan because of random page access. However, there are cases when
> Index Scan could be cheaper.
> 1) If the heap row is wide and the index contains all the required
> columns, Index Only Scan can be cheaper than Sequential Scan because
> of lesser volume.
> 2) If the query predicate is selective enough and matches the index,
> Index Scan might be significantly cheaper. One may say that if the
> query predicate is selective enough then there are not that many
> matching rows, so aggregating them either way isn't a big deal anyway.
> However, given that data volumes are growing tremendously, it's not
> hard to imagine that the index selected a small fraction of table
> rows, but they are still enough to be costly for aggregating.
> Therefore, I think there are significant cases where matching GROUP BY
> clauses to the order of the input path could give a substantial
> improvement over Hash Aggregate.
>
> While there are some particular use-cases by Jian He, I hope that
> above could give some rationale.
I've assembled patches in this thread into one patchset.
0001 The patch fixing asymmetry in setting EquivalenceClass.ec_sortref
by Andrei [1]. I've revised comments and wrote the commit message.
0002 The patch for handling duplicates of SortGroupClause. I didn't
get the sense of Andrei implementation. It seems to care about
duplicate pointers in group clauses list. But the question is the
equal SortGroupClause's comprising different pointers. I think we
should group duplicate SortGroupClause's together as
preprocess_groupclause() used to do. Reimplemented patch to do so.
0003 Rename PathKeyInfo to GroupByOrdering by Andres [3]. I only
revised comments and wrote the commit message.
0004 Turn back preprocess_groupclause() for the reason I described upthread [4].
Any thoughts?
Links.
1. https://www.postgresql.org/message-id/17037754-f187-4138-8285-0e2bfebd0dea%40postgrespro.ru
2. https://www.postgresql.org/message-id/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
3. https://www.postgresql.org/message-id/db0fc3a4-966c-4cec-a136-94024d39212d%40postgrespro.ru
4. https://www.postgresql.org/message-id/CAPpHfdvyWLMGwvxaf%3D7KAp-z-4mxbSH8ti2f6mNOQv5metZFzg%40mail.g...
------
Regards,
Alexander Korotkov
Supabase
Attachments:
[application/octet-stream] v2-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch (7.0K, ../../CAPpHfdvbxK3nszdk6YXWpV7p+P+pGOQhpKTxJ3xME3uD+qZmXg@mail.gmail.com/2-v2-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch)
download | inline diff:
From 67c4ab916acf94a136179107f6d8a84834758445 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Mon, 27 May 2024 14:57:43 +0300
Subject: [PATCH v2 3/4] Rename PathKeyInfo to GroupByOrdering
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
The PathKeyInfo data structure was used to store the particular ordering of
group-by pathkeys and corresponding clauses. It turns out that PathKeyInfo
is not the best name for that purpose. This commit renames this data structure
to GroupByOrdering, and revises its comment.
Discussion: https://postgr.es/m/db0fc3a4-966c-4cec-a136-94024d39212d%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 18 +++++++++---------
src/backend/optimizer/plan/planner.c | 8 ++++----
src/include/nodes/pathnodes.h | 13 ++++++++++---
src/tools/pgindent/typedefs.list | 2 +-
4 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e8ff2297697..85bfa080680 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -462,7 +462,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* pathkeys_are_duplicate
* Check if give pathkeys are already contained the list of
- * PathKeyInfo's.
+ * GroupByOrdering's.
*/
static bool
pathkeys_are_duplicate(List *infos, List *pathkeys)
@@ -471,7 +471,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
foreach(lc, infos)
{
- PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+ GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
return true;
@@ -483,7 +483,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
*
- * Returns a list of PathKeyInfo items, each representing an interesting
+ * Returns a list of GroupByOrdering items, each representing an interesting
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
@@ -498,13 +498,13 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
{
Query *parse = root->parse;
List *infos = NIL;
- PathKeyInfo *info;
+ GroupByOrdering *info;
List *pathkeys = root->group_pathkeys;
List *clauses = root->processed_groupClause;
/* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
infos = lappend(infos, info);
@@ -540,7 +540,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -565,7 +565,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -575,7 +575,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
#ifdef USE_ASSERT_CHECKING
{
- PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ GroupByOrdering *pinfo = linitial_node(GroupByOrdering, infos);
ListCell *lc;
/* Test consistency of info structures */
@@ -584,7 +584,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
ListCell *lc1,
*lc2;
- info = lfirst_node(PathKeyInfo, lc);
+ info = lfirst_node(GroupByOrdering, lc);
Assert(list_length(info->clauses) == list_length(pinfo->clauses));
Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index e723f72db64..5739275aa70 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6917,7 +6917,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -6998,7 +6998,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7249,7 +7249,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7305,7 +7305,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 14ef296ab72..78489398294 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1468,14 +1468,21 @@ typedef struct PathKey
} PathKey;
/*
- * Combines the information about pathkeys and the associated clauses.
+ * Contains an order of group-by clauses and the corresponding list of
+ * pathkeys.
+ *
+ * The elements of 'clauses' list should have the same order as the head of
+ * 'pathkeys' list. The tleSortGroupRef of the clause should be equal to
+ * ec_sortref of the pathkey equivalence class. If there are redundant
+ * clauses with the same tleSortGroupRef, they must be grouped together.
*/
-typedef struct PathKeyInfo
+typedef struct GroupByOrdering
{
NodeTag type;
+
List *pathkeys;
List *clauses;
-} PathKeyInfo;
+} GroupByOrdering;
/*
* VolatileFunctionStatus -- allows nodes to cache their
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d427a1c16a5..4f57078d133 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1056,6 +1056,7 @@ GrantRoleStmt
GrantStmt
GrantTargetType
Group
+GroupByOrdering
GroupClause
GroupPath
GroupPathExtraData
@@ -2067,7 +2068,6 @@ PathClauseUsage
PathCostComparison
PathHashStack
PathKey
-PathKeyInfo
PathKeysComparison
PathTarget
PatternInfo
--
2.39.3 (Apple Git-145)
[application/octet-stream] v2-0004-Restore-preprocess_groupclause.patch (13.1K, ../../CAPpHfdvbxK3nszdk6YXWpV7p+P+pGOQhpKTxJ3xME3uD+qZmXg@mail.gmail.com/3-v2-0004-Restore-preprocess_groupclause.patch)
download | inline diff:
From e182d34d9128b9628bcb2c4b2df3868b65ae241b Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:51:31 +0300
Subject: [PATCH v2 4/4] Restore preprocess_groupclause()
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
It eliminated preprocess_groupclause(), which was intended to match items
between GROUP BY and ORDER BY. Instead, get_useful_group_keys_orderings()
function generates orderings of GROUP BY elements at the time of grouping
paths generation. The get_useful_group_keys_orderings() function takes into
account 3 orderings of GROUP BY pathkeys and clauses: original order as written
in GROUP BY, matching ORDER BY clauses as much as possible, and matching the
input path as much as possible. Given that even before 0452b461b,
preprocess_groupclause() could change the original order of GROUP BY clauses
we don't need to consider it apart from ordering matching ORDER BY clauses.
This commit restores preprocess_groupclause() to provide an ordering of
GROUP BY elements matching ORDER BY before generation of paths. The new
version of preprocess_groupclause() takes into account an incremental sort.
The get_useful_group_keys_orderings() function now takes into 2 orderings of
GROUP BY elements: the order generated preprocess_groupclause() and the order
matching the input path as much as possible.
Discussion: https://postgr.es/m/CAPpHfdvyWLMGwvxaf%3D7KAp-z-4mxbSH8ti2f6mNOQv5metZFzg%40mail.gmail.com
Author: Alexander Korotkov
---
src/backend/optimizer/path/pathkeys.c | 55 +--------
src/backend/optimizer/plan/planner.c | 108 +++++++++++++++---
src/include/nodes/pathnodes.h | 6 +-
.../regress/expected/partition_aggregate.out | 6 +-
4 files changed, 108 insertions(+), 67 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 85bfa080680..301a4b1ba99 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -459,26 +459,6 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
return n;
}
-/*
- * pathkeys_are_duplicate
- * Check if give pathkeys are already contained the list of
- * GroupByOrdering's.
- */
-static bool
-pathkeys_are_duplicate(List *infos, List *pathkeys)
-{
- ListCell *lc;
-
- foreach(lc, infos)
- {
- GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
-
- if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
- return true;
- }
- return false;
-}
-
/*
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
@@ -487,11 +467,11 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
- * The function considers (and keeps) multiple GROUP BY orderings:
+ * The function considers (and keeps) following GROUP BY orderings:
*
- * - the original ordering, as specified by the GROUP BY clause,
- * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
- * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ * - GROUP BY keys as ordered by preprocess_groupclause() to match target
+ * ORDER BY clause (as much as possible),
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible).
*/
List *
get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
@@ -538,32 +518,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
if (n > 0 &&
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
- !pathkeys_are_duplicate(infos, pathkeys))
- {
- info = makeNode(GroupByOrdering);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause).
- */
- if (root->sort_pathkeys &&
- !pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys))
- {
- int n;
-
- n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
- &clauses,
- root->num_groupby_pathkeys);
-
- if (n > 0 &&
- (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
- !pathkeys_are_duplicate(infos, pathkeys))
+ compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
{
info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 5739275aa70..c4306b9b9ba 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -137,7 +137,7 @@ static double preprocess_limit(PlannerInfo *root,
double tuple_fraction,
int64 *offset_est, int64 *count_est);
static void remove_useless_groupby_columns(PlannerInfo *root);
-static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
+static List *preprocess_groupclause(PlannerInfo *root, List *force);
static List *extract_rollup_sets(List *groupingSets);
static List *reorder_grouping_sets(List *groupingSets, List *sortclause);
static void standard_qp_callback(PlannerInfo *root, void *extra);
@@ -1422,7 +1422,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction,
else if (parse->groupClause)
{
/* Preprocess regular GROUP BY clause, if any */
- root->processed_groupClause = list_copy(parse->groupClause);
+ root->processed_groupClause = preprocess_groupclause(root, NIL);
/* Remove any redundant GROUP BY columns */
remove_useless_groupby_columns(root);
}
@@ -2169,7 +2169,7 @@ preprocess_grouping_sets(PlannerInfo *root)
* The groupClauses for hashed grouping sets are built later on.)
*/
if (gs->set)
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
else
rollup->groupClause = NIL;
@@ -2821,24 +2821,106 @@ remove_useless_groupby_columns(PlannerInfo *root)
}
/*
- * groupclause_apply_groupingset
- * Apply the order of GROUP BY clauses defined by grouping sets. Items
- * not in the grouping set are skipped.
+ * preprocess_groupclause - do preparatory work on GROUP BY clause
+ *
+ * The idea here is to adjust the ordering of the GROUP BY elements
+ * (which in itself is semantically insignificant) to match ORDER BY,
+ * thereby allowing a single sort operation to both implement the ORDER BY
+ * requirement and set up for a Unique step that implements GROUP BY.
+ * We also consider partial match between GROUP BY and ORDER BY elements,
+ * which could allow to implement ORDER BY using the incremental sort.
+ *
+ * We also consider other orderings of the GROUP BY elements, which could
+ * match the sort ordering of other possible plans (eg an indexscan) and
+ * thereby reduce cost. This is implemented during the generation of grouping
+ * paths. See get_useful_group_keys_orderings() for details.
+ *
+ * Note: we need no comparable processing of the distinctClause because
+ * the parser already enforced that that matches ORDER BY.
+ *
+ * Note: we return a fresh List, but its elements are the same
+ * SortGroupClauses appearing in parse->groupClause. This is important
+ * because later processing may modify the processed_groupClause list.
+ *
+ * For grouping sets, the order of items is instead forced to agree with that
+ * of the grouping set (and items not in the grouping set are skipped). The
+ * work of sorting the order of grouping set elements to match the ORDER BY if
+ * possible is done elsewhere.
*/
static List *
-groupclause_apply_groupingset(PlannerInfo *root, List *gset)
+preprocess_groupclause(PlannerInfo *root, List *force)
{
Query *parse = root->parse;
List *new_groupclause = NIL;
ListCell *sl;
+ ListCell *gl;
- foreach(sl, gset)
+ /* For grouping sets, we need to force the ordering */
+ if (force)
{
- Index ref = lfirst_int(sl);
- SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+ foreach(sl, force)
+ {
+ Index ref = lfirst_int(sl);
+ SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+
+ new_groupclause = lappend(new_groupclause, cl);
+ }
- new_groupclause = lappend(new_groupclause, cl);
+ return new_groupclause;
}
+
+ /* If no ORDER BY, nothing useful to do here */
+ if (parse->sortClause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Scan the ORDER BY clause and construct a list of matching GROUP BY
+ * items, but only as far as we can make a matching prefix.
+ *
+ * This code assumes that the sortClause contains no duplicate items.
+ */
+ foreach(sl, parse->sortClause)
+ {
+ SortGroupClause *sc = lfirst_node(SortGroupClause, sl);
+
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (equal(gc, sc))
+ {
+ new_groupclause = lappend(new_groupclause, gc);
+ break;
+ }
+ }
+ if (gl == NULL)
+ break; /* no match, so stop scanning */
+ }
+
+
+ /* If no match at all, no point in reordering GROUP BY */
+ if (new_groupclause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Add any remaining GROUP BY items to the new list. We don't require a
+ * complete match, because even partial match allows ORDER BY to be
+ * implemented using incremental sort. Also, give up if there are any
+ * non-sortable GROUP BY items, since then there's no hope anyway.
+ */
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (list_member_ptr(new_groupclause, gc))
+ continue; /* it matched an ORDER BY item */
+ if (!OidIsValid(gc->sortop)) /* give up, GROUP BY can't be sorted */
+ return list_copy(parse->groupClause);
+ new_groupclause = lappend(new_groupclause, gc);
+ }
+
+ /* Success --- install the rearranged GROUP BY list */
+ Assert(list_length(parse->groupClause) == list_length(new_groupclause));
return new_groupclause;
}
@@ -4170,7 +4252,7 @@ consider_groupingsets_paths(PlannerInfo *root,
{
rollup = makeNode(RollupData);
- rollup->groupClause = groupclause_apply_groupingset(root, gset);
+ rollup->groupClause = preprocess_groupclause(root, gset);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
@@ -4359,7 +4441,7 @@ consider_groupingsets_paths(PlannerInfo *root,
Assert(gs->set != NIL);
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 78489398294..af8de4213de 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -426,7 +426,11 @@ struct PlannerInfo
* items to be proven redundant, implying that there is only one group
* containing all the query's rows. Hence, if you want to check whether
* GROUP BY was specified, test for nonempty parse->groupClause, not for
- * nonempty processed_groupClause.
+ * nonempty processed_groupClause. Optimiser chooses specific order of
+ * group-by clauses during the upper paths generation process, attempting
+ * to use different strategies to minimize number of sorts or engage
+ * incremental sort. See preprocess_groupclause() and
+ * get_useful_group_keys_orderings() for details.
*
* Currently, when grouping sets are specified we do not attempt to
* optimize the groupClause, so that processed_groupClause will be
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index 1b900fddf8e..5f2c0cf5786 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -901,15 +901,15 @@ SELECT a, c, sum(b), avg(c), count(*) FROM pagg_tab_m GROUP BY (a+b)/2, 2, 1 HAV
Sort Key: pagg_tab_m.a, pagg_tab_m.c, (sum(pagg_tab_m.b))
-> Append
-> HashAggregate
- Group Key: ((pagg_tab_m.a + pagg_tab_m.b) / 2), pagg_tab_m.c, pagg_tab_m.a
+ Group Key: pagg_tab_m.a, pagg_tab_m.c, ((pagg_tab_m.a + pagg_tab_m.b) / 2)
Filter: ((sum(pagg_tab_m.b) = 50) AND (avg(pagg_tab_m.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p1 pagg_tab_m
-> HashAggregate
- Group Key: ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2), pagg_tab_m_1.c, pagg_tab_m_1.a
+ Group Key: pagg_tab_m_1.a, pagg_tab_m_1.c, ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2)
Filter: ((sum(pagg_tab_m_1.b) = 50) AND (avg(pagg_tab_m_1.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p2 pagg_tab_m_1
-> HashAggregate
- Group Key: ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2), pagg_tab_m_2.c, pagg_tab_m_2.a
+ Group Key: pagg_tab_m_2.a, pagg_tab_m_2.c, ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2)
Filter: ((sum(pagg_tab_m_2.b) = 50) AND (avg(pagg_tab_m_2.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p3 pagg_tab_m_2
(15 rows)
--
2.39.3 (Apple Git-145)
[application/octet-stream] v2-0002-Teach-group_keys_reorder_by_pathkeys-about-redund.patch (4.2K, ../../CAPpHfdvbxK3nszdk6YXWpV7p+P+pGOQhpKTxJ3xME3uD+qZmXg@mail.gmail.com/4-v2-0002-Teach-group_keys_reorder_by_pathkeys-about-redund.patch)
download | inline diff:
From b5ea90e50faf8a7a8fc2329a35da5f66fbe3d35b Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:13:11 +0300
Subject: [PATCH v2 2/4] Teach group_keys_reorder_by_pathkeys() about redundant
SortGroupClause's
We don't have strict guarantees that there shouldn't be redundant
SortGroupClause's. Although there is no confirmed way to generate
duplicate SortGroupClause's, this commit teaches
group_keys_reorder_by_pathkeys() to pull all the SortGroupClause matching to
pathkey at once.
This commit also introduces checking invariants of generated orderings
in get_useful_group_keys_orderings for assert-enabled builds.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 69 +++++++++++++++++++++------
1 file changed, 55 insertions(+), 14 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7858a56fae0..e8ff2297697 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -401,7 +401,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
+ bool found = false;
/*
* Pathkeys are built in a way that allows simply comparing pointers.
@@ -416,28 +416,40 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* Since 1349d27 pathkey coming from underlying node can be in the
* root->group_pathkeys but not in the processed_groupClause. So, we
- * should be careful here.
+ * should be careful here. Also, there could be redundant
+ * SortGroupClause's. So, we need to pull of all the matching
+ * SortGroupClause's, but ensure there is at least one.
*/
- sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
- if (!sgc)
- /* The grouping clause does not cover this pathkey */
- break;
+ foreach_ptr(SortGroupClause, sgc, *group_clauses)
+ {
+ if (sgc->tleSortGroupRef == pathkey->pk_eclass->ec_sortref)
+ {
+ /*
+ * Sort group clause should have an ordering operator as long
+ * as there is an associated pathkey.
+ */
+ Assert(OidIsValid(sgc->sortop));
- /*
- * Sort group clause should have an ordering operator as long as there
- * is an associated pathkey.
- */
- Assert(OidIsValid(sgc->sortop));
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ found = true;
+ }
+ }
+
+ /* Check if the grouping clause does not cover this pathkey */
+ if (!found)
+ break;
new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
- new_group_clauses = lappend(new_group_clauses, sgc);
+
}
/* remember the number of pathkeys with a matching GROUP BY key */
n = list_length(new_group_pathkeys);
- /* append the remaining group pathkeys (will be treated as not sorted) */
+ /*
+ * Append the remaining group pathkeys (will be treated as not sorted) and
+ * grouping clauses.
+ */
*group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
*group_pathkeys);
*group_clauses = list_concat_unique_ptr(new_group_clauses,
@@ -561,6 +573,35 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
+#ifdef USE_ASSERT_CHECKING
+ {
+ PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ ListCell *lc;
+
+ /* Test consistency of info structures */
+ for_each_from(lc, infos, 1)
+ {
+ ListCell *lc1,
+ *lc2;
+
+ info = lfirst_node(PathKeyInfo, lc);
+
+ Assert(list_length(info->clauses) == list_length(pinfo->clauses));
+ Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
+ Assert(list_difference(info->clauses, pinfo->clauses) == NIL);
+ Assert(list_difference_ptr(info->pathkeys, pinfo->pathkeys) == NIL);
+
+ forboth(lc1, info->clauses, lc2, info->pathkeys)
+ {
+ SortGroupClause *sgc = lfirst_node(SortGroupClause, lc1);
+ PathKey *pk = lfirst_node(PathKey, lc2);
+
+ if (pk->pk_eclass->ec_sortref != sgc->tleSortGroupRef)
+ elog(ERROR, "Order of group-by clauses doesn't correspond incoming sort order");
+ }
+ }
+ }
+#endif
return infos;
}
--
2.39.3 (Apple Git-145)
[application/octet-stream] v2-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch (8.8K, ../../CAPpHfdvbxK3nszdk6YXWpV7p+P+pGOQhpKTxJ3xME3uD+qZmXg@mail.gmail.com/5-v2-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch)
download | inline diff:
From 4e57d213a9619d04b527ee24810a6daf45fe028d Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:10:34 +0300
Subject: [PATCH v2 1/4] Fix asymmetry in setting EquivalenceClass.ec_sortref
0452b461bc made get_eclass_for_sort_expr() always set
EquivalenceClass.ec_sortref if it's not done yet. This leads to an asymmetric
situation when whoever first looks for the EquivalenceClass sets the
ec_sortref. It is also counterintuitive that get_eclass_for_sort_expr()
performs modification of data structures.
This commit makes make_pathkeys_for_sortclauses_extended() responsible for
setting EquivalenceClass.ec_sortref. Now we set the
EquivalenceClass.ec_sortref's needed to explore alternative GROUP BY ordering
specifically during turning GROUP BY clauses into pathkeys.
Discussion: https://postgr.es/m/17037754-f187-4138-8285-0e2bfebd0dea%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/equivclass.c | 13 +------
src/backend/optimizer/path/pathkeys.c | 18 ++++++++-
src/backend/optimizer/plan/planner.c | 16 ++++++--
src/include/optimizer/paths.h | 3 +-
src/test/regress/expected/aggregates.out | 47 ++++++++++++++++++++++++
src/test/regress/sql/aggregates.sql | 14 +++++++
6 files changed, 92 insertions(+), 19 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 21ce1ae2e13..51d806326eb 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,18 +652,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from a WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 8b258cbef92..7858a56fae0 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -1355,7 +1355,8 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
&sortclauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
/* It's caller error if not all clauses were sortable */
Assert(sortable);
return result;
@@ -1379,13 +1380,17 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
* to remove any clauses that can be proven redundant via the eclass logic.
* Even though we'll have to hash in that case, we might as well not hash
* redundant columns.)
+ *
+ * If set_ec_sortref is true, then the ec_sortref field of the pathkey's
+ * EquivalenceClass is set, if not initialized beforehand.
*/
List *
make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable)
+ bool *sortable,
+ bool set_ec_sortref)
{
List *pathkeys = NIL;
ListCell *l;
@@ -1409,6 +1414,15 @@ make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
sortcl->nulls_first,
sortcl->tleSortGroupRef,
true);
+ if (pathkey->pk_eclass->ec_sortref == 0 && set_ec_sortref)
+ {
+ /*
+ * Copy the sortref if it wasn't set yet. That may happen if the
+ * ec was constructed from a WHERE clause, i.e. it doesn't have a
+ * target reference at all.
+ */
+ pathkey->pk_eclass->ec_sortref = sortcl->tleSortGroupRef;
+ }
/* Canonical form eliminates redundant ordering keys */
if (!pathkey_is_redundant(pathkey, pathkeys))
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 032818423f6..e723f72db64 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -3395,12 +3395,17 @@ standard_qp_callback(PlannerInfo *root, void *extra)
*/
bool sortable;
+ /*
+ * Covert group clauses into pathkeys. Set the ec_sortref field of
+ * the EquivalenceClass'es if it's not set yet.
+ */
root->group_pathkeys =
make_pathkeys_for_sortclauses_extended(root,
&root->processed_groupClause,
tlist,
true,
- &sortable);
+ &sortable,
+ true);
if (!sortable)
{
/* Can't sort; no point in considering aggregate ordering either */
@@ -3450,7 +3455,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&root->processed_distinctClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->distinct_pathkeys = NIL;
}
@@ -3476,7 +3482,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&groupClauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->setop_pathkeys = NIL;
}
@@ -6061,7 +6068,8 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc,
&wc->partitionClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
Assert(sortable);
}
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 914d9bdef58..5e88c0224a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -239,7 +239,8 @@ extern List *make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable);
+ bool *sortable,
+ bool set_ec_sortref);
extern void initialize_mergeclause_eclasses(PlannerInfo *root,
RestrictInfo *restrictinfo);
extern void update_mergeclause_eclasses(PlannerInfo *root,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 2442342e9d1..1c1ca7573ad 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2917,6 +2917,53 @@ GROUP BY c1.w, c1.z;
5.0000000000000000
(2 rows)
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c1.x, c1.y
+ -> Incremental Sort
+ Sort Key: c1.x, c1.y
+ Presorted Key: c1.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c2.x, c1.y
+ -> Incremental Sort
+ Sort Key: c2.x, c1.y
+ Presorted Key: c2.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 61a3424c845..1a18ca3d8fe 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1263,6 +1263,20 @@ SELECT avg(c1.f ORDER BY c1.x, c1.y)
FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* About 0001:,Having overviewed it, I don't see any issues (but I'm the author), except grammatical ones - but I'm not a native to judge it.,Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear to me. It may be better to write something like: 'building pathkeys by the list of grouping clauses'.,,0002:,The part under USE_ASSERT_CHECKING looks good to me. But the code in group_keys_reorder_by_pathkeys looks suspicious: of course, we do some doubtful work without any possible way to reproduce, but if we envision some duplicated elements in the group_clauses, we should avoid usage of the list_concat_unique_ptr. What's more, why do you not exit from foreach_ptr immediately after SortGroupClause has been found? I think the new_group_clauses should be consistent with the new_group_pathkeys.,,0003:,Looks good,,0004:,I was also thinking about reintroducing the preprocess_groupclause because with the re-arrangement of GROUP-BY clauses according to incoming pathkeys, it doesn't make sense to have a user-defined order—at least while cost_sort doesn't differ costs for alternative column orderings.,So, I'm okay with the code. But why don't you use the same approach with foreach_ptr as before?
@ 2024-05-29 08:08 Andrei Lepikhov <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-05-29 08:08 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; Tom Lane <[email protected]>; +Cc: Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 5/27/24 19:41, Alexander Korotkov wrote:
> On Tue, Apr 23, 2024 at 1:19 AM Alexander Korotkov <[email protected]> wrote:
>> While there are some particular use-cases by Jian He, I hope that
>> above could give some rationale.
>
> I've assembled patches in this thread into one patchset.
> 0001 The patch fixing asymmetry in setting EquivalenceClass.ec_sortref
> by Andrei [1]. I've revised comments and wrote the commit message.
> 0002 The patch for handling duplicates of SortGroupClause. I didn't
> get the sense of Andrei implementation. It seems to care about
> duplicate pointers in group clauses list. But the question is the
> equal SortGroupClause's comprising different pointers. I think we
> should group duplicate SortGroupClause's together as
> preprocess_groupclause() used to do. Reimplemented patch to do so.
> 0003 Rename PathKeyInfo to GroupByOrdering by Andres [3]. I only
> revised comments and wrote the commit message.
> 0004 Turn back preprocess_groupclause() for the reason I described upthread [4].
>
> Any thoughts?
About 0001:
Having overviewed it, I don't see any issues (but I'm the author),
except grammatical ones - but I'm not a native to judge it.
Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear
to me. It may be better to write something like: 'building pathkeys by
the list of grouping clauses'.
0002:
The part under USE_ASSERT_CHECKING looks good to me. But the code in
group_keys_reorder_by_pathkeys looks suspicious: of course, we do some
doubtful work without any possible way to reproduce, but if we envision
some duplicated elements in the group_clauses, we should avoid usage of
the list_concat_unique_ptr. What's more, why do you not exit from
foreach_ptr immediately after SortGroupClause has been found? I think
the new_group_clauses should be consistent with the new_group_pathkeys.
0003:
Looks good
0004:
I was also thinking about reintroducing the preprocess_groupclause
because with the re-arrangement of GROUP-BY clauses according to
incoming pathkeys, it doesn't make sense to have a user-defined order—at
least while cost_sort doesn't differ costs for alternative column orderings.
So, I'm okay with the code. But why don't you use the same approach with
foreach_ptr as before?
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: About 0001:,Having overviewed it, I don't see any issues (but I'm the author), except grammatical ones - but I'm not a native to judge it.,Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear to me. It may be better to write something like: 'building pathkeys by the list of grouping clauses'.,,0002:,The part under USE_ASSERT_CHECKING looks good to me. But the code in group_keys_reorder_by_pathkeys looks suspicious: of course, we do some doubtful work without any possible way to reproduce, but if we envision some duplicated elements in the group_clauses, we should avoid usage of the list_concat_unique_ptr. What's more, why do you not exit from foreach_ptr immediately after SortGroupClause has been found? I think the new_group_clauses should be consistent with the new_group_pathkeys.,,0003:,Looks good,,0004:,I was also thinking about reintroducing the preprocess_groupclause because with the re-arrangement of GROUP-BY clauses according to incoming pathkeys, it d...
@ 2024-05-29 12:53 Alexander Korotkov <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-05-29 12:53 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi, Andrei!
Thank you for your feedback.
On Wed, May 29, 2024 at 11:08 AM Andrei Lepikhov
<[email protected]> wrote:
> On 5/27/24 19:41, Alexander Korotkov wrote:
> > On Tue, Apr 23, 2024 at 1:19 AM Alexander Korotkov <[email protected]> wrote:
> >> While there are some particular use-cases by Jian He, I hope that
> >> above could give some rationale.
> >
> > I've assembled patches in this thread into one patchset.
> > 0001 The patch fixing asymmetry in setting EquivalenceClass.ec_sortref
> > by Andrei [1]. I've revised comments and wrote the commit message.
> > 0002 The patch for handling duplicates of SortGroupClause. I didn't
> > get the sense of Andrei implementation. It seems to care about
> > duplicate pointers in group clauses list. But the question is the
> > equal SortGroupClause's comprising different pointers. I think we
> > should group duplicate SortGroupClause's together as
> > preprocess_groupclause() used to do. Reimplemented patch to do so.
> > 0003 Rename PathKeyInfo to GroupByOrdering by Andres [3]. I only
> > revised comments and wrote the commit message.
> > 0004 Turn back preprocess_groupclause() for the reason I described upthread [4].
> >
> > Any thoughts?
> About 0001:
> Having overviewed it, I don't see any issues (but I'm the author),
> except grammatical ones - but I'm not a native to judge it.
> Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear
> to me. It may be better to write something like: 'building pathkeys by
> the list of grouping clauses'.
OK, thank you. I'll run once again for the grammar issues.
> 0002:
> The part under USE_ASSERT_CHECKING looks good to me. But the code in
> group_keys_reorder_by_pathkeys looks suspicious: of course, we do some
> doubtful work without any possible way to reproduce, but if we envision
> some duplicated elements in the group_clauses, we should avoid usage of
> the list_concat_unique_ptr.
As I understand Tom, there is a risk that clauses list may contain
multiple instances of equivalent SortGroupClause, not duplicate
pointers.
> What's more, why do you not exit from
> foreach_ptr immediately after SortGroupClause has been found? I think
> the new_group_clauses should be consistent with the new_group_pathkeys.
I wanted this to be consistent with preprocess_groupclause(), where
duplicate SortGroupClause'es are grouped together. Otherwise, we
could just delete redundant SortGroupClause'es.
> 0003:
> Looks good
>
> 0004:
> I was also thinking about reintroducing the preprocess_groupclause
> because with the re-arrangement of GROUP-BY clauses according to
> incoming pathkeys, it doesn't make sense to have a user-defined order—at
> least while cost_sort doesn't differ costs for alternative column orderings.
> So, I'm okay with the code. But why don't you use the same approach with
> foreach_ptr as before?
I restored the function as it was before 0452b461bc with minimal edits
to support the incremental sort. I think it would be more valuable to
keep the difference with pg16 code small rather than refactor to
simplify existing code.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-30 04:22 Andrei Lepikhov <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Andrei Lepikhov @ 2024-05-30 04:22 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On 5/29/24 19:53, Alexander Korotkov wrote:
> Hi, Andrei!
>
> Thank you for your feedback.
>
> On Wed, May 29, 2024 at 11:08 AM Andrei Lepikhov
> <[email protected]> wrote:
>> On 5/27/24 19:41, Alexander Korotkov wrote:
>>> Any thoughts?
>> About 0001:
>> Having overviewed it, I don't see any issues (but I'm the author),
>> except grammatical ones - but I'm not a native to judge it.
>> Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear
>> to me. It may be better to write something like: 'building pathkeys by
>> the list of grouping clauses'.
>
> OK, thank you. I'll run once again for the grammar issues.
>
>> 0002:
>> The part under USE_ASSERT_CHECKING looks good to me. But the code in
>> group_keys_reorder_by_pathkeys looks suspicious: of course, we do some
>> doubtful work without any possible way to reproduce, but if we envision
>> some duplicated elements in the group_clauses, we should avoid usage of
>> the list_concat_unique_ptr.
>
> As I understand Tom, there is a risk that clauses list may contain
> multiple instances of equivalent SortGroupClause, not duplicate
> pointers.
Maybe. I just implemented the worst-case scenario with the intention of
maximum safety. So, it's up to you.
>
>> What's more, why do you not exit from
>> foreach_ptr immediately after SortGroupClause has been found? I think
>> the new_group_clauses should be consistent with the new_group_pathkeys.
>
> I wanted this to be consistent with preprocess_groupclause(), where
> duplicate SortGroupClause'es are grouped together. Otherwise, we
> could just delete redundant SortGroupClause'es.
Hm, preprocess_groupclause is called before the standard_qp_callback
forms the 'canonical form' of group_pathkeys and such behaviour needed.
But the code that chooses the reordering strategy uses already processed
grouping clauses, where we don't have duplicates in the first
num_groupby_pathkeys elements, do we?
>> 0004:
>> I was also thinking about reintroducing the preprocess_groupclause
>> because with the re-arrangement of GROUP-BY clauses according to
>> incoming pathkeys, it doesn't make sense to have a user-defined order—at
>> least while cost_sort doesn't differ costs for alternative column orderings.
>> So, I'm okay with the code. But why don't you use the same approach with
>> foreach_ptr as before?
>
> I restored the function as it was before 0452b461bc with minimal edits
> to support the incremental sort. I think it would be more valuable to
> keep the difference with pg16 code small rather than refactor to
> simplify existing code.
Got it
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-05-31 00:12 Alexander Korotkov <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-05-31 00:12 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi!
On Thu, May 30, 2024 at 7:22 AM Andrei Lepikhov
<[email protected]> wrote:
> On 5/29/24 19:53, Alexander Korotkov wrote:
> > Thank you for your feedback.
> >
> > On Wed, May 29, 2024 at 11:08 AM Andrei Lepikhov
> > <[email protected]> wrote:
> >> On 5/27/24 19:41, Alexander Korotkov wrote:
> >>> Any thoughts?
> >> About 0001:
> >> Having overviewed it, I don't see any issues (but I'm the author),
> >> except grammatical ones - but I'm not a native to judge it.
> >> Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear
> >> to me. It may be better to write something like: 'building pathkeys by
> >> the list of grouping clauses'.
> >
> > OK, thank you. I'll run once again for the grammar issues.
I've revised some grammar including the sentence you've proposed.
> >> 0002:
> >> The part under USE_ASSERT_CHECKING looks good to me. But the code in
> >> group_keys_reorder_by_pathkeys looks suspicious: of course, we do some
> >> doubtful work without any possible way to reproduce, but if we envision
> >> some duplicated elements in the group_clauses, we should avoid usage of
> >> the list_concat_unique_ptr.
> >
> > As I understand Tom, there is a risk that clauses list may contain
> > multiple instances of equivalent SortGroupClause, not duplicate
> > pointers.
> Maybe. I just implemented the worst-case scenario with the intention of
> maximum safety. So, it's up to you.
> >
> >> What's more, why do you not exit from
> >> foreach_ptr immediately after SortGroupClause has been found? I think
> >> the new_group_clauses should be consistent with the new_group_pathkeys.
> >
> > I wanted this to be consistent with preprocess_groupclause(), where
> > duplicate SortGroupClause'es are grouped together. Otherwise, we
> > could just delete redundant SortGroupClause'es.
> Hm, preprocess_groupclause is called before the standard_qp_callback
> forms the 'canonical form' of group_pathkeys and such behaviour needed.
> But the code that chooses the reordering strategy uses already processed
> grouping clauses, where we don't have duplicates in the first
> num_groupby_pathkeys elements, do we?
Yep, it seems that we work with group clauses which were processed by
standard_qp_callback(). In turn standard_qp_callback() called
make_pathkeys_for_sortclauses_extended() with remove_redundant ==
true. So, there shouldn't be duplicates. And it's unclear what
should we be protected from.
I leaved 0002 with just asserts. And extracted questionable changes into 0005.
------
Regards,
Alexander Korotkov
Supabase
Attachments:
[application/octet-stream] v3-0004-Restore-preprocess_groupclause.patch (13.1K, ../../CAPpHfdtAOT8ZN8FjeP2eBRrCnW8BKb0V0PFzfSuChdNVk84QQw@mail.gmail.com/2-v3-0004-Restore-preprocess_groupclause.patch)
download | inline diff:
From c1b657dbb5cbdb19004ba6ab93757f1762d305c2 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:51:31 +0300
Subject: [PATCH v3 4/5] Restore preprocess_groupclause()
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
It eliminated preprocess_groupclause(), which was intended to match items
between GROUP BY and ORDER BY. Instead, get_useful_group_keys_orderings()
function generates orderings of GROUP BY elements at the time of grouping
paths generation. The get_useful_group_keys_orderings() function takes into
account 3 orderings of GROUP BY pathkeys and clauses: original order as written
in GROUP BY, matching ORDER BY clauses as much as possible, and matching the
input path as much as possible. Given that even before 0452b461b,
preprocess_groupclause() could change the original order of GROUP BY clauses
we don't need to consider it apart from ordering matching ORDER BY clauses.
This commit restores preprocess_groupclause() to provide an ordering of
GROUP BY elements matching ORDER BY before generation of paths. The new
version of preprocess_groupclause() takes into account an incremental sort.
The get_useful_group_keys_orderings() function now takes into 2 orderings of
GROUP BY elements: the order generated preprocess_groupclause() and the order
matching the input path as much as possible.
Discussion: https://postgr.es/m/CAPpHfdvyWLMGwvxaf%3D7KAp-z-4mxbSH8ti2f6mNOQv5metZFzg%40mail.gmail.com
Author: Alexander Korotkov
---
src/backend/optimizer/path/pathkeys.c | 55 +--------
src/backend/optimizer/plan/planner.c | 108 +++++++++++++++---
src/include/nodes/pathnodes.h | 6 +-
.../regress/expected/partition_aggregate.out | 6 +-
4 files changed, 108 insertions(+), 67 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1cc8b5da49a..4287811abe4 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -447,26 +447,6 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
return n;
}
-/*
- * pathkeys_are_duplicate
- * Check if give pathkeys are already contained the list of
- * GroupByOrdering's.
- */
-static bool
-pathkeys_are_duplicate(List *infos, List *pathkeys)
-{
- ListCell *lc;
-
- foreach(lc, infos)
- {
- GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
-
- if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
- return true;
- }
- return false;
-}
-
/*
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
@@ -475,11 +455,11 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
- * The function considers (and keeps) multiple GROUP BY orderings:
+ * The function considers (and keeps) following GROUP BY orderings:
*
- * - the original ordering, as specified by the GROUP BY clause,
- * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
- * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ * - GROUP BY keys as ordered by preprocess_groupclause() to match target
+ * ORDER BY clause (as much as possible),
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible).
*/
List *
get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
@@ -526,32 +506,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
if (n > 0 &&
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
- !pathkeys_are_duplicate(infos, pathkeys))
- {
- info = makeNode(GroupByOrdering);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause).
- */
- if (root->sort_pathkeys &&
- !pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys))
- {
- int n;
-
- n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
- &clauses,
- root->num_groupby_pathkeys);
-
- if (n > 0 &&
- (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
- !pathkeys_are_duplicate(infos, pathkeys))
+ compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
{
info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 1eeedc121ed..d6a1fb8e7d5 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -137,7 +137,7 @@ static double preprocess_limit(PlannerInfo *root,
double tuple_fraction,
int64 *offset_est, int64 *count_est);
static void remove_useless_groupby_columns(PlannerInfo *root);
-static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
+static List *preprocess_groupclause(PlannerInfo *root, List *force);
static List *extract_rollup_sets(List *groupingSets);
static List *reorder_grouping_sets(List *groupingSets, List *sortclause);
static void standard_qp_callback(PlannerInfo *root, void *extra);
@@ -1422,7 +1422,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction,
else if (parse->groupClause)
{
/* Preprocess regular GROUP BY clause, if any */
- root->processed_groupClause = list_copy(parse->groupClause);
+ root->processed_groupClause = preprocess_groupclause(root, NIL);
/* Remove any redundant GROUP BY columns */
remove_useless_groupby_columns(root);
}
@@ -2169,7 +2169,7 @@ preprocess_grouping_sets(PlannerInfo *root)
* The groupClauses for hashed grouping sets are built later on.)
*/
if (gs->set)
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
else
rollup->groupClause = NIL;
@@ -2821,24 +2821,106 @@ remove_useless_groupby_columns(PlannerInfo *root)
}
/*
- * groupclause_apply_groupingset
- * Apply the order of GROUP BY clauses defined by grouping sets. Items
- * not in the grouping set are skipped.
+ * preprocess_groupclause - do preparatory work on GROUP BY clause
+ *
+ * The idea here is to adjust the ordering of the GROUP BY elements
+ * (which in itself is semantically insignificant) to match ORDER BY,
+ * thereby allowing a single sort operation to both implement the ORDER BY
+ * requirement and set up for a Unique step that implements GROUP BY.
+ * We also consider partial match between GROUP BY and ORDER BY elements,
+ * which could allow to implement ORDER BY using the incremental sort.
+ *
+ * We also consider other orderings of the GROUP BY elements, which could
+ * match the sort ordering of other possible plans (eg an indexscan) and
+ * thereby reduce cost. This is implemented during the generation of grouping
+ * paths. See get_useful_group_keys_orderings() for details.
+ *
+ * Note: we need no comparable processing of the distinctClause because
+ * the parser already enforced that that matches ORDER BY.
+ *
+ * Note: we return a fresh List, but its elements are the same
+ * SortGroupClauses appearing in parse->groupClause. This is important
+ * because later processing may modify the processed_groupClause list.
+ *
+ * For grouping sets, the order of items is instead forced to agree with that
+ * of the grouping set (and items not in the grouping set are skipped). The
+ * work of sorting the order of grouping set elements to match the ORDER BY if
+ * possible is done elsewhere.
*/
static List *
-groupclause_apply_groupingset(PlannerInfo *root, List *gset)
+preprocess_groupclause(PlannerInfo *root, List *force)
{
Query *parse = root->parse;
List *new_groupclause = NIL;
ListCell *sl;
+ ListCell *gl;
- foreach(sl, gset)
+ /* For grouping sets, we need to force the ordering */
+ if (force)
{
- Index ref = lfirst_int(sl);
- SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+ foreach(sl, force)
+ {
+ Index ref = lfirst_int(sl);
+ SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+
+ new_groupclause = lappend(new_groupclause, cl);
+ }
- new_groupclause = lappend(new_groupclause, cl);
+ return new_groupclause;
}
+
+ /* If no ORDER BY, nothing useful to do here */
+ if (parse->sortClause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Scan the ORDER BY clause and construct a list of matching GROUP BY
+ * items, but only as far as we can make a matching prefix.
+ *
+ * This code assumes that the sortClause contains no duplicate items.
+ */
+ foreach(sl, parse->sortClause)
+ {
+ SortGroupClause *sc = lfirst_node(SortGroupClause, sl);
+
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (equal(gc, sc))
+ {
+ new_groupclause = lappend(new_groupclause, gc);
+ break;
+ }
+ }
+ if (gl == NULL)
+ break; /* no match, so stop scanning */
+ }
+
+
+ /* If no match at all, no point in reordering GROUP BY */
+ if (new_groupclause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Add any remaining GROUP BY items to the new list. We don't require a
+ * complete match, because even partial match allows ORDER BY to be
+ * implemented using incremental sort. Also, give up if there are any
+ * non-sortable GROUP BY items, since then there's no hope anyway.
+ */
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (list_member_ptr(new_groupclause, gc))
+ continue; /* it matched an ORDER BY item */
+ if (!OidIsValid(gc->sortop)) /* give up, GROUP BY can't be sorted */
+ return list_copy(parse->groupClause);
+ new_groupclause = lappend(new_groupclause, gc);
+ }
+
+ /* Success --- install the rearranged GROUP BY list */
+ Assert(list_length(parse->groupClause) == list_length(new_groupclause));
return new_groupclause;
}
@@ -4170,7 +4252,7 @@ consider_groupingsets_paths(PlannerInfo *root,
{
rollup = makeNode(RollupData);
- rollup->groupClause = groupclause_apply_groupingset(root, gset);
+ rollup->groupClause = preprocess_groupclause(root, gset);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
@@ -4359,7 +4441,7 @@ consider_groupingsets_paths(PlannerInfo *root,
Assert(gs->set != NIL);
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 78489398294..af8de4213de 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -426,7 +426,11 @@ struct PlannerInfo
* items to be proven redundant, implying that there is only one group
* containing all the query's rows. Hence, if you want to check whether
* GROUP BY was specified, test for nonempty parse->groupClause, not for
- * nonempty processed_groupClause.
+ * nonempty processed_groupClause. Optimiser chooses specific order of
+ * group-by clauses during the upper paths generation process, attempting
+ * to use different strategies to minimize number of sorts or engage
+ * incremental sort. See preprocess_groupclause() and
+ * get_useful_group_keys_orderings() for details.
*
* Currently, when grouping sets are specified we do not attempt to
* optimize the groupClause, so that processed_groupClause will be
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index 1b900fddf8e..5f2c0cf5786 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -901,15 +901,15 @@ SELECT a, c, sum(b), avg(c), count(*) FROM pagg_tab_m GROUP BY (a+b)/2, 2, 1 HAV
Sort Key: pagg_tab_m.a, pagg_tab_m.c, (sum(pagg_tab_m.b))
-> Append
-> HashAggregate
- Group Key: ((pagg_tab_m.a + pagg_tab_m.b) / 2), pagg_tab_m.c, pagg_tab_m.a
+ Group Key: pagg_tab_m.a, pagg_tab_m.c, ((pagg_tab_m.a + pagg_tab_m.b) / 2)
Filter: ((sum(pagg_tab_m.b) = 50) AND (avg(pagg_tab_m.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p1 pagg_tab_m
-> HashAggregate
- Group Key: ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2), pagg_tab_m_1.c, pagg_tab_m_1.a
+ Group Key: pagg_tab_m_1.a, pagg_tab_m_1.c, ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2)
Filter: ((sum(pagg_tab_m_1.b) = 50) AND (avg(pagg_tab_m_1.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p2 pagg_tab_m_1
-> HashAggregate
- Group Key: ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2), pagg_tab_m_2.c, pagg_tab_m_2.a
+ Group Key: pagg_tab_m_2.a, pagg_tab_m_2.c, ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2)
Filter: ((sum(pagg_tab_m_2.b) = 50) AND (avg(pagg_tab_m_2.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p3 pagg_tab_m_2
(15 rows)
--
2.39.3 (Apple Git-145)
[application/octet-stream] v3-0005-Teach-group_keys_reorder_by_pathkeys-about-redund.patch (3.2K, ../../CAPpHfdtAOT8ZN8FjeP2eBRrCnW8BKb0V0PFzfSuChdNVk84QQw@mail.gmail.com/3-v3-0005-Teach-group_keys_reorder_by_pathkeys-about-redund.patch)
download | inline diff:
From d76a85b614f5d27c603af812799104a006076cbe Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:13:11 +0300
Subject: [PATCH v3 5/5] Teach group_keys_reorder_by_pathkeys() about redundant
SortGroupClause's
We don't have strict guarantees that there shouldn't be redundant
SortGroupClause's. Although there is no confirmed way to generate
duplicate SortGroupClause's, this commit teaches
group_keys_reorder_by_pathkeys() to pull all the SortGroupClause matching to
pathkey at once.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 41 ++++++++++++++++++---------
1 file changed, 27 insertions(+), 14 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4287811abe4..6ae595781d8 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -401,7 +401,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
+ bool found = false;
/*
* Pathkeys are built in a way that allows simply comparing pointers.
@@ -416,28 +416,40 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* Since 1349d27 pathkey coming from underlying node can be in the
* root->group_pathkeys but not in the processed_groupClause. So, we
- * should be careful here.
+ * should be careful here. Also, there could be redundant
+ * SortGroupClause's. So, we need to pull of all the matching
+ * SortGroupClause's, but ensure there is at least one.
*/
- sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
- if (!sgc)
- /* The grouping clause does not cover this pathkey */
- break;
+ foreach_ptr(SortGroupClause, sgc, *group_clauses)
+ {
+ if (sgc->tleSortGroupRef == pathkey->pk_eclass->ec_sortref)
+ {
+ /*
+ * Sort group clause should have an ordering operator as long
+ * as there is an associated pathkey.
+ */
+ Assert(OidIsValid(sgc->sortop));
- /*
- * Sort group clause should have an ordering operator as long as there
- * is an associated pathkey.
- */
- Assert(OidIsValid(sgc->sortop));
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ found = true;
+ }
+ }
+
+ /* Check if the grouping clause does not cover this pathkey */
+ if (!found)
+ break;
new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
- new_group_clauses = lappend(new_group_clauses, sgc);
+
}
/* remember the number of pathkeys with a matching GROUP BY key */
n = list_length(new_group_pathkeys);
- /* append the remaining group pathkeys (will be treated as not sorted) */
+ /*
+ * Append the remaining group pathkeys (will be treated as not sorted) and
+ * grouping clauses.
+ */
*group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
*group_pathkeys);
*group_clauses = list_concat_unique_ptr(new_group_clauses,
@@ -545,6 +557,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
#endif
+
return infos;
}
--
2.39.3 (Apple Git-145)
[application/octet-stream] v3-0002-Add-invariants-check-to-get_useful_group_keys_ord.patch (1.8K, ../../CAPpHfdtAOT8ZN8FjeP2eBRrCnW8BKb0V0PFzfSuChdNVk84QQw@mail.gmail.com/4-v3-0002-Add-invariants-check-to-get_useful_group_keys_ord.patch)
download | inline diff:
From b77abf6c882778321ca10a63b11a5d789518fc01 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 30 May 2024 22:38:53 +0300
Subject: [PATCH v3 2/5] Add invariants check to
get_useful_group_keys_orderings()
This commit introduces invariants checking of generated orderings
in get_useful_group_keys_orderings for assert-enabled builds.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 29 +++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 837e612121f..4e13fdcb23c 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -561,6 +561,35 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
+#ifdef USE_ASSERT_CHECKING
+ {
+ PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ ListCell *lc;
+
+ /* Test consistency of info structures */
+ for_each_from(lc, infos, 1)
+ {
+ ListCell *lc1,
+ *lc2;
+
+ info = lfirst_node(PathKeyInfo, lc);
+
+ Assert(list_length(info->clauses) == list_length(pinfo->clauses));
+ Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
+ Assert(list_difference(info->clauses, pinfo->clauses) == NIL);
+ Assert(list_difference_ptr(info->pathkeys, pinfo->pathkeys) == NIL);
+
+ forboth(lc1, info->clauses, lc2, info->pathkeys)
+ {
+ SortGroupClause *sgc = lfirst_node(SortGroupClause, lc1);
+ PathKey *pk = lfirst_node(PathKey, lc2);
+
+ if (pk->pk_eclass->ec_sortref != sgc->tleSortGroupRef)
+ elog(ERROR, "Order of group-by clauses doesn't correspond incoming sort order");
+ }
+ }
+ }
+#endif
return infos;
}
--
2.39.3 (Apple Git-145)
[application/octet-stream] v3-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch (7.0K, ../../CAPpHfdtAOT8ZN8FjeP2eBRrCnW8BKb0V0PFzfSuChdNVk84QQw@mail.gmail.com/5-v3-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch)
download | inline diff:
From c805a94738e51d5ee21ad51fe1ed52c595e99758 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Mon, 27 May 2024 14:57:43 +0300
Subject: [PATCH v3 3/5] Rename PathKeyInfo to GroupByOrdering
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
The PathKeyInfo data structure was used to store the particular ordering of
group-by pathkeys and corresponding clauses. It turns out that PathKeyInfo
is not the best name for that purpose. This commit renames this data structure
to GroupByOrdering, and revises its comment.
Discussion: https://postgr.es/m/db0fc3a4-966c-4cec-a136-94024d39212d%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 18 +++++++++---------
src/backend/optimizer/plan/planner.c | 8 ++++----
src/include/nodes/pathnodes.h | 13 ++++++++++---
src/tools/pgindent/typedefs.list | 2 +-
4 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4e13fdcb23c..1cc8b5da49a 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -450,7 +450,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* pathkeys_are_duplicate
* Check if give pathkeys are already contained the list of
- * PathKeyInfo's.
+ * GroupByOrdering's.
*/
static bool
pathkeys_are_duplicate(List *infos, List *pathkeys)
@@ -459,7 +459,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
foreach(lc, infos)
{
- PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+ GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
return true;
@@ -471,7 +471,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
*
- * Returns a list of PathKeyInfo items, each representing an interesting
+ * Returns a list of GroupByOrdering items, each representing an interesting
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
@@ -486,13 +486,13 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
{
Query *parse = root->parse;
List *infos = NIL;
- PathKeyInfo *info;
+ GroupByOrdering *info;
List *pathkeys = root->group_pathkeys;
List *clauses = root->processed_groupClause;
/* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
infos = lappend(infos, info);
@@ -528,7 +528,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -553,7 +553,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -563,7 +563,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
#ifdef USE_ASSERT_CHECKING
{
- PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ GroupByOrdering *pinfo = linitial_node(GroupByOrdering, infos);
ListCell *lc;
/* Test consistency of info structures */
@@ -572,7 +572,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
ListCell *lc1,
*lc2;
- info = lfirst_node(PathKeyInfo, lc);
+ info = lfirst_node(GroupByOrdering, lc);
Assert(list_length(info->clauses) == list_length(pinfo->clauses));
Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 669903b9059..1eeedc121ed 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6917,7 +6917,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -6998,7 +6998,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7249,7 +7249,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7305,7 +7305,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 14ef296ab72..78489398294 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1468,14 +1468,21 @@ typedef struct PathKey
} PathKey;
/*
- * Combines the information about pathkeys and the associated clauses.
+ * Contains an order of group-by clauses and the corresponding list of
+ * pathkeys.
+ *
+ * The elements of 'clauses' list should have the same order as the head of
+ * 'pathkeys' list. The tleSortGroupRef of the clause should be equal to
+ * ec_sortref of the pathkey equivalence class. If there are redundant
+ * clauses with the same tleSortGroupRef, they must be grouped together.
*/
-typedef struct PathKeyInfo
+typedef struct GroupByOrdering
{
NodeTag type;
+
List *pathkeys;
List *clauses;
-} PathKeyInfo;
+} GroupByOrdering;
/*
* VolatileFunctionStatus -- allows nodes to cache their
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d427a1c16a5..4f57078d133 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1056,6 +1056,7 @@ GrantRoleStmt
GrantStmt
GrantTargetType
Group
+GroupByOrdering
GroupClause
GroupPath
GroupPathExtraData
@@ -2067,7 +2068,6 @@ PathClauseUsage
PathCostComparison
PathHashStack
PathKey
-PathKeyInfo
PathKeysComparison
PathTarget
PatternInfo
--
2.39.3 (Apple Git-145)
[application/octet-stream] v3-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch (8.8K, ../../CAPpHfdtAOT8ZN8FjeP2eBRrCnW8BKb0V0PFzfSuChdNVk84QQw@mail.gmail.com/6-v3-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch)
download | inline diff:
From eef9b88438eb4c3cc40d98f4367a5c6b4ec4a920 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:10:34 +0300
Subject: [PATCH v3 1/5] Fix asymmetry in setting EquivalenceClass.ec_sortref
0452b461bc made get_eclass_for_sort_expr() always set
EquivalenceClass.ec_sortref if it's not done yet. This leads to an asymmetric
situation when whoever first looks for the EquivalenceClass sets the
ec_sortref. It is also counterintuitive that get_eclass_for_sort_expr()
performs modification of data structures.
This commit makes make_pathkeys_for_sortclauses_extended() responsible for
setting EquivalenceClass.ec_sortref. Now we set the
EquivalenceClass.ec_sortref's needed to explore alternative GROUP BY ordering
specifically during building pathkeys by the list of grouping clauses.
Discussion: https://postgr.es/m/17037754-f187-4138-8285-0e2bfebd0dea%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/equivclass.c | 13 +------
src/backend/optimizer/path/pathkeys.c | 18 ++++++++-
src/backend/optimizer/plan/planner.c | 16 ++++++--
src/include/optimizer/paths.h | 3 +-
src/test/regress/expected/aggregates.out | 47 ++++++++++++++++++++++++
src/test/regress/sql/aggregates.sql | 14 +++++++
6 files changed, 92 insertions(+), 19 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 21ce1ae2e13..51d806326eb 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,18 +652,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from a WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 8b258cbef92..837e612121f 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -1355,7 +1355,8 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
&sortclauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
/* It's caller error if not all clauses were sortable */
Assert(sortable);
return result;
@@ -1379,13 +1380,17 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
* to remove any clauses that can be proven redundant via the eclass logic.
* Even though we'll have to hash in that case, we might as well not hash
* redundant columns.)
+ *
+ * If set_ec_sortref is true then sets the uninitialized value of ec_sortref
+ * of the pathkey's EquivalenceClass.
*/
List *
make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable)
+ bool *sortable,
+ bool set_ec_sortref)
{
List *pathkeys = NIL;
ListCell *l;
@@ -1409,6 +1414,15 @@ make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
sortcl->nulls_first,
sortcl->tleSortGroupRef,
true);
+ if (pathkey->pk_eclass->ec_sortref == 0 && set_ec_sortref)
+ {
+ /*
+ * Copy the sortref if it wasn't set yet. That may happen if the
+ * EquivalenceClass was constructed from a WHERE clause, i.e. it
+ * doesn't have a target reference at all.
+ */
+ pathkey->pk_eclass->ec_sortref = sortcl->tleSortGroupRef;
+ }
/* Canonical form eliminates redundant ordering keys */
if (!pathkey_is_redundant(pathkey, pathkeys))
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 032818423f6..669903b9059 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -3395,12 +3395,17 @@ standard_qp_callback(PlannerInfo *root, void *extra)
*/
bool sortable;
+ /*
+ * Covert group clauses into pathkeys. Set the ec_sortref field of
+ * EquivalenceClass'es if it's not set yet.
+ */
root->group_pathkeys =
make_pathkeys_for_sortclauses_extended(root,
&root->processed_groupClause,
tlist,
true,
- &sortable);
+ &sortable,
+ true);
if (!sortable)
{
/* Can't sort; no point in considering aggregate ordering either */
@@ -3450,7 +3455,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&root->processed_distinctClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->distinct_pathkeys = NIL;
}
@@ -3476,7 +3482,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&groupClauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->setop_pathkeys = NIL;
}
@@ -6061,7 +6068,8 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc,
&wc->partitionClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
Assert(sortable);
}
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 914d9bdef58..5e88c0224a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -239,7 +239,8 @@ extern List *make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable);
+ bool *sortable,
+ bool set_ec_sortref);
extern void initialize_mergeclause_eclasses(PlannerInfo *root,
RestrictInfo *restrictinfo);
extern void update_mergeclause_eclasses(PlannerInfo *root,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 2442342e9d1..1c1ca7573ad 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2917,6 +2917,53 @@ GROUP BY c1.w, c1.z;
5.0000000000000000
(2 rows)
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c1.x, c1.y
+ -> Incremental Sort
+ Sort Key: c1.x, c1.y
+ Presorted Key: c1.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c2.x, c1.y
+ -> Incremental Sort
+ Sort Key: c2.x, c1.y
+ Presorted Key: c2.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 61a3424c845..1a18ca3d8fe 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1263,6 +1263,20 @@ SELECT avg(c1.f ORDER BY c1.x, c1.y)
FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-06-02 07:55 jian he <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: jian he @ 2024-06-02 07:55 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Fri, May 31, 2024 at 8:12 AM Alexander Korotkov <[email protected]> wrote:
>
> I've revised some grammar including the sentence you've proposed.
>
-static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
+static List *preprocess_groupclause(PlannerInfo *root, List *force);
changing preprocess_groupclause the second argument
from "force" to "gset" would be more intuitive, I think.
`elog(ERROR, "Order of group-by clauses doesn't correspond incoming
sort order");`
I think this error message makes people wonder what "incoming sort order" is.
BTW, "correspond", generally people use "correspond to".
I did some minor cosmetic changes, mainly changing foreach to foreach_node.
Please check the attachment.
Attachments:
[text/x-patch] minor_change_on_v3_Restore-preprocess_groupclause.diff (2.4K, ../../CACJufxHVvFdeLWDfp9MBEUvpizau4Vw83hN8wjOSegeptk5w8A@mail.gmail.com/2-minor_change_on_v3_Restore-preprocess_groupclause.diff)
download | inline diff:
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index d6a1fb8e..801d9f6d 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2852,15 +2852,13 @@ preprocess_groupclause(PlannerInfo *root, List *force)
{
Query *parse = root->parse;
List *new_groupclause = NIL;
- ListCell *sl;
- ListCell *gl;
+ ListCell *gl;
/* For grouping sets, we need to force the ordering */
- if (force)
+ if (force != NIL)
{
- foreach(sl, force)
+ foreach_int(ref, force)
{
- Index ref = lfirst_int(sl);
SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
new_groupclause = lappend(new_groupclause, cl);
@@ -2879,10 +2877,8 @@ preprocess_groupclause(PlannerInfo *root, List *force)
*
* This code assumes that the sortClause contains no duplicate items.
*/
- foreach(sl, parse->sortClause)
+ foreach_node(SortGroupClause, sc, parse->sortClause)
{
- SortGroupClause *sc = lfirst_node(SortGroupClause, sl);
-
foreach(gl, parse->groupClause)
{
SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
@@ -2908,10 +2904,8 @@ preprocess_groupclause(PlannerInfo *root, List *force)
* implemented using incremental sort. Also, give up if there are any
* non-sortable GROUP BY items, since then there's no hope anyway.
*/
- foreach(gl, parse->groupClause)
+ foreach_node(SortGroupClause,gc, parse->groupClause)
{
- SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
-
if (list_member_ptr(new_groupclause, gc))
continue; /* it matched an ORDER BY item */
if (!OidIsValid(gc->sortop)) /* give up, GROUP BY can't be sorted */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index af8de421..2ba297c1 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -426,7 +426,7 @@ struct PlannerInfo
* items to be proven redundant, implying that there is only one group
* containing all the query's rows. Hence, if you want to check whether
* GROUP BY was specified, test for nonempty parse->groupClause, not for
- * nonempty processed_groupClause. Optimiser chooses specific order of
+ * nonempty processed_groupClause. Optimizer chooses specific order of
* group-by clauses during the upper paths generation process, attempting
* to use different strategies to minimize number of sorts or engage
* incremental sort. See preprocess_groupclause() and
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-06-02 13:18 Alexander Korotkov <[email protected]>
parent: jian he <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2024-06-02 13:18 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; Pavel Borisov <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi!
On Sun, Jun 2, 2024 at 10:55 AM jian he <[email protected]> wrote:
>
> On Fri, May 31, 2024 at 8:12 AM Alexander Korotkov <[email protected]> wrote:
> >
> > I've revised some grammar including the sentence you've proposed.
> >
>
> -static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
> +static List *preprocess_groupclause(PlannerInfo *root, List *force);
>
> changing preprocess_groupclause the second argument
> from "force" to "gset" would be more intuitive, I think.
Probably, but my intention is to restore preprocess_groupclause() as
it was before 0452b461bc with minimal edits to support incremental
sort. I'd rather avoid refactoring if this area for now.
> `elog(ERROR, "Order of group-by clauses doesn't correspond incoming
> sort order");`
>
> I think this error message makes people wonder what "incoming sort order" is.
> BTW, "correspond", generally people use "correspond to".
Thank you. On the second thought, I think it would be better to turn
this into an assertion like the checks before.
> I did some minor cosmetic changes, mainly changing foreach to foreach_node.
> Please check the attachment.
I would avoid refactoring of preprocess_groupclause() for the reason
described above. But I picked the grammar fix for PlannerInfo's
comment.
------
Regards,
Alexander Korotkov
Supabase
Attachments:
[application/octet-stream] v4-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch (7.0K, ../../CAPpHfdsTUWnrNFRTenQpmv=JQ7yFdmoMbEADMSg8SNuHOZDFwQ@mail.gmail.com/2-v4-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch)
download | inline diff:
From 74734378e209cf37124cb12b7d80717d49d56d42 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Mon, 27 May 2024 14:57:43 +0300
Subject: [PATCH v4 3/5] Rename PathKeyInfo to GroupByOrdering
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
The PathKeyInfo data structure was used to store the particular ordering of
group-by pathkeys and corresponding clauses. It turns out that PathKeyInfo
is not the best name for that purpose. This commit renames this data structure
to GroupByOrdering, and revises its comment.
Discussion: https://postgr.es/m/db0fc3a4-966c-4cec-a136-94024d39212d%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 18 +++++++++---------
src/backend/optimizer/plan/planner.c | 8 ++++----
src/include/nodes/pathnodes.h | 13 ++++++++++---
src/tools/pgindent/typedefs.list | 2 +-
4 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1ed79797329..c98486e292c 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -450,7 +450,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* pathkeys_are_duplicate
* Check if give pathkeys are already contained the list of
- * PathKeyInfo's.
+ * GroupByOrdering's.
*/
static bool
pathkeys_are_duplicate(List *infos, List *pathkeys)
@@ -459,7 +459,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
foreach(lc, infos)
{
- PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+ GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
return true;
@@ -471,7 +471,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
*
- * Returns a list of PathKeyInfo items, each representing an interesting
+ * Returns a list of GroupByOrdering items, each representing an interesting
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
@@ -486,13 +486,13 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
{
Query *parse = root->parse;
List *infos = NIL;
- PathKeyInfo *info;
+ GroupByOrdering *info;
List *pathkeys = root->group_pathkeys;
List *clauses = root->processed_groupClause;
/* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
infos = lappend(infos, info);
@@ -528,7 +528,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -553,7 +553,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -563,7 +563,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
#ifdef USE_ASSERT_CHECKING
{
- PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ GroupByOrdering *pinfo = linitial_node(GroupByOrdering, infos);
ListCell *lc;
/* Test consistency of info structures */
@@ -572,7 +572,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
ListCell *lc1,
*lc2;
- info = lfirst_node(PathKeyInfo, lc);
+ info = lfirst_node(GroupByOrdering, lc);
Assert(list_length(info->clauses) == list_length(pinfo->clauses));
Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 669903b9059..1eeedc121ed 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6917,7 +6917,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -6998,7 +6998,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7249,7 +7249,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7305,7 +7305,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 14ef296ab72..78489398294 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1468,14 +1468,21 @@ typedef struct PathKey
} PathKey;
/*
- * Combines the information about pathkeys and the associated clauses.
+ * Contains an order of group-by clauses and the corresponding list of
+ * pathkeys.
+ *
+ * The elements of 'clauses' list should have the same order as the head of
+ * 'pathkeys' list. The tleSortGroupRef of the clause should be equal to
+ * ec_sortref of the pathkey equivalence class. If there are redundant
+ * clauses with the same tleSortGroupRef, they must be grouped together.
*/
-typedef struct PathKeyInfo
+typedef struct GroupByOrdering
{
NodeTag type;
+
List *pathkeys;
List *clauses;
-} PathKeyInfo;
+} GroupByOrdering;
/*
* VolatileFunctionStatus -- allows nodes to cache their
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d427a1c16a5..4f57078d133 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1056,6 +1056,7 @@ GrantRoleStmt
GrantStmt
GrantTargetType
Group
+GroupByOrdering
GroupClause
GroupPath
GroupPathExtraData
@@ -2067,7 +2068,6 @@ PathClauseUsage
PathCostComparison
PathHashStack
PathKey
-PathKeyInfo
PathKeysComparison
PathTarget
PatternInfo
--
2.39.3 (Apple Git-145)
[application/octet-stream] v4-0004-Restore-preprocess_groupclause.patch (13.1K, ../../CAPpHfdsTUWnrNFRTenQpmv=JQ7yFdmoMbEADMSg8SNuHOZDFwQ@mail.gmail.com/3-v4-0004-Restore-preprocess_groupclause.patch)
download | inline diff:
From 087791dc5de8626337b9a7ab6099c8c7794a763c Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:51:31 +0300
Subject: [PATCH v4 4/5] Restore preprocess_groupclause()
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
It eliminated preprocess_groupclause(), which was intended to match items
between GROUP BY and ORDER BY. Instead, get_useful_group_keys_orderings()
function generates orderings of GROUP BY elements at the time of grouping
paths generation. The get_useful_group_keys_orderings() function takes into
account 3 orderings of GROUP BY pathkeys and clauses: original order as written
in GROUP BY, matching ORDER BY clauses as much as possible, and matching the
input path as much as possible. Given that even before 0452b461b,
preprocess_groupclause() could change the original order of GROUP BY clauses
we don't need to consider it apart from ordering matching ORDER BY clauses.
This commit restores preprocess_groupclause() to provide an ordering of
GROUP BY elements matching ORDER BY before generation of paths. The new
version of preprocess_groupclause() takes into account an incremental sort.
The get_useful_group_keys_orderings() function now takes into 2 orderings of
GROUP BY elements: the order generated preprocess_groupclause() and the order
matching the input path as much as possible.
Discussion: https://postgr.es/m/CAPpHfdvyWLMGwvxaf%3D7KAp-z-4mxbSH8ti2f6mNOQv5metZFzg%40mail.gmail.com
Author: Alexander Korotkov
---
src/backend/optimizer/path/pathkeys.c | 55 +--------
src/backend/optimizer/plan/planner.c | 108 +++++++++++++++---
src/include/nodes/pathnodes.h | 6 +-
.../regress/expected/partition_aggregate.out | 6 +-
4 files changed, 108 insertions(+), 67 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index c98486e292c..793c8ad2930 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -447,26 +447,6 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
return n;
}
-/*
- * pathkeys_are_duplicate
- * Check if give pathkeys are already contained the list of
- * GroupByOrdering's.
- */
-static bool
-pathkeys_are_duplicate(List *infos, List *pathkeys)
-{
- ListCell *lc;
-
- foreach(lc, infos)
- {
- GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
-
- if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
- return true;
- }
- return false;
-}
-
/*
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
@@ -475,11 +455,11 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
- * The function considers (and keeps) multiple GROUP BY orderings:
+ * The function considers (and keeps) following GROUP BY orderings:
*
- * - the original ordering, as specified by the GROUP BY clause,
- * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
- * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ * - GROUP BY keys as ordered by preprocess_groupclause() to match target
+ * ORDER BY clause (as much as possible),
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible).
*/
List *
get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
@@ -526,32 +506,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
if (n > 0 &&
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
- !pathkeys_are_duplicate(infos, pathkeys))
- {
- info = makeNode(GroupByOrdering);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause).
- */
- if (root->sort_pathkeys &&
- !pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys))
- {
- int n;
-
- n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
- &clauses,
- root->num_groupby_pathkeys);
-
- if (n > 0 &&
- (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
- !pathkeys_are_duplicate(infos, pathkeys))
+ compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
{
info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 1eeedc121ed..d6a1fb8e7d5 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -137,7 +137,7 @@ static double preprocess_limit(PlannerInfo *root,
double tuple_fraction,
int64 *offset_est, int64 *count_est);
static void remove_useless_groupby_columns(PlannerInfo *root);
-static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
+static List *preprocess_groupclause(PlannerInfo *root, List *force);
static List *extract_rollup_sets(List *groupingSets);
static List *reorder_grouping_sets(List *groupingSets, List *sortclause);
static void standard_qp_callback(PlannerInfo *root, void *extra);
@@ -1422,7 +1422,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction,
else if (parse->groupClause)
{
/* Preprocess regular GROUP BY clause, if any */
- root->processed_groupClause = list_copy(parse->groupClause);
+ root->processed_groupClause = preprocess_groupclause(root, NIL);
/* Remove any redundant GROUP BY columns */
remove_useless_groupby_columns(root);
}
@@ -2169,7 +2169,7 @@ preprocess_grouping_sets(PlannerInfo *root)
* The groupClauses for hashed grouping sets are built later on.)
*/
if (gs->set)
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
else
rollup->groupClause = NIL;
@@ -2821,24 +2821,106 @@ remove_useless_groupby_columns(PlannerInfo *root)
}
/*
- * groupclause_apply_groupingset
- * Apply the order of GROUP BY clauses defined by grouping sets. Items
- * not in the grouping set are skipped.
+ * preprocess_groupclause - do preparatory work on GROUP BY clause
+ *
+ * The idea here is to adjust the ordering of the GROUP BY elements
+ * (which in itself is semantically insignificant) to match ORDER BY,
+ * thereby allowing a single sort operation to both implement the ORDER BY
+ * requirement and set up for a Unique step that implements GROUP BY.
+ * We also consider partial match between GROUP BY and ORDER BY elements,
+ * which could allow to implement ORDER BY using the incremental sort.
+ *
+ * We also consider other orderings of the GROUP BY elements, which could
+ * match the sort ordering of other possible plans (eg an indexscan) and
+ * thereby reduce cost. This is implemented during the generation of grouping
+ * paths. See get_useful_group_keys_orderings() for details.
+ *
+ * Note: we need no comparable processing of the distinctClause because
+ * the parser already enforced that that matches ORDER BY.
+ *
+ * Note: we return a fresh List, but its elements are the same
+ * SortGroupClauses appearing in parse->groupClause. This is important
+ * because later processing may modify the processed_groupClause list.
+ *
+ * For grouping sets, the order of items is instead forced to agree with that
+ * of the grouping set (and items not in the grouping set are skipped). The
+ * work of sorting the order of grouping set elements to match the ORDER BY if
+ * possible is done elsewhere.
*/
static List *
-groupclause_apply_groupingset(PlannerInfo *root, List *gset)
+preprocess_groupclause(PlannerInfo *root, List *force)
{
Query *parse = root->parse;
List *new_groupclause = NIL;
ListCell *sl;
+ ListCell *gl;
- foreach(sl, gset)
+ /* For grouping sets, we need to force the ordering */
+ if (force)
{
- Index ref = lfirst_int(sl);
- SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+ foreach(sl, force)
+ {
+ Index ref = lfirst_int(sl);
+ SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+
+ new_groupclause = lappend(new_groupclause, cl);
+ }
- new_groupclause = lappend(new_groupclause, cl);
+ return new_groupclause;
}
+
+ /* If no ORDER BY, nothing useful to do here */
+ if (parse->sortClause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Scan the ORDER BY clause and construct a list of matching GROUP BY
+ * items, but only as far as we can make a matching prefix.
+ *
+ * This code assumes that the sortClause contains no duplicate items.
+ */
+ foreach(sl, parse->sortClause)
+ {
+ SortGroupClause *sc = lfirst_node(SortGroupClause, sl);
+
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (equal(gc, sc))
+ {
+ new_groupclause = lappend(new_groupclause, gc);
+ break;
+ }
+ }
+ if (gl == NULL)
+ break; /* no match, so stop scanning */
+ }
+
+
+ /* If no match at all, no point in reordering GROUP BY */
+ if (new_groupclause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Add any remaining GROUP BY items to the new list. We don't require a
+ * complete match, because even partial match allows ORDER BY to be
+ * implemented using incremental sort. Also, give up if there are any
+ * non-sortable GROUP BY items, since then there's no hope anyway.
+ */
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (list_member_ptr(new_groupclause, gc))
+ continue; /* it matched an ORDER BY item */
+ if (!OidIsValid(gc->sortop)) /* give up, GROUP BY can't be sorted */
+ return list_copy(parse->groupClause);
+ new_groupclause = lappend(new_groupclause, gc);
+ }
+
+ /* Success --- install the rearranged GROUP BY list */
+ Assert(list_length(parse->groupClause) == list_length(new_groupclause));
return new_groupclause;
}
@@ -4170,7 +4252,7 @@ consider_groupingsets_paths(PlannerInfo *root,
{
rollup = makeNode(RollupData);
- rollup->groupClause = groupclause_apply_groupingset(root, gset);
+ rollup->groupClause = preprocess_groupclause(root, gset);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
@@ -4359,7 +4441,7 @@ consider_groupingsets_paths(PlannerInfo *root,
Assert(gs->set != NIL);
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 78489398294..2ba297c1172 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -426,7 +426,11 @@ struct PlannerInfo
* items to be proven redundant, implying that there is only one group
* containing all the query's rows. Hence, if you want to check whether
* GROUP BY was specified, test for nonempty parse->groupClause, not for
- * nonempty processed_groupClause.
+ * nonempty processed_groupClause. Optimizer chooses specific order of
+ * group-by clauses during the upper paths generation process, attempting
+ * to use different strategies to minimize number of sorts or engage
+ * incremental sort. See preprocess_groupclause() and
+ * get_useful_group_keys_orderings() for details.
*
* Currently, when grouping sets are specified we do not attempt to
* optimize the groupClause, so that processed_groupClause will be
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index 1b900fddf8e..5f2c0cf5786 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -901,15 +901,15 @@ SELECT a, c, sum(b), avg(c), count(*) FROM pagg_tab_m GROUP BY (a+b)/2, 2, 1 HAV
Sort Key: pagg_tab_m.a, pagg_tab_m.c, (sum(pagg_tab_m.b))
-> Append
-> HashAggregate
- Group Key: ((pagg_tab_m.a + pagg_tab_m.b) / 2), pagg_tab_m.c, pagg_tab_m.a
+ Group Key: pagg_tab_m.a, pagg_tab_m.c, ((pagg_tab_m.a + pagg_tab_m.b) / 2)
Filter: ((sum(pagg_tab_m.b) = 50) AND (avg(pagg_tab_m.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p1 pagg_tab_m
-> HashAggregate
- Group Key: ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2), pagg_tab_m_1.c, pagg_tab_m_1.a
+ Group Key: pagg_tab_m_1.a, pagg_tab_m_1.c, ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2)
Filter: ((sum(pagg_tab_m_1.b) = 50) AND (avg(pagg_tab_m_1.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p2 pagg_tab_m_1
-> HashAggregate
- Group Key: ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2), pagg_tab_m_2.c, pagg_tab_m_2.a
+ Group Key: pagg_tab_m_2.a, pagg_tab_m_2.c, ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2)
Filter: ((sum(pagg_tab_m_2.b) = 50) AND (avg(pagg_tab_m_2.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p3 pagg_tab_m_2
(15 rows)
--
2.39.3 (Apple Git-145)
[application/octet-stream] v4-0005-Teach-group_keys_reorder_by_pathkeys-about-redund.patch (3.2K, ../../CAPpHfdsTUWnrNFRTenQpmv=JQ7yFdmoMbEADMSg8SNuHOZDFwQ@mail.gmail.com/4-v4-0005-Teach-group_keys_reorder_by_pathkeys-about-redund.patch)
download | inline diff:
From 0e33f7c66682401180e037ec0fd8564401241c0a Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:13:11 +0300
Subject: [PATCH v4 5/5] Teach group_keys_reorder_by_pathkeys() about redundant
SortGroupClause's
We don't have strict guarantees that there shouldn't be redundant
SortGroupClause's. Although there is no confirmed way to generate
duplicate SortGroupClause's, this commit teaches
group_keys_reorder_by_pathkeys() to pull all the SortGroupClause matching to
pathkey at once.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 41 ++++++++++++++++++---------
1 file changed, 27 insertions(+), 14 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 793c8ad2930..0b7e9cd48db 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -401,7 +401,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
+ bool found = false;
/*
* Pathkeys are built in a way that allows simply comparing pointers.
@@ -416,28 +416,40 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* Since 1349d27 pathkey coming from underlying node can be in the
* root->group_pathkeys but not in the processed_groupClause. So, we
- * should be careful here.
+ * should be careful here. Also, there could be redundant
+ * SortGroupClause's. So, we need to pull of all the matching
+ * SortGroupClause's, but ensure there is at least one.
*/
- sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
- if (!sgc)
- /* The grouping clause does not cover this pathkey */
- break;
+ foreach_ptr(SortGroupClause, sgc, *group_clauses)
+ {
+ if (sgc->tleSortGroupRef == pathkey->pk_eclass->ec_sortref)
+ {
+ /*
+ * Sort group clause should have an ordering operator as long
+ * as there is an associated pathkey.
+ */
+ Assert(OidIsValid(sgc->sortop));
- /*
- * Sort group clause should have an ordering operator as long as there
- * is an associated pathkey.
- */
- Assert(OidIsValid(sgc->sortop));
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ found = true;
+ }
+ }
+
+ /* Check if the grouping clause does not cover this pathkey */
+ if (!found)
+ break;
new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
- new_group_clauses = lappend(new_group_clauses, sgc);
+
}
/* remember the number of pathkeys with a matching GROUP BY key */
n = list_length(new_group_pathkeys);
- /* append the remaining group pathkeys (will be treated as not sorted) */
+ /*
+ * Append the remaining group pathkeys (will be treated as not sorted) and
+ * grouping clauses.
+ */
*group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
*group_pathkeys);
*group_clauses = list_concat_unique_ptr(new_group_clauses,
@@ -544,6 +556,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
#endif
+
return infos;
}
--
2.39.3 (Apple Git-145)
[application/octet-stream] v4-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch (8.8K, ../../CAPpHfdsTUWnrNFRTenQpmv=JQ7yFdmoMbEADMSg8SNuHOZDFwQ@mail.gmail.com/5-v4-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch)
download | inline diff:
From eef9b88438eb4c3cc40d98f4367a5c6b4ec4a920 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:10:34 +0300
Subject: [PATCH v4 1/5] Fix asymmetry in setting EquivalenceClass.ec_sortref
0452b461bc made get_eclass_for_sort_expr() always set
EquivalenceClass.ec_sortref if it's not done yet. This leads to an asymmetric
situation when whoever first looks for the EquivalenceClass sets the
ec_sortref. It is also counterintuitive that get_eclass_for_sort_expr()
performs modification of data structures.
This commit makes make_pathkeys_for_sortclauses_extended() responsible for
setting EquivalenceClass.ec_sortref. Now we set the
EquivalenceClass.ec_sortref's needed to explore alternative GROUP BY ordering
specifically during building pathkeys by the list of grouping clauses.
Discussion: https://postgr.es/m/17037754-f187-4138-8285-0e2bfebd0dea%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/equivclass.c | 13 +------
src/backend/optimizer/path/pathkeys.c | 18 ++++++++-
src/backend/optimizer/plan/planner.c | 16 ++++++--
src/include/optimizer/paths.h | 3 +-
src/test/regress/expected/aggregates.out | 47 ++++++++++++++++++++++++
src/test/regress/sql/aggregates.sql | 14 +++++++
6 files changed, 92 insertions(+), 19 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 21ce1ae2e13..51d806326eb 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,18 +652,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from a WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 8b258cbef92..837e612121f 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -1355,7 +1355,8 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
&sortclauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
/* It's caller error if not all clauses were sortable */
Assert(sortable);
return result;
@@ -1379,13 +1380,17 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
* to remove any clauses that can be proven redundant via the eclass logic.
* Even though we'll have to hash in that case, we might as well not hash
* redundant columns.)
+ *
+ * If set_ec_sortref is true then sets the uninitialized value of ec_sortref
+ * of the pathkey's EquivalenceClass.
*/
List *
make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable)
+ bool *sortable,
+ bool set_ec_sortref)
{
List *pathkeys = NIL;
ListCell *l;
@@ -1409,6 +1414,15 @@ make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
sortcl->nulls_first,
sortcl->tleSortGroupRef,
true);
+ if (pathkey->pk_eclass->ec_sortref == 0 && set_ec_sortref)
+ {
+ /*
+ * Copy the sortref if it wasn't set yet. That may happen if the
+ * EquivalenceClass was constructed from a WHERE clause, i.e. it
+ * doesn't have a target reference at all.
+ */
+ pathkey->pk_eclass->ec_sortref = sortcl->tleSortGroupRef;
+ }
/* Canonical form eliminates redundant ordering keys */
if (!pathkey_is_redundant(pathkey, pathkeys))
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 032818423f6..669903b9059 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -3395,12 +3395,17 @@ standard_qp_callback(PlannerInfo *root, void *extra)
*/
bool sortable;
+ /*
+ * Covert group clauses into pathkeys. Set the ec_sortref field of
+ * EquivalenceClass'es if it's not set yet.
+ */
root->group_pathkeys =
make_pathkeys_for_sortclauses_extended(root,
&root->processed_groupClause,
tlist,
true,
- &sortable);
+ &sortable,
+ true);
if (!sortable)
{
/* Can't sort; no point in considering aggregate ordering either */
@@ -3450,7 +3455,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&root->processed_distinctClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->distinct_pathkeys = NIL;
}
@@ -3476,7 +3482,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&groupClauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->setop_pathkeys = NIL;
}
@@ -6061,7 +6068,8 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc,
&wc->partitionClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
Assert(sortable);
}
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 914d9bdef58..5e88c0224a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -239,7 +239,8 @@ extern List *make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable);
+ bool *sortable,
+ bool set_ec_sortref);
extern void initialize_mergeclause_eclasses(PlannerInfo *root,
RestrictInfo *restrictinfo);
extern void update_mergeclause_eclasses(PlannerInfo *root,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 2442342e9d1..1c1ca7573ad 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2917,6 +2917,53 @@ GROUP BY c1.w, c1.z;
5.0000000000000000
(2 rows)
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c1.x, c1.y
+ -> Incremental Sort
+ Sort Key: c1.x, c1.y
+ Presorted Key: c1.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c2.x, c1.y
+ -> Incremental Sort
+ Sort Key: c2.x, c1.y
+ Presorted Key: c2.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 61a3424c845..1a18ca3d8fe 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1263,6 +1263,20 @@ SELECT avg(c1.f ORDER BY c1.x, c1.y)
FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--
2.39.3 (Apple Git-145)
[application/octet-stream] v4-0002-Add-invariants-check-to-get_useful_group_keys_ord.patch (1.7K, ../../CAPpHfdsTUWnrNFRTenQpmv=JQ7yFdmoMbEADMSg8SNuHOZDFwQ@mail.gmail.com/6-v4-0002-Add-invariants-check-to-get_useful_group_keys_ord.patch)
download | inline diff:
From 61cd7d2c4439a5971a9f00930b08011f7e881814 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 30 May 2024 22:38:53 +0300
Subject: [PATCH v4 2/5] Add invariants check to
get_useful_group_keys_orderings()
This commit introduces invariants checking of generated orderings
in get_useful_group_keys_orderings for assert-enabled builds.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 28 +++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 837e612121f..1ed79797329 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -561,6 +561,34 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
+#ifdef USE_ASSERT_CHECKING
+ {
+ PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ ListCell *lc;
+
+ /* Test consistency of info structures */
+ for_each_from(lc, infos, 1)
+ {
+ ListCell *lc1,
+ *lc2;
+
+ info = lfirst_node(PathKeyInfo, lc);
+
+ Assert(list_length(info->clauses) == list_length(pinfo->clauses));
+ Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
+ Assert(list_difference(info->clauses, pinfo->clauses) == NIL);
+ Assert(list_difference_ptr(info->pathkeys, pinfo->pathkeys) == NIL);
+
+ forboth(lc1, info->clauses, lc2, info->pathkeys)
+ {
+ SortGroupClause *sgc = lfirst_node(SortGroupClause, lc1);
+ PathKey *pk = lfirst_node(PathKey, lc2);
+
+ Assert(pk->pk_eclass->ec_sortref == sgc->tleSortGroupRef);
+ }
+ }
+ }
+#endif
return infos;
}
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-06-03 15:54 Pavel Borisov <[email protected]>
parent: Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Pavel Borisov @ 2024-06-03 15:54 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: jian he <[email protected]>; Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
Hi, Alexander!
On Sun, 2 Jun 2024 at 17:18, Alexander Korotkov <[email protected]>
wrote:
> Hi!
>
> On Sun, Jun 2, 2024 at 10:55 AM jian he <[email protected]>
> wrote:
> >
> > On Fri, May 31, 2024 at 8:12 AM Alexander Korotkov <[email protected]>
> wrote:
> > >
> > > I've revised some grammar including the sentence you've proposed.
> > >
> >
> > -static List *groupclause_apply_groupingset(PlannerInfo *root, List
> *force);
> > +static List *preprocess_groupclause(PlannerInfo *root, List *force);
> >
> > changing preprocess_groupclause the second argument
> > from "force" to "gset" would be more intuitive, I think.
>
> Probably, but my intention is to restore preprocess_groupclause() as
> it was before 0452b461bc with minimal edits to support incremental
> sort. I'd rather avoid refactoring if this area for now.
>
> > `elog(ERROR, "Order of group-by clauses doesn't correspond incoming
> > sort order");`
> >
> > I think this error message makes people wonder what "incoming sort
> order" is.
> > BTW, "correspond", generally people use "correspond to".
>
> Thank you. On the second thought, I think it would be better to turn
> this into an assertion like the checks before.
>
> > I did some minor cosmetic changes, mainly changing foreach to
> foreach_node.
> > Please check the attachment.
>
> I would avoid refactoring of preprocess_groupclause() for the reason
> described above. But I picked the grammar fix for PlannerInfo's
> comment.
>
Thank you for working on this patchset.
0001: Patch looks right
Comments:
Covert -> Convert
sets the uninitialized value of ec_sortref of the pathkey's
EquivalenceClass -> sets the value of the pathkey's EquivalenceClass unless
it's already initialized
wasn't set yet -> hasn't been set yet
0002: additional assert checking only. Looks right.
0003: pure renaming, looks good.
0004: Restores pre 0452b461bc state to preprocess_groupclause with removed
partial_match fallback. Looks right. I haven't checked the comments
provided they are restored from pre 0452b461bc state.
0005: Looks right.
Regards,
Pavel Borisov
Supabase
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: POC: GROUP BY optimization
@ 2024-06-03 22:47 Alexander Korotkov <[email protected]>
parent: Pavel Borisov <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2024-06-03 22:47 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: jian he <[email protected]>; Andrei Lepikhov <[email protected]>; Tom Lane <[email protected]>; Richard Guo <[email protected]>; vignesh C <[email protected]>; PostgreSQL Developers <[email protected]>; Tomas Vondra <[email protected]>; Teodor Sigaev <[email protected]>; David Rowley <[email protected]>; a.rybakina <[email protected]>
On Mon, Jun 3, 2024 at 6:55 PM Pavel Borisov <[email protected]> wrote:
> On Sun, 2 Jun 2024 at 17:18, Alexander Korotkov <[email protected]> wrote:
>>
>> Hi!
>>
>> On Sun, Jun 2, 2024 at 10:55 AM jian he <[email protected]> wrote:
>> >
>> > On Fri, May 31, 2024 at 8:12 AM Alexander Korotkov <[email protected]> wrote:
>> > >
>> > > I've revised some grammar including the sentence you've proposed.
>> > >
>> >
>> > -static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
>> > +static List *preprocess_groupclause(PlannerInfo *root, List *force);
>> >
>> > changing preprocess_groupclause the second argument
>> > from "force" to "gset" would be more intuitive, I think.
>>
>> Probably, but my intention is to restore preprocess_groupclause() as
>> it was before 0452b461bc with minimal edits to support incremental
>> sort. I'd rather avoid refactoring if this area for now.
>>
>> > `elog(ERROR, "Order of group-by clauses doesn't correspond incoming
>> > sort order");`
>> >
>> > I think this error message makes people wonder what "incoming sort order" is.
>> > BTW, "correspond", generally people use "correspond to".
>>
>> Thank you. On the second thought, I think it would be better to turn
>> this into an assertion like the checks before.
>>
>> > I did some minor cosmetic changes, mainly changing foreach to foreach_node.
>> > Please check the attachment.
>>
>> I would avoid refactoring of preprocess_groupclause() for the reason
>> described above. But I picked the grammar fix for PlannerInfo's
>> comment.
>
>
> Thank you for working on this patchset.
>
> 0001: Patch looks right
>
> Comments:
>
> Covert -> Convert
> sets the uninitialized value of ec_sortref of the pathkey's EquivalenceClass -> sets the value of the pathkey's EquivalenceClass unless it's already initialized
> wasn't set yet -> hasn't been set yet
>
> 0002: additional assert checking only. Looks right.
>
> 0003: pure renaming, looks good.
>
> 0004: Restores pre 0452b461bc state to preprocess_groupclause with removed partial_match fallback. Looks right. I haven't checked the comments provided they are restored from pre 0452b461bc state.
>
> 0005: Looks right.
Thank you. Revised according to your comments. I think 0001-0004
comprising a good refactoring addressing concerns from Tom [1].
0001 Removes from get_eclass_for_sort_expr() unnatural responsibility
of setting EquivalenceClass.ec_sortref. Instead this field is set in
make_pathkeys_for_sortclauses_extended() while processing of group
clauses.
0002 Provides additional asserts. That should helping to defend
against lurking bugs.
0003 Fixes unsuitable name of data structure.
0004 Restores pre 0452b461bc state to preprocess_groupclause() and
lowers the number of paths to consider.
It would be good to explicitly hear Tom about this. But I'm quite
sure these patches makes situation better not worse. I'm going to
push them if no objections.
I'm putting 0005 aside. It's unclear why and how there could be
duplicate SortGroupClauses at that stage. Also, it's unclear why
existing code handles them bad. So, this should wait a comment from
Tom.
Links.
1. https://www.postgresql.org/message-id/266850.1712879082%40sss.pgh.pa.us
------
Regards,
Alexander Korotkov
Supabase
Attachments:
[application/octet-stream] v5-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch (8.8K, ../../CAPpHfdvS6Cnzz44_LES+z+erpch2yL8_m1MJ2aYLYhvGq6Vh2A@mail.gmail.com/2-v5-0001-Fix-asymmetry-in-setting-EquivalenceClass.ec_sort.patch)
download | inline diff:
From eef9b88438eb4c3cc40d98f4367a5c6b4ec4a920 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:10:34 +0300
Subject: [PATCH v5 1/5] Fix asymmetry in setting EquivalenceClass.ec_sortref
0452b461bc made get_eclass_for_sort_expr() always set
EquivalenceClass.ec_sortref if it's not done yet. This leads to an asymmetric
situation when whoever first looks for the EquivalenceClass sets the
ec_sortref. It is also counterintuitive that get_eclass_for_sort_expr()
performs modification of data structures.
This commit makes make_pathkeys_for_sortclauses_extended() responsible for
setting EquivalenceClass.ec_sortref. Now we set the
EquivalenceClass.ec_sortref's needed to explore alternative GROUP BY ordering
specifically during building pathkeys by the list of grouping clauses.
Discussion: https://postgr.es/m/17037754-f187-4138-8285-0e2bfebd0dea%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/equivclass.c | 13 +------
src/backend/optimizer/path/pathkeys.c | 18 ++++++++-
src/backend/optimizer/plan/planner.c | 16 ++++++--
src/include/optimizer/paths.h | 3 +-
src/test/regress/expected/aggregates.out | 47 ++++++++++++++++++++++++
src/test/regress/sql/aggregates.sql | 14 +++++++
6 files changed, 92 insertions(+), 19 deletions(-)
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 21ce1ae2e13..51d806326eb 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -652,18 +652,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from a WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 8b258cbef92..837e612121f 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -1355,7 +1355,8 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
&sortclauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
/* It's caller error if not all clauses were sortable */
Assert(sortable);
return result;
@@ -1379,13 +1380,17 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
* to remove any clauses that can be proven redundant via the eclass logic.
* Even though we'll have to hash in that case, we might as well not hash
* redundant columns.)
+ *
+ * If set_ec_sortref is true then sets the uninitialized value of ec_sortref
+ * of the pathkey's EquivalenceClass.
*/
List *
make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable)
+ bool *sortable,
+ bool set_ec_sortref)
{
List *pathkeys = NIL;
ListCell *l;
@@ -1409,6 +1414,15 @@ make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
sortcl->nulls_first,
sortcl->tleSortGroupRef,
true);
+ if (pathkey->pk_eclass->ec_sortref == 0 && set_ec_sortref)
+ {
+ /*
+ * Copy the sortref if it wasn't set yet. That may happen if the
+ * EquivalenceClass was constructed from a WHERE clause, i.e. it
+ * doesn't have a target reference at all.
+ */
+ pathkey->pk_eclass->ec_sortref = sortcl->tleSortGroupRef;
+ }
/* Canonical form eliminates redundant ordering keys */
if (!pathkey_is_redundant(pathkey, pathkeys))
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 032818423f6..669903b9059 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -3395,12 +3395,17 @@ standard_qp_callback(PlannerInfo *root, void *extra)
*/
bool sortable;
+ /*
+ * Covert group clauses into pathkeys. Set the ec_sortref field of
+ * EquivalenceClass'es if it's not set yet.
+ */
root->group_pathkeys =
make_pathkeys_for_sortclauses_extended(root,
&root->processed_groupClause,
tlist,
true,
- &sortable);
+ &sortable,
+ true);
if (!sortable)
{
/* Can't sort; no point in considering aggregate ordering either */
@@ -3450,7 +3455,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&root->processed_distinctClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->distinct_pathkeys = NIL;
}
@@ -3476,7 +3482,8 @@ standard_qp_callback(PlannerInfo *root, void *extra)
&groupClauses,
tlist,
false,
- &sortable);
+ &sortable,
+ false);
if (!sortable)
root->setop_pathkeys = NIL;
}
@@ -6061,7 +6068,8 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc,
&wc->partitionClause,
tlist,
true,
- &sortable);
+ &sortable,
+ false);
Assert(sortable);
}
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 914d9bdef58..5e88c0224a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -239,7 +239,8 @@ extern List *make_pathkeys_for_sortclauses_extended(PlannerInfo *root,
List **sortclauses,
List *tlist,
bool remove_redundant,
- bool *sortable);
+ bool *sortable,
+ bool set_ec_sortref);
extern void initialize_mergeclause_eclasses(PlannerInfo *root,
RestrictInfo *restrictinfo);
extern void update_mergeclause_eclasses(PlannerInfo *root,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 2442342e9d1..1c1ca7573ad 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -2917,6 +2917,53 @@ GROUP BY c1.w, c1.z;
5.0000000000000000
(2 rows)
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c1.x, c1.y
+ -> Incremental Sort
+ Sort Key: c1.x, c1.y
+ Presorted Key: c1.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+ QUERY PLAN
+-----------------------------------------------------
+ Group
+ Group Key: c2.x, c1.y
+ -> Incremental Sort
+ Sort Key: c2.x, c1.y
+ Presorted Key: c2.x
+ -> Merge Join
+ Merge Cond: (c1.x = c2.x)
+ -> Sort
+ Sort Key: c1.x
+ -> Seq Scan on group_agg_pk c1
+ -> Sort
+ Sort Key: c2.x
+ -> Seq Scan on group_agg_pk c2
+(13 rows)
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 61a3424c845..1a18ca3d8fe 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1263,6 +1263,20 @@ SELECT avg(c1.f ORDER BY c1.x, c1.y)
FROM group_agg_pk c1 JOIN group_agg_pk c2 ON c1.x = c2.x
GROUP BY c1.w, c1.z;
+-- Pathkeys, built in a subtree, can be used to optimize GROUP-BY clause
+-- ordering. Also, here we check that it doesn't depend on the initial clause
+-- order in the GROUP-BY list.
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c1.x,c2.x;
+EXPLAIN (COSTS OFF)
+SELECT c1.y,c1.x FROM group_agg_pk c1
+ JOIN group_agg_pk c2
+ ON c1.x = c2.x
+GROUP BY c1.y,c2.x,c1.x;
+
RESET enable_nestloop;
RESET enable_hashjoin;
DROP TABLE group_agg_pk;
--
2.39.3 (Apple Git-145)
[application/octet-stream] v5-0002-Add-invariants-check-to-get_useful_group_keys_ord.patch (1.7K, ../../CAPpHfdvS6Cnzz44_LES+z+erpch2yL8_m1MJ2aYLYhvGq6Vh2A@mail.gmail.com/3-v5-0002-Add-invariants-check-to-get_useful_group_keys_ord.patch)
download | inline diff:
From 61cd7d2c4439a5971a9f00930b08011f7e881814 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Thu, 30 May 2024 22:38:53 +0300
Subject: [PATCH v5 2/5] Add invariants check to
get_useful_group_keys_orderings()
This commit introduces invariants checking of generated orderings
in get_useful_group_keys_orderings for assert-enabled builds.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 28 +++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 837e612121f..1ed79797329 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -561,6 +561,34 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
+#ifdef USE_ASSERT_CHECKING
+ {
+ PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ ListCell *lc;
+
+ /* Test consistency of info structures */
+ for_each_from(lc, infos, 1)
+ {
+ ListCell *lc1,
+ *lc2;
+
+ info = lfirst_node(PathKeyInfo, lc);
+
+ Assert(list_length(info->clauses) == list_length(pinfo->clauses));
+ Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
+ Assert(list_difference(info->clauses, pinfo->clauses) == NIL);
+ Assert(list_difference_ptr(info->pathkeys, pinfo->pathkeys) == NIL);
+
+ forboth(lc1, info->clauses, lc2, info->pathkeys)
+ {
+ SortGroupClause *sgc = lfirst_node(SortGroupClause, lc1);
+ PathKey *pk = lfirst_node(PathKey, lc2);
+
+ Assert(pk->pk_eclass->ec_sortref == sgc->tleSortGroupRef);
+ }
+ }
+ }
+#endif
return infos;
}
--
2.39.3 (Apple Git-145)
[application/octet-stream] v5-0004-Restore-preprocess_groupclause.patch (13.1K, ../../CAPpHfdvS6Cnzz44_LES+z+erpch2yL8_m1MJ2aYLYhvGq6Vh2A@mail.gmail.com/4-v5-0004-Restore-preprocess_groupclause.patch)
download | inline diff:
From 087791dc5de8626337b9a7ab6099c8c7794a763c Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:51:31 +0300
Subject: [PATCH v5 4/5] Restore preprocess_groupclause()
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
It eliminated preprocess_groupclause(), which was intended to match items
between GROUP BY and ORDER BY. Instead, get_useful_group_keys_orderings()
function generates orderings of GROUP BY elements at the time of grouping
paths generation. The get_useful_group_keys_orderings() function takes into
account 3 orderings of GROUP BY pathkeys and clauses: original order as written
in GROUP BY, matching ORDER BY clauses as much as possible, and matching the
input path as much as possible. Given that even before 0452b461b,
preprocess_groupclause() could change the original order of GROUP BY clauses
we don't need to consider it apart from ordering matching ORDER BY clauses.
This commit restores preprocess_groupclause() to provide an ordering of
GROUP BY elements matching ORDER BY before generation of paths. The new
version of preprocess_groupclause() takes into account an incremental sort.
The get_useful_group_keys_orderings() function now takes into 2 orderings of
GROUP BY elements: the order generated preprocess_groupclause() and the order
matching the input path as much as possible.
Discussion: https://postgr.es/m/CAPpHfdvyWLMGwvxaf%3D7KAp-z-4mxbSH8ti2f6mNOQv5metZFzg%40mail.gmail.com
Author: Alexander Korotkov
---
src/backend/optimizer/path/pathkeys.c | 55 +--------
src/backend/optimizer/plan/planner.c | 108 +++++++++++++++---
src/include/nodes/pathnodes.h | 6 +-
.../regress/expected/partition_aggregate.out | 6 +-
4 files changed, 108 insertions(+), 67 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index c98486e292c..793c8ad2930 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -447,26 +447,6 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
return n;
}
-/*
- * pathkeys_are_duplicate
- * Check if give pathkeys are already contained the list of
- * GroupByOrdering's.
- */
-static bool
-pathkeys_are_duplicate(List *infos, List *pathkeys)
-{
- ListCell *lc;
-
- foreach(lc, infos)
- {
- GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
-
- if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
- return true;
- }
- return false;
-}
-
/*
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
@@ -475,11 +455,11 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
- * The function considers (and keeps) multiple GROUP BY orderings:
+ * The function considers (and keeps) following GROUP BY orderings:
*
- * - the original ordering, as specified by the GROUP BY clause,
- * - GROUP BY keys reordered to match 'path' ordering (as much as possible),
- * - GROUP BY keys to match target ORDER BY clause (as much as possible).
+ * - GROUP BY keys as ordered by preprocess_groupclause() to match target
+ * ORDER BY clause (as much as possible),
+ * - GROUP BY keys reordered to match 'path' ordering (as much as possible).
*/
List *
get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
@@ -526,32 +506,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
if (n > 0 &&
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
- !pathkeys_are_duplicate(infos, pathkeys))
- {
- info = makeNode(GroupByOrdering);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause).
- */
- if (root->sort_pathkeys &&
- !pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys))
- {
- int n;
-
- n = group_keys_reorder_by_pathkeys(root->sort_pathkeys, &pathkeys,
- &clauses,
- root->num_groupby_pathkeys);
-
- if (n > 0 &&
- (enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
- !pathkeys_are_duplicate(infos, pathkeys))
+ compare_pathkeys(pathkeys, root->group_pathkeys) != PATHKEYS_EQUAL)
{
info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 1eeedc121ed..d6a1fb8e7d5 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -137,7 +137,7 @@ static double preprocess_limit(PlannerInfo *root,
double tuple_fraction,
int64 *offset_est, int64 *count_est);
static void remove_useless_groupby_columns(PlannerInfo *root);
-static List *groupclause_apply_groupingset(PlannerInfo *root, List *force);
+static List *preprocess_groupclause(PlannerInfo *root, List *force);
static List *extract_rollup_sets(List *groupingSets);
static List *reorder_grouping_sets(List *groupingSets, List *sortclause);
static void standard_qp_callback(PlannerInfo *root, void *extra);
@@ -1422,7 +1422,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction,
else if (parse->groupClause)
{
/* Preprocess regular GROUP BY clause, if any */
- root->processed_groupClause = list_copy(parse->groupClause);
+ root->processed_groupClause = preprocess_groupclause(root, NIL);
/* Remove any redundant GROUP BY columns */
remove_useless_groupby_columns(root);
}
@@ -2169,7 +2169,7 @@ preprocess_grouping_sets(PlannerInfo *root)
* The groupClauses for hashed grouping sets are built later on.)
*/
if (gs->set)
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
else
rollup->groupClause = NIL;
@@ -2821,24 +2821,106 @@ remove_useless_groupby_columns(PlannerInfo *root)
}
/*
- * groupclause_apply_groupingset
- * Apply the order of GROUP BY clauses defined by grouping sets. Items
- * not in the grouping set are skipped.
+ * preprocess_groupclause - do preparatory work on GROUP BY clause
+ *
+ * The idea here is to adjust the ordering of the GROUP BY elements
+ * (which in itself is semantically insignificant) to match ORDER BY,
+ * thereby allowing a single sort operation to both implement the ORDER BY
+ * requirement and set up for a Unique step that implements GROUP BY.
+ * We also consider partial match between GROUP BY and ORDER BY elements,
+ * which could allow to implement ORDER BY using the incremental sort.
+ *
+ * We also consider other orderings of the GROUP BY elements, which could
+ * match the sort ordering of other possible plans (eg an indexscan) and
+ * thereby reduce cost. This is implemented during the generation of grouping
+ * paths. See get_useful_group_keys_orderings() for details.
+ *
+ * Note: we need no comparable processing of the distinctClause because
+ * the parser already enforced that that matches ORDER BY.
+ *
+ * Note: we return a fresh List, but its elements are the same
+ * SortGroupClauses appearing in parse->groupClause. This is important
+ * because later processing may modify the processed_groupClause list.
+ *
+ * For grouping sets, the order of items is instead forced to agree with that
+ * of the grouping set (and items not in the grouping set are skipped). The
+ * work of sorting the order of grouping set elements to match the ORDER BY if
+ * possible is done elsewhere.
*/
static List *
-groupclause_apply_groupingset(PlannerInfo *root, List *gset)
+preprocess_groupclause(PlannerInfo *root, List *force)
{
Query *parse = root->parse;
List *new_groupclause = NIL;
ListCell *sl;
+ ListCell *gl;
- foreach(sl, gset)
+ /* For grouping sets, we need to force the ordering */
+ if (force)
{
- Index ref = lfirst_int(sl);
- SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+ foreach(sl, force)
+ {
+ Index ref = lfirst_int(sl);
+ SortGroupClause *cl = get_sortgroupref_clause(ref, parse->groupClause);
+
+ new_groupclause = lappend(new_groupclause, cl);
+ }
- new_groupclause = lappend(new_groupclause, cl);
+ return new_groupclause;
}
+
+ /* If no ORDER BY, nothing useful to do here */
+ if (parse->sortClause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Scan the ORDER BY clause and construct a list of matching GROUP BY
+ * items, but only as far as we can make a matching prefix.
+ *
+ * This code assumes that the sortClause contains no duplicate items.
+ */
+ foreach(sl, parse->sortClause)
+ {
+ SortGroupClause *sc = lfirst_node(SortGroupClause, sl);
+
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (equal(gc, sc))
+ {
+ new_groupclause = lappend(new_groupclause, gc);
+ break;
+ }
+ }
+ if (gl == NULL)
+ break; /* no match, so stop scanning */
+ }
+
+
+ /* If no match at all, no point in reordering GROUP BY */
+ if (new_groupclause == NIL)
+ return list_copy(parse->groupClause);
+
+ /*
+ * Add any remaining GROUP BY items to the new list. We don't require a
+ * complete match, because even partial match allows ORDER BY to be
+ * implemented using incremental sort. Also, give up if there are any
+ * non-sortable GROUP BY items, since then there's no hope anyway.
+ */
+ foreach(gl, parse->groupClause)
+ {
+ SortGroupClause *gc = lfirst_node(SortGroupClause, gl);
+
+ if (list_member_ptr(new_groupclause, gc))
+ continue; /* it matched an ORDER BY item */
+ if (!OidIsValid(gc->sortop)) /* give up, GROUP BY can't be sorted */
+ return list_copy(parse->groupClause);
+ new_groupclause = lappend(new_groupclause, gc);
+ }
+
+ /* Success --- install the rearranged GROUP BY list */
+ Assert(list_length(parse->groupClause) == list_length(new_groupclause));
return new_groupclause;
}
@@ -4170,7 +4252,7 @@ consider_groupingsets_paths(PlannerInfo *root,
{
rollup = makeNode(RollupData);
- rollup->groupClause = groupclause_apply_groupingset(root, gset);
+ rollup->groupClause = preprocess_groupclause(root, gset);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
@@ -4359,7 +4441,7 @@ consider_groupingsets_paths(PlannerInfo *root,
Assert(gs->set != NIL);
- rollup->groupClause = groupclause_apply_groupingset(root, gs->set);
+ rollup->groupClause = preprocess_groupclause(root, gs->set);
rollup->gsets_data = list_make1(gs);
rollup->gsets = remap_to_groupclause_idx(rollup->groupClause,
rollup->gsets_data,
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 78489398294..2ba297c1172 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -426,7 +426,11 @@ struct PlannerInfo
* items to be proven redundant, implying that there is only one group
* containing all the query's rows. Hence, if you want to check whether
* GROUP BY was specified, test for nonempty parse->groupClause, not for
- * nonempty processed_groupClause.
+ * nonempty processed_groupClause. Optimizer chooses specific order of
+ * group-by clauses during the upper paths generation process, attempting
+ * to use different strategies to minimize number of sorts or engage
+ * incremental sort. See preprocess_groupclause() and
+ * get_useful_group_keys_orderings() for details.
*
* Currently, when grouping sets are specified we do not attempt to
* optimize the groupClause, so that processed_groupClause will be
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index 1b900fddf8e..5f2c0cf5786 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -901,15 +901,15 @@ SELECT a, c, sum(b), avg(c), count(*) FROM pagg_tab_m GROUP BY (a+b)/2, 2, 1 HAV
Sort Key: pagg_tab_m.a, pagg_tab_m.c, (sum(pagg_tab_m.b))
-> Append
-> HashAggregate
- Group Key: ((pagg_tab_m.a + pagg_tab_m.b) / 2), pagg_tab_m.c, pagg_tab_m.a
+ Group Key: pagg_tab_m.a, pagg_tab_m.c, ((pagg_tab_m.a + pagg_tab_m.b) / 2)
Filter: ((sum(pagg_tab_m.b) = 50) AND (avg(pagg_tab_m.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p1 pagg_tab_m
-> HashAggregate
- Group Key: ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2), pagg_tab_m_1.c, pagg_tab_m_1.a
+ Group Key: pagg_tab_m_1.a, pagg_tab_m_1.c, ((pagg_tab_m_1.a + pagg_tab_m_1.b) / 2)
Filter: ((sum(pagg_tab_m_1.b) = 50) AND (avg(pagg_tab_m_1.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p2 pagg_tab_m_1
-> HashAggregate
- Group Key: ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2), pagg_tab_m_2.c, pagg_tab_m_2.a
+ Group Key: pagg_tab_m_2.a, pagg_tab_m_2.c, ((pagg_tab_m_2.a + pagg_tab_m_2.b) / 2)
Filter: ((sum(pagg_tab_m_2.b) = 50) AND (avg(pagg_tab_m_2.c) > '25'::numeric))
-> Seq Scan on pagg_tab_m_p3 pagg_tab_m_2
(15 rows)
--
2.39.3 (Apple Git-145)
[application/octet-stream] v5-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch (7.0K, ../../CAPpHfdvS6Cnzz44_LES+z+erpch2yL8_m1MJ2aYLYhvGq6Vh2A@mail.gmail.com/5-v5-0003-Rename-PathKeyInfo-to-GroupByOrdering.patch)
download | inline diff:
From 74734378e209cf37124cb12b7d80717d49d56d42 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Mon, 27 May 2024 14:57:43 +0300
Subject: [PATCH v5 3/5] Rename PathKeyInfo to GroupByOrdering
0452b461bc made optimizer explore alternative orderings of group-by pathkeys.
The PathKeyInfo data structure was used to store the particular ordering of
group-by pathkeys and corresponding clauses. It turns out that PathKeyInfo
is not the best name for that purpose. This commit renames this data structure
to GroupByOrdering, and revises its comment.
Discussion: https://postgr.es/m/db0fc3a4-966c-4cec-a136-94024d39212d%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 18 +++++++++---------
src/backend/optimizer/plan/planner.c | 8 ++++----
src/include/nodes/pathnodes.h | 13 ++++++++++---
src/tools/pgindent/typedefs.list | 2 +-
4 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1ed79797329..c98486e292c 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -450,7 +450,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* pathkeys_are_duplicate
* Check if give pathkeys are already contained the list of
- * PathKeyInfo's.
+ * GroupByOrdering's.
*/
static bool
pathkeys_are_duplicate(List *infos, List *pathkeys)
@@ -459,7 +459,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
foreach(lc, infos)
{
- PathKeyInfo *info = lfirst_node(PathKeyInfo, lc);
+ GroupByOrdering *info = lfirst_node(GroupByOrdering, lc);
if (compare_pathkeys(pathkeys, info->pathkeys) == PATHKEYS_EQUAL)
return true;
@@ -471,7 +471,7 @@ pathkeys_are_duplicate(List *infos, List *pathkeys)
* get_useful_group_keys_orderings
* Determine which orderings of GROUP BY keys are potentially interesting.
*
- * Returns a list of PathKeyInfo items, each representing an interesting
+ * Returns a list of GroupByOrdering items, each representing an interesting
* ordering of GROUP BY keys. Each item stores pathkeys and clauses in the
* matching order.
*
@@ -486,13 +486,13 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
{
Query *parse = root->parse;
List *infos = NIL;
- PathKeyInfo *info;
+ GroupByOrdering *info;
List *pathkeys = root->group_pathkeys;
List *clauses = root->processed_groupClause;
/* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
infos = lappend(infos, info);
@@ -528,7 +528,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == root->num_groupby_pathkeys) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -553,7 +553,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
(enable_incremental_sort || n == list_length(root->sort_pathkeys)) &&
!pathkeys_are_duplicate(infos, pathkeys))
{
- info = makeNode(PathKeyInfo);
+ info = makeNode(GroupByOrdering);
info->pathkeys = pathkeys;
info->clauses = clauses;
@@ -563,7 +563,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
#ifdef USE_ASSERT_CHECKING
{
- PathKeyInfo *pinfo = linitial_node(PathKeyInfo, infos);
+ GroupByOrdering *pinfo = linitial_node(GroupByOrdering, infos);
ListCell *lc;
/* Test consistency of info structures */
@@ -572,7 +572,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
ListCell *lc1,
*lc2;
- info = lfirst_node(PathKeyInfo, lc);
+ info = lfirst_node(GroupByOrdering, lc);
Assert(list_length(info->clauses) == list_length(pinfo->clauses));
Assert(list_length(info->pathkeys) == list_length(pinfo->pathkeys));
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 669903b9059..1eeedc121ed 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6917,7 +6917,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -6998,7 +6998,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7249,7 +7249,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
path = path_save;
@@ -7305,7 +7305,7 @@ create_partial_grouping_paths(PlannerInfo *root,
/* process all potentially interesting grouping reorderings */
foreach(lc2, pathkey_orderings)
{
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
+ GroupByOrdering *info = (GroupByOrdering *) lfirst(lc2);
/* restore the path (we replace it in the loop) */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 14ef296ab72..78489398294 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1468,14 +1468,21 @@ typedef struct PathKey
} PathKey;
/*
- * Combines the information about pathkeys and the associated clauses.
+ * Contains an order of group-by clauses and the corresponding list of
+ * pathkeys.
+ *
+ * The elements of 'clauses' list should have the same order as the head of
+ * 'pathkeys' list. The tleSortGroupRef of the clause should be equal to
+ * ec_sortref of the pathkey equivalence class. If there are redundant
+ * clauses with the same tleSortGroupRef, they must be grouped together.
*/
-typedef struct PathKeyInfo
+typedef struct GroupByOrdering
{
NodeTag type;
+
List *pathkeys;
List *clauses;
-} PathKeyInfo;
+} GroupByOrdering;
/*
* VolatileFunctionStatus -- allows nodes to cache their
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d427a1c16a5..4f57078d133 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1056,6 +1056,7 @@ GrantRoleStmt
GrantStmt
GrantTargetType
Group
+GroupByOrdering
GroupClause
GroupPath
GroupPathExtraData
@@ -2067,7 +2068,6 @@ PathClauseUsage
PathCostComparison
PathHashStack
PathKey
-PathKeyInfo
PathKeysComparison
PathTarget
PatternInfo
--
2.39.3 (Apple Git-145)
[application/octet-stream] v5-0005-Teach-group_keys_reorder_by_pathkeys-about-redund.patch (3.2K, ../../CAPpHfdvS6Cnzz44_LES+z+erpch2yL8_m1MJ2aYLYhvGq6Vh2A@mail.gmail.com/6-v5-0005-Teach-group_keys_reorder_by_pathkeys-about-redund.patch)
download | inline diff:
From 0e33f7c66682401180e037ec0fd8564401241c0a Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 21 May 2024 03:13:11 +0300
Subject: [PATCH v5 5/5] Teach group_keys_reorder_by_pathkeys() about redundant
SortGroupClause's
We don't have strict guarantees that there shouldn't be redundant
SortGroupClause's. Although there is no confirmed way to generate
duplicate SortGroupClause's, this commit teaches
group_keys_reorder_by_pathkeys() to pull all the SortGroupClause matching to
pathkey at once.
Discussion: https://postgr.es/m/a663f0f6-cbf6-49aa-af2e-234dc6768a07%40postgrespro.ru
Author: Andrei Lepikhov
---
src/backend/optimizer/path/pathkeys.c | 41 ++++++++++++++++++---------
1 file changed, 27 insertions(+), 14 deletions(-)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 793c8ad2930..0b7e9cd48db 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -401,7 +401,7 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
+ bool found = false;
/*
* Pathkeys are built in a way that allows simply comparing pointers.
@@ -416,28 +416,40 @@ group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
/*
* Since 1349d27 pathkey coming from underlying node can be in the
* root->group_pathkeys but not in the processed_groupClause. So, we
- * should be careful here.
+ * should be careful here. Also, there could be redundant
+ * SortGroupClause's. So, we need to pull of all the matching
+ * SortGroupClause's, but ensure there is at least one.
*/
- sgc = get_sortgroupref_clause_noerr(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
- if (!sgc)
- /* The grouping clause does not cover this pathkey */
- break;
+ foreach_ptr(SortGroupClause, sgc, *group_clauses)
+ {
+ if (sgc->tleSortGroupRef == pathkey->pk_eclass->ec_sortref)
+ {
+ /*
+ * Sort group clause should have an ordering operator as long
+ * as there is an associated pathkey.
+ */
+ Assert(OidIsValid(sgc->sortop));
- /*
- * Sort group clause should have an ordering operator as long as there
- * is an associated pathkey.
- */
- Assert(OidIsValid(sgc->sortop));
+ new_group_clauses = lappend(new_group_clauses, sgc);
+ found = true;
+ }
+ }
+
+ /* Check if the grouping clause does not cover this pathkey */
+ if (!found)
+ break;
new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
- new_group_clauses = lappend(new_group_clauses, sgc);
+
}
/* remember the number of pathkeys with a matching GROUP BY key */
n = list_length(new_group_pathkeys);
- /* append the remaining group pathkeys (will be treated as not sorted) */
+ /*
+ * Append the remaining group pathkeys (will be treated as not sorted) and
+ * grouping clauses.
+ */
*group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
*group_pathkeys);
*group_clauses = list_concat_unique_ptr(new_group_clauses,
@@ -544,6 +556,7 @@ get_useful_group_keys_orderings(PlannerInfo *root, Path *path)
}
}
#endif
+
return infos;
}
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 67+ messages in thread
end of thread, other threads:[~2024-06-03 22:47 UTC | newest]
Thread overview: 67+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-06-02 13:38 Support for CREATE MODULE? Jim Mlodgenski <[email protected]>
2021-06-02 13:58 ` Tom Lane <[email protected]>
2021-06-02 14:43 ` Jim Mlodgenski <[email protected]>
2021-06-02 15:07 ` Bruce Momjian <[email protected]>
2021-06-02 15:14 ` Tom Lane <[email protected]>
2021-06-02 15:11 ` Joe Conway <[email protected]>
2021-06-02 15:22 ` Joel Jacobson <[email protected]>
2021-06-03 12:49 ` Peter Eisentraut <[email protected]>
2021-06-04 19:31 ` Jim Mlodgenski <[email protected]>
2021-06-02 19:08 ` Alvaro Herrera <[email protected]>
2021-06-04 20:09 ` Pavel Stehule <[email protected]>
2021-08-02 17:42 [PATCH v2 2/3] Calculate MaxBackends earlier in PostmasterMain(). Nathan Bossart <[email protected]>
2021-08-02 17:42 [PATCH v1 2/3] Calculate MaxBackends earlier in PostmasterMain(). Nathan Bossart <[email protected]>
2022-07-18 18:06 Re: support for CREATE MODULE Nathan Bossart <[email protected]>
2024-01-13 15:00 Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-01-14 12:14 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-01-15 00:19 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-01-15 06:42 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-01-15 07:56 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-01-16 02:48 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-01-16 15:05 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-01-18 09:48 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-01-18 14:18 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-01-19 14:21 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-01-24 17:37 ` Re: POC: GROUP BY optimization Nathan Bossart <[email protected]>
2024-01-24 19:44 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-01-26 14:37 ` Re: POC: GROUP BY optimization vignesh C <[email protected]>
2024-02-02 02:02 ` Re: POC: GROUP BY optimization Tom Lane <[email protected]>
2024-02-02 03:32 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-02-02 04:06 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-02-02 04:40 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-02-21 08:08 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-02-21 10:19 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-02-21 15:08 ` Re: POC: GROUP BY optimization Maxim Orlov <[email protected]>
2024-02-22 02:09 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-02-22 04:18 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-02-22 06:35 ` Re: POC: GROUP BY optimization Richard Guo <[email protected]>
2024-02-22 07:04 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-04-11 23:44 ` Re: POC: GROUP BY optimization Tom Lane <[email protected]>
2024-04-12 05:05 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-04-16 05:59 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-04-17 03:11 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-04-18 05:01 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-04-18 08:54 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-04-18 10:57 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-04-19 10:44 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-04-22 08:56 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-04-22 22:19 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-04-24 06:25 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-04-24 11:17 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-04-29 02:50 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-05-16 07:47 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-05-20 08:54 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-05-21 09:01 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-05-24 00:00 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-05-27 12:41 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-05-29 08:08 ` About 0001:,Having overviewed it, I don't see any issues (but I'm the author), except grammatical ones - but I'm not a native to judge it.,Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear to me. It may be better to write something like: 'building pathkeys by the list of grouping clauses'.,,0002:,The part under USE_ASSERT_CHECKING looks good to me. But the code in group_keys_reorder_by_pathkeys looks suspicious: of course, we do some doubtful work without any possible way to reproduce, but if we envision some duplicated elements in the group_clauses, we should avoid usage of the list_concat_unique_ptr. What's more, why do you not exit from foreach_ptr immediately after SortGroupClause has been found? I think the new_group_clauses should be consistent with the new_group_pathkeys.,,0003:,Looks good,,0004:,I was also thinking about reintroducing the preprocess_groupclause because with the re-arrangement of GROUP-BY clauses according to incoming pathkeys, it doesn't make sense to have a user-defined order—at least while cost_sort doesn't differ costs for alternative column orderings.,So, I'm okay with the code. But why don't you use the same approach with foreach_ptr as before? Andrei Lepikhov <[email protected]>
2024-05-29 12:53 ` Re: About 0001:,Having overviewed it, I don't see any issues (but I'm the author), except grammatical ones - but I'm not a native to judge it.,Also, the sentence 'turning GROUP BY clauses into pathkeys' is unclear to me. It may be better to write something like: 'building pathkeys by the list of grouping clauses'.,,0002:,The part under USE_ASSERT_CHECKING looks good to me. But the code in group_keys_reorder_by_pathkeys looks suspicious: of course, we do some doubtful work without any possible way to reproduce, but if we envision some duplicated elements in the group_clauses, we should avoid usage of the list_concat_unique_ptr. What's more, why do you not exit from foreach_ptr immediately after SortGroupClause has been found? I think the new_group_clauses should be consistent with the new_group_pathkeys.,,0003:,Looks good,,0004:,I was also thinking about reintroducing the preprocess_groupclause because with the re-arrangement of GROUP-BY clauses according to incoming pathkeys, it d... Alexander Korotkov <[email protected]>
2024-05-30 04:22 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-05-31 00:12 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-06-02 07:55 ` Re: POC: GROUP BY optimization jian he <[email protected]>
2024-06-02 13:18 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-06-03 15:54 ` Re: POC: GROUP BY optimization Pavel Borisov <[email protected]>
2024-06-03 22:47 ` Re: POC: GROUP BY optimization Alexander Korotkov <[email protected]>
2024-04-22 03:51 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-02-02 03:42 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
2024-01-15 09:46 ` Re: POC: GROUP BY optimization Andrei Lepikhov <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox