public inbox for [email protected]
help / color / mirror / Atom feedFrom: Lepikhov Andrei <[email protected]>
To: [email protected]
Cc: Евгений Бредня <[email protected]>
Subject: Optimize planner memory consumption for huge arrays
Date: Mon, 04 Sep 2023 12:25:44 +0700
Message-ID: <[email protected]> (raw)
Hi, hackers,
Looking at the planner behaviour with the memory consumption patch [1], I figured out that arrays increase memory consumption by the optimizer significantly. See init.sql in attachment.
The point here is that the planner does small memory allocations for each element during estimation. As a result, it looks like the planner consumes about 250 bytes for each integer element.
It is maybe not a problem most of the time. However, in the case of partitions, memory consumption multiplies by each partition. Such a corner case looks weird, but the fix is simple. So, why not?
The diff in the attachment is proof of concept showing how to reduce wasting of memory. Having benchmarked a bit, I didn't find any overhead.
[1] Report planning memory in EXPLAIN ANALYZE
https://www.postgresql.org/message-id/flat/CAExHW5sZA=5LJ_ZPpRO-w09ck8z9p7eaYAqq3Ks9GDfhrxeWBw@mail....
--
Regards,
Andrey Lepikhov
Attachments:
[application/sql] init.sql (344B, ../[email protected]/2-init.sql)
download
[application/octet-stream] array_compact_memusage.diff (1.5K, ../[email protected]/3-array_compact_memusage.diff)
download | inline diff:
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index fe37e65af0..963464c9ad 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -280,6 +280,7 @@ eqsel_internal(PG_FUNCTION_ARGS, bool negate)
selec = var_eq_non_const(&vardata, operator, collation, other,
varonleft, negate);
+ pfree(other);
ReleaseVariableStats(vardata);
return selec;
@@ -1960,15 +1961,15 @@ scalararraysel(PlannerInfo *root,
{
List *args;
Selectivity s2;
-
- args = list_make2(leftop,
- makeConst(nominal_element_type,
- -1,
- nominal_element_collation,
- elmlen,
- elem_values[i],
- elem_nulls[i],
- elmbyval));
+ Const *c = makeConst(nominal_element_type,
+ -1,
+ nominal_element_collation,
+ elmlen,
+ elem_values[i],
+ elem_nulls[i],
+ elmbyval);
+
+ args = list_make2(leftop, c);
if (is_join_clause)
s2 = DatumGetFloat8(FunctionCall5Coll(&oprselproc,
clause->inputcollid,
@@ -1984,7 +1985,8 @@ scalararraysel(PlannerInfo *root,
ObjectIdGetDatum(operator),
PointerGetDatum(args),
Int32GetDatum(varRelid)));
-
+ list_free(args);
+ pfree(c);
if (useOr)
{
s1 = s1 + s2 - s1 * s2;
@@ -2051,6 +2053,7 @@ scalararraysel(PlannerInfo *root,
ObjectIdGetDatum(operator),
PointerGetDatum(args),
Int32GetDatum(varRelid)));
+ list_free(args);
if (useOr)
{
view thread (4+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Optimize planner memory consumption for huge arrays
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox