public inbox for [email protected]  

help / color / mirror / Atom feed

From: Dilip Kumar <[email protected]>
To: Lepikhov Andrei <[email protected]>
Cc: [email protected]
Cc: Евгений Бредня <[email protected]>
Subject: Re: Optimize planner memory consumption for huge arrays
Date: Mon, 4 Sep 2023 14:07:03 +0530
Message-ID: <CAFiTN-sghJuMxwH+uAjvm4_L5-RvhH9T9FciPTnLCpKQRFGfiA@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>

On Mon, Sep 4, 2023 at 11:58 AM Lepikhov Andrei
<[email protected]> wrote:
>
> Hi, hackers,
>
> Looking at the planner behaviour with the memory consumption patch [1], I figured out that arrays increase memory consumption by the optimizer significantly. See init.sql in attachment.
> The point here is that the planner does small memory allocations for each element during estimation. As a result, it looks like the planner consumes about 250 bytes for each integer element.
>
> It is maybe not a problem most of the time. However, in the case of partitions, memory consumption multiplies by each partition. Such a corner case looks weird, but the fix is simple. So, why not?
>
> The diff in the attachment is proof of concept showing how to reduce wasting of memory. Having benchmarked a bit, I didn't find any overhead.

+ Const *c = makeConst(nominal_element_type,
+ -1,
+ nominal_element_collation,
+ elmlen,
+ elem_values[i],
+ elem_nulls[i],
+ elmbyval);
+
+ args = list_make2(leftop, c);
  if (is_join_clause)
  s2 = DatumGetFloat8(FunctionCall5Coll(&oprselproc,
    clause->inputcollid,
@@ -1984,7 +1985,8 @@ scalararraysel(PlannerInfo *root,
    ObjectIdGetDatum(operator),
    PointerGetDatum(args),
    Int32GetDatum(varRelid)));
-
+ list_free(args);
+ pfree(c);

Maybe you can just use list_free_deep, instead of storing the constant
in a separate variable.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

view thread (4+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Optimize planner memory consumption for huge arrays
  In-Reply-To: <CAFiTN-sghJuMxwH+uAjvm4_L5-RvhH9T9FciPTnLCpKQRFGfiA@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox