Feedback-ID: ie8b149e1:Fastmail
User-Agent: Cyrus-JMAP/3.9.0-alpha0-701-g9b2f44d3ee-fm-20230823.001-g9b2f44d3
Mime-Version: 1.0
Message-Id: <8498b480-51f8-4827-ac3a-c0c87872810c@app.fastmail.com>
In-Reply-To: 
 <CAFiTN-sghJuMxwH+uAjvm4_L5-RvhH9T9FciPTnLCpKQRFGfiA@mail.gmail.com>
References: <em9939439a-441a-4b27-a977-ebdf5987dc49@7d14f008.com>
 <CAFiTN-sghJuMxwH+uAjvm4_L5-RvhH9T9FciPTnLCpKQRFGfiA@mail.gmail.com>
Date: Mon, 04 Sep 2023 17:19:05 +0700
From: "Lepikhov Andrei" <a.lepikhov@postgrespro.ru>
To: "Dilip Kumar" <dilipbalaut@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org,
 =?UTF-8?Q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9_=D0=91=D1=80=D0=B5?=
 =?UTF-8?Q?=D0=B4=D0=BD=D1=8F?= <e.brednya@postgrespro.ru>
Subject: Re: Optimize planner memory consumption for huge arrays
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/8498b480-51f8-4827-ac3a-c0c87872810c%40app.fastmail.com>
Precedence: bulk


On Mon, Sep 4, 2023, at 3:37 PM, Dilip Kumar wrote:
> On Mon, Sep 4, 2023 at 11:58=E2=80=AFAM Lepikhov Andrei
> <a.lepikhov@postgrespro.ru> wrote:
>>
>> Hi, hackers,
>>
>> Looking at the planner behaviour with the memory consumption patch [1=
], I figured out that arrays increase memory consumption by the optimize=
r significantly. See init.sql in attachment.
>> The point here is that the planner does small memory allocations for =
each element during estimation. As a result, it looks like the planner c=
onsumes about 250 bytes for each integer element.
>>
>> It is maybe not a problem most of the time. However, in the case of p=
artitions, memory consumption multiplies by each partition. Such a corne=
r case looks weird, but the fix is simple. So, why not?
>>
>> The diff in the attachment is proof of concept showing how to reduce =
wasting of memory. Having benchmarked a bit, I didn't find any overhead.
>
> + Const *c =3D makeConst(nominal_element_type,
> + -1,
> + nominal_element_collation,
> + elmlen,
> + elem_values[i],
> + elem_nulls[i],
> + elmbyval);
> +
> + args =3D list_make2(leftop, c);
>   if (is_join_clause)
>   s2 =3D DatumGetFloat8(FunctionCall5Coll(&oprselproc,
>     clause->inputcollid,
> @@ -1984,7 +1985,8 @@ scalararraysel(PlannerInfo *root,
>     ObjectIdGetDatum(operator),
>     PointerGetDatum(args),
>     Int32GetDatum(varRelid)));
> -
> + list_free(args);
> + pfree(c);
>
> Maybe you can just use list_free_deep, instead of storing the constant
> in a separate variable.
As I see, the first element in the array is leftop, which is used on oth=
er iterations. So, we can't use list_free_deep here.

--=20
Regards,
Andrei Lepikhov