pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFrom: Jehan-Guillaume de Rorthais <jgdr@dalibo.com>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Memory leak from ExecutorState context?
Date: Thu, 2 Mar 2023 23:57:21 +0100
Message-ID: <20230302235721.54af8258@karst> (raw)
In-Reply-To: <dbae24d7-0dda-18aa-5e08-8138ac1caef9@enterprisedb.com>
References: <20230228190643.1e368315@karst>
<45d453c8-b2d3-b477-36eb-32fdf4455f3c@enterprisedb.com>
<20230301184840.0a897a80@karst>
<3013398b-316c-638f-2a73-3783e8e2ef02@enterprisedb.com>
<20230302001827.66e95dc3@karst>
<41c5766d-ed71-b70c-bbbc-d3396c462d62@enterprisedb.com>
<20230302130838.717e888d@karst>
<77a96d42-00cb-2448-465a-aa1e92d00cac@enterprisedb.com>
<20230302191530.781909fe@karst>
<dbae24d7-0dda-18aa-5e08-8138ac1caef9@enterprisedb.com>
On Thu, 2 Mar 2023 19:53:14 +0100
Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote:
...
> > There was some thoughts about how to make a better usage of the memory. As
> > memory is exploding way beyond work_mem, at least, avoid to waste it with
> > too many buffers of BufFile. So you expand either the work_mem or the
> > number of batch, depending on what move is smarter. TJis is explained and
> > tested here:
> >
> > https://www.postgresql.org/message-id/20190421161434.4hedytsadpbnglgk%40development
> > https://www.postgresql.org/message-id/20190422030927.3huxq7gghms4kmf4%40development
> >
> > And then, another patch to overflow each batch to a dedicated temp file and
> > stay inside work_mem (v4-per-slice-overflow-file.patch):
> >
> > https://www.postgresql.org/message-id/20190428141901.5dsbge2ka3rxmpk6%40development
> >
> > Then, nothing more on the discussion about this last patch. So I guess it
> > just went cold.
>
> I think a contributing factor was that the OP did not respond for a
> couple months, so the thread went cold.
>
> > For what it worth, these two patches seems really interesting to me. Do you
> > need any help to revive it?
>
> I think another reason why that thread went nowhere were some that we've
> been exploring a different (and likely better) approach to fix this by
> falling back to a nested loop for the "problematic" batches.
>
> As proposed in this thread:
>
> https://www.postgresql.org/message-id/20190421161434.4hedytsadpbnglgk%40development
Unless I'm wrong, you are linking to the same «frustrated as heck!» discussion,
for your patch v2-0001-account-for-size-of-BatchFile-structure-in-hashJo.patch
(balancing between increasing batches *and* work_mem).
No sign of turning "problematic" batches to nested loop. Did I miss something?
Do you have a link close to your hand about such algo/patch test by any chance?
> I was hoping we'd solve this by the BNL, but if we didn't get that in 4
> years, maybe we shouldn't stall and get at least an imperfect stop-gap
> solution ...
I'll keep searching tomorrow about existing BLN discussions (is it block level
nested loops?).
Regards,
view thread (60+ messages) latest in thread
Message-ID: <20230302235721.54af8258@karst>
Permalink: ../20230302235721.54af8258@karst/
Also on: postgresql.org/message-id/20230302235721.54af8258@karst
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: jgdr@dalibo.com, tomas.vondra@enterprisedb.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Memory leak from ExecutorState context?
In-Reply-To: <20230302235721.54af8258@karst>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox