pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedMemory leak on subquery as scalar operand
18+ messages / 6 participants
[nested] [flat]
Commits from this thread:
2023-09-27 9dce22033d 2cf50585e5 aef521849b 75a20a4b4b 10912f7d4f 3b991f81c4 llvmjit: Use explicit LLVMContextRef for inlining
2023-09-27 ef668d8bf5 llvmjit: Make llvm_types_module variable static
2023-09-27 2dad308e73 llvmjit: Remove unnecessary types
* Memory leak on subquery as scalar operand
@ 2022-10-30 17:13 Lauri Laanmets <pcspets@gmail.com>
0 siblings, 2 replies; 18+ messages in thread
From: Lauri Laanmets @ 2022-10-30 17:13 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
There seems to be a memory leak in PostgreSQL for Linux (doesn't happen on
Windows). At least in versions 12 to 14 (and not in 9.5). It can be
reproduced with the following SQL script on an empty database.
* When repeated, Linux process starts using more and more RAM until
OOM-killer will kill it.
* 'pg_backend_memory_contexts' doesn't show any increase.
* (gdb) p MemoryContextStats(TopMemoryContext) - doesn't show any increase.
-/
CREATE TABLE IF NOT EXISTS public.leak_test
(
id integer NOT NULL,
CONSTRAINT leak_test_pkey PRIMARY KEY (id)
);
INSERT INTO leak_test(id)
SELECT id
FROM generate_series(1,100000) id
ON CONFLICT DO NOTHING;
SELECT
id,
(SELECT count(*) FROM leak_test x WHERE x.id=l.id) as x_result,
(SELECT count(*) FROM leak_test y WHERE y.id=l.id) as y_result
/* Leaks memory around 80 kB on each query, but only if two sub-queries
are used. */
FROM leak_test l;
/-
Regards
Lauri
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-10-31 13:40 Tom Lane <tgl@sss.pgh.pa.us>
parent: Lauri Laanmets <pcspets@gmail.com>
1 sibling, 1 reply; 18+ messages in thread
From: Tom Lane @ 2022-10-31 13:40 UTC (permalink / raw)
To: lauri.laanmets@eesti.ee; +Cc: pgsql-bugs@lists.postgresql.org
Lauri Laanmets <pcspets@gmail.com> writes:
> There seems to be a memory leak in PostgreSQL for Linux (doesn't happen on
> Windows). At least in versions 12 to 14 (and not in 9.5). It can be
> reproduced with the following SQL script on an empty database.
Thanks for the test case! Unfortunately, I can't see any ill effects
with it in currently released versions (I tried 12, 14, 15, and HEAD).
Which minor releases did you try? Do you have any non-default
settings or build options?
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-10-31 14:16 Lauri Laanmets <pcspets@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 18+ messages in thread
From: Lauri Laanmets @ 2022-10-31 14:16 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: Tom Lane <tgl@sss.pgh.pa.us>
I have tested it on version 12 that comes with Ubuntu 20.04.5 LTS. Then
upgraded to "PostgreSQL 14.5 (Ubuntu 14.5-2.pgdg20.04+2) on
x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0,
64-bit" from "deb http://apt.postgresql.org/pub/repos/apt focal-pgdg main"
but the behaviour is the same - if my example query is repeatedly executed
on an open session, the postgres process will start using 1 MB more RAM
after around every 14 times of execution. Until OOM-killer kills it.
I haven't changed any default settings and it can be reproduced also by
pgAdmin Query Tool that executes the query under 'postgres' user. Although
it's a bit tiresome to make so many executions manually.
It is interesting that it doesn't happen on Windows. Might be that it's
caused by the OS or system in general but I have no idea what to try or
test next.
The machine is: Linux 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22
UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Regards
Lauri
Kontakt Tom Lane (<tgl@sss.pgh.pa.us>) kirjutas kuupäeval E, 31. oktoober
2022 kell 15:49:
> Lauri Laanmets <pcspets@gmail.com> writes:
> > There seems to be a memory leak in PostgreSQL for Linux (doesn't happen
> on
> > Windows). At least in versions 12 to 14 (and not in 9.5). It can be
> > reproduced with the following SQL script on an empty database.
>
> Thanks for the test case! Unfortunately, I can't see any ill effects
> with it in currently released versions (I tried 12, 14, 15, and HEAD).
> Which minor releases did you try? Do you have any non-default
> settings or build options?
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-10-31 23:20 David Rowley <dgrowleyml@gmail.com>
parent: Lauri Laanmets <pcspets@gmail.com>
1 sibling, 1 reply; 18+ messages in thread
From: David Rowley @ 2022-10-31 23:20 UTC (permalink / raw)
To: lauri.laanmets@eesti.ee; +Cc: pgsql-bugs@lists.postgresql.org
On Mon, 31 Oct 2022 at 23:03, Lauri Laanmets <pcspets@gmail.com> wrote:
> There seems to be a memory leak in PostgreSQL for Linux (doesn't happen on Windows). At least in versions 12 to 14 (and not in 9.5). It can be reproduced with the following SQL script on an empty database.
>
> * When repeated, Linux process starts using more and more RAM until OOM-killer will kill it.
> * 'pg_backend_memory_contexts' doesn't show any increase.
> * (gdb) p MemoryContextStats(TopMemoryContext) - doesn't show any increase.
Does it still OOM if you do:
SET jit TO off; ?
David
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-01 00:44 Tom Lane <tgl@sss.pgh.pa.us>
parent: David Rowley <dgrowleyml@gmail.com>
0 siblings, 2 replies; 18+ messages in thread
From: Tom Lane @ 2022-11-01 00:44 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
David Rowley <dgrowleyml@gmail.com> writes:
> Does it still OOM if you do:
> SET jit TO off; ?
Ah, bingo --- I can reproduce the behavior if I use a JIT-enabled
build. So this is related to the known problems with leakage of
compiled objects. It's pretty odd though that it requires two
sub-selects to cause the problem.
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-01 03:48 David Rowley <dgrowleyml@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 18+ messages in thread
From: David Rowley @ 2022-11-01 03:48 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
On Tue, 1 Nov 2022 at 13:44, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> It's pretty odd though that it requires two
> sub-selects to cause the problem.
Perhaps that's just what it takes to bump the costs above the JIT threshold.
David
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-01 04:10 Tom Lane <tgl@sss.pgh.pa.us>
parent: David Rowley <dgrowleyml@gmail.com>
0 siblings, 1 reply; 18+ messages in thread
From: Tom Lane @ 2022-11-01 04:10 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
David Rowley <dgrowleyml@gmail.com> writes:
> On Tue, 1 Nov 2022 at 13:44, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It's pretty odd though that it requires two
>> sub-selects to cause the problem.
> Perhaps that's just what it takes to bump the costs above the JIT threshold.
I see JIT being invoked either way:
regression=# explain verbose SELECT
id,
(SELECT count(*) FROM leak_test x WHERE x.id=l.id) as x_result,
(SELECT count(*) FROM leak_test y WHERE y.id=l.id) as y_result
FROM leak_test l;
QUERY PLAN
------------------------------------------------------------------------------------------------------------
Seq Scan on public.leak_test l (cost=0.00..865943.00 rows=100000 width=20)
Output: l.id, (SubPlan 1), (SubPlan 2)
SubPlan 1
-> Aggregate (cost=4.31..4.32 rows=1 width=8)
Output: count(*)
-> Index Only Scan using leak_test_pkey on public.leak_test x (cost=0.29..4.31 rows=1 width=0)
Output: x.id
Index Cond: (x.id = l.id)
SubPlan 2
-> Aggregate (cost=4.31..4.32 rows=1 width=8)
Output: count(*)
-> Index Only Scan using leak_test_pkey on public.leak_test y (cost=0.29..4.31 rows=1 width=0)
Output: y.id
Index Cond: (y.id = l.id)
JIT:
Functions: 12
Options: Inlining true, Optimization true, Expressions true, Deforming true
(17 rows)
regression=# explain verbose SELECT
id,
(SELECT count(*) FROM leak_test x WHERE x.id=l.id) as x_result FROM leak_test l;
QUERY PLAN
------------------------------------------------------------------------------------------------------------
Seq Scan on public.leak_test l (cost=0.00..433693.00 rows=100000 width=12)
Output: l.id, (SubPlan 1)
SubPlan 1
-> Aggregate (cost=4.31..4.32 rows=1 width=8)
Output: count(*)
-> Index Only Scan using leak_test_pkey on public.leak_test x (cost=0.29..4.31 rows=1 width=0)
Output: x.id
Index Cond: (x.id = l.id)
JIT:
Functions: 7
Options: Inlining false, Optimization false, Expressions true, Deforming true
(11 rows)
Maybe the different "inlining" choice makes a difference?
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-01 04:26 David Rowley <dgrowleyml@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 18+ messages in thread
From: David Rowley @ 2022-11-01 04:26 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
On Tue, 1 Nov 2022 at 17:10, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Maybe the different "inlining" choice makes a difference?
The single subquery version also crashes for me, so perhaps it's just
the amount of memory that's being used and when the OOM killer is
triggering.
It crashes even when I set jit_inline_above_cost and
jit_optimize_above_cost above the query's cost.
postgres=# set jit_optimize_above_cost=10000000;
SET
postgres=# set jit_inline_above_cost=10000000;
SET
postgres=# set jit=0;
SET
postgres=# explain verbose SELECT
id,
(SELECT count(*) FROM leak_test y WHERE y.id=l.id) as y_result
FROM leak_test l;
QUERY PLAN
------------------------------------------------------------------------------------------------------------
Seq Scan on public.leak_test l (cost=0.00..941723.86 rows=112965 width=12)
Output: l.id, (SubPlan 1)
SubPlan 1
-> Aggregate (cost=8.31..8.32 rows=1 width=8)
Output: count(*)
-> Index Only Scan using leak_test_pkey on
public.leak_test y (cost=0.29..8.31 rows=1 width=0)
Output: y.id
Index Cond: (y.id = l.id)
(8 rows)
postgres=# set jit=1;
SET
postgres=# explain verbose SELECT
id,
(SELECT count(*) FROM leak_test y WHERE y.id=l.id) as y_result
FROM leak_test l;
server closed the connection unexpectedly
David
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-01 04:52 Tom Lane <tgl@sss.pgh.pa.us>
parent: David Rowley <dgrowleyml@gmail.com>
0 siblings, 1 reply; 18+ messages in thread
From: Tom Lane @ 2022-11-01 04:52 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
David Rowley <dgrowleyml@gmail.com> writes:
> The single subquery version also crashes for me, so perhaps it's just
> the amount of memory that's being used and when the OOM killer is
> triggering.
> It crashes even when I set jit_inline_above_cost and
> jit_optimize_above_cost above the query's cost.
Hmm, maybe we're not seeing the same thing? For me, the behavior
seems similar to what the OP reported: there's a per-query leakage
but it's less than 100kB per query. It'd take more than a handful
of repetitions to get to an OOM failure. This is with LLVM 13.0.1
on RHEL 8.6.
Also, as far as I can see there is no leak with the single-subquery
version. The process's reported VIRT consumption bounces around a
fair amount, but it doesn't go above 300MB, even after ~10min in
a tight plpgsql DO loop. VIRT bounces around a lot with the
two-subquery version too, actually, but there does seem to be a
general uptrend there. I did not have the patience to wait for
actual OOM; it looked like it'd take a good long while, tens
of minutes at least.
If the behavior varies across LLVM versions, as is now seeming
a bit likely, it might be their bug not ours.
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-01 05:51 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 18+ messages in thread
From: Andres Freund @ 2022-11-01 05:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
Hi,
On 2022-11-01 00:52:16 -0400, Tom Lane wrote:
> David Rowley <dgrowleyml@gmail.com> writes:
> > The single subquery version also crashes for me, so perhaps it's just
> > the amount of memory that's being used and when the OOM killer is
> > triggering.
> > It crashes even when I set jit_inline_above_cost and
> > jit_optimize_above_cost above the query's cost.
>
> Hmm, maybe we're not seeing the same thing?
I can't reproduce that either.
> For me, the behavior seems similar to what the OP reported: there's a
> per-query leakage but it's less than 100kB per query. It'd take more than a
> handful of repetitions to get to an OOM failure. This is with LLVM 13.0.1
> on RHEL 8.6.
This I can reproduce. Here's an updated patchset addressing this. This query,
for some reason, leaks a lot more aggressively than what I've seen in the
past, so I needed to reduce the amount of time until an llvm context is
recycled substantially. A bit of benchmarking showed no negative consequences
of going to 100 uses till recycling, even with absurd settings
(i.e. jit_*_cost = 0), but did show impact on lower values.
Greetings,
Andres Freund
Attachments:
v3-0001-wip-llvmjit-Use-explicit-LLVMContextRef.patch (text/x-diff, 0B) download
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2022-11-02 08:26 Lauri Laanmets <pcspets@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 18+ messages in thread
From: Lauri Laanmets @ 2022-11-02 08:26 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-bugs@lists.postgresql.org
> Does it still OOM if you do:
> SET jit TO off; ?
Indeed, it doesn't leak if JIT is OFF. Thank you for a workaround hint!
Regards
Lauri
Kontakt Tom Lane (<tgl@sss.pgh.pa.us>) kirjutas kuupäeval T, 1. november
2022 kell 02:44:
> David Rowley <dgrowleyml@gmail.com> writes:
> > Does it still OOM if you do:
> > SET jit TO off; ?
>
> Ah, bingo --- I can reproduce the behavior if I use a JIT-enabled
> build. So this is related to the known problems with leakage of
> compiled objects. It's pretty odd though that it requires two
> sub-selects to cause the problem.
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2023-02-22 19:43 Justin Pryzby <pryzby@telsasoft.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 18+ messages in thread
From: Justin Pryzby @ 2023-02-22 19:43 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee; pgsql-bugs@lists.postgresql.org
On Mon, Oct 31, 2022 at 10:51:32PM -0700, Andres Freund wrote:
> Hi,
>
> On 2022-11-01 00:52:16 -0400, Tom Lane wrote:
> > David Rowley <dgrowleyml@gmail.com> writes:
> > > The single subquery version also crashes for me, so perhaps it's just
> > > the amount of memory that's being used and when the OOM killer is
> > > triggering.
> > > It crashes even when I set jit_inline_above_cost and
> > > jit_optimize_above_cost above the query's cost.
> >
> > Hmm, maybe we're not seeing the same thing?
>
> I can't reproduce that either.
@David: does the query really crash as you described - with neither jit
inlining nor optimization ?
Is it due to OOM or something else ?
Does it crash with/without Andres' patches ?
What version were you testing ?
--
Justin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2023-05-02 13:33 Daniel Gustafsson <daniel@yesql.se>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 18+ messages in thread
From: Daniel Gustafsson @ 2023-05-02 13:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee, pgsql-bugs <pgsql-bugs@lists.postgresql.org>
> On 1 Nov 2022, at 06:51, Andres Freund <andres@anarazel.de> wrote:
>> David Rowley <dgrowleyml@gmail.com> writes:
>> For me, the behavior seems similar to what the OP reported: there's a
>> per-query leakage but it's less than 100kB per query. It'd take more than a
>> handful of repetitions to get to an OOM failure. This is with LLVM 13.0.1
>> on RHEL 8.6.
>
> This I can reproduce. Here's an updated patchset addressing this. This query,
> for some reason, leaks a lot more aggressively than what I've seen in the
> past, so I needed to reduce the amount of time until an llvm context is
> recycled substantially. A bit of benchmarking showed no negative consequences
> of going to 100 uses till recycling, even with absurd settings
> (i.e. jit_*_cost = 0), but did show impact on lower values.
I had a look at this and I concur with the findings in this thread. I didn't
do any benchmarking but running various tests I was unable to trigger an OOM.
Will do more testing and stressing of it. A few small comments on the
patchset:
From reading it seems that patch 0002 and 0003 can be committed regardless of
the other patches in this series. Were they included because they were found
while looking at this, or is there a deeper connection I'm missing?
0004:
The commit message states: "That incurs some overhead, so only do so after
10000 JITed queries.", but I fail to see how that's implemented. There is
currently recreation after 100 reuses, was the intention to have a different
number here or is this just a leftover from an earlier patch-version?
+ /*
+ * The LLVM Context used by this JIT context. An LLVM context is reused
+ * across many compilations, but occasionally reset to prevent it using
+ * too much memory due to more and more types accumulating.
+ */
+ LLVMContextRef llvm_context;
llvm_context is added as a member in LLVMJitContext but is never set or read,
the static llvmjit.c:llvm_context is still used for everything. Is this a
lefover or was the plan to move this to LLVMJitContext?
+ * FIXME: should split the handling of llvm_triple / llvm_layout out
+ * of llvm_create_types() - that doesn't need to be redone.
Agreed, that seems wasteful. AFAICT there would not be any reason to recreate
this once set?
+ /*
+ * Consider as cleaned up even if we skip doing so below, that way we can
+ * verify the tracking is correct (see llvm_shutdown()).
+ */
+ llvm_jit_context_in_use_count--;
Since this doesn't actually release even if llvm_jit_context_in_use_count goes
to zero here, this hunk in llvm_release_context is seemingly a bit at odds with
the following from jit/README: "If it is desirable to release resources
earlier, jit_release_context() can be used". If we want to verify the usage
tracking in llvm_shutdown it's hard to see how we could much else, but maybe a
note in the README that not all resources are guaranteed to be released could
be in order?
Regarding llvm_release_context we currently have this:
static void
llvm_release_context(JitContext *context)
{
LLVMJitContext *llvm_context = (LLVMJitContext *) context;
..which shadows the "static LLVMContextRef llvm_context;" declared in this
patchset, we should probably rename the local var in llvm_release_context.
(this is referred to in the 0004 commit message.)
The attached 0005 is a WIP attempt to address a few of the FIXME's in this
patchset on top of your the 0001-0004.
--
Daniel Gustafsson
Attachments: download all (5 files, .tar.gz)
v4-0005-Address-review-comments.patch (application/octet-stream, 7.2K) download
From e7e56f0f7b36ea973fa472aadd16dbfe6db7a6ec Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 2 May 2023 14:58:44 +0200
Subject: [PATCH v4 5/5] Address review comments
---
src/backend/jit/llvm/llvmjit.c | 155 +++++++++++++++++++--------------
1 file changed, 89 insertions(+), 66 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index f0c70d68ab..8fe68a64ba 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -112,6 +112,8 @@ static void llvm_compile_module(LLVMJitContext *context);
static void llvm_optimize_module(LLVMJitContext *context, LLVMModuleRef module);
static void llvm_create_types(void);
+static void llvm_set_target(void);
+static void llvm_recreate_llvm_context(void);
static uint64_t llvm_resolve_symbol(const char *name, void *ctx);
#if LLVM_VERSION_MAJOR > 11
@@ -133,6 +135,61 @@ _PG_jit_provider_init(JitProviderCallbacks *cb)
cb->compile_expr = llvm_compile_expr;
}
+
+/*
+ * Every now and then create a new LLVMContextRef. Unfortunately, during
+ * every round of inlining, types may "leak" (they can still be found/used
+ * via the context, but new types will be created the next time in
+ * inlining is performed). To prevent that from slowly accumulating
+ * problematic amounts of memory, recreate the LLVMContextRef we use. We
+ * don't want to do so too often, as that implies some overhead
+ * (particularly re-loading the module summaries / modules is fairly
+ * expensive).
+ */
+static void
+llvm_recreate_llvm_context(void)
+{
+ Assert(llvm_context != NULL);
+
+ if (llvm_context)
+ {
+ /*
+ * We can only safely recreate the LLVM context if no other code is being
+ * JITed, otherwise we'd release the types in use for that.
+ */
+ if (llvm_jit_context_in_use_count > 0)
+ {
+ llvm_llvm_context_reuse_count++;
+ return;
+ }
+
+ if (llvm_llvm_context_reuse_count <= LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
+ {
+ llvm_llvm_context_reuse_count++;
+ return;
+ }
+ }
+
+ /*
+ * Need to reset the modules that the inlining code caches before
+ * disposing of the context. LLVM modules exist within a specific LLVM
+ * context, therefore disposing of the context before resetting the
+ * cache would lead to dangling pointers to modules.
+ */
+ llvm_inline_reset_caches();
+
+ LLVMContextDispose(llvm_context);
+ llvm_context = LLVMContextCreate();
+
+ /*
+ * Re-build cached type information, so code generation code can rely
+ * on that information to be present (also prevents the variables to
+ * be dangling references).
+ */
+ llvm_create_types();
+}
+
+
/*
* Create a context for JITing work.
*
@@ -149,54 +206,7 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
- /*
- * Every now and then create a new LLVMContextRef. Unfortunately, during
- * every round of inlining, types may "leak" (they can still be found/used
- * via the context, but new types will be created the next time in
- * inlining is performed). To prevent that from slowly accumulating
- * problematic amounts of memory, recreate the LLVMContextRef we use. We
- * don't want to do so too often, as that implies some overhead
- * (particularly re-loading the module summaries / modules is fairly
- * expensive).
- *
- * We can only safely recreate the LLVM context if no other code is being
- * JITed, otherwise we'd release the types in use for that.
- *
- * FIXME: Extract into helper function.
- */
- if (llvm_jit_context_in_use_count == 0 &&
- llvm_llvm_context_reuse_count > LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
- {
- llvm_llvm_context_reuse_count = 0;
-
- Assert(llvm_context != NULL);
-
- /*
- * Need to reset the modules that the inlining code caches before
- * disposing of the context. LLVM modules exist within a specific LLVM
- * context, therefore disposing of the context before resetting the
- * cache would lead to dangling pointers to modules.
- */
- llvm_inline_reset_caches();
-
- if (llvm_context != NULL)
- LLVMContextDispose(llvm_context);
- llvm_context = LLVMContextCreate();
-
- /*
- * Re-build cached type information, so code generation code can rely
- * on that information to be present (also prevents the variables to
- * be dangling references).
- *
- * FIXME: should split the handling of llvm_triple / llvm_layout out
- * of llvm_create_types() - that doesn't need to be redone.
- */
- llvm_create_types();
- }
- else
- {
- llvm_llvm_context_reuse_count++;
- }
+ llvm_recreate_llvm_context();
ResourceOwnerEnlargeJIT(CurrentResourceOwner);
@@ -219,7 +229,7 @@ llvm_create_context(int jitFlags)
static void
llvm_release_context(JitContext *context)
{
- LLVMJitContext *llvm_context = (LLVMJitContext *) context;
+ LLVMJitContext *llvm_jit_context = (LLVMJitContext *) context;
ListCell *lc;
/*
@@ -238,13 +248,13 @@ llvm_release_context(JitContext *context)
llvm_enter_fatal_on_oom();
- if (llvm_context->module)
+ if (llvm_jit_context->module)
{
- LLVMDisposeModule(llvm_context->module);
- llvm_context->module = NULL;
+ LLVMDisposeModule(llvm_jit_context->module);
+ llvm_jit_context->module = NULL;
}
- foreach(lc, llvm_context->handles)
+ foreach(lc, llvm_jit_context->handles)
{
LLVMJitHandle *jit_handle = (LLVMJitHandle *) lfirst(lc);
@@ -274,8 +284,8 @@ llvm_release_context(JitContext *context)
pfree(jit_handle);
}
- list_free(llvm_context->handles);
- llvm_context->handles = NIL;
+ list_free(llvm_jit_context->handles);
+ llvm_jit_context->handles = NIL;
}
/*
@@ -853,7 +863,8 @@ llvm_session_initialize(void)
LLVMInitializeNativeAsmPrinter();
LLVMInitializeNativeAsmParser();
- llvm_context = LLVMContextCreate();
+ if (llvm_context == NULL)
+ llvm_context = LLVMContextCreate();
/*
* When targeting an LLVM version with opaque pointers enabled by
@@ -871,6 +882,11 @@ llvm_session_initialize(void)
*/
llvm_create_types();
+ /*
+ * Extract target information from loaded module.
+ */
+ llvm_set_target();
+
if (LLVMGetTargetFromTriple(llvm_triple, &llvm_targetref, &error) != 0)
{
elog(FATAL, "failed to query triple %s", error);
@@ -1040,6 +1056,23 @@ load_return_type(LLVMModuleRef mod, const char *name)
return typ;
}
+/*
+ * Load triple & layout from clang emitted file so we're guaranteed to be
+ * compatible.
+ */
+static void
+llvm_set_target(void)
+{
+ if (!llvm_types_module)
+ elog(ERROR, "failed to extract target information, llvmjit_types.c not loaded");
+
+ if (llvm_triple == NULL)
+ llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
+
+ if (llvm_layout == NULL)
+ llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+}
+
/*
* Load required information, types, function signatures from llvmjit_types.c
* and make them available in global variables.
@@ -1069,16 +1102,6 @@ llvm_create_types(void)
}
LLVMDisposeMemoryBuffer(buf);
- /*
- * Load triple & layout from clang emitted file so we're guaranteed to be
- * compatible.
- */
- if (llvm_triple == NULL)
- {
- llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
- llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
- }
-
TypeSizeT = llvm_pg_var_type("TypeSizeT");
TypeParamBool = load_return_type(llvm_types_module, "FunctionReturningBool");
TypeStorageBool = llvm_pg_var_type("TypeStorageBool");
--
2.32.1 (Apple Git-133)
=
v4-0004-wip-llvmjit-Plug-memory-leak-when-performing-inli.patch (application/octet-stream, 7.4K) download
From 762de3b68e669df1ca8d8471b00f78186d87aad8 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 21:44:40 +0200
Subject: [PATCH v4 4/5] wip: llvmjit: Plug memory leak when performing
inlining.
When performing inlining LLVM unfortunately "leaks" types (the types
survive and are usable, but a new round of inlining will recreate new
types that are structurally equivalent). I could not come up with a
backpatchable fix for that, so address the problem by dropping &
recreating the LLVMContextRef that contains all IR related data -
which releases all the types.
That incurs some overhead, so only do so after 10000 JITed queries.
Needs a good bit more work:
- confusing naming between different types of contexts (PG side
LLVMJitContext, and LLVM side LLVMContextRef)
- add detailed explanation of leak issue
- consider only incrementing the counter triggering the recreation of
the memory context during inlining.
- several FIXMEs in code
Reported-By: Justin Pryzby <pryzby@telsasoft.com>
Reported-By: Kurt Roeckx <kurt@roeckx.be>
Reported-By: Jaime Casanova <jcasanov@systemguards.com.ec>
Author: Andres Freund
Discussion: https://postgr.es/m/7acc8678-df5f-4923-9cf6-e843131ae89d@www.fastmail.com
Discussion: https://postgr.es/m/20201218235607.GC30237@telsasoft.com
---
src/backend/jit/llvm/llvmjit.c | 76 ++++++++++++++++++++++++-
src/backend/jit/llvm/llvmjit_inline.cpp | 12 ++++
src/include/jit/llvmjit.h | 8 +++
3 files changed, 94 insertions(+), 2 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index 6e0664745d..f0c70d68ab 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -42,6 +42,10 @@
#include "utils/memutils.h"
#include "utils/resowner_private.h"
+
+#define LLVMJIT_LLVM_CONTEXT_REUSE_MAX 100
+
+
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
{
@@ -81,6 +85,10 @@ static LLVMModuleRef llvm_types_module = NULL;
static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
+/* number of LLVMJitContexts that currently are in use */
+static size_t llvm_jit_context_in_use_count = 0;
+/* how many times has the current LLVMContextRef been used */
+static size_t llvm_llvm_context_reuse_count = 0;
static const char *llvm_triple = NULL;
static const char *llvm_layout = NULL;
static LLVMContextRef llvm_context;
@@ -141,6 +149,55 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
+ /*
+ * Every now and then create a new LLVMContextRef. Unfortunately, during
+ * every round of inlining, types may "leak" (they can still be found/used
+ * via the context, but new types will be created the next time in
+ * inlining is performed). To prevent that from slowly accumulating
+ * problematic amounts of memory, recreate the LLVMContextRef we use. We
+ * don't want to do so too often, as that implies some overhead
+ * (particularly re-loading the module summaries / modules is fairly
+ * expensive).
+ *
+ * We can only safely recreate the LLVM context if no other code is being
+ * JITed, otherwise we'd release the types in use for that.
+ *
+ * FIXME: Extract into helper function.
+ */
+ if (llvm_jit_context_in_use_count == 0 &&
+ llvm_llvm_context_reuse_count > LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
+ {
+ llvm_llvm_context_reuse_count = 0;
+
+ Assert(llvm_context != NULL);
+
+ /*
+ * Need to reset the modules that the inlining code caches before
+ * disposing of the context. LLVM modules exist within a specific LLVM
+ * context, therefore disposing of the context before resetting the
+ * cache would lead to dangling pointers to modules.
+ */
+ llvm_inline_reset_caches();
+
+ if (llvm_context != NULL)
+ LLVMContextDispose(llvm_context);
+ llvm_context = LLVMContextCreate();
+
+ /*
+ * Re-build cached type information, so code generation code can rely
+ * on that information to be present (also prevents the variables to
+ * be dangling references).
+ *
+ * FIXME: should split the handling of llvm_triple / llvm_layout out
+ * of llvm_create_types() - that doesn't need to be redone.
+ */
+ llvm_create_types();
+ }
+ else
+ {
+ llvm_llvm_context_reuse_count++;
+ }
+
ResourceOwnerEnlargeJIT(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
@@ -151,6 +208,8 @@ llvm_create_context(int jitFlags)
context->base.resowner = CurrentResourceOwner;
ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ llvm_jit_context_in_use_count++;
+
return context;
}
@@ -163,6 +222,12 @@ llvm_release_context(JitContext *context)
LLVMJitContext *llvm_context = (LLVMJitContext *) context;
ListCell *lc;
+ /*
+ * Consider as cleaned up even if we skip doing so below, that way we can
+ * verify the tracking is correct (see llvm_shutdown()).
+ */
+ llvm_jit_context_in_use_count--;
+
/*
* When this backend is exiting, don't clean up LLVM. As an error might
* have occurred from within LLVM, we do not want to risk reentering. All
@@ -901,6 +966,10 @@ llvm_shutdown(int code, Datum arg)
return;
}
+ if (llvm_jit_context_in_use_count != 0)
+ elog(PANIC, "LLVMJitContext in use count not 0 at exit (is %zu)",
+ llvm_jit_context_in_use_count);
+
#if LLVM_VERSION_MAJOR > 11
{
if (llvm_opt3_orc)
@@ -1004,8 +1073,11 @@ llvm_create_types(void)
* Load triple & layout from clang emitted file so we're guaranteed to be
* compatible.
*/
- llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
- llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+ if (llvm_triple == NULL)
+ {
+ llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
+ llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+ }
TypeSizeT = llvm_pg_var_type("TypeSizeT");
TypeParamBool = load_return_type(llvm_types_module, "FunctionReturningBool");
diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp b/src/backend/jit/llvm/llvmjit_inline.cpp
index cbd17a9fef..d92d7f3c88 100644
--- a/src/backend/jit/llvm/llvmjit_inline.cpp
+++ b/src/backend/jit/llvm/llvmjit_inline.cpp
@@ -152,6 +152,18 @@ summaries_for_guid(const InlineSearchPath& path, llvm::GlobalValue::GUID guid);
#define ilog(...) (void) 0
#endif
+/*
+ * Reset inlining related state. This needs to be called before the currently
+ * used LLVMContextRef is disposed (and a new one create), otherwise we would
+ * have dangling references to deleted modules.
+ */
+void
+llvm_inline_reset_caches(void)
+{
+ module_cache->clear();
+ summary_cache->clear();
+}
+
/*
* Perform inlining of external function references in M based on a simple
* cost based analysis.
diff --git a/src/include/jit/llvmjit.h b/src/include/jit/llvmjit.h
index 17b3b7593f..6d90a16f79 100644
--- a/src/include/jit/llvmjit.h
+++ b/src/include/jit/llvmjit.h
@@ -42,6 +42,13 @@ typedef struct LLVMJitContext
/* number of modules created */
size_t module_generation;
+ /*
+ * The LLVM Context used by this JIT context. An LLVM context is reused
+ * across many compilations, but occasionally reset to prevent it using
+ * too much memory due to more and more types accumulating.
+ */
+ LLVMContextRef llvm_context;
+
/* current, "open for write", module */
LLVMModuleRef module;
@@ -99,6 +106,7 @@ extern LLVMValueRef llvm_function_reference(LLVMJitContext *context,
LLVMModuleRef mod,
FunctionCallInfo fcinfo);
+extern void llvm_inline_reset_caches(void);
extern void llvm_inline(LLVMModuleRef mod);
/*
--
2.32.1 (Apple Git-133)
=
v4-0003-wip-llvmjit-Make-llvm_types_module-variable-stati.patch (application/octet-stream, 1.4K) download
From 38dee75a349eaf168b2dc5db207569af1da5dfad Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:44:20 +0200
Subject: [PATCH v4 3/5] wip: llvmjit: Make llvm_types_module variable static.
Not sure why I did that differently in b059d2f45685a.
---
src/backend/jit/llvm/llvmjit.c | 2 +-
src/include/jit/llvmjit.h | 3 ---
2 files changed, 1 insertion(+), 4 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index e26b56f0dd..6e0664745d 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -77,7 +77,7 @@ LLVMTypeRef StructAggStatePerTransData;
LLVMValueRef AttributeTemplate;
-LLVMModuleRef llvm_types_module = NULL;
+static LLVMModuleRef llvm_types_module = NULL;
static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
diff --git a/src/include/jit/llvmjit.h b/src/include/jit/llvmjit.h
index 551b585464..17b3b7593f 100644
--- a/src/include/jit/llvmjit.h
+++ b/src/include/jit/llvmjit.h
@@ -55,9 +55,6 @@ typedef struct LLVMJitContext
List *handles;
} LLVMJitContext;
-/* llvm module containing information about types */
-extern PGDLLIMPORT LLVMModuleRef llvm_types_module;
-
/* type and struct definitions */
extern PGDLLIMPORT LLVMTypeRef TypeParamBool;
extern PGDLLIMPORT LLVMTypeRef TypePGFunction;
--
2.32.1 (Apple Git-133)
=
v4-0002-wip-llvmjit-remove-unnecessary-types.patch (application/octet-stream, 1.3K) download
From 33c38651ca8cd786e15e1a5c8d40990caadf0398 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:43:37 +0200
Subject: [PATCH v4 2/5] wip: llvmjit: remove unnecessary types.
---
src/backend/jit/llvm/llvmjit.c | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index d9b0aec5af..e26b56f0dd 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -61,23 +61,12 @@ LLVMTypeRef TypeParamBool;
LLVMTypeRef TypeStorageBool;
LLVMTypeRef TypePGFunction;
LLVMTypeRef StructNullableDatum;
-LLVMTypeRef StructHeapTupleFieldsField3;
-LLVMTypeRef StructHeapTupleFields;
-LLVMTypeRef StructHeapTupleHeaderData;
-LLVMTypeRef StructHeapTupleDataChoice;
LLVMTypeRef StructHeapTupleData;
-LLVMTypeRef StructMinimalTupleData;
-LLVMTypeRef StructItemPointerData;
-LLVMTypeRef StructBlockId;
-LLVMTypeRef StructFormPgAttribute;
-LLVMTypeRef StructTupleConstr;
LLVMTypeRef StructTupleDescData;
LLVMTypeRef StructTupleTableSlot;
LLVMTypeRef StructHeapTupleTableSlot;
LLVMTypeRef StructMinimalTupleTableSlot;
LLVMTypeRef StructMemoryContextData;
-LLVMTypeRef StructPGFinfoRecord;
-LLVMTypeRef StructFmgrInfo;
LLVMTypeRef StructFunctionCallInfoData;
LLVMTypeRef StructExprContext;
LLVMTypeRef StructExprEvalStep;
--
2.32.1 (Apple Git-133)
=
v4-0001-wip-llvmjit-Use-explicit-LLVMContextRef.patch (application/octet-stream, 23.4K) download
From 3e9e191f30c87df311310bb021fb1ee6bdfca3c2 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:43:03 +0200
Subject: [PATCH v4 1/5] wip: llvmjit: Use explicit LLVMContextRef.
This is a prerequisite for a later commit fixing a memory leak due to
slow accumulation of types in LLVM.
---
src/backend/jit/llvm/llvmjit.c | 9 ++--
src/backend/jit/llvm/llvmjit_deform.c | 60 +++++++++++++------------
src/backend/jit/llvm/llvmjit_expr.c | 58 +++++++++++++-----------
src/backend/jit/llvm/llvmjit_inline.cpp | 21 ++++-----
src/include/jit/llvmjit_emit.h | 30 ++++++++-----
5 files changed, 98 insertions(+), 80 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index a8b73a9cf1..d9b0aec5af 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -94,6 +94,7 @@ static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
static const char *llvm_triple = NULL;
static const char *llvm_layout = NULL;
+static LLVMContextRef llvm_context;
static LLVMTargetRef llvm_targetref;
@@ -238,7 +239,7 @@ llvm_mutable_module(LLVMJitContext *context)
{
context->compiled = false;
context->module_generation = llvm_generation++;
- context->module = LLVMModuleCreateWithName("pg");
+ context->module = LLVMModuleCreateWithNameInContext("pg", llvm_context);
LLVMSetTarget(context->module, llvm_triple);
LLVMSetDataLayout(context->module, llvm_layout);
}
@@ -798,6 +799,8 @@ llvm_session_initialize(void)
LLVMInitializeNativeAsmPrinter();
LLVMInitializeNativeAsmParser();
+ llvm_context = LLVMContextCreate();
+
/*
* When targeting an LLVM version with opaque pointers enabled by
* default, turn them off for the context we build our code in. We don't
@@ -1002,9 +1005,9 @@ llvm_create_types(void)
}
/* eagerly load contents, going to need it all */
- if (LLVMParseBitcode2(buf, &llvm_types_module))
+ if (LLVMParseBitcodeInContext2(llvm_context, buf, &llvm_types_module))
{
- elog(ERROR, "LLVMParseBitcode2 of %s failed", path);
+ elog(ERROR, "LLVMParseBitcodeInContext2 of %s failed", path);
}
LLVMDisposeMemoryBuffer(buf);
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 6b15588da6..68e0ff354d 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -37,6 +37,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
char *funcname;
LLVMModuleRef mod;
+ LLVMContextRef lc;
LLVMBuilderRef b;
LLVMTypeRef deform_sig;
@@ -99,6 +100,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
return NULL;
mod = llvm_mutable_module(context);
+ lc = LLVMGetModuleContext(mod);
funcname = llvm_expand_funcname(context, "deform");
@@ -133,8 +135,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
param_types[0] = l_ptr(StructTupleTableSlot);
- deform_sig = LLVMFunctionType(LLVMVoidType(), param_types,
- lengthof(param_types), 0);
+ deform_sig = LLVMFunctionType(LLVMVoidTypeInContext(lc),
+ param_types, lengthof(param_types), 0);
}
v_deform_fn = LLVMAddFunction(mod, funcname, deform_sig);
LLVMSetLinkage(v_deform_fn, LLVMInternalLinkage);
@@ -142,17 +144,17 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
llvm_copy_attributes(AttributeTemplate, v_deform_fn);
b_entry =
- LLVMAppendBasicBlock(v_deform_fn, "entry");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "entry");
b_adjust_unavail_cols =
- LLVMAppendBasicBlock(v_deform_fn, "adjust_unavail_cols");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "adjust_unavail_cols");
b_find_start =
- LLVMAppendBasicBlock(v_deform_fn, "find_startblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "find_startblock");
b_out =
- LLVMAppendBasicBlock(v_deform_fn, "outblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "outblock");
b_dead =
- LLVMAppendBasicBlock(v_deform_fn, "deadblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "deadblock");
- b = LLVMCreateBuilder();
+ b = LLVMCreateBuilderInContext(lc);
attcheckattnoblocks = palloc(sizeof(LLVMBasicBlockRef) * natts);
attstartblocks = palloc(sizeof(LLVMBasicBlockRef) * natts);
@@ -221,7 +223,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMBuildStructGEP(b, v_tuplep,
FIELDNO_HEAPTUPLEHEADERDATA_BITS,
""),
- l_ptr(LLVMInt8Type()),
+ l_ptr(LLVMInt8TypeInContext(lc)),
"t_bits");
v_infomask1 =
l_load_struct_gep(b, v_tuplep,
@@ -236,14 +238,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
v_hasnulls =
LLVMBuildICmp(b, LLVMIntNE,
LLVMBuildAnd(b,
- l_int16_const(HEAP_HASNULL),
+ l_int16_const(lc, HEAP_HASNULL),
v_infomask1, ""),
- l_int16_const(0),
+ l_int16_const(lc, 0),
"hasnulls");
/* t_infomask2 & HEAP_NATTS_MASK */
v_maxatt = LLVMBuildAnd(b,
- l_int16_const(HEAP_NATTS_MASK),
+ l_int16_const(lc, HEAP_NATTS_MASK),
v_infomask2,
"maxatt");
@@ -256,13 +258,13 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
l_load_struct_gep(b, v_tuplep,
FIELDNO_HEAPTUPLEHEADERDATA_HOFF,
""),
- LLVMInt32Type(), "t_hoff");
+ LLVMInt32TypeInContext(lc), "t_hoff");
v_tupdata_base =
LLVMBuildGEP(b,
LLVMBuildBitCast(b,
v_tuplep,
- l_ptr(LLVMInt8Type()),
+ l_ptr(LLVMInt8TypeInContext(lc)),
""),
&v_hoff, 1,
"v_tupdata_base");
@@ -319,7 +321,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMBuildCondBr(b,
LLVMBuildICmp(b, LLVMIntULT,
v_maxatt,
- l_int16_const(natts),
+ l_int16_const(lc, natts),
""),
b_adjust_unavail_cols,
b_find_start);
@@ -328,8 +330,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMPositionBuilderAtEnd(b, b_adjust_unavail_cols);
v_params[0] = v_slot;
- v_params[1] = LLVMBuildZExt(b, v_maxatt, LLVMInt32Type(), "");
- v_params[2] = l_int32_const(natts);
+ v_params[1] = LLVMBuildZExt(b, v_maxatt, LLVMInt32TypeInContext(lc), "");
+ v_params[2] = l_int32_const(lc, natts);
LLVMBuildCall(b, llvm_pg_func(mod, "slot_getmissingattrs"),
v_params, lengthof(v_params), "");
LLVMBuildBr(b, b_find_start);
@@ -352,7 +354,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
- LLVMValueRef v_attno = l_int16_const(attnum);
+ LLVMValueRef v_attno = l_int16_const(lc, attnum);
LLVMAddCase(v_switch, v_attno, attcheckattnoblocks[attnum]);
}
@@ -375,7 +377,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
Form_pg_attribute att = TupleDescAttr(desc, attnum);
LLVMValueRef v_incby;
int alignto;
- LLVMValueRef l_attno = l_int16_const(attnum);
+ LLVMValueRef l_attno = l_int16_const(lc, attnum);
LLVMValueRef v_attdatap;
LLVMValueRef v_resultp;
@@ -436,14 +438,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
else
b_next = attcheckattnoblocks[attnum + 1];
- v_nullbyteno = l_int32_const(attnum >> 3);
- v_nullbytemask = l_int8_const(1 << ((attnum) & 0x07));
+ v_nullbyteno = l_int32_const(lc, attnum >> 3);
+ v_nullbytemask = l_int8_const(lc, 1 << ((attnum) & 0x07));
v_nullbyte = l_load_gep1(b, v_bits, v_nullbyteno, "attnullbyte");
v_nullbit = LLVMBuildICmp(b,
LLVMIntEQ,
LLVMBuildAnd(b, v_nullbyte, v_nullbytemask, ""),
- l_int8_const(0),
+ l_int8_const(lc, 0),
"attisnull");
v_attisnull = LLVMBuildAnd(b, v_hasnulls, v_nullbit, "");
@@ -454,7 +456,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
/* store null-byte */
LLVMBuildStore(b,
- l_int8_const(1),
+ l_int8_const(lc, 1),
LLVMBuildGEP(b, v_tts_nulls, &l_attno, 1, ""));
/* store zero datum */
LLVMBuildStore(b,
@@ -524,7 +526,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
l_load_gep1(b, v_tupdata_base, v_off, "padbyte");
v_ispad =
LLVMBuildICmp(b, LLVMIntEQ,
- v_possible_padbyte, l_int8_const(0),
+ v_possible_padbyte, l_int8_const(lc, 0),
"ispadbyte");
LLVMBuildCondBr(b, v_ispad,
attalignblocks[attnum],
@@ -639,7 +641,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
v_resultp = LLVMBuildGEP(b, v_tts_values, &l_attno, 1, "");
/* store null-byte (false) */
- LLVMBuildStore(b, l_int8_const(0),
+ LLVMBuildStore(b, l_int8_const(lc, 0),
LLVMBuildGEP(b, v_tts_nulls, &l_attno, 1, ""));
/*
@@ -650,7 +652,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
{
LLVMValueRef v_tmp_loaddata;
LLVMTypeRef vartypep =
- LLVMPointerType(LLVMIntType(att->attlen * 8), 0);
+ LLVMPointerType(LLVMIntTypeInContext(lc, att->attlen * 8), 0);
v_tmp_loaddata =
LLVMBuildPointerCast(b, v_attdatap, vartypep, "");
@@ -739,11 +741,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMValueRef v_off = LLVMBuildLoad(b, v_offp, "");
LLVMValueRef v_flags;
- LLVMBuildStore(b, l_int16_const(natts), v_nvalidp);
- v_off = LLVMBuildTrunc(b, v_off, LLVMInt32Type(), "");
+ LLVMBuildStore(b, l_int16_const(lc, natts), v_nvalidp);
+ v_off = LLVMBuildTrunc(b, v_off, LLVMInt32TypeInContext(lc), "");
LLVMBuildStore(b, v_off, v_slotoffp);
v_flags = LLVMBuildLoad(b, v_flagsp, "tts_flags");
- v_flags = LLVMBuildOr(b, v_flags, l_int16_const(TTS_FLAG_SLOW), "");
+ v_flags = LLVMBuildOr(b, v_flags, l_int16_const(lc, TTS_FLAG_SLOW), "");
LLVMBuildStore(b, v_flags, v_flagsp);
LLVMBuildRetVoid(b);
}
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
index a1b33d6e6c..61089ebdfc 100644
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -84,6 +84,7 @@ llvm_compile_expr(ExprState *state)
LLVMBuilderRef b;
LLVMModuleRef mod;
+ LLVMContextRef lc;
LLVMValueRef eval_fn;
LLVMBasicBlockRef entry;
LLVMBasicBlockRef *opblocks;
@@ -143,8 +144,9 @@ llvm_compile_expr(ExprState *state)
INSTR_TIME_SET_CURRENT(starttime);
mod = llvm_mutable_module(context);
+ lc = LLVMGetModuleContext(mod);
- b = LLVMCreateBuilder();
+ b = LLVMCreateBuilderInContext(lc);
funcname = llvm_expand_funcname(context, "evalexpr");
@@ -155,7 +157,7 @@ llvm_compile_expr(ExprState *state)
LLVMSetVisibility(eval_fn, LLVMDefaultVisibility);
llvm_copy_attributes(AttributeTemplate, eval_fn);
- entry = LLVMAppendBasicBlock(eval_fn, "entry");
+ entry = LLVMAppendBasicBlockInContext(lc, eval_fn, "entry");
/* build state */
v_state = LLVMGetParam(eval_fn, 0);
@@ -301,7 +303,7 @@ llvm_compile_expr(ExprState *state)
"");
LLVMBuildCondBr(b,
LLVMBuildICmp(b, LLVMIntUGE, v_nvalid,
- l_int16_const(op->d.fetch.last_var),
+ l_int16_const(lc, op->d.fetch.last_var),
""),
opblocks[opno + 1], b_fetch);
@@ -335,7 +337,7 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef params[2];
params[0] = v_slot;
- params[1] = l_int32_const(op->d.fetch.last_var);
+ params[1] = l_int32_const(lc, op->d.fetch.last_var);
LLVMBuildCall(b,
llvm_pg_func(mod, "slot_getsomeattrs_int"),
@@ -372,7 +374,7 @@ llvm_compile_expr(ExprState *state)
v_nulls = v_scannulls;
}
- v_attnum = l_int32_const(op->d.var.attnum);
+ v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, v_values, v_attnum, "");
isnull = l_load_gep1(b, v_nulls, v_attnum, "");
LLVMBuildStore(b, value, v_resvaluep);
@@ -438,12 +440,12 @@ llvm_compile_expr(ExprState *state)
}
/* load data */
- v_attnum = l_int32_const(op->d.assign_var.attnum);
+ v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
v_value = l_load_gep1(b, v_values, v_attnum, "");
v_isnull = l_load_gep1(b, v_nulls, v_attnum, "");
/* compute addresses of targets */
- v_resultnum = l_int32_const(op->d.assign_var.resultnum);
+ v_resultnum = l_int32_const(lc, op->d.assign_var.resultnum);
v_rvaluep = LLVMBuildGEP(b, v_resultvalues,
&v_resultnum, 1, "");
v_risnullp = LLVMBuildGEP(b, v_resultnulls,
@@ -472,7 +474,7 @@ llvm_compile_expr(ExprState *state)
v_isnull = LLVMBuildLoad(b, v_tmpisnullp, "");
/* compute addresses of targets */
- v_resultnum = l_int32_const(resultnum);
+ v_resultnum = l_int32_const(lc, resultnum);
v_rvaluep =
LLVMBuildGEP(b, v_resultvalues, &v_resultnum, 1, "");
v_risnullp =
@@ -1691,7 +1693,7 @@ llvm_compile_expr(ExprState *state)
v_cmpresult =
LLVMBuildTrunc(b,
LLVMBuildLoad(b, v_resvaluep, ""),
- LLVMInt32Type(), "");
+ LLVMInt32TypeInContext(lc), "");
switch (rctype)
{
@@ -1717,7 +1719,7 @@ llvm_compile_expr(ExprState *state)
v_result = LLVMBuildICmp(b,
predicate,
v_cmpresult,
- l_int32_const(0),
+ l_int32_const(lc, 0),
"");
v_result = LLVMBuildZExt(b, v_result, TypeSizeT, "");
@@ -1860,7 +1862,7 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef value,
isnull;
- v_aggno = l_int32_const(op->d.aggref.aggno);
+ v_aggno = l_int32_const(lc, op->d.aggref.aggno);
/* load agg value / null */
value = l_load_gep1(b, v_aggvalues, v_aggno, "aggvalue");
@@ -1894,7 +1896,7 @@ llvm_compile_expr(ExprState *state)
* expression). So load it from memory each time round.
*/
v_wfuncnop = l_ptr_const(&wfunc->wfuncno,
- l_ptr(LLVMInt32Type()));
+ l_ptr(LLVMInt32TypeInContext(lc)));
v_wfuncno = LLVMBuildLoad(b, v_wfuncnop, "v_wfuncno");
/* load window func value / null */
@@ -2002,7 +2004,7 @@ llvm_compile_expr(ExprState *state)
/* strict function, check for NULL args */
for (int argno = 0; argno < nargs; argno++)
{
- LLVMValueRef v_argno = l_int32_const(argno);
+ LLVMValueRef v_argno = l_int32_const(lc, argno);
LLVMValueRef v_argisnull;
LLVMBasicBlockRef b_argnotnull;
@@ -2059,7 +2061,7 @@ llvm_compile_expr(ExprState *state)
FIELDNO_AGGSTATE_ALL_PERGROUPS,
"aggstate.all_pergroups");
- v_setoff = l_int32_const(op->d.agg_plain_pergroup_nullcheck.setoff);
+ v_setoff = l_int32_const(lc, op->d.agg_plain_pergroup_nullcheck.setoff);
v_pergroup_allaggs = l_load_gep1(b, v_allpergroupsp, v_setoff, "");
@@ -2128,8 +2130,8 @@ llvm_compile_expr(ExprState *state)
l_load_struct_gep(b, v_aggstatep,
FIELDNO_AGGSTATE_ALL_PERGROUPS,
"aggstate.all_pergroups");
- v_setoff = l_int32_const(op->d.agg_trans.setoff);
- v_transno = l_int32_const(op->d.agg_trans.transno);
+ v_setoff = l_int32_const(lc, op->d.agg_trans.setoff);
+ v_transno = l_int32_const(lc, op->d.agg_trans.transno);
v_pergroupp =
LLVMBuildGEP(b,
l_load_gep1(b, v_allpergroupsp, v_setoff, ""),
@@ -2232,7 +2234,7 @@ llvm_compile_expr(ExprState *state)
/* set aggstate globals */
LLVMBuildStore(b, v_aggcontext, v_curaggcontext);
- LLVMBuildStore(b, l_int32_const(op->d.agg_trans.setno),
+ LLVMBuildStore(b, l_int32_const(lc, op->d.agg_trans.setno),
v_current_setp);
LLVMBuildStore(b, v_pertransp, v_current_pertransp);
@@ -2468,11 +2470,14 @@ BuildV1Call(LLVMJitContext *context, LLVMBuilderRef b,
LLVMModuleRef mod, FunctionCallInfo fcinfo,
LLVMValueRef *v_fcinfo_isnull)
{
+ LLVMContextRef lc;
LLVMValueRef v_fn;
LLVMValueRef v_fcinfo_isnullp;
LLVMValueRef v_retval;
LLVMValueRef v_fcinfo;
+ lc = LLVMGetModuleContext(mod);
+
v_fn = llvm_function_reference(context, b, mod, fcinfo);
v_fcinfo = l_ptr_const(fcinfo, l_ptr(StructFunctionCallInfoData));
@@ -2494,12 +2499,12 @@ BuildV1Call(LLVMJitContext *context, LLVMBuilderRef b,
LLVMValueRef v_lifetime = create_LifetimeEnd(mod);
LLVMValueRef params[2];
- params[0] = l_int64_const(sizeof(NullableDatum) * fcinfo->nargs);
- params[1] = l_ptr_const(fcinfo->args, l_ptr(LLVMInt8Type()));
+ params[0] = l_int64_const(lc, sizeof(NullableDatum) * fcinfo->nargs);
+ params[1] = l_ptr_const(fcinfo->args, l_ptr(LLVMInt8TypeInContext(lc)));
LLVMBuildCall(b, v_lifetime, params, lengthof(params), "");
- params[0] = l_int64_const(sizeof(fcinfo->isnull));
- params[1] = l_ptr_const(&fcinfo->isnull, l_ptr(LLVMInt8Type()));
+ params[0] = l_int64_const(lc, sizeof(fcinfo->isnull));
+ params[1] = l_ptr_const(&fcinfo->isnull, l_ptr(LLVMInt8TypeInContext(lc)));
LLVMBuildCall(b, v_lifetime, params, lengthof(params), "");
}
@@ -2545,6 +2550,7 @@ create_LifetimeEnd(LLVMModuleRef mod)
LLVMTypeRef sig;
LLVMValueRef fn;
LLVMTypeRef param_types[2];
+ LLVMContextRef lc;
/* LLVM 5+ has a variadic pointer argument */
#if LLVM_VERSION_MAJOR < 5
@@ -2557,12 +2563,12 @@ create_LifetimeEnd(LLVMModuleRef mod)
if (fn)
return fn;
- param_types[0] = LLVMInt64Type();
- param_types[1] = l_ptr(LLVMInt8Type());
+ lc = LLVMGetModuleContext(mod);
+ param_types[0] = LLVMInt64TypeInContext(lc);
+ param_types[1] = l_ptr(LLVMInt8TypeInContext(lc));
- sig = LLVMFunctionType(LLVMVoidType(),
- param_types, lengthof(param_types),
- false);
+ sig = LLVMFunctionType(LLVMVoidTypeInContext(lc), param_types,
+ lengthof(param_types), false);
fn = LLVMAddFunction(mod, nm, sig);
LLVMSetFunctionCallConv(fn, LLVMCCallConv);
diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp b/src/backend/jit/llvm/llvmjit_inline.cpp
index c765add856..cbd17a9fef 100644
--- a/src/backend/jit/llvm/llvmjit_inline.cpp
+++ b/src/backend/jit/llvm/llvmjit_inline.cpp
@@ -114,12 +114,12 @@ typedef llvm::StringMap<std::unique_ptr<llvm::ModuleSummaryIndex> > SummaryCache
llvm::ManagedStatic<SummaryCache> summary_cache;
-static std::unique_ptr<ImportMapTy> llvm_build_inline_plan(llvm::Module *mod);
+static std::unique_ptr<ImportMapTy> llvm_build_inline_plan(LLVMContextRef lc, llvm::Module *mod);
static void llvm_execute_inline_plan(llvm::Module *mod,
ImportMapTy *globalsToInline);
-static llvm::Module* load_module_cached(llvm::StringRef modPath);
-static std::unique_ptr<llvm::Module> load_module(llvm::StringRef Identifier);
+static llvm::Module* load_module_cached(LLVMContextRef c, llvm::StringRef modPath);
+static std::unique_ptr<llvm::Module> load_module(LLVMContextRef c, llvm::StringRef Identifier);
static std::unique_ptr<llvm::ModuleSummaryIndex> llvm_load_summary(llvm::StringRef path);
@@ -159,9 +159,10 @@ summaries_for_guid(const InlineSearchPath& path, llvm::GlobalValue::GUID guid);
void
llvm_inline(LLVMModuleRef M)
{
+ LLVMContextRef lc = LLVMGetModuleContext(M);
llvm::Module *mod = llvm::unwrap(M);
- std::unique_ptr<ImportMapTy> globalsToInline = llvm_build_inline_plan(mod);
+ std::unique_ptr<ImportMapTy> globalsToInline = llvm_build_inline_plan(lc, mod);
if (!globalsToInline)
return;
llvm_execute_inline_plan(mod, globalsToInline.get());
@@ -172,7 +173,7 @@ llvm_inline(LLVMModuleRef M)
* mod.
*/
static std::unique_ptr<ImportMapTy>
-llvm_build_inline_plan(llvm::Module *mod)
+llvm_build_inline_plan(LLVMContextRef lc, llvm::Module *mod)
{
std::unique_ptr<ImportMapTy> globalsToInline(new ImportMapTy());
FunctionInlineStates functionStates;
@@ -271,7 +272,7 @@ llvm_build_inline_plan(llvm::Module *mod)
continue;
}
- defMod = load_module_cached(modPath);
+ defMod = load_module_cached(lc, modPath);
if (defMod->materializeMetadata())
elog(FATAL, "failed to materialize metadata");
@@ -466,20 +467,20 @@ llvm_execute_inline_plan(llvm::Module *mod, ImportMapTy *globalsToInline)
* the cache state would get corrupted.
*/
static llvm::Module*
-load_module_cached(llvm::StringRef modPath)
+load_module_cached(LLVMContextRef lc, llvm::StringRef modPath)
{
auto it = module_cache->find(modPath);
if (it == module_cache->end())
{
it = module_cache->insert(
- std::make_pair(modPath, load_module(modPath))).first;
+ std::make_pair(modPath, load_module(lc, modPath))).first;
}
return it->second.get();
}
static std::unique_ptr<llvm::Module>
-load_module(llvm::StringRef Identifier)
+load_module(LLVMContextRef lc, llvm::StringRef Identifier)
{
LLVMMemoryBufferRef buf;
LLVMModuleRef mod;
@@ -491,7 +492,7 @@ load_module(llvm::StringRef Identifier)
if (LLVMCreateMemoryBufferWithContentsOfFile(path, &buf, &msg))
elog(FATAL, "failed to open bitcode file \"%s\": %s",
path, msg);
- if (LLVMGetBitcodeModuleInContext2(LLVMGetGlobalContext(), buf, &mod))
+ if (LLVMGetBitcodeModuleInContext2(lc, buf, &mod))
elog(FATAL, "failed to parse bitcode in file \"%s\"", path);
/*
diff --git a/src/include/jit/llvmjit_emit.h b/src/include/jit/llvmjit_emit.h
index 0745dcac9c..5e74543be4 100644
--- a/src/include/jit/llvmjit_emit.h
+++ b/src/include/jit/llvmjit_emit.h
@@ -44,36 +44,36 @@ l_ptr(LLVMTypeRef t)
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int8_const(int8 i)
+l_int8_const(LLVMContextRef lc, int8 i)
{
- return LLVMConstInt(LLVMInt8Type(), i, false);
+ return LLVMConstInt(LLVMInt8TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int16_const(int16 i)
+l_int16_const(LLVMContextRef lc, int16 i)
{
- return LLVMConstInt(LLVMInt16Type(), i, false);
+ return LLVMConstInt(LLVMInt16TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int32_const(int32 i)
+l_int32_const(LLVMContextRef lc, int32 i)
{
- return LLVMConstInt(LLVMInt32Type(), i, false);
+ return LLVMConstInt(LLVMInt32TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int64_const(int64 i)
+l_int64_const(LLVMContextRef lc, int64 i)
{
- return LLVMConstInt(LLVMInt64Type(), i, false);
+ return LLVMConstInt(LLVMInt64TypeInContext(lc), i, false);
}
/*
@@ -137,12 +137,15 @@ l_bb_before_v(LLVMBasicBlockRef r, const char *fmt,...)
{
char buf[512];
va_list args;
+ LLVMContextRef lc;
va_start(args, fmt);
vsnprintf(buf, sizeof(buf), fmt, args);
va_end(args);
- return LLVMInsertBasicBlock(r, buf);
+ lc = LLVMGetTypeContext(LLVMTypeOf(LLVMGetBasicBlockParent(r)));
+
+ return LLVMInsertBasicBlockInContext(lc, r, buf);
}
/* separate, because pg_attribute_printf(2, 3) can't appear in definition */
@@ -157,12 +160,15 @@ l_bb_append_v(LLVMValueRef f, const char *fmt,...)
{
char buf[512];
va_list args;
+ LLVMContextRef lc;
va_start(args, fmt);
vsnprintf(buf, sizeof(buf), fmt, args);
va_end(args);
- return LLVMAppendBasicBlock(f, buf);
+ lc = LLVMGetTypeContext(LLVMTypeOf(f));
+
+ return LLVMAppendBasicBlockInContext(lc, f, buf);
}
/*
@@ -174,7 +180,7 @@ l_callsite_ro(LLVMValueRef f)
const char argname[] = "readonly";
LLVMAttributeRef ref;
- ref = LLVMCreateStringAttribute(LLVMGetGlobalContext(),
+ ref = LLVMCreateStringAttribute(LLVMGetTypeContext(LLVMTypeOf(f)),
argname,
sizeof(argname) - 1,
NULL, 0);
@@ -194,7 +200,7 @@ l_callsite_alwaysinline(LLVMValueRef f)
id = LLVMGetEnumAttributeKindForName(argname,
sizeof(argname) - 1);
- attr = LLVMCreateEnumAttribute(LLVMGetGlobalContext(), id, 0);
+ attr = LLVMCreateEnumAttribute(LLVMGetTypeContext(LLVMTypeOf(f)), id, 0);
LLVMAddCallSiteAttribute(f, LLVMAttributeFunctionIndex, attr);
}
--
2.32.1 (Apple Git-133)
=
* Re: Memory leak on subquery as scalar operand
@ 2023-05-16 13:31 Daniel Gustafsson <daniel@yesql.se>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 18+ messages in thread
From: Daniel Gustafsson @ 2023-05-16 13:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee, pgsql-bugs <pgsql-bugs@lists.postgresql.org>
> On 2 May 2023, at 15:33, Daniel Gustafsson <daniel@yesql.se> wrote:
> The attached 0005 is a WIP attempt to address a few of the FIXME's in this
> patchset on top of your the 0001-0004.
I've attached a rebased v5 which has the above fixes as well as a few smaller
ones missed in v4. Testing through the reports linked to from the open item
it's clear that while there is still a continuous rise in resident size, this
slows it down dramatically with the size levelling out which unpatched master
doesn't. Reducing LLVMJIT_LLVM_CONTEXT_REUSE_MAX back even further pushes the
levelling off further as well, making me wonder if this is something worth
using a GUC for given that we're stuck with the destructive LLVM linker for the
foreseeable future?
--
Daniel Gustafsson
Attachments: download all (5 files, .tar.gz)
v5-0005-Address-review-comments.patch (application/octet-stream, 7.4K) download
From d9be6f3e888211a75613738b7af4652e3b345f67 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 2 May 2023 14:58:44 +0200
Subject: [PATCH v5 5/5] Address review comments
---
src/backend/jit/llvm/llvmjit.c | 160 +++++++++++++++++++--------------
1 file changed, 94 insertions(+), 66 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index f0c70d68ab..0713f12139 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -112,6 +112,8 @@ static void llvm_compile_module(LLVMJitContext *context);
static void llvm_optimize_module(LLVMJitContext *context, LLVMModuleRef module);
static void llvm_create_types(void);
+static void llvm_set_target(void);
+static void llvm_recreate_llvm_context(void);
static uint64_t llvm_resolve_symbol(const char *name, void *ctx);
#if LLVM_VERSION_MAJOR > 11
@@ -133,6 +135,61 @@ _PG_jit_provider_init(JitProviderCallbacks *cb)
cb->compile_expr = llvm_compile_expr;
}
+
+/*
+ * Every now and then create a new LLVMContextRef. Unfortunately, during
+ * every round of inlining, types may "leak" (they can still be found/used
+ * via the context, but new types will be created the next time in
+ * inlining is performed). To prevent that from slowly accumulating
+ * problematic amounts of memory, recreate the LLVMContextRef we use. We
+ * don't want to do so too often, as that implies some overhead
+ * (particularly re-loading the module summaries / modules is fairly
+ * expensive).
+ */
+static void
+llvm_recreate_llvm_context(void)
+{
+ if (!llvm_context)
+ elog(ERROR, "Trying to recreate a non-existing context");
+
+
+ /*
+ * We can only safely recreate the LLVM context if no other code is being
+ * JITed, otherwise we'd release the types in use for that.
+ */
+ if (llvm_jit_context_in_use_count > 0)
+ {
+ llvm_llvm_context_reuse_count++;
+ return;
+ }
+
+ if (llvm_llvm_context_reuse_count <= LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
+ {
+ llvm_llvm_context_reuse_count++;
+ return;
+ }
+
+ /*
+ * Need to reset the modules that the inlining code caches before
+ * disposing of the context. LLVM modules exist within a specific LLVM
+ * context, therefore disposing of the context before resetting the
+ * cache would lead to dangling pointers to modules.
+ */
+ llvm_inline_reset_caches();
+
+ LLVMContextDispose(llvm_context);
+ llvm_context = LLVMContextCreate();
+ llvm_llvm_context_reuse_count = 0;
+
+ /*
+ * Re-build cached type information, so code generation code can rely
+ * on that information to be present (also prevents the variables to
+ * be dangling references).
+ */
+ llvm_create_types();
+}
+
+
/*
* Create a context for JITing work.
*
@@ -149,54 +206,7 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
- /*
- * Every now and then create a new LLVMContextRef. Unfortunately, during
- * every round of inlining, types may "leak" (they can still be found/used
- * via the context, but new types will be created the next time in
- * inlining is performed). To prevent that from slowly accumulating
- * problematic amounts of memory, recreate the LLVMContextRef we use. We
- * don't want to do so too often, as that implies some overhead
- * (particularly re-loading the module summaries / modules is fairly
- * expensive).
- *
- * We can only safely recreate the LLVM context if no other code is being
- * JITed, otherwise we'd release the types in use for that.
- *
- * FIXME: Extract into helper function.
- */
- if (llvm_jit_context_in_use_count == 0 &&
- llvm_llvm_context_reuse_count > LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
- {
- llvm_llvm_context_reuse_count = 0;
-
- Assert(llvm_context != NULL);
-
- /*
- * Need to reset the modules that the inlining code caches before
- * disposing of the context. LLVM modules exist within a specific LLVM
- * context, therefore disposing of the context before resetting the
- * cache would lead to dangling pointers to modules.
- */
- llvm_inline_reset_caches();
-
- if (llvm_context != NULL)
- LLVMContextDispose(llvm_context);
- llvm_context = LLVMContextCreate();
-
- /*
- * Re-build cached type information, so code generation code can rely
- * on that information to be present (also prevents the variables to
- * be dangling references).
- *
- * FIXME: should split the handling of llvm_triple / llvm_layout out
- * of llvm_create_types() - that doesn't need to be redone.
- */
- llvm_create_types();
- }
- else
- {
- llvm_llvm_context_reuse_count++;
- }
+ llvm_recreate_llvm_context();
ResourceOwnerEnlargeJIT(CurrentResourceOwner);
@@ -219,7 +229,7 @@ llvm_create_context(int jitFlags)
static void
llvm_release_context(JitContext *context)
{
- LLVMJitContext *llvm_context = (LLVMJitContext *) context;
+ LLVMJitContext *llvm_jit_context = (LLVMJitContext *) context;
ListCell *lc;
/*
@@ -238,13 +248,13 @@ llvm_release_context(JitContext *context)
llvm_enter_fatal_on_oom();
- if (llvm_context->module)
+ if (llvm_jit_context->module)
{
- LLVMDisposeModule(llvm_context->module);
- llvm_context->module = NULL;
+ LLVMDisposeModule(llvm_jit_context->module);
+ llvm_jit_context->module = NULL;
}
- foreach(lc, llvm_context->handles)
+ foreach(lc, llvm_jit_context->handles)
{
LLVMJitHandle *jit_handle = (LLVMJitHandle *) lfirst(lc);
@@ -274,8 +284,8 @@ llvm_release_context(JitContext *context)
pfree(jit_handle);
}
- list_free(llvm_context->handles);
- llvm_context->handles = NIL;
+ list_free(llvm_jit_context->handles);
+ llvm_jit_context->handles = NIL;
}
/*
@@ -853,7 +863,13 @@ llvm_session_initialize(void)
LLVMInitializeNativeAsmPrinter();
LLVMInitializeNativeAsmParser();
- llvm_context = LLVMContextCreate();
+ if (llvm_context == NULL)
+ {
+ llvm_context = LLVMContextCreate();
+
+ llvm_jit_context_in_use_count = 0;
+ llvm_llvm_context_reuse_count = 0;
+ }
/*
* When targeting an LLVM version with opaque pointers enabled by
@@ -871,6 +887,11 @@ llvm_session_initialize(void)
*/
llvm_create_types();
+ /*
+ * Extract target information from loaded module.
+ */
+ llvm_set_target();
+
if (LLVMGetTargetFromTriple(llvm_triple, &llvm_targetref, &error) != 0)
{
elog(FATAL, "failed to query triple %s", error);
@@ -1040,6 +1061,23 @@ load_return_type(LLVMModuleRef mod, const char *name)
return typ;
}
+/*
+ * Load triple & layout from clang emitted file so we're guaranteed to be
+ * compatible.
+ */
+static void
+llvm_set_target(void)
+{
+ if (!llvm_types_module)
+ elog(ERROR, "failed to extract target information, llvmjit_types.c not loaded");
+
+ if (llvm_triple == NULL)
+ llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
+
+ if (llvm_layout == NULL)
+ llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+}
+
/*
* Load required information, types, function signatures from llvmjit_types.c
* and make them available in global variables.
@@ -1069,16 +1107,6 @@ llvm_create_types(void)
}
LLVMDisposeMemoryBuffer(buf);
- /*
- * Load triple & layout from clang emitted file so we're guaranteed to be
- * compatible.
- */
- if (llvm_triple == NULL)
- {
- llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
- llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
- }
-
TypeSizeT = llvm_pg_var_type("TypeSizeT");
TypeParamBool = load_return_type(llvm_types_module, "FunctionReturningBool");
TypeStorageBool = llvm_pg_var_type("TypeStorageBool");
--
2.32.1 (Apple Git-133)
=
v5-0004-wip-llvmjit-Plug-memory-leak-when-performing-inli.patch (application/octet-stream, 7.4K) download
From e894efc9f2b4d2cab2f44d9ad60fdb2d9c2ea61b Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 21:44:40 +0200
Subject: [PATCH v5 4/5] wip: llvmjit: Plug memory leak when performing
inlining.
When performing inlining LLVM unfortunately "leaks" types (the types
survive and are usable, but a new round of inlining will recreate new
types that are structurally equivalent). I could not come up with a
backpatchable fix for that, so address the problem by dropping &
recreating the LLVMContextRef that contains all IR related data -
which releases all the types.
That incurs some overhead, so only do so after 10000 JITed queries.
Needs a good bit more work:
- confusing naming between different types of contexts (PG side
LLVMJitContext, and LLVM side LLVMContextRef)
- add detailed explanation of leak issue
- consider only incrementing the counter triggering the recreation of
the memory context during inlining.
- several FIXMEs in code
Reported-By: Justin Pryzby <pryzby@telsasoft.com>
Reported-By: Kurt Roeckx <kurt@roeckx.be>
Reported-By: Jaime Casanova <jcasanov@systemguards.com.ec>
Author: Andres Freund
Discussion: https://postgr.es/m/7acc8678-df5f-4923-9cf6-e843131ae89d@www.fastmail.com
Discussion: https://postgr.es/m/20201218235607.GC30237@telsasoft.com
---
src/backend/jit/llvm/llvmjit.c | 76 ++++++++++++++++++++++++-
src/backend/jit/llvm/llvmjit_inline.cpp | 12 ++++
src/include/jit/llvmjit.h | 8 +++
3 files changed, 94 insertions(+), 2 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index 6e0664745d..f0c70d68ab 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -42,6 +42,10 @@
#include "utils/memutils.h"
#include "utils/resowner_private.h"
+
+#define LLVMJIT_LLVM_CONTEXT_REUSE_MAX 100
+
+
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
{
@@ -81,6 +85,10 @@ static LLVMModuleRef llvm_types_module = NULL;
static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
+/* number of LLVMJitContexts that currently are in use */
+static size_t llvm_jit_context_in_use_count = 0;
+/* how many times has the current LLVMContextRef been used */
+static size_t llvm_llvm_context_reuse_count = 0;
static const char *llvm_triple = NULL;
static const char *llvm_layout = NULL;
static LLVMContextRef llvm_context;
@@ -141,6 +149,55 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
+ /*
+ * Every now and then create a new LLVMContextRef. Unfortunately, during
+ * every round of inlining, types may "leak" (they can still be found/used
+ * via the context, but new types will be created the next time in
+ * inlining is performed). To prevent that from slowly accumulating
+ * problematic amounts of memory, recreate the LLVMContextRef we use. We
+ * don't want to do so too often, as that implies some overhead
+ * (particularly re-loading the module summaries / modules is fairly
+ * expensive).
+ *
+ * We can only safely recreate the LLVM context if no other code is being
+ * JITed, otherwise we'd release the types in use for that.
+ *
+ * FIXME: Extract into helper function.
+ */
+ if (llvm_jit_context_in_use_count == 0 &&
+ llvm_llvm_context_reuse_count > LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
+ {
+ llvm_llvm_context_reuse_count = 0;
+
+ Assert(llvm_context != NULL);
+
+ /*
+ * Need to reset the modules that the inlining code caches before
+ * disposing of the context. LLVM modules exist within a specific LLVM
+ * context, therefore disposing of the context before resetting the
+ * cache would lead to dangling pointers to modules.
+ */
+ llvm_inline_reset_caches();
+
+ if (llvm_context != NULL)
+ LLVMContextDispose(llvm_context);
+ llvm_context = LLVMContextCreate();
+
+ /*
+ * Re-build cached type information, so code generation code can rely
+ * on that information to be present (also prevents the variables to
+ * be dangling references).
+ *
+ * FIXME: should split the handling of llvm_triple / llvm_layout out
+ * of llvm_create_types() - that doesn't need to be redone.
+ */
+ llvm_create_types();
+ }
+ else
+ {
+ llvm_llvm_context_reuse_count++;
+ }
+
ResourceOwnerEnlargeJIT(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
@@ -151,6 +208,8 @@ llvm_create_context(int jitFlags)
context->base.resowner = CurrentResourceOwner;
ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ llvm_jit_context_in_use_count++;
+
return context;
}
@@ -163,6 +222,12 @@ llvm_release_context(JitContext *context)
LLVMJitContext *llvm_context = (LLVMJitContext *) context;
ListCell *lc;
+ /*
+ * Consider as cleaned up even if we skip doing so below, that way we can
+ * verify the tracking is correct (see llvm_shutdown()).
+ */
+ llvm_jit_context_in_use_count--;
+
/*
* When this backend is exiting, don't clean up LLVM. As an error might
* have occurred from within LLVM, we do not want to risk reentering. All
@@ -901,6 +966,10 @@ llvm_shutdown(int code, Datum arg)
return;
}
+ if (llvm_jit_context_in_use_count != 0)
+ elog(PANIC, "LLVMJitContext in use count not 0 at exit (is %zu)",
+ llvm_jit_context_in_use_count);
+
#if LLVM_VERSION_MAJOR > 11
{
if (llvm_opt3_orc)
@@ -1004,8 +1073,11 @@ llvm_create_types(void)
* Load triple & layout from clang emitted file so we're guaranteed to be
* compatible.
*/
- llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
- llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+ if (llvm_triple == NULL)
+ {
+ llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
+ llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+ }
TypeSizeT = llvm_pg_var_type("TypeSizeT");
TypeParamBool = load_return_type(llvm_types_module, "FunctionReturningBool");
diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp b/src/backend/jit/llvm/llvmjit_inline.cpp
index cbd17a9fef..d92d7f3c88 100644
--- a/src/backend/jit/llvm/llvmjit_inline.cpp
+++ b/src/backend/jit/llvm/llvmjit_inline.cpp
@@ -152,6 +152,18 @@ summaries_for_guid(const InlineSearchPath& path, llvm::GlobalValue::GUID guid);
#define ilog(...) (void) 0
#endif
+/*
+ * Reset inlining related state. This needs to be called before the currently
+ * used LLVMContextRef is disposed (and a new one create), otherwise we would
+ * have dangling references to deleted modules.
+ */
+void
+llvm_inline_reset_caches(void)
+{
+ module_cache->clear();
+ summary_cache->clear();
+}
+
/*
* Perform inlining of external function references in M based on a simple
* cost based analysis.
diff --git a/src/include/jit/llvmjit.h b/src/include/jit/llvmjit.h
index 17b3b7593f..6d90a16f79 100644
--- a/src/include/jit/llvmjit.h
+++ b/src/include/jit/llvmjit.h
@@ -42,6 +42,13 @@ typedef struct LLVMJitContext
/* number of modules created */
size_t module_generation;
+ /*
+ * The LLVM Context used by this JIT context. An LLVM context is reused
+ * across many compilations, but occasionally reset to prevent it using
+ * too much memory due to more and more types accumulating.
+ */
+ LLVMContextRef llvm_context;
+
/* current, "open for write", module */
LLVMModuleRef module;
@@ -99,6 +106,7 @@ extern LLVMValueRef llvm_function_reference(LLVMJitContext *context,
LLVMModuleRef mod,
FunctionCallInfo fcinfo);
+extern void llvm_inline_reset_caches(void);
extern void llvm_inline(LLVMModuleRef mod);
/*
--
2.32.1 (Apple Git-133)
=
v5-0003-wip-llvmjit-Make-llvm_types_module-variable-stati.patch (application/octet-stream, 1.4K) download
From afbfd8499e12efa536a818c00b64b0cdda385a55 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:44:20 +0200
Subject: [PATCH v5 3/5] wip: llvmjit: Make llvm_types_module variable static.
Not sure why I did that differently in b059d2f45685a.
---
src/backend/jit/llvm/llvmjit.c | 2 +-
src/include/jit/llvmjit.h | 3 ---
2 files changed, 1 insertion(+), 4 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index e26b56f0dd..6e0664745d 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -77,7 +77,7 @@ LLVMTypeRef StructAggStatePerTransData;
LLVMValueRef AttributeTemplate;
-LLVMModuleRef llvm_types_module = NULL;
+static LLVMModuleRef llvm_types_module = NULL;
static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
diff --git a/src/include/jit/llvmjit.h b/src/include/jit/llvmjit.h
index 551b585464..17b3b7593f 100644
--- a/src/include/jit/llvmjit.h
+++ b/src/include/jit/llvmjit.h
@@ -55,9 +55,6 @@ typedef struct LLVMJitContext
List *handles;
} LLVMJitContext;
-/* llvm module containing information about types */
-extern PGDLLIMPORT LLVMModuleRef llvm_types_module;
-
/* type and struct definitions */
extern PGDLLIMPORT LLVMTypeRef TypeParamBool;
extern PGDLLIMPORT LLVMTypeRef TypePGFunction;
--
2.32.1 (Apple Git-133)
=
v5-0002-wip-llvmjit-remove-unnecessary-types.patch (application/octet-stream, 1.3K) download
From 93eca3d5ec0fd4a7c04b1eb35540cd59cf3f028c Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:43:37 +0200
Subject: [PATCH v5 2/5] wip: llvmjit: remove unnecessary types.
---
src/backend/jit/llvm/llvmjit.c | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index d9b0aec5af..e26b56f0dd 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -61,23 +61,12 @@ LLVMTypeRef TypeParamBool;
LLVMTypeRef TypeStorageBool;
LLVMTypeRef TypePGFunction;
LLVMTypeRef StructNullableDatum;
-LLVMTypeRef StructHeapTupleFieldsField3;
-LLVMTypeRef StructHeapTupleFields;
-LLVMTypeRef StructHeapTupleHeaderData;
-LLVMTypeRef StructHeapTupleDataChoice;
LLVMTypeRef StructHeapTupleData;
-LLVMTypeRef StructMinimalTupleData;
-LLVMTypeRef StructItemPointerData;
-LLVMTypeRef StructBlockId;
-LLVMTypeRef StructFormPgAttribute;
-LLVMTypeRef StructTupleConstr;
LLVMTypeRef StructTupleDescData;
LLVMTypeRef StructTupleTableSlot;
LLVMTypeRef StructHeapTupleTableSlot;
LLVMTypeRef StructMinimalTupleTableSlot;
LLVMTypeRef StructMemoryContextData;
-LLVMTypeRef StructPGFinfoRecord;
-LLVMTypeRef StructFmgrInfo;
LLVMTypeRef StructFunctionCallInfoData;
LLVMTypeRef StructExprContext;
LLVMTypeRef StructExprEvalStep;
--
2.32.1 (Apple Git-133)
=
v5-0001-wip-llvmjit-Use-explicit-LLVMContextRef.patch (application/octet-stream, 23.4K) download
From b2abdc178d0d3d20918582f80cc7ea0417175855 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:43:03 +0200
Subject: [PATCH v5 1/5] wip: llvmjit: Use explicit LLVMContextRef.
This is a prerequisite for a later commit fixing a memory leak due to
slow accumulation of types in LLVM.
---
src/backend/jit/llvm/llvmjit.c | 9 ++--
src/backend/jit/llvm/llvmjit_deform.c | 60 +++++++++++++------------
src/backend/jit/llvm/llvmjit_expr.c | 58 +++++++++++++-----------
src/backend/jit/llvm/llvmjit_inline.cpp | 21 ++++-----
src/include/jit/llvmjit_emit.h | 30 ++++++++-----
5 files changed, 98 insertions(+), 80 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index a8b73a9cf1..d9b0aec5af 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -94,6 +94,7 @@ static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
static const char *llvm_triple = NULL;
static const char *llvm_layout = NULL;
+static LLVMContextRef llvm_context;
static LLVMTargetRef llvm_targetref;
@@ -238,7 +239,7 @@ llvm_mutable_module(LLVMJitContext *context)
{
context->compiled = false;
context->module_generation = llvm_generation++;
- context->module = LLVMModuleCreateWithName("pg");
+ context->module = LLVMModuleCreateWithNameInContext("pg", llvm_context);
LLVMSetTarget(context->module, llvm_triple);
LLVMSetDataLayout(context->module, llvm_layout);
}
@@ -798,6 +799,8 @@ llvm_session_initialize(void)
LLVMInitializeNativeAsmPrinter();
LLVMInitializeNativeAsmParser();
+ llvm_context = LLVMContextCreate();
+
/*
* When targeting an LLVM version with opaque pointers enabled by
* default, turn them off for the context we build our code in. We don't
@@ -1002,9 +1005,9 @@ llvm_create_types(void)
}
/* eagerly load contents, going to need it all */
- if (LLVMParseBitcode2(buf, &llvm_types_module))
+ if (LLVMParseBitcodeInContext2(llvm_context, buf, &llvm_types_module))
{
- elog(ERROR, "LLVMParseBitcode2 of %s failed", path);
+ elog(ERROR, "LLVMParseBitcodeInContext2 of %s failed", path);
}
LLVMDisposeMemoryBuffer(buf);
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 6b15588da6..68e0ff354d 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -37,6 +37,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
char *funcname;
LLVMModuleRef mod;
+ LLVMContextRef lc;
LLVMBuilderRef b;
LLVMTypeRef deform_sig;
@@ -99,6 +100,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
return NULL;
mod = llvm_mutable_module(context);
+ lc = LLVMGetModuleContext(mod);
funcname = llvm_expand_funcname(context, "deform");
@@ -133,8 +135,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
param_types[0] = l_ptr(StructTupleTableSlot);
- deform_sig = LLVMFunctionType(LLVMVoidType(), param_types,
- lengthof(param_types), 0);
+ deform_sig = LLVMFunctionType(LLVMVoidTypeInContext(lc),
+ param_types, lengthof(param_types), 0);
}
v_deform_fn = LLVMAddFunction(mod, funcname, deform_sig);
LLVMSetLinkage(v_deform_fn, LLVMInternalLinkage);
@@ -142,17 +144,17 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
llvm_copy_attributes(AttributeTemplate, v_deform_fn);
b_entry =
- LLVMAppendBasicBlock(v_deform_fn, "entry");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "entry");
b_adjust_unavail_cols =
- LLVMAppendBasicBlock(v_deform_fn, "adjust_unavail_cols");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "adjust_unavail_cols");
b_find_start =
- LLVMAppendBasicBlock(v_deform_fn, "find_startblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "find_startblock");
b_out =
- LLVMAppendBasicBlock(v_deform_fn, "outblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "outblock");
b_dead =
- LLVMAppendBasicBlock(v_deform_fn, "deadblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "deadblock");
- b = LLVMCreateBuilder();
+ b = LLVMCreateBuilderInContext(lc);
attcheckattnoblocks = palloc(sizeof(LLVMBasicBlockRef) * natts);
attstartblocks = palloc(sizeof(LLVMBasicBlockRef) * natts);
@@ -221,7 +223,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMBuildStructGEP(b, v_tuplep,
FIELDNO_HEAPTUPLEHEADERDATA_BITS,
""),
- l_ptr(LLVMInt8Type()),
+ l_ptr(LLVMInt8TypeInContext(lc)),
"t_bits");
v_infomask1 =
l_load_struct_gep(b, v_tuplep,
@@ -236,14 +238,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
v_hasnulls =
LLVMBuildICmp(b, LLVMIntNE,
LLVMBuildAnd(b,
- l_int16_const(HEAP_HASNULL),
+ l_int16_const(lc, HEAP_HASNULL),
v_infomask1, ""),
- l_int16_const(0),
+ l_int16_const(lc, 0),
"hasnulls");
/* t_infomask2 & HEAP_NATTS_MASK */
v_maxatt = LLVMBuildAnd(b,
- l_int16_const(HEAP_NATTS_MASK),
+ l_int16_const(lc, HEAP_NATTS_MASK),
v_infomask2,
"maxatt");
@@ -256,13 +258,13 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
l_load_struct_gep(b, v_tuplep,
FIELDNO_HEAPTUPLEHEADERDATA_HOFF,
""),
- LLVMInt32Type(), "t_hoff");
+ LLVMInt32TypeInContext(lc), "t_hoff");
v_tupdata_base =
LLVMBuildGEP(b,
LLVMBuildBitCast(b,
v_tuplep,
- l_ptr(LLVMInt8Type()),
+ l_ptr(LLVMInt8TypeInContext(lc)),
""),
&v_hoff, 1,
"v_tupdata_base");
@@ -319,7 +321,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMBuildCondBr(b,
LLVMBuildICmp(b, LLVMIntULT,
v_maxatt,
- l_int16_const(natts),
+ l_int16_const(lc, natts),
""),
b_adjust_unavail_cols,
b_find_start);
@@ -328,8 +330,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMPositionBuilderAtEnd(b, b_adjust_unavail_cols);
v_params[0] = v_slot;
- v_params[1] = LLVMBuildZExt(b, v_maxatt, LLVMInt32Type(), "");
- v_params[2] = l_int32_const(natts);
+ v_params[1] = LLVMBuildZExt(b, v_maxatt, LLVMInt32TypeInContext(lc), "");
+ v_params[2] = l_int32_const(lc, natts);
LLVMBuildCall(b, llvm_pg_func(mod, "slot_getmissingattrs"),
v_params, lengthof(v_params), "");
LLVMBuildBr(b, b_find_start);
@@ -352,7 +354,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
- LLVMValueRef v_attno = l_int16_const(attnum);
+ LLVMValueRef v_attno = l_int16_const(lc, attnum);
LLVMAddCase(v_switch, v_attno, attcheckattnoblocks[attnum]);
}
@@ -375,7 +377,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
Form_pg_attribute att = TupleDescAttr(desc, attnum);
LLVMValueRef v_incby;
int alignto;
- LLVMValueRef l_attno = l_int16_const(attnum);
+ LLVMValueRef l_attno = l_int16_const(lc, attnum);
LLVMValueRef v_attdatap;
LLVMValueRef v_resultp;
@@ -436,14 +438,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
else
b_next = attcheckattnoblocks[attnum + 1];
- v_nullbyteno = l_int32_const(attnum >> 3);
- v_nullbytemask = l_int8_const(1 << ((attnum) & 0x07));
+ v_nullbyteno = l_int32_const(lc, attnum >> 3);
+ v_nullbytemask = l_int8_const(lc, 1 << ((attnum) & 0x07));
v_nullbyte = l_load_gep1(b, v_bits, v_nullbyteno, "attnullbyte");
v_nullbit = LLVMBuildICmp(b,
LLVMIntEQ,
LLVMBuildAnd(b, v_nullbyte, v_nullbytemask, ""),
- l_int8_const(0),
+ l_int8_const(lc, 0),
"attisnull");
v_attisnull = LLVMBuildAnd(b, v_hasnulls, v_nullbit, "");
@@ -454,7 +456,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
/* store null-byte */
LLVMBuildStore(b,
- l_int8_const(1),
+ l_int8_const(lc, 1),
LLVMBuildGEP(b, v_tts_nulls, &l_attno, 1, ""));
/* store zero datum */
LLVMBuildStore(b,
@@ -524,7 +526,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
l_load_gep1(b, v_tupdata_base, v_off, "padbyte");
v_ispad =
LLVMBuildICmp(b, LLVMIntEQ,
- v_possible_padbyte, l_int8_const(0),
+ v_possible_padbyte, l_int8_const(lc, 0),
"ispadbyte");
LLVMBuildCondBr(b, v_ispad,
attalignblocks[attnum],
@@ -639,7 +641,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
v_resultp = LLVMBuildGEP(b, v_tts_values, &l_attno, 1, "");
/* store null-byte (false) */
- LLVMBuildStore(b, l_int8_const(0),
+ LLVMBuildStore(b, l_int8_const(lc, 0),
LLVMBuildGEP(b, v_tts_nulls, &l_attno, 1, ""));
/*
@@ -650,7 +652,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
{
LLVMValueRef v_tmp_loaddata;
LLVMTypeRef vartypep =
- LLVMPointerType(LLVMIntType(att->attlen * 8), 0);
+ LLVMPointerType(LLVMIntTypeInContext(lc, att->attlen * 8), 0);
v_tmp_loaddata =
LLVMBuildPointerCast(b, v_attdatap, vartypep, "");
@@ -739,11 +741,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMValueRef v_off = LLVMBuildLoad(b, v_offp, "");
LLVMValueRef v_flags;
- LLVMBuildStore(b, l_int16_const(natts), v_nvalidp);
- v_off = LLVMBuildTrunc(b, v_off, LLVMInt32Type(), "");
+ LLVMBuildStore(b, l_int16_const(lc, natts), v_nvalidp);
+ v_off = LLVMBuildTrunc(b, v_off, LLVMInt32TypeInContext(lc), "");
LLVMBuildStore(b, v_off, v_slotoffp);
v_flags = LLVMBuildLoad(b, v_flagsp, "tts_flags");
- v_flags = LLVMBuildOr(b, v_flags, l_int16_const(TTS_FLAG_SLOW), "");
+ v_flags = LLVMBuildOr(b, v_flags, l_int16_const(lc, TTS_FLAG_SLOW), "");
LLVMBuildStore(b, v_flags, v_flagsp);
LLVMBuildRetVoid(b);
}
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
index 2c3d64ea6e..13e1481639 100644
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -84,6 +84,7 @@ llvm_compile_expr(ExprState *state)
LLVMBuilderRef b;
LLVMModuleRef mod;
+ LLVMContextRef lc;
LLVMValueRef eval_fn;
LLVMBasicBlockRef entry;
LLVMBasicBlockRef *opblocks;
@@ -143,8 +144,9 @@ llvm_compile_expr(ExprState *state)
INSTR_TIME_SET_CURRENT(starttime);
mod = llvm_mutable_module(context);
+ lc = LLVMGetModuleContext(mod);
- b = LLVMCreateBuilder();
+ b = LLVMCreateBuilderInContext(lc);
funcname = llvm_expand_funcname(context, "evalexpr");
@@ -155,7 +157,7 @@ llvm_compile_expr(ExprState *state)
LLVMSetVisibility(eval_fn, LLVMDefaultVisibility);
llvm_copy_attributes(AttributeTemplate, eval_fn);
- entry = LLVMAppendBasicBlock(eval_fn, "entry");
+ entry = LLVMAppendBasicBlockInContext(lc, eval_fn, "entry");
/* build state */
v_state = LLVMGetParam(eval_fn, 0);
@@ -301,7 +303,7 @@ llvm_compile_expr(ExprState *state)
"");
LLVMBuildCondBr(b,
LLVMBuildICmp(b, LLVMIntUGE, v_nvalid,
- l_int16_const(op->d.fetch.last_var),
+ l_int16_const(lc, op->d.fetch.last_var),
""),
opblocks[opno + 1], b_fetch);
@@ -335,7 +337,7 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef params[2];
params[0] = v_slot;
- params[1] = l_int32_const(op->d.fetch.last_var);
+ params[1] = l_int32_const(lc, op->d.fetch.last_var);
LLVMBuildCall(b,
llvm_pg_func(mod, "slot_getsomeattrs_int"),
@@ -372,7 +374,7 @@ llvm_compile_expr(ExprState *state)
v_nulls = v_scannulls;
}
- v_attnum = l_int32_const(op->d.var.attnum);
+ v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, v_values, v_attnum, "");
isnull = l_load_gep1(b, v_nulls, v_attnum, "");
LLVMBuildStore(b, value, v_resvaluep);
@@ -438,12 +440,12 @@ llvm_compile_expr(ExprState *state)
}
/* load data */
- v_attnum = l_int32_const(op->d.assign_var.attnum);
+ v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
v_value = l_load_gep1(b, v_values, v_attnum, "");
v_isnull = l_load_gep1(b, v_nulls, v_attnum, "");
/* compute addresses of targets */
- v_resultnum = l_int32_const(op->d.assign_var.resultnum);
+ v_resultnum = l_int32_const(lc, op->d.assign_var.resultnum);
v_rvaluep = LLVMBuildGEP(b, v_resultvalues,
&v_resultnum, 1, "");
v_risnullp = LLVMBuildGEP(b, v_resultnulls,
@@ -472,7 +474,7 @@ llvm_compile_expr(ExprState *state)
v_isnull = LLVMBuildLoad(b, v_tmpisnullp, "");
/* compute addresses of targets */
- v_resultnum = l_int32_const(resultnum);
+ v_resultnum = l_int32_const(lc, resultnum);
v_rvaluep =
LLVMBuildGEP(b, v_resultvalues, &v_resultnum, 1, "");
v_risnullp =
@@ -1691,7 +1693,7 @@ llvm_compile_expr(ExprState *state)
v_cmpresult =
LLVMBuildTrunc(b,
LLVMBuildLoad(b, v_resvaluep, ""),
- LLVMInt32Type(), "");
+ LLVMInt32TypeInContext(lc), "");
switch (rctype)
{
@@ -1717,7 +1719,7 @@ llvm_compile_expr(ExprState *state)
v_result = LLVMBuildICmp(b,
predicate,
v_cmpresult,
- l_int32_const(0),
+ l_int32_const(lc, 0),
"");
v_result = LLVMBuildZExt(b, v_result, TypeSizeT, "");
@@ -1860,7 +1862,7 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef value,
isnull;
- v_aggno = l_int32_const(op->d.aggref.aggno);
+ v_aggno = l_int32_const(lc, op->d.aggref.aggno);
/* load agg value / null */
value = l_load_gep1(b, v_aggvalues, v_aggno, "aggvalue");
@@ -1894,7 +1896,7 @@ llvm_compile_expr(ExprState *state)
* expression). So load it from memory each time round.
*/
v_wfuncnop = l_ptr_const(&wfunc->wfuncno,
- l_ptr(LLVMInt32Type()));
+ l_ptr(LLVMInt32TypeInContext(lc)));
v_wfuncno = LLVMBuildLoad(b, v_wfuncnop, "v_wfuncno");
/* load window func value / null */
@@ -2002,7 +2004,7 @@ llvm_compile_expr(ExprState *state)
/* strict function, check for NULL args */
for (int argno = 0; argno < nargs; argno++)
{
- LLVMValueRef v_argno = l_int32_const(argno);
+ LLVMValueRef v_argno = l_int32_const(lc, argno);
LLVMValueRef v_argisnull;
LLVMBasicBlockRef b_argnotnull;
@@ -2059,7 +2061,7 @@ llvm_compile_expr(ExprState *state)
FIELDNO_AGGSTATE_ALL_PERGROUPS,
"aggstate.all_pergroups");
- v_setoff = l_int32_const(op->d.agg_plain_pergroup_nullcheck.setoff);
+ v_setoff = l_int32_const(lc, op->d.agg_plain_pergroup_nullcheck.setoff);
v_pergroup_allaggs = l_load_gep1(b, v_allpergroupsp, v_setoff, "");
@@ -2128,8 +2130,8 @@ llvm_compile_expr(ExprState *state)
l_load_struct_gep(b, v_aggstatep,
FIELDNO_AGGSTATE_ALL_PERGROUPS,
"aggstate.all_pergroups");
- v_setoff = l_int32_const(op->d.agg_trans.setoff);
- v_transno = l_int32_const(op->d.agg_trans.transno);
+ v_setoff = l_int32_const(lc, op->d.agg_trans.setoff);
+ v_transno = l_int32_const(lc, op->d.agg_trans.transno);
v_pergroupp =
LLVMBuildGEP(b,
l_load_gep1(b, v_allpergroupsp, v_setoff, ""),
@@ -2232,7 +2234,7 @@ llvm_compile_expr(ExprState *state)
/* set aggstate globals */
LLVMBuildStore(b, v_aggcontext, v_curaggcontext);
- LLVMBuildStore(b, l_int32_const(op->d.agg_trans.setno),
+ LLVMBuildStore(b, l_int32_const(lc, op->d.agg_trans.setno),
v_current_setp);
LLVMBuildStore(b, v_pertransp, v_current_pertransp);
@@ -2468,11 +2470,14 @@ BuildV1Call(LLVMJitContext *context, LLVMBuilderRef b,
LLVMModuleRef mod, FunctionCallInfo fcinfo,
LLVMValueRef *v_fcinfo_isnull)
{
+ LLVMContextRef lc;
LLVMValueRef v_fn;
LLVMValueRef v_fcinfo_isnullp;
LLVMValueRef v_retval;
LLVMValueRef v_fcinfo;
+ lc = LLVMGetModuleContext(mod);
+
v_fn = llvm_function_reference(context, b, mod, fcinfo);
v_fcinfo = l_ptr_const(fcinfo, l_ptr(StructFunctionCallInfoData));
@@ -2494,12 +2499,12 @@ BuildV1Call(LLVMJitContext *context, LLVMBuilderRef b,
LLVMValueRef v_lifetime = create_LifetimeEnd(mod);
LLVMValueRef params[2];
- params[0] = l_int64_const(sizeof(NullableDatum) * fcinfo->nargs);
- params[1] = l_ptr_const(fcinfo->args, l_ptr(LLVMInt8Type()));
+ params[0] = l_int64_const(lc, sizeof(NullableDatum) * fcinfo->nargs);
+ params[1] = l_ptr_const(fcinfo->args, l_ptr(LLVMInt8TypeInContext(lc)));
LLVMBuildCall(b, v_lifetime, params, lengthof(params), "");
- params[0] = l_int64_const(sizeof(fcinfo->isnull));
- params[1] = l_ptr_const(&fcinfo->isnull, l_ptr(LLVMInt8Type()));
+ params[0] = l_int64_const(lc, sizeof(fcinfo->isnull));
+ params[1] = l_ptr_const(&fcinfo->isnull, l_ptr(LLVMInt8TypeInContext(lc)));
LLVMBuildCall(b, v_lifetime, params, lengthof(params), "");
}
@@ -2545,6 +2550,7 @@ create_LifetimeEnd(LLVMModuleRef mod)
LLVMTypeRef sig;
LLVMValueRef fn;
LLVMTypeRef param_types[2];
+ LLVMContextRef lc;
/* LLVM 5+ has a variadic pointer argument */
#if LLVM_VERSION_MAJOR < 5
@@ -2557,12 +2563,12 @@ create_LifetimeEnd(LLVMModuleRef mod)
if (fn)
return fn;
- param_types[0] = LLVMInt64Type();
- param_types[1] = l_ptr(LLVMInt8Type());
+ lc = LLVMGetModuleContext(mod);
+ param_types[0] = LLVMInt64TypeInContext(lc);
+ param_types[1] = l_ptr(LLVMInt8TypeInContext(lc));
- sig = LLVMFunctionType(LLVMVoidType(),
- param_types, lengthof(param_types),
- false);
+ sig = LLVMFunctionType(LLVMVoidTypeInContext(lc), param_types,
+ lengthof(param_types), false);
fn = LLVMAddFunction(mod, nm, sig);
LLVMSetFunctionCallConv(fn, LLVMCCallConv);
diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp b/src/backend/jit/llvm/llvmjit_inline.cpp
index c765add856..cbd17a9fef 100644
--- a/src/backend/jit/llvm/llvmjit_inline.cpp
+++ b/src/backend/jit/llvm/llvmjit_inline.cpp
@@ -114,12 +114,12 @@ typedef llvm::StringMap<std::unique_ptr<llvm::ModuleSummaryIndex> > SummaryCache
llvm::ManagedStatic<SummaryCache> summary_cache;
-static std::unique_ptr<ImportMapTy> llvm_build_inline_plan(llvm::Module *mod);
+static std::unique_ptr<ImportMapTy> llvm_build_inline_plan(LLVMContextRef lc, llvm::Module *mod);
static void llvm_execute_inline_plan(llvm::Module *mod,
ImportMapTy *globalsToInline);
-static llvm::Module* load_module_cached(llvm::StringRef modPath);
-static std::unique_ptr<llvm::Module> load_module(llvm::StringRef Identifier);
+static llvm::Module* load_module_cached(LLVMContextRef c, llvm::StringRef modPath);
+static std::unique_ptr<llvm::Module> load_module(LLVMContextRef c, llvm::StringRef Identifier);
static std::unique_ptr<llvm::ModuleSummaryIndex> llvm_load_summary(llvm::StringRef path);
@@ -159,9 +159,10 @@ summaries_for_guid(const InlineSearchPath& path, llvm::GlobalValue::GUID guid);
void
llvm_inline(LLVMModuleRef M)
{
+ LLVMContextRef lc = LLVMGetModuleContext(M);
llvm::Module *mod = llvm::unwrap(M);
- std::unique_ptr<ImportMapTy> globalsToInline = llvm_build_inline_plan(mod);
+ std::unique_ptr<ImportMapTy> globalsToInline = llvm_build_inline_plan(lc, mod);
if (!globalsToInline)
return;
llvm_execute_inline_plan(mod, globalsToInline.get());
@@ -172,7 +173,7 @@ llvm_inline(LLVMModuleRef M)
* mod.
*/
static std::unique_ptr<ImportMapTy>
-llvm_build_inline_plan(llvm::Module *mod)
+llvm_build_inline_plan(LLVMContextRef lc, llvm::Module *mod)
{
std::unique_ptr<ImportMapTy> globalsToInline(new ImportMapTy());
FunctionInlineStates functionStates;
@@ -271,7 +272,7 @@ llvm_build_inline_plan(llvm::Module *mod)
continue;
}
- defMod = load_module_cached(modPath);
+ defMod = load_module_cached(lc, modPath);
if (defMod->materializeMetadata())
elog(FATAL, "failed to materialize metadata");
@@ -466,20 +467,20 @@ llvm_execute_inline_plan(llvm::Module *mod, ImportMapTy *globalsToInline)
* the cache state would get corrupted.
*/
static llvm::Module*
-load_module_cached(llvm::StringRef modPath)
+load_module_cached(LLVMContextRef lc, llvm::StringRef modPath)
{
auto it = module_cache->find(modPath);
if (it == module_cache->end())
{
it = module_cache->insert(
- std::make_pair(modPath, load_module(modPath))).first;
+ std::make_pair(modPath, load_module(lc, modPath))).first;
}
return it->second.get();
}
static std::unique_ptr<llvm::Module>
-load_module(llvm::StringRef Identifier)
+load_module(LLVMContextRef lc, llvm::StringRef Identifier)
{
LLVMMemoryBufferRef buf;
LLVMModuleRef mod;
@@ -491,7 +492,7 @@ load_module(llvm::StringRef Identifier)
if (LLVMCreateMemoryBufferWithContentsOfFile(path, &buf, &msg))
elog(FATAL, "failed to open bitcode file \"%s\": %s",
path, msg);
- if (LLVMGetBitcodeModuleInContext2(LLVMGetGlobalContext(), buf, &mod))
+ if (LLVMGetBitcodeModuleInContext2(lc, buf, &mod))
elog(FATAL, "failed to parse bitcode in file \"%s\"", path);
/*
diff --git a/src/include/jit/llvmjit_emit.h b/src/include/jit/llvmjit_emit.h
index 0745dcac9c..5e74543be4 100644
--- a/src/include/jit/llvmjit_emit.h
+++ b/src/include/jit/llvmjit_emit.h
@@ -44,36 +44,36 @@ l_ptr(LLVMTypeRef t)
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int8_const(int8 i)
+l_int8_const(LLVMContextRef lc, int8 i)
{
- return LLVMConstInt(LLVMInt8Type(), i, false);
+ return LLVMConstInt(LLVMInt8TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int16_const(int16 i)
+l_int16_const(LLVMContextRef lc, int16 i)
{
- return LLVMConstInt(LLVMInt16Type(), i, false);
+ return LLVMConstInt(LLVMInt16TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int32_const(int32 i)
+l_int32_const(LLVMContextRef lc, int32 i)
{
- return LLVMConstInt(LLVMInt32Type(), i, false);
+ return LLVMConstInt(LLVMInt32TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int64_const(int64 i)
+l_int64_const(LLVMContextRef lc, int64 i)
{
- return LLVMConstInt(LLVMInt64Type(), i, false);
+ return LLVMConstInt(LLVMInt64TypeInContext(lc), i, false);
}
/*
@@ -137,12 +137,15 @@ l_bb_before_v(LLVMBasicBlockRef r, const char *fmt,...)
{
char buf[512];
va_list args;
+ LLVMContextRef lc;
va_start(args, fmt);
vsnprintf(buf, sizeof(buf), fmt, args);
va_end(args);
- return LLVMInsertBasicBlock(r, buf);
+ lc = LLVMGetTypeContext(LLVMTypeOf(LLVMGetBasicBlockParent(r)));
+
+ return LLVMInsertBasicBlockInContext(lc, r, buf);
}
/* separate, because pg_attribute_printf(2, 3) can't appear in definition */
@@ -157,12 +160,15 @@ l_bb_append_v(LLVMValueRef f, const char *fmt,...)
{
char buf[512];
va_list args;
+ LLVMContextRef lc;
va_start(args, fmt);
vsnprintf(buf, sizeof(buf), fmt, args);
va_end(args);
- return LLVMAppendBasicBlock(f, buf);
+ lc = LLVMGetTypeContext(LLVMTypeOf(f));
+
+ return LLVMAppendBasicBlockInContext(lc, f, buf);
}
/*
@@ -174,7 +180,7 @@ l_callsite_ro(LLVMValueRef f)
const char argname[] = "readonly";
LLVMAttributeRef ref;
- ref = LLVMCreateStringAttribute(LLVMGetGlobalContext(),
+ ref = LLVMCreateStringAttribute(LLVMGetTypeContext(LLVMTypeOf(f)),
argname,
sizeof(argname) - 1,
NULL, 0);
@@ -194,7 +200,7 @@ l_callsite_alwaysinline(LLVMValueRef f)
id = LLVMGetEnumAttributeKindForName(argname,
sizeof(argname) - 1);
- attr = LLVMCreateEnumAttribute(LLVMGetGlobalContext(), id, 0);
+ attr = LLVMCreateEnumAttribute(LLVMGetTypeContext(LLVMTypeOf(f)), id, 0);
LLVMAddCallSiteAttribute(f, LLVMAttributeFunctionIndex, attr);
}
--
2.32.1 (Apple Git-133)
=
* Re: Memory leak on subquery as scalar operand
@ 2023-09-13 13:24 Daniel Gustafsson <daniel@yesql.se>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 18+ messages in thread
From: Daniel Gustafsson @ 2023-09-13 13:24 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee, pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Attached is a v6 rebase of this patchset which curbs a memory leak in llvmjit
by explicitly using an LLVMContextRef for types which is dropped and recreated
at intervals to free unused types. The attached graph plots the memory usage
of a backend continuously running the query from the OP in this thread. The
patched version (running with all JIT costs at zero to get all inlining etc)
goes to a levelled off memory usage where master just continues to grow until
terminated. There is more to do on llvmjit memory usage, but this is clearly a
win for queries which otherwise accumulte type leaks potentially ending with an
OOM.
0001 and 0002 are tangentially related, but are mainly of cleanup character.
0003 contains the LLVMContextRef work which is the meat of the patchset.
I think it would be good to get this in early in the v17 cycle such that we
have time to revisit the herustic if need be.
Thoughts?
--
Daniel Gustafsson
Attachments: download all (4 files, .tar.gz)
v6-0003-llvmjit-Use-explicit-LLVMContextRef-for-inlining.patch (application/octet-stream, 32.4K) download
From 772c2016f63c1c2c1c84e89adc8ae504682d5ddc Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Mon, 11 Sep 2023 10:16:42 +0200
Subject: [PATCH v6 3/3] llvmjit: Use explicit LLVMContextRef for inlining
When performing inlining LLVM unfortunately "leaks" types (the
types survive and are usable, but a new round of inlining will
recreate new structurally equivalent types). This accumulation
will over time amount to a memory leak which for some queries
can be large enough to trigger the OOM process killer.
To avoid accumulation of types, all IR related data is stored
in an LLVMContextRef which is dropped and recreated in order
to release all types. Dropping and recreating incurs overhead,
so it will be done only after 100 queries. This is a heuristic
which might be revisited, but until we can get the size of the
context from LLVM we are flying a bit blind.
This issue has been reported several times, there may be more
references to it in the archives on top of the threads linked
below.
Reported-By: Justin Pryzby <pryzby@telsasoft.com>
Reported-By: Kurt Roeckx <kurt@roeckx.be>
Reported-By: Jaime Casanova <jcasanov@systemguards.com.ec>
Reported-By: Lauri Laanmets <pcspets@gmail.com>
Author: Andres Freund and Daniel Gustafsson
Discussion: https://postgr.es/m/7acc8678-df5f-4923-9cf6-e843131ae89d@www.fastmail.com
Discussion: https://postgr.es/m/20201218235607.GC30237@telsasoft.com
Discussion: https://postgr.es/m/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA@mail.gmail.com
---
src/backend/jit/llvm/llvmjit.c | 139 +++++++++++++++++++++---
src/backend/jit/llvm/llvmjit_deform.c | 60 +++++-----
src/backend/jit/llvm/llvmjit_expr.c | 58 +++++-----
src/backend/jit/llvm/llvmjit_inline.cpp | 33 ++++--
src/include/jit/llvmjit.h | 8 ++
src/include/jit/llvmjit_emit.h | 30 +++--
6 files changed, 234 insertions(+), 94 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index caf261a559..4dfaf79743 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -42,6 +42,8 @@
#include "utils/memutils.h"
#include "utils/resowner_private.h"
+#define LLVMJIT_LLVM_CONTEXT_REUSE_MAX 100
+
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
{
@@ -81,8 +83,15 @@ static LLVMModuleRef llvm_types_module = NULL;
static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
+
+/* number of LLVMJitContexts that currently are in use */
+static size_t llvm_jit_context_in_use_count = 0;
+
+/* how many times has the current LLVMContextRef been used */
+static size_t llvm_llvm_context_reuse_count = 0;
static const char *llvm_triple = NULL;
static const char *llvm_layout = NULL;
+static LLVMContextRef llvm_context;
static LLVMTargetRef llvm_targetref;
@@ -103,6 +112,8 @@ static void llvm_compile_module(LLVMJitContext *context);
static void llvm_optimize_module(LLVMJitContext *context, LLVMModuleRef module);
static void llvm_create_types(void);
+static void llvm_set_target(void);
+static void llvm_recreate_llvm_context(void);
static uint64_t llvm_resolve_symbol(const char *name, void *ctx);
#if LLVM_VERSION_MAJOR > 11
@@ -124,6 +135,63 @@ _PG_jit_provider_init(JitProviderCallbacks *cb)
cb->compile_expr = llvm_compile_expr;
}
+
+/*
+ * Every now and then create a new LLVMContextRef. Unfortunately, during every
+ * round of inlining, types may "leak" (they can still be found/used via the
+ * context, but new types will be created the next time in inlining is
+ * performed). To prevent that from slowly accumulating problematic amounts of
+ * memory, recreate the LLVMContextRef we use. We don't want to do so too
+ * often, as that implies some overhead (particularly re-loading the module
+ * summaries / modules is fairly expensive). A future TODO would be to make
+ * this more finegrained and only drop/recreate the LLVMContextRef when we know
+ * there has been inlining. If we can get the size of the context from LLVM
+ * then that might be a better way to determine when to drop/recreate rather
+ * then the usagecount heuristic currently employed.
+ */
+static void
+llvm_recreate_llvm_context(void)
+{
+ if (!llvm_context)
+ elog(ERROR, "Trying to recreate a non-existing context");
+
+ /*
+ * We can only safely recreate the LLVM context if no other code is being
+ * JITed, otherwise we'd release the types in use for that.
+ */
+ if (llvm_jit_context_in_use_count > 0)
+ {
+ llvm_llvm_context_reuse_count++;
+ return;
+ }
+
+ if (llvm_llvm_context_reuse_count <= LLVMJIT_LLVM_CONTEXT_REUSE_MAX)
+ {
+ llvm_llvm_context_reuse_count++;
+ return;
+ }
+
+ /*
+ * Need to reset the modules that the inlining code caches before
+ * disposing of the context. LLVM modules exist within a specific LLVM
+ * context, therefore disposing of the context before resetting the cache
+ * would lead to dangling pointers to modules.
+ */
+ llvm_inline_reset_caches();
+
+ LLVMContextDispose(llvm_context);
+ llvm_context = LLVMContextCreate();
+ llvm_llvm_context_reuse_count = 0;
+
+ /*
+ * Re-build cached type information, so code generation code can rely on
+ * that information to be present (also prevents the variables to be
+ * dangling references).
+ */
+ llvm_create_types();
+}
+
+
/*
* Create a context for JITing work.
*
@@ -140,6 +208,8 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
+ llvm_recreate_llvm_context();
+
ResourceOwnerEnlargeJIT(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
@@ -150,6 +220,8 @@ llvm_create_context(int jitFlags)
context->base.resowner = CurrentResourceOwner;
ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ llvm_jit_context_in_use_count++;
+
return context;
}
@@ -159,9 +231,15 @@ llvm_create_context(int jitFlags)
static void
llvm_release_context(JitContext *context)
{
- LLVMJitContext *llvm_context = (LLVMJitContext *) context;
+ LLVMJitContext *llvm_jit_context = (LLVMJitContext *) context;
ListCell *lc;
+ /*
+ * Consider as cleaned up even if we skip doing so below, that way we can
+ * verify the tracking is correct (see llvm_shutdown()).
+ */
+ llvm_jit_context_in_use_count--;
+
/*
* When this backend is exiting, don't clean up LLVM. As an error might
* have occurred from within LLVM, we do not want to risk reentering. All
@@ -172,13 +250,13 @@ llvm_release_context(JitContext *context)
llvm_enter_fatal_on_oom();
- if (llvm_context->module)
+ if (llvm_jit_context->module)
{
- LLVMDisposeModule(llvm_context->module);
- llvm_context->module = NULL;
+ LLVMDisposeModule(llvm_jit_context->module);
+ llvm_jit_context->module = NULL;
}
- foreach(lc, llvm_context->handles)
+ foreach(lc, llvm_jit_context->handles)
{
LLVMJitHandle *jit_handle = (LLVMJitHandle *) lfirst(lc);
@@ -208,8 +286,8 @@ llvm_release_context(JitContext *context)
pfree(jit_handle);
}
- list_free(llvm_context->handles);
- llvm_context->handles = NIL;
+ list_free(llvm_jit_context->handles);
+ llvm_jit_context->handles = NIL;
llvm_leave_fatal_on_oom();
}
@@ -229,7 +307,7 @@ llvm_mutable_module(LLVMJitContext *context)
{
context->compiled = false;
context->module_generation = llvm_generation++;
- context->module = LLVMModuleCreateWithName("pg");
+ context->module = LLVMModuleCreateWithNameInContext("pg", llvm_context);
LLVMSetTarget(context->module, llvm_triple);
LLVMSetDataLayout(context->module, llvm_layout);
}
@@ -787,6 +865,14 @@ llvm_session_initialize(void)
LLVMInitializeNativeAsmPrinter();
LLVMInitializeNativeAsmParser();
+ if (llvm_context == NULL)
+ {
+ llvm_context = LLVMContextCreate();
+
+ llvm_jit_context_in_use_count = 0;
+ llvm_llvm_context_reuse_count = 0;
+ }
+
/*
* When targeting an LLVM version with opaque pointers enabled by default,
* turn them off for the context we build our code in. We don't need to
@@ -803,6 +889,11 @@ llvm_session_initialize(void)
*/
llvm_create_types();
+ /*
+ * Extract target information from loaded module.
+ */
+ llvm_set_target();
+
if (LLVMGetTargetFromTriple(llvm_triple, &llvm_targetref, &error) != 0)
{
elog(FATAL, "failed to query triple %s", error);
@@ -898,6 +989,10 @@ llvm_shutdown(int code, Datum arg)
return;
}
+ if (llvm_jit_context_in_use_count != 0)
+ elog(PANIC, "LLVMJitContext in use count not 0 at exit (is %zu)",
+ llvm_jit_context_in_use_count);
+
#if LLVM_VERSION_MAJOR > 11
{
if (llvm_opt3_orc)
@@ -968,6 +1063,23 @@ load_return_type(LLVMModuleRef mod, const char *name)
return typ;
}
+/*
+ * Load triple & layout from clang emitted file so we're guaranteed to be
+ * compatible.
+ */
+static void
+llvm_set_target(void)
+{
+ if (!llvm_types_module)
+ elog(ERROR, "failed to extract target information, llvmjit_types.c not loaded");
+
+ if (llvm_triple == NULL)
+ llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
+
+ if (llvm_layout == NULL)
+ llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+}
+
/*
* Load required information, types, function signatures from llvmjit_types.c
* and make them available in global variables.
@@ -991,19 +1103,12 @@ llvm_create_types(void)
}
/* eagerly load contents, going to need it all */
- if (LLVMParseBitcode2(buf, &llvm_types_module))
+ if (LLVMParseBitcodeInContext2(llvm_context, buf, &llvm_types_module))
{
- elog(ERROR, "LLVMParseBitcode2 of %s failed", path);
+ elog(ERROR, "LLVMParseBitcodeInContext2 of %s failed", path);
}
LLVMDisposeMemoryBuffer(buf);
- /*
- * Load triple & layout from clang emitted file so we're guaranteed to be
- * compatible.
- */
- llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
- llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
-
TypeSizeT = llvm_pg_var_type("TypeSizeT");
TypeParamBool = load_return_type(llvm_types_module, "FunctionReturningBool");
TypeStorageBool = llvm_pg_var_type("TypeStorageBool");
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 15d4a7b431..a729143c47 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -37,6 +37,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
char *funcname;
LLVMModuleRef mod;
+ LLVMContextRef lc;
LLVMBuilderRef b;
LLVMTypeRef deform_sig;
@@ -99,6 +100,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
return NULL;
mod = llvm_mutable_module(context);
+ lc = LLVMGetModuleContext(mod);
funcname = llvm_expand_funcname(context, "deform");
@@ -133,8 +135,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
param_types[0] = l_ptr(StructTupleTableSlot);
- deform_sig = LLVMFunctionType(LLVMVoidType(), param_types,
- lengthof(param_types), 0);
+ deform_sig = LLVMFunctionType(LLVMVoidTypeInContext(lc),
+ param_types, lengthof(param_types), 0);
}
v_deform_fn = LLVMAddFunction(mod, funcname, deform_sig);
LLVMSetLinkage(v_deform_fn, LLVMInternalLinkage);
@@ -142,17 +144,17 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
llvm_copy_attributes(AttributeTemplate, v_deform_fn);
b_entry =
- LLVMAppendBasicBlock(v_deform_fn, "entry");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "entry");
b_adjust_unavail_cols =
- LLVMAppendBasicBlock(v_deform_fn, "adjust_unavail_cols");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "adjust_unavail_cols");
b_find_start =
- LLVMAppendBasicBlock(v_deform_fn, "find_startblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "find_startblock");
b_out =
- LLVMAppendBasicBlock(v_deform_fn, "outblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "outblock");
b_dead =
- LLVMAppendBasicBlock(v_deform_fn, "deadblock");
+ LLVMAppendBasicBlockInContext(lc, v_deform_fn, "deadblock");
- b = LLVMCreateBuilder();
+ b = LLVMCreateBuilderInContext(lc);
attcheckattnoblocks = palloc(sizeof(LLVMBasicBlockRef) * natts);
attstartblocks = palloc(sizeof(LLVMBasicBlockRef) * natts);
@@ -221,7 +223,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMBuildStructGEP(b, v_tuplep,
FIELDNO_HEAPTUPLEHEADERDATA_BITS,
""),
- l_ptr(LLVMInt8Type()),
+ l_ptr(LLVMInt8TypeInContext(lc)),
"t_bits");
v_infomask1 =
l_load_struct_gep(b, v_tuplep,
@@ -236,14 +238,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
v_hasnulls =
LLVMBuildICmp(b, LLVMIntNE,
LLVMBuildAnd(b,
- l_int16_const(HEAP_HASNULL),
+ l_int16_const(lc, HEAP_HASNULL),
v_infomask1, ""),
- l_int16_const(0),
+ l_int16_const(lc, 0),
"hasnulls");
/* t_infomask2 & HEAP_NATTS_MASK */
v_maxatt = LLVMBuildAnd(b,
- l_int16_const(HEAP_NATTS_MASK),
+ l_int16_const(lc, HEAP_NATTS_MASK),
v_infomask2,
"maxatt");
@@ -256,13 +258,13 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
l_load_struct_gep(b, v_tuplep,
FIELDNO_HEAPTUPLEHEADERDATA_HOFF,
""),
- LLVMInt32Type(), "t_hoff");
+ LLVMInt32TypeInContext(lc), "t_hoff");
v_tupdata_base =
LLVMBuildGEP(b,
LLVMBuildBitCast(b,
v_tuplep,
- l_ptr(LLVMInt8Type()),
+ l_ptr(LLVMInt8TypeInContext(lc)),
""),
&v_hoff, 1,
"v_tupdata_base");
@@ -319,7 +321,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMBuildCondBr(b,
LLVMBuildICmp(b, LLVMIntULT,
v_maxatt,
- l_int16_const(natts),
+ l_int16_const(lc, natts),
""),
b_adjust_unavail_cols,
b_find_start);
@@ -328,8 +330,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMPositionBuilderAtEnd(b, b_adjust_unavail_cols);
v_params[0] = v_slot;
- v_params[1] = LLVMBuildZExt(b, v_maxatt, LLVMInt32Type(), "");
- v_params[2] = l_int32_const(natts);
+ v_params[1] = LLVMBuildZExt(b, v_maxatt, LLVMInt32TypeInContext(lc), "");
+ v_params[2] = l_int32_const(lc, natts);
LLVMBuildCall(b, llvm_pg_func(mod, "slot_getmissingattrs"),
v_params, lengthof(v_params), "");
LLVMBuildBr(b, b_find_start);
@@ -352,7 +354,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
- LLVMValueRef v_attno = l_int16_const(attnum);
+ LLVMValueRef v_attno = l_int16_const(lc, attnum);
LLVMAddCase(v_switch, v_attno, attcheckattnoblocks[attnum]);
}
@@ -375,7 +377,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
Form_pg_attribute att = TupleDescAttr(desc, attnum);
LLVMValueRef v_incby;
int alignto;
- LLVMValueRef l_attno = l_int16_const(attnum);
+ LLVMValueRef l_attno = l_int16_const(lc, attnum);
LLVMValueRef v_attdatap;
LLVMValueRef v_resultp;
@@ -436,14 +438,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
else
b_next = attcheckattnoblocks[attnum + 1];
- v_nullbyteno = l_int32_const(attnum >> 3);
- v_nullbytemask = l_int8_const(1 << ((attnum) & 0x07));
+ v_nullbyteno = l_int32_const(lc, attnum >> 3);
+ v_nullbytemask = l_int8_const(lc, 1 << ((attnum) & 0x07));
v_nullbyte = l_load_gep1(b, v_bits, v_nullbyteno, "attnullbyte");
v_nullbit = LLVMBuildICmp(b,
LLVMIntEQ,
LLVMBuildAnd(b, v_nullbyte, v_nullbytemask, ""),
- l_int8_const(0),
+ l_int8_const(lc, 0),
"attisnull");
v_attisnull = LLVMBuildAnd(b, v_hasnulls, v_nullbit, "");
@@ -454,7 +456,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
/* store null-byte */
LLVMBuildStore(b,
- l_int8_const(1),
+ l_int8_const(lc, 1),
LLVMBuildGEP(b, v_tts_nulls, &l_attno, 1, ""));
/* store zero datum */
LLVMBuildStore(b,
@@ -524,7 +526,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
l_load_gep1(b, v_tupdata_base, v_off, "padbyte");
v_ispad =
LLVMBuildICmp(b, LLVMIntEQ,
- v_possible_padbyte, l_int8_const(0),
+ v_possible_padbyte, l_int8_const(lc, 0),
"ispadbyte");
LLVMBuildCondBr(b, v_ispad,
attalignblocks[attnum],
@@ -639,7 +641,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
v_resultp = LLVMBuildGEP(b, v_tts_values, &l_attno, 1, "");
/* store null-byte (false) */
- LLVMBuildStore(b, l_int8_const(0),
+ LLVMBuildStore(b, l_int8_const(lc, 0),
LLVMBuildGEP(b, v_tts_nulls, &l_attno, 1, ""));
/*
@@ -650,7 +652,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
{
LLVMValueRef v_tmp_loaddata;
LLVMTypeRef vartypep =
- LLVMPointerType(LLVMIntType(att->attlen * 8), 0);
+ LLVMPointerType(LLVMIntTypeInContext(lc, att->attlen * 8), 0);
v_tmp_loaddata =
LLVMBuildPointerCast(b, v_attdatap, vartypep, "");
@@ -739,11 +741,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
LLVMValueRef v_off = LLVMBuildLoad(b, v_offp, "");
LLVMValueRef v_flags;
- LLVMBuildStore(b, l_int16_const(natts), v_nvalidp);
- v_off = LLVMBuildTrunc(b, v_off, LLVMInt32Type(), "");
+ LLVMBuildStore(b, l_int16_const(lc, natts), v_nvalidp);
+ v_off = LLVMBuildTrunc(b, v_off, LLVMInt32TypeInContext(lc), "");
LLVMBuildStore(b, v_off, v_slotoffp);
v_flags = LLVMBuildLoad(b, v_flagsp, "tts_flags");
- v_flags = LLVMBuildOr(b, v_flags, l_int16_const(TTS_FLAG_SLOW), "");
+ v_flags = LLVMBuildOr(b, v_flags, l_int16_const(lc, TTS_FLAG_SLOW), "");
LLVMBuildStore(b, v_flags, v_flagsp);
LLVMBuildRetVoid(b);
}
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
index 2ac335e238..4b51aa1ce0 100644
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -84,6 +84,7 @@ llvm_compile_expr(ExprState *state)
LLVMBuilderRef b;
LLVMModuleRef mod;
+ LLVMContextRef lc;
LLVMValueRef eval_fn;
LLVMBasicBlockRef entry;
LLVMBasicBlockRef *opblocks;
@@ -145,8 +146,9 @@ llvm_compile_expr(ExprState *state)
INSTR_TIME_SET_CURRENT(starttime);
mod = llvm_mutable_module(context);
+ lc = LLVMGetModuleContext(mod);
- b = LLVMCreateBuilder();
+ b = LLVMCreateBuilderInContext(lc);
funcname = llvm_expand_funcname(context, "evalexpr");
@@ -157,7 +159,7 @@ llvm_compile_expr(ExprState *state)
LLVMSetVisibility(eval_fn, LLVMDefaultVisibility);
llvm_copy_attributes(AttributeTemplate, eval_fn);
- entry = LLVMAppendBasicBlock(eval_fn, "entry");
+ entry = LLVMAppendBasicBlockInContext(lc, eval_fn, "entry");
/* build state */
v_state = LLVMGetParam(eval_fn, 0);
@@ -303,7 +305,7 @@ llvm_compile_expr(ExprState *state)
"");
LLVMBuildCondBr(b,
LLVMBuildICmp(b, LLVMIntUGE, v_nvalid,
- l_int16_const(op->d.fetch.last_var),
+ l_int16_const(lc, op->d.fetch.last_var),
""),
opblocks[opno + 1], b_fetch);
@@ -341,7 +343,7 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef params[2];
params[0] = v_slot;
- params[1] = l_int32_const(op->d.fetch.last_var);
+ params[1] = l_int32_const(lc, op->d.fetch.last_var);
LLVMBuildCall(b,
llvm_pg_func(mod, "slot_getsomeattrs_int"),
@@ -378,7 +380,7 @@ llvm_compile_expr(ExprState *state)
v_nulls = v_scannulls;
}
- v_attnum = l_int32_const(op->d.var.attnum);
+ v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, v_values, v_attnum, "");
isnull = l_load_gep1(b, v_nulls, v_attnum, "");
LLVMBuildStore(b, value, v_resvaluep);
@@ -444,12 +446,12 @@ llvm_compile_expr(ExprState *state)
}
/* load data */
- v_attnum = l_int32_const(op->d.assign_var.attnum);
+ v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
v_value = l_load_gep1(b, v_values, v_attnum, "");
v_isnull = l_load_gep1(b, v_nulls, v_attnum, "");
/* compute addresses of targets */
- v_resultnum = l_int32_const(op->d.assign_var.resultnum);
+ v_resultnum = l_int32_const(lc, op->d.assign_var.resultnum);
v_rvaluep = LLVMBuildGEP(b, v_resultvalues,
&v_resultnum, 1, "");
v_risnullp = LLVMBuildGEP(b, v_resultnulls,
@@ -478,7 +480,7 @@ llvm_compile_expr(ExprState *state)
v_isnull = LLVMBuildLoad(b, v_tmpisnullp, "");
/* compute addresses of targets */
- v_resultnum = l_int32_const(resultnum);
+ v_resultnum = l_int32_const(lc, resultnum);
v_rvaluep =
LLVMBuildGEP(b, v_resultvalues, &v_resultnum, 1, "");
v_risnullp =
@@ -1703,7 +1705,7 @@ llvm_compile_expr(ExprState *state)
v_cmpresult =
LLVMBuildTrunc(b,
LLVMBuildLoad(b, v_resvaluep, ""),
- LLVMInt32Type(), "");
+ LLVMInt32TypeInContext(lc), "");
switch (rctype)
{
@@ -1729,7 +1731,7 @@ llvm_compile_expr(ExprState *state)
v_result = LLVMBuildICmp(b,
predicate,
v_cmpresult,
- l_int32_const(0),
+ l_int32_const(lc, 0),
"");
v_result = LLVMBuildZExt(b, v_result, TypeSizeT, "");
@@ -1872,7 +1874,7 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef value,
isnull;
- v_aggno = l_int32_const(op->d.aggref.aggno);
+ v_aggno = l_int32_const(lc, op->d.aggref.aggno);
/* load agg value / null */
value = l_load_gep1(b, v_aggvalues, v_aggno, "aggvalue");
@@ -1906,7 +1908,7 @@ llvm_compile_expr(ExprState *state)
* expression). So load it from memory each time round.
*/
v_wfuncnop = l_ptr_const(&wfunc->wfuncno,
- l_ptr(LLVMInt32Type()));
+ l_ptr(LLVMInt32TypeInContext(lc)));
v_wfuncno = LLVMBuildLoad(b, v_wfuncnop, "v_wfuncno");
/* load window func value / null */
@@ -2014,7 +2016,7 @@ llvm_compile_expr(ExprState *state)
/* strict function, check for NULL args */
for (int argno = 0; argno < nargs; argno++)
{
- LLVMValueRef v_argno = l_int32_const(argno);
+ LLVMValueRef v_argno = l_int32_const(lc, argno);
LLVMValueRef v_argisnull;
LLVMBasicBlockRef b_argnotnull;
@@ -2071,7 +2073,7 @@ llvm_compile_expr(ExprState *state)
FIELDNO_AGGSTATE_ALL_PERGROUPS,
"aggstate.all_pergroups");
- v_setoff = l_int32_const(op->d.agg_plain_pergroup_nullcheck.setoff);
+ v_setoff = l_int32_const(lc, op->d.agg_plain_pergroup_nullcheck.setoff);
v_pergroup_allaggs = l_load_gep1(b, v_allpergroupsp, v_setoff, "");
@@ -2139,8 +2141,8 @@ llvm_compile_expr(ExprState *state)
l_load_struct_gep(b, v_aggstatep,
FIELDNO_AGGSTATE_ALL_PERGROUPS,
"aggstate.all_pergroups");
- v_setoff = l_int32_const(op->d.agg_trans.setoff);
- v_transno = l_int32_const(op->d.agg_trans.transno);
+ v_setoff = l_int32_const(lc, op->d.agg_trans.setoff);
+ v_transno = l_int32_const(lc, op->d.agg_trans.transno);
v_pergroupp =
LLVMBuildGEP(b,
l_load_gep1(b, v_allpergroupsp, v_setoff, ""),
@@ -2243,7 +2245,7 @@ llvm_compile_expr(ExprState *state)
/* set aggstate globals */
LLVMBuildStore(b, v_aggcontext, v_curaggcontext);
- LLVMBuildStore(b, l_int32_const(op->d.agg_trans.setno),
+ LLVMBuildStore(b, l_int32_const(lc, op->d.agg_trans.setno),
v_current_setp);
LLVMBuildStore(b, v_pertransp, v_current_pertransp);
@@ -2479,11 +2481,14 @@ BuildV1Call(LLVMJitContext *context, LLVMBuilderRef b,
LLVMModuleRef mod, FunctionCallInfo fcinfo,
LLVMValueRef *v_fcinfo_isnull)
{
+ LLVMContextRef lc;
LLVMValueRef v_fn;
LLVMValueRef v_fcinfo_isnullp;
LLVMValueRef v_retval;
LLVMValueRef v_fcinfo;
+ lc = LLVMGetModuleContext(mod);
+
v_fn = llvm_function_reference(context, b, mod, fcinfo);
v_fcinfo = l_ptr_const(fcinfo, l_ptr(StructFunctionCallInfoData));
@@ -2505,12 +2510,12 @@ BuildV1Call(LLVMJitContext *context, LLVMBuilderRef b,
LLVMValueRef v_lifetime = create_LifetimeEnd(mod);
LLVMValueRef params[2];
- params[0] = l_int64_const(sizeof(NullableDatum) * fcinfo->nargs);
- params[1] = l_ptr_const(fcinfo->args, l_ptr(LLVMInt8Type()));
+ params[0] = l_int64_const(lc, sizeof(NullableDatum) * fcinfo->nargs);
+ params[1] = l_ptr_const(fcinfo->args, l_ptr(LLVMInt8TypeInContext(lc)));
LLVMBuildCall(b, v_lifetime, params, lengthof(params), "");
- params[0] = l_int64_const(sizeof(fcinfo->isnull));
- params[1] = l_ptr_const(&fcinfo->isnull, l_ptr(LLVMInt8Type()));
+ params[0] = l_int64_const(lc, sizeof(fcinfo->isnull));
+ params[1] = l_ptr_const(&fcinfo->isnull, l_ptr(LLVMInt8TypeInContext(lc)));
LLVMBuildCall(b, v_lifetime, params, lengthof(params), "");
}
@@ -2556,6 +2561,7 @@ create_LifetimeEnd(LLVMModuleRef mod)
LLVMTypeRef sig;
LLVMValueRef fn;
LLVMTypeRef param_types[2];
+ LLVMContextRef lc;
/* LLVM 5+ has a variadic pointer argument */
#if LLVM_VERSION_MAJOR < 5
@@ -2568,12 +2574,12 @@ create_LifetimeEnd(LLVMModuleRef mod)
if (fn)
return fn;
- param_types[0] = LLVMInt64Type();
- param_types[1] = l_ptr(LLVMInt8Type());
+ lc = LLVMGetModuleContext(mod);
+ param_types[0] = LLVMInt64TypeInContext(lc);
+ param_types[1] = l_ptr(LLVMInt8TypeInContext(lc));
- sig = LLVMFunctionType(LLVMVoidType(),
- param_types, lengthof(param_types),
- false);
+ sig = LLVMFunctionType(LLVMVoidTypeInContext(lc), param_types,
+ lengthof(param_types), false);
fn = LLVMAddFunction(mod, nm, sig);
LLVMSetFunctionCallConv(fn, LLVMCCallConv);
diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp b/src/backend/jit/llvm/llvmjit_inline.cpp
index c765add856..d92d7f3c88 100644
--- a/src/backend/jit/llvm/llvmjit_inline.cpp
+++ b/src/backend/jit/llvm/llvmjit_inline.cpp
@@ -114,12 +114,12 @@ typedef llvm::StringMap<std::unique_ptr<llvm::ModuleSummaryIndex> > SummaryCache
llvm::ManagedStatic<SummaryCache> summary_cache;
-static std::unique_ptr<ImportMapTy> llvm_build_inline_plan(llvm::Module *mod);
+static std::unique_ptr<ImportMapTy> llvm_build_inline_plan(LLVMContextRef lc, llvm::Module *mod);
static void llvm_execute_inline_plan(llvm::Module *mod,
ImportMapTy *globalsToInline);
-static llvm::Module* load_module_cached(llvm::StringRef modPath);
-static std::unique_ptr<llvm::Module> load_module(llvm::StringRef Identifier);
+static llvm::Module* load_module_cached(LLVMContextRef c, llvm::StringRef modPath);
+static std::unique_ptr<llvm::Module> load_module(LLVMContextRef c, llvm::StringRef Identifier);
static std::unique_ptr<llvm::ModuleSummaryIndex> llvm_load_summary(llvm::StringRef path);
@@ -152,6 +152,18 @@ summaries_for_guid(const InlineSearchPath& path, llvm::GlobalValue::GUID guid);
#define ilog(...) (void) 0
#endif
+/*
+ * Reset inlining related state. This needs to be called before the currently
+ * used LLVMContextRef is disposed (and a new one create), otherwise we would
+ * have dangling references to deleted modules.
+ */
+void
+llvm_inline_reset_caches(void)
+{
+ module_cache->clear();
+ summary_cache->clear();
+}
+
/*
* Perform inlining of external function references in M based on a simple
* cost based analysis.
@@ -159,9 +171,10 @@ summaries_for_guid(const InlineSearchPath& path, llvm::GlobalValue::GUID guid);
void
llvm_inline(LLVMModuleRef M)
{
+ LLVMContextRef lc = LLVMGetModuleContext(M);
llvm::Module *mod = llvm::unwrap(M);
- std::unique_ptr<ImportMapTy> globalsToInline = llvm_build_inline_plan(mod);
+ std::unique_ptr<ImportMapTy> globalsToInline = llvm_build_inline_plan(lc, mod);
if (!globalsToInline)
return;
llvm_execute_inline_plan(mod, globalsToInline.get());
@@ -172,7 +185,7 @@ llvm_inline(LLVMModuleRef M)
* mod.
*/
static std::unique_ptr<ImportMapTy>
-llvm_build_inline_plan(llvm::Module *mod)
+llvm_build_inline_plan(LLVMContextRef lc, llvm::Module *mod)
{
std::unique_ptr<ImportMapTy> globalsToInline(new ImportMapTy());
FunctionInlineStates functionStates;
@@ -271,7 +284,7 @@ llvm_build_inline_plan(llvm::Module *mod)
continue;
}
- defMod = load_module_cached(modPath);
+ defMod = load_module_cached(lc, modPath);
if (defMod->materializeMetadata())
elog(FATAL, "failed to materialize metadata");
@@ -466,20 +479,20 @@ llvm_execute_inline_plan(llvm::Module *mod, ImportMapTy *globalsToInline)
* the cache state would get corrupted.
*/
static llvm::Module*
-load_module_cached(llvm::StringRef modPath)
+load_module_cached(LLVMContextRef lc, llvm::StringRef modPath)
{
auto it = module_cache->find(modPath);
if (it == module_cache->end())
{
it = module_cache->insert(
- std::make_pair(modPath, load_module(modPath))).first;
+ std::make_pair(modPath, load_module(lc, modPath))).first;
}
return it->second.get();
}
static std::unique_ptr<llvm::Module>
-load_module(llvm::StringRef Identifier)
+load_module(LLVMContextRef lc, llvm::StringRef Identifier)
{
LLVMMemoryBufferRef buf;
LLVMModuleRef mod;
@@ -491,7 +504,7 @@ load_module(llvm::StringRef Identifier)
if (LLVMCreateMemoryBufferWithContentsOfFile(path, &buf, &msg))
elog(FATAL, "failed to open bitcode file \"%s\": %s",
path, msg);
- if (LLVMGetBitcodeModuleInContext2(LLVMGetGlobalContext(), buf, &mod))
+ if (LLVMGetBitcodeModuleInContext2(lc, buf, &mod))
elog(FATAL, "failed to parse bitcode in file \"%s\"", path);
/*
diff --git a/src/include/jit/llvmjit.h b/src/include/jit/llvmjit.h
index 17b3b7593f..6d90a16f79 100644
--- a/src/include/jit/llvmjit.h
+++ b/src/include/jit/llvmjit.h
@@ -42,6 +42,13 @@ typedef struct LLVMJitContext
/* number of modules created */
size_t module_generation;
+ /*
+ * The LLVM Context used by this JIT context. An LLVM context is reused
+ * across many compilations, but occasionally reset to prevent it using
+ * too much memory due to more and more types accumulating.
+ */
+ LLVMContextRef llvm_context;
+
/* current, "open for write", module */
LLVMModuleRef module;
@@ -99,6 +106,7 @@ extern LLVMValueRef llvm_function_reference(LLVMJitContext *context,
LLVMModuleRef mod,
FunctionCallInfo fcinfo);
+extern void llvm_inline_reset_caches(void);
extern void llvm_inline(LLVMModuleRef mod);
/*
diff --git a/src/include/jit/llvmjit_emit.h b/src/include/jit/llvmjit_emit.h
index 0745dcac9c..5e74543be4 100644
--- a/src/include/jit/llvmjit_emit.h
+++ b/src/include/jit/llvmjit_emit.h
@@ -44,36 +44,36 @@ l_ptr(LLVMTypeRef t)
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int8_const(int8 i)
+l_int8_const(LLVMContextRef lc, int8 i)
{
- return LLVMConstInt(LLVMInt8Type(), i, false);
+ return LLVMConstInt(LLVMInt8TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int16_const(int16 i)
+l_int16_const(LLVMContextRef lc, int16 i)
{
- return LLVMConstInt(LLVMInt16Type(), i, false);
+ return LLVMConstInt(LLVMInt16TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int32_const(int32 i)
+l_int32_const(LLVMContextRef lc, int32 i)
{
- return LLVMConstInt(LLVMInt32Type(), i, false);
+ return LLVMConstInt(LLVMInt32TypeInContext(lc), i, false);
}
/*
* Emit constant integer.
*/
static inline LLVMValueRef
-l_int64_const(int64 i)
+l_int64_const(LLVMContextRef lc, int64 i)
{
- return LLVMConstInt(LLVMInt64Type(), i, false);
+ return LLVMConstInt(LLVMInt64TypeInContext(lc), i, false);
}
/*
@@ -137,12 +137,15 @@ l_bb_before_v(LLVMBasicBlockRef r, const char *fmt,...)
{
char buf[512];
va_list args;
+ LLVMContextRef lc;
va_start(args, fmt);
vsnprintf(buf, sizeof(buf), fmt, args);
va_end(args);
- return LLVMInsertBasicBlock(r, buf);
+ lc = LLVMGetTypeContext(LLVMTypeOf(LLVMGetBasicBlockParent(r)));
+
+ return LLVMInsertBasicBlockInContext(lc, r, buf);
}
/* separate, because pg_attribute_printf(2, 3) can't appear in definition */
@@ -157,12 +160,15 @@ l_bb_append_v(LLVMValueRef f, const char *fmt,...)
{
char buf[512];
va_list args;
+ LLVMContextRef lc;
va_start(args, fmt);
vsnprintf(buf, sizeof(buf), fmt, args);
va_end(args);
- return LLVMAppendBasicBlock(f, buf);
+ lc = LLVMGetTypeContext(LLVMTypeOf(f));
+
+ return LLVMAppendBasicBlockInContext(lc, f, buf);
}
/*
@@ -174,7 +180,7 @@ l_callsite_ro(LLVMValueRef f)
const char argname[] = "readonly";
LLVMAttributeRef ref;
- ref = LLVMCreateStringAttribute(LLVMGetGlobalContext(),
+ ref = LLVMCreateStringAttribute(LLVMGetTypeContext(LLVMTypeOf(f)),
argname,
sizeof(argname) - 1,
NULL, 0);
@@ -194,7 +200,7 @@ l_callsite_alwaysinline(LLVMValueRef f)
id = LLVMGetEnumAttributeKindForName(argname,
sizeof(argname) - 1);
- attr = LLVMCreateEnumAttribute(LLVMGetGlobalContext(), id, 0);
+ attr = LLVMCreateEnumAttribute(LLVMGetTypeContext(LLVMTypeOf(f)), id, 0);
LLVMAddCallSiteAttribute(f, LLVMAttributeFunctionIndex, attr);
}
--
2.32.1 (Apple Git-133)
=
v6-0002-llvmjit-Make-llvm_types_module-variable-static.patch (application/octet-stream, 1.6K) download
From 15f4e97bb1ffd712024b3e902b8a99802898f05e Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:44:20 +0200
Subject: [PATCH v6 2/3] llvmjit: Make llvm_types_module variable static
Commit b059d2f45685a introduced llvm_types_module and accidentally
exported it. As there is no usecase for accessing this variable
externally, this makes it static.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/20221101055132.pjjsvlkeo4stbjkq@awork3.anarazel.de
---
src/backend/jit/llvm/llvmjit.c | 2 +-
src/include/jit/llvmjit.h | 3 ---
2 files changed, 1 insertion(+), 4 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index fc0ccea329..caf261a559 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -77,7 +77,7 @@ LLVMTypeRef StructAggStatePerTransData;
LLVMValueRef AttributeTemplate;
-LLVMModuleRef llvm_types_module = NULL;
+static LLVMModuleRef llvm_types_module = NULL;
static bool llvm_session_initialized = false;
static size_t llvm_generation = 0;
diff --git a/src/include/jit/llvmjit.h b/src/include/jit/llvmjit.h
index 551b585464..17b3b7593f 100644
--- a/src/include/jit/llvmjit.h
+++ b/src/include/jit/llvmjit.h
@@ -55,9 +55,6 @@ typedef struct LLVMJitContext
List *handles;
} LLVMJitContext;
-/* llvm module containing information about types */
-extern PGDLLIMPORT LLVMModuleRef llvm_types_module;
-
/* type and struct definitions */
extern PGDLLIMPORT LLVMTypeRef TypeParamBool;
extern PGDLLIMPORT LLVMTypeRef TypePGFunction;
--
2.32.1 (Apple Git-133)
=
v6-0001-llvmjit-Remove-unnecessary-types.patch (application/octet-stream, 1.6K) download
From 1b79cda6152e3cf61b576d0b1e82f8bb44adc5e9 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 25 Apr 2023 14:43:37 +0200
Subject: [PATCH v6 1/3] llvmjit: Remove unnecessary types
These types were added in fb46ac26fe but hasn't been used, so
remove until there is a need for them.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/20221101055132.pjjsvlkeo4stbjkq@awork3.anarazel.de
---
src/backend/jit/llvm/llvmjit.c | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index 09650e2c70..fc0ccea329 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -61,23 +61,12 @@ LLVMTypeRef TypeParamBool;
LLVMTypeRef TypeStorageBool;
LLVMTypeRef TypePGFunction;
LLVMTypeRef StructNullableDatum;
-LLVMTypeRef StructHeapTupleFieldsField3;
-LLVMTypeRef StructHeapTupleFields;
-LLVMTypeRef StructHeapTupleHeaderData;
-LLVMTypeRef StructHeapTupleDataChoice;
LLVMTypeRef StructHeapTupleData;
-LLVMTypeRef StructMinimalTupleData;
-LLVMTypeRef StructItemPointerData;
-LLVMTypeRef StructBlockId;
-LLVMTypeRef StructFormPgAttribute;
-LLVMTypeRef StructTupleConstr;
LLVMTypeRef StructTupleDescData;
LLVMTypeRef StructTupleTableSlot;
LLVMTypeRef StructHeapTupleTableSlot;
LLVMTypeRef StructMinimalTupleTableSlot;
LLVMTypeRef StructMemoryContextData;
-LLVMTypeRef StructPGFinfoRecord;
-LLVMTypeRef StructFmgrInfo;
LLVMTypeRef StructFunctionCallInfoData;
LLVMTypeRef StructExprContext;
LLVMTypeRef StructExprEvalStep;
--
2.32.1 (Apple Git-133)
=
* Re: Memory leak on subquery as scalar operand
@ 2023-09-27 12:07 Daniel Gustafsson <daniel@yesql.se>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 2 replies; 18+ messages in thread
From: Daniel Gustafsson @ 2023-09-27 12:07 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee, pgsql-bugs <pgsql-bugs@lists.postgresql.org>
> On 13 Sep 2023, at 15:24, Daniel Gustafsson <daniel@yesql.se> wrote:
> I think it would be good to get this in early in the v17 cycle such that we
> have time to revisit the herustic if need be.
With an offlist +1 from Andres, and another couple of rounds of read-throughs
and jit_above_cost=0 check-worlds I went ahead and applied this to master. The
buildfarm has green builds for LLVM enabled builds on Linux, FreeBSD and macOS.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2023-09-27 20:45 Andres Freund <andres@anarazel.de>
parent: Daniel Gustafsson <daniel@yesql.se>
1 sibling, 0 replies; 18+ messages in thread
From: Andres Freund @ 2023-09-27 20:45 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee, pgsql-bugs <pgsql-bugs@lists.postgresql.org>
On 2023-09-27 14:07:16 +0200, Daniel Gustafsson wrote:
> > On 13 Sep 2023, at 15:24, Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > I think it would be good to get this in early in the v17 cycle such that we
> > have time to revisit the herustic if need be.
>
> With an offlist +1 from Andres, and another couple of rounds of read-throughs
> and jit_above_cost=0 check-worlds I went ahead and applied this to master. The
> buildfarm has green builds for LLVM enabled builds on Linux, FreeBSD and macOS.
Thanks a lot or working on this!
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Memory leak on subquery as scalar operand
@ 2023-11-17 15:31 Daniel Gustafsson <daniel@yesql.se>
parent: Daniel Gustafsson <daniel@yesql.se>
1 sibling, 0 replies; 18+ messages in thread
From: Daniel Gustafsson @ 2023-11-17 15:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Rowley <dgrowleyml@gmail.com>; lauri.laanmets@eesti.ee, pgsql-bugs <pgsql-bugs@lists.postgresql.org>
> On 27 Sep 2023, at 14:07, Daniel Gustafsson <daniel@yesql.se> wrote:
>
>> On 13 Sep 2023, at 15:24, Daniel Gustafsson <daniel@yesql.se> wrote:
>
>> I think it would be good to get this in early in the v17 cycle such that we
>> have time to revisit the herustic if need be.
>
> With an offlist +1 from Andres, and another couple of rounds of read-throughs
> and jit_above_cost=0 check-worlds I went ahead and applied this to master. The
> buildfarm has green builds for LLVM enabled builds on Linux, FreeBSD and macOS.
This, as well as the prerequisite backpatch of bab150045bd9 to 12 and 13 has
now been backpatched to all supported branches. By now there has been green
builds by LLVM enabled animals in the buildfarm for all branches.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 18+ messages in thread
end of thread, other threads:[~2023-11-17 15:31 UTC | newest]
Thread overview: 18+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-10-30 17:13 Memory leak on subquery as scalar operand Lauri Laanmets <pcspets@gmail.com>
2022-10-31 13:40 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-10-31 14:16 ` Lauri Laanmets <pcspets@gmail.com>
2022-10-31 23:20 ` David Rowley <dgrowleyml@gmail.com>
2022-11-01 00:44 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-11-01 03:48 ` David Rowley <dgrowleyml@gmail.com>
2022-11-01 04:10 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-11-01 04:26 ` David Rowley <dgrowleyml@gmail.com>
2022-11-01 04:52 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-11-01 05:51 ` Andres Freund <andres@anarazel.de>
2023-02-22 19:43 ` Justin Pryzby <pryzby@telsasoft.com>
2023-05-02 13:33 ` Daniel Gustafsson <daniel@yesql.se>
2023-05-16 13:31 ` Daniel Gustafsson <daniel@yesql.se>
2023-09-13 13:24 ` Daniel Gustafsson <daniel@yesql.se>
2023-09-27 12:07 ` Daniel Gustafsson <daniel@yesql.se>
2023-09-27 20:45 ` Andres Freund <andres@anarazel.de>
2023-11-17 15:31 ` Daniel Gustafsson <daniel@yesql.se>
2022-11-02 08:26 ` Lauri Laanmets <pcspets@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox