pgjdbc/pgjdbc GitHub issues and pull requests (mirror)
help / color / mirror / Atom feed[pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
49+ messages / 5 participants
[nested] [flat]
* [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 13:56 "embuc (@embuc)" <[email protected]>
0 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 13:56 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
**Describe the issue**
After update to driver version > 42.6.0 (tested also both released version up to latest released 42.7.1) we started to notice memory leaks connected to re-deploys of web-apps in Apache Tomcat (ver 9.0.80). I inspected heap dumps and found the instances of org.postgresql.util.LazyCleaner retained in Metaspace i.e. with path to GC roots with excluded phantom/weak/soft/etc references. Regression seem to have happened with release of 42.6.0 version. Downgraded to 42.5.4 - after excessive testing - no leaks in Metaspace and everything was released correctly. Perhaps this has to do something with [fix 2817](https://github.com/pgjdbc/pgjdbc/pull/2817) and/or [fix 2847 ](https://github.com/pgjdbc/pgjdbc/pull/2847)? I didn't have time to investigate more but it should be hopefully enough to point the experts in right direction.
**Driver Version?**
42.6.0
**Java Version?**
17
**OS Version?**
Ubuntu 22.04
**PostgreSQL Version?**
14
**To Reproduce**
Steps to reproduce the behaviour:
Re-deploy .war files in Apache Tomcat.
**Expected behaviour**
GC should be able to clean Metaspace when classes are unloaded (as with deploy).
And what actually happens:
Old Classloader is bound with reference from LazyCleaner thus preventing GC to remove it.
**Logs**
If possible PostgreSQL logs surrounding the occurrence of the issue (N/A)
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 14:08 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-09 14:08 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Thank you for the report. It would help if you could provide a reproducer or a heapdump.
`LazyCleaner` thread should terminate after (30000ms of inactivity, `pgjdbc.config.cleanup.thread.ttl` system property), so it should not generate a long-lasting leak.
Do you think you redeploy multiple times within 30sec window?
As you are using Java 17, we could probably make a patch that uses Java's `Cleaner` for Java 9+, so it is more robust.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 14:22 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 14:22 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
No problem, however I am not at liberty to provide heap dumps (policy). I tried re-deploys in various configurations - as you suggested rather fast, with multiple apps shorter span than 30 seconds but even extremes on the other end such as waiting for 2h (default tcp_idle timeout) but as long as Tomcat is not restarted it self, the reference seems to survive.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 14:30 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-09 14:30 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
I wonder if https://github.com/pgjdbc/pgjdbc/blob/aa235e14c2213d1386100e62206006d83082d914/pgjdbc/src/main/java/... ever returns true ?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 14:54 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-09 14:54 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@embuc are there any error messages from tomcat suggesting there will be a leak ?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 15:16 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 15:16 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@davecramer Unfortunately none whatsoever. It took me a while to track it down to this driver - I had more probable suspects to check first to say at least. Tomcat crashes eventually (CI/CD dev servers with somewhat limited RAM) - OOM is thrown and deploys fail.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 15:28 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-09 15:28 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@embuc Not sure this https://cwiki.apache.org/confluence/display/tomcat/MemoryLeakProtection is still relevant. Apparently there is a FindLeaks button. Does that exist ?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 15:35 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 15:35 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Oh, such a blast from the past, haven't used that for more than a decade - lo and behold -it's still there and I had to try it. It does confirm that: '...classes from previous runs are still loaded in memory, thus causing a memory leak (use a profiler to confirm)...' Which corresponds with my findings (it simply states which contexts contain leaks).
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 15:47 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-09 15:47 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Ok, so nothing useful.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 15:59 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 15:59 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Unfortunately, no. I can not share heap dumps but here is an excerpt from GC path traversal of what is retained (after phantom/weak.. are pruned):
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 16:02 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-09 16:02 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@vlsi probably the best thing to do is use Java's Cleaner for anything above 8.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 16:26 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-09 16:26 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@embuc , can you find `org.postgresql.util.LazyCleaner` instance in the heap dump and show its values?
The question is whether `LazyCleaner#first` value is `non-null` and if it is `non-null`, then what keeps the value in the heap (show paths to GC roots for the `LazyCleaner.first.referent`)
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 18:14 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 18:14 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@vlsi sure, `LazyCleaner#first` seems to be `non-null` indeed, however referent is sparse on info, i can only see it as an `Object`:
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 18:20 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-09 18:20 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Can you show "paths to GC roots" for the referent anyway?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 18:29 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 18:29 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
It doesn't seems to be available, I can see a lot of threads like this however:
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 18:43 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-09 18:43 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
So the remove is not completing during a reload.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 18:45 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-09 18:45 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> It doesn't seems to be availabl
Can you elaborate? Can you open the dump in Eclipse Memory Analyzer?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-09 19:17 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-09 19:17 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Yes, it was easier in MAT - I couldn't find it in Yourkit. Here it is (_all_ references to GC root):
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-10 10:35 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-10 10:35 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
It appears that Object.wait is not returning during a reload.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-10 10:40 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-10 10:40 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
We need a reproducible case. If the object is not referenced, then the phantom reference should be cleared at some point and it should enqueue the reference so the thread should terminate.
if the reference does not clear, it might be caused by **lack of GC activity**. In other words, if GC does not happen, then `PhantomReferences` do not clear
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-10 11:31 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-10 11:31 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Do we know what tomcat does when it reloads? I'm guessing it's quite complicated.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-10 11:35 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-10 11:35 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
It would be interesting to see if Java's Cleaner has the same issue. Although looking at the implementation it does exactly the same thing ` Cleanable ref = (Cleanable) queue.remove(60 * 1000L);`
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-10 19:20 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-10 19:20 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> It doesn't seems to be available, I can see a lot of threads like this however: 
One thing that would be interesting to know is size of the `queue`
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 10:16 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-11 10:16 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> > It doesn't seems to be available, I can see a lot of threads like this however: 
>
> One thing that would be interesting to know is size of the `queue`
I'm not 100% sure which of these retained instances one should look at but on random if I inspect them in MAT i see that queue is not null but it has length 0?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 10:19 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-11 10:19 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
So that is consistent with @vlsi suggestion that gc() didn't run. There's nothing in the queue for us to remove. Which explains they leak
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 10:20 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-11 10:20 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
you could try building the jar using https://github.com/pgjdbc/pgjdbc/pull/3090 but if gc() isn't running I fail to see how using Cleaner will work
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 12:07 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-11 12:07 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@embuc, just for reference, if the `LazyCleaner` queue is not empty, then it means there are unclosed connections.
In other words, it means the application leaks connections (or it still uses the connections).
If the application closed connections properly, there would be nothing to clean in the first place.
You might debug that by adding `logUnclosedConnections=true` connection parameters, and then check logs or `PgConnectionCleaningAction.openStackTrace` field for the stacktrace which opened the connection that was eventually left unclosed.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 12:46 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-11 12:46 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> you could try building the jar using #3090 but if gc() isn't running I fail to see how using Cleaner will work
@davecramer I built the snapshot with #3090 and tried but I get same thing :/ GC is definitely running but I can see that it frees up about ~15MB of Metaspace but thats not enough (when close the limit) and the OOM is thrown.
@vlsi the queue seems to be empty on all instances I checked.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 12:58 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-11 12:58 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> queue seems to be empty on all instances I checked
If the queue is empty the thread should terminate, and it should no longer hold the resources.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 13:02 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-11 13:02 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> > you could try building the jar using #3090 but if gc() isn't running I fail to see how using Cleaner will work
>
> @davecramer I built the snapshot with #3090 and tried but I get same thing :/ GC is definitely running but I can see that it frees up about ~15MB of Metaspace but thats not enough (when close the limit) and the OOM is thrown. @vlsi the queue seems to be empty on all instances I checked.
What this tells us is that the issue is not with our code as this uses java 9+ Cleaner.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 13:23 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-11 13:23 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> > queue seems to be empty on all instances I checked
>
> If the queue is empty the thread should terminate, and it should no longer hold the resources.
I think you are right about an application leak. Something is holding a reference to the connections so gc() is not collecting them and they aren't in the queue.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 13:59 ` "bokken (@bokken)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: bokken (@bokken) @ 2024-01-11 13:59 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
If there is a leak, the heap dump should show the connection objects with
references to the problematic class loader. Perhaps there are other details
on the connection objects which could provide a hint of where they are
leaked from.
On Thu, Jan 11, 2024 at 7:23 AM Dave Cramer ***@***.***>
wrote:
> queue seems to be empty on all instances I checked
>
> If the queue is empty the thread should terminate, and it should no longer
> hold the resources.
>
> I think you are right about an application leak. Something is holding a
> reference to the connections so gc() is not collecting them and they aren't
> in the queue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pgjdbc/pgjdbc/issues/3089#issuecomment-1887155340;,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAW3U3OGL52XVVRVOGJCCYLYN7RVLAVCNFSM6AAAAABBTEMBAC...;
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 15:15 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-11 15:15 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
I looked around a bit now that i have cloned the code and omitting the Cleaner (Java's and/or LazyCleaner) and calling the close directly on the queryExecutor works. Or should I say does not block the GC during the hot reload. I was thinking that this would be interesting to explore: "The cleaning action could be a lambda but all too easily will capture the object reference, by referring to fields of the object being cleaned, preventing the object from becoming phantom reachable. Using a static nested class, as above, will avoid accidentally retaining the object reference.
Cleaning actions should be prepared to be invoked concurrently with other cleaning actions. Typically the cleaning actions should be very quick to execute and not block. If the cleaning action blocks, it may delay processing other cleaning actions registered to the same cleaner" <- This is from the API docs.
Edit: Just to be clear after this wall of text: 42.7.2-SNAPSHOT with calling close on queryExecutor (in PgConnection#close()) works as expected regarding the memory leak.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 16:02 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-11 16:02 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
If I read the above correctly, then pgjdbc does everything right, and it was application that holded the reference to the connection.
In that case, I am leaning towards closing the issue
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 16:16 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-11 16:16 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Perhaps i dont follow but i recompiled pgjdbc with omitting the code that seems to block the GC and it works correctly. I can't see how that has to do with application holding reference?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 16:20 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-11 16:20 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
So here is what should happen:
gc() is called and adds any references that can be cleaned up to the PhantomReferenceQueue. For whatever reason this is not occurring since the queue is empty. However we have added the connections to the linked list so the thread keeps running waiting for either connection.close to be called or gc() to add them to the queue. By. removing that code you are bypassing the second step. I'd be curious how many connections are still open after a reload
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 16:54 ` "embuc (@embuc)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: embuc (@embuc) @ 2024-01-11 16:54 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
No connections seem to survive re-deploy if that tells you anything - after the new applications are deployed there are a few opened connections (this is completely normal for our environment due to batch jobs etc.).
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-11 17:22 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2024-01-11 17:22 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
I have no systems that reproduce the issue.
There is no reproducer, and the reporter does not provide information for the analysis.
I have no desire to spend time trying to reproduce it.
So I unsubscribe.
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-01-18 13:16 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-01-18 13:16 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
unreproducible
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-12-02 11:13 ` "bataroland (@bataroland)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: bataroland (@bataroland) @ 2024-12-02 11:13 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
This problem appears on: JBoss EAP 7.4, but I think on ALL JBoss?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-12-02 11:56 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-12-02 11:56 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> This problem appears on: JBoss EAP 7.0.9, but I think on ALL JBoss?
Still occurring on the latest version of the driver ?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-12-02 16:09 ` "bataroland (@bataroland)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: bataroland (@bataroland) @ 2024-12-02 16:09 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@davecramer We are using 42.7.3
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-12-12 06:49 ` "bataroland (@bataroland)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: bataroland (@bataroland) @ 2024-12-12 06:49 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@davecramer At early of 2025, we are try to investigate the problem, and If needs any information about to identify the problem, then we will share any information as we can.
FYI: EDB JDBC drivers impacted too, cause EDB forks from the pgjdbc driver (we are using that).
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2024-12-12 10:15 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2024-12-12 10:15 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> @davecramer At early of 2025, we are try to investigate the problem, and If needs any information about to identify the problem, then we will share any information as we can.
>
> FYI: EDB JDBC drivers impacted too, cause EDB forks from the pgjdbc driver (we are using that).
Interesting,
Thanks look forward to it.
Dave
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2025-01-29 15:20 ` "bataroland (@bataroland)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: bataroland (@bataroland) @ 2025-01-29 15:20 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
### Analysis of the PostgreSQL JDBC Cleaner Thread Issue in PostgreSQL JDBC on Application Servers
After extensive analysis, we identified a classloader leak issue related to the `PostgreSQL-JDBC-Cleaner` thread in PostgreSQL JDBC when used with application servers. This issue is similar to the one described in [this StackOverflow answer](https://stackoverflow.com/a/78171021/9004863), but in our case, it occurs on a different application server (JBoss).
#### **Root Cause of the Issue**
The core problem lies in how the `PostgreSQL-JDBC-Cleaner` thread inherits its execution context from the thread that creates it. Specifically, when a new thread is started in Java, it inherits an `AccessControlContext` from its parent thread. This means that the `PostgreSQL-JDBC-Cleaner` thread may retain a reference to the `ProtectionDomain` of the web application that initially triggered its creation. As a result the web application’s classloader remains referenced, preventing garbage collection (GC) from reclaiming it.
In PostgreSQL JDBC, the `org.postgresql.util.LazyCleaner` class manages a background cleaner thread named **"PostgreSQL-JDBC-Cleaner"**. When this thread is created, the following method is invoked:
```java
# JDK8
java.lang.Thread#init(java.lang.ThreadGroup, java.lang.Runnable, java.lang.String, long, java.security.AccessControlContext, boolean)
```
This method sets an internal field called `inheritedAccessControlContext`, which holds an instance of `java.security.AccessControlContext` inherited from the creator thread. If the parent thread belongs to a web application, its classloader will be retained in memory through this field. Consequently, the classloader cannot be garbage collected until all `org.postgresql.util.LazyCleaner.Node` instances have been removed and cleaned.
#### **How the Issue Manifests in Application Servers**
The impact of this issue depends on when the `PostgreSQL-JDBC-Cleaner` thread is initialized:
- **Scenario 1 (Worst Case): The Cleaner Thread is created after a Web Application gets a connection**
- If a web application is deployed and establishes a database connection via JNDI, the `PostgreSQL-JDBC-Cleaner` thread is initialized within the application's context.
- Since the `PostgreSQL-JDBC-Cleaner` thread holds a reference to the web application’s classloader, undeploying the application leaves a dangling reference, causing a classloader leak.
- Over time, repeated (re)deployments can accumulate these leaked classloaders, leading to increased memory usage and after some time OOM.
- **Scenario 2 (Better Case): The Cleaner Thread is created before any Web Application is deployed**
- If a test connection is made **immediately after the application server starts** (before any web application is deployed), the `PostgreSQL-JDBC-Cleaner` thread is initialized within the application server’s context instead of a specific web application’s classloader.
- In this case, subsequent web applications will not be referenced by the cleaner thread, and their classloaders can be garbage collected.
- However, connection pools automatically removes idle connections after **30 minutes**. If this happens and the application gets a connection from the pool, then a new `PostgreSQL-JDBC-Cleaner` thread may be created under the web application’s classloader, reintroducing the classloader leak.
#### **Why the Problem Persists**
- The PostgreSQL JDBC driver is registered **globally** in the application server.
- Connection pooling exacerbates the problem:
- If the application server manages database connections through a connection pool, the `PostgreSQL-JDBC-Cleaner` thread may outlive the applications using it.
- If the application itself manages the connections instead of relying on the server’s connection pool, the problem would not occur.
- In a production application server, it is unlikely that **all** pooled connections would be closed. As a result, the leaked classloader remains in memory indefinitely unless the server is restarted.
#### **Steps to Reproduce the Issue**
1. Configure a **PostgreSQL JDBC datasource** in the application server.
2. Deploy a web application that retrieves the datasource via **JNDI**.
3. Execute a simple SQL statement within the application (e.g., `SELECT version();`).
4. Undeploy the web application.
5. Create a **heap dump** of the application server’s memory.
1. If a `PostgreSQL-JDBC-Cleaner` thread was created from a thread owned by the web application’s classloader, the web application’s classloader will remain in memory and **cannot be garbage collected** unless:
- The server is completely shut down.
- All database connections are explicitly closed and cleaned.
2. If a `PostgreSQL-JDBC-Cleaner` thread was created **before any web application deployment**, then until the `pgjdbc.config.cleanup.thread.ttl` expires, only the application server’s classloader is referenced in `inheritedAccessControlContext`, preventing web application classloader leaks.
#### **Conclusion**
The `PostgreSQL-JDBC-Cleaner` thread in PostgreSQL JDBC introduces a potential **classloader leak** when used with application servers that manage connection pooling. The issue arises because the cleaner thread inherits the `AccessControlContext` of the thread that creates it.
- If the cleaner thread is initialized **after a web application gets a connection**, as a result, the web application’s classloader remains referenced indefinitely, leading to **memory leaks**.
- If the cleaner thread is initialized **before any web application deploys**, the issue can be avoided, but idle connection removal mechanisms can reintroduce it over time.
To mitigate this issue, possible workarounds include:
- Ensuring that the `PostgreSQL-JDBC-Cleaner` thread is **initialized only once at the driver init** before any applications deploy.
#### Used and tested JDBC drivers
- Postgres JDBC 42.7.5
- EDB JDBC 42.7.3.2
Collab: @bodzso
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2025-01-29 20:39 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2025-01-29 20:39 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
wow!, nice thanks for the detective work and detailed report
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2025-02-27 05:57 ` "bataroland (@bataroland)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: bataroland (@bataroland) @ 2025-02-27 05:57 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@davecramer any news about that? Do you need any further information?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2025-02-27 12:57 ` "davecramer (@davecramer)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: davecramer (@davecramer) @ 2025-02-27 12:57 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
No news, just need more round tuit's. Any chance you could provide a PR ?
^ permalink raw reply [nested|flat] 49+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext
@ 2026-03-31 06:06 ` "vlsi (@vlsi)" <[email protected]>
47 siblings, 0 replies; 49+ messages in thread
From: vlsi (@vlsi) @ 2026-03-31 06:06 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
`Thread.inheritedAccessControlContext` leak was fixed in https://github.com/pgjdbc/pgjdbc/pull/3886 (pgjdbc 42.7.10+)
^ permalink raw reply [nested|flat] 49+ messages in thread
end of thread, other threads:[~2026-03-31 06:06 UTC | newest]
Thread overview: 49+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-01-09 13:56 [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext "embuc (@embuc)" <[email protected]>
2024-01-09 14:08 ` "vlsi (@vlsi)" <[email protected]>
2024-01-09 14:22 ` "embuc (@embuc)" <[email protected]>
2024-01-09 14:30 ` "davecramer (@davecramer)" <[email protected]>
2024-01-09 14:54 ` "davecramer (@davecramer)" <[email protected]>
2024-01-09 15:16 ` "embuc (@embuc)" <[email protected]>
2024-01-09 15:28 ` "davecramer (@davecramer)" <[email protected]>
2024-01-09 15:35 ` "embuc (@embuc)" <[email protected]>
2024-01-09 15:47 ` "davecramer (@davecramer)" <[email protected]>
2024-01-09 15:59 ` "embuc (@embuc)" <[email protected]>
2024-01-09 16:02 ` "davecramer (@davecramer)" <[email protected]>
2024-01-09 16:26 ` "vlsi (@vlsi)" <[email protected]>
2024-01-09 18:14 ` "embuc (@embuc)" <[email protected]>
2024-01-09 18:20 ` "vlsi (@vlsi)" <[email protected]>
2024-01-09 18:29 ` "embuc (@embuc)" <[email protected]>
2024-01-09 18:43 ` "davecramer (@davecramer)" <[email protected]>
2024-01-09 18:45 ` "vlsi (@vlsi)" <[email protected]>
2024-01-09 19:17 ` "embuc (@embuc)" <[email protected]>
2024-01-10 10:35 ` "davecramer (@davecramer)" <[email protected]>
2024-01-10 10:40 ` "vlsi (@vlsi)" <[email protected]>
2024-01-10 11:31 ` "davecramer (@davecramer)" <[email protected]>
2024-01-10 11:35 ` "davecramer (@davecramer)" <[email protected]>
2024-01-10 19:20 ` "davecramer (@davecramer)" <[email protected]>
2024-01-11 10:16 ` "embuc (@embuc)" <[email protected]>
2024-01-11 10:19 ` "davecramer (@davecramer)" <[email protected]>
2024-01-11 10:20 ` "davecramer (@davecramer)" <[email protected]>
2024-01-11 12:07 ` "vlsi (@vlsi)" <[email protected]>
2024-01-11 12:46 ` "embuc (@embuc)" <[email protected]>
2024-01-11 12:58 ` "vlsi (@vlsi)" <[email protected]>
2024-01-11 13:02 ` "davecramer (@davecramer)" <[email protected]>
2024-01-11 13:23 ` "davecramer (@davecramer)" <[email protected]>
2024-01-11 13:59 ` "bokken (@bokken)" <[email protected]>
2024-01-11 15:15 ` "embuc (@embuc)" <[email protected]>
2024-01-11 16:02 ` "vlsi (@vlsi)" <[email protected]>
2024-01-11 16:16 ` "embuc (@embuc)" <[email protected]>
2024-01-11 16:20 ` "davecramer (@davecramer)" <[email protected]>
2024-01-11 16:54 ` "embuc (@embuc)" <[email protected]>
2024-01-11 17:22 ` "vlsi (@vlsi)" <[email protected]>
2024-01-18 13:16 ` "davecramer (@davecramer)" <[email protected]>
2024-12-02 11:13 ` "bataroland (@bataroland)" <[email protected]>
2024-12-02 11:56 ` "davecramer (@davecramer)" <[email protected]>
2024-12-02 16:09 ` "bataroland (@bataroland)" <[email protected]>
2024-12-12 06:49 ` "bataroland (@bataroland)" <[email protected]>
2024-12-12 10:15 ` "davecramer (@davecramer)" <[email protected]>
2025-01-29 15:20 ` "bataroland (@bataroland)" <[email protected]>
2025-01-29 20:39 ` "davecramer (@davecramer)" <[email protected]>
2025-02-27 05:57 ` "bataroland (@bataroland)" <[email protected]>
2025-02-27 12:57 ` "davecramer (@davecramer)" <[email protected]>
2026-03-31 06:06 ` "vlsi (@vlsi)" <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox