Subject: [pgjdbc/pgjdbc] issue #3089: Metaspace Memory leak: Thread.inheritedAccessControlContext

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 13:56:41 +0000

**Describe the issue**
After update to driver version > 42.6.0 (tested also both released version up to latest released 42.7.1) we started to notice memory leaks connected to re-deploys of web-apps in Apache Tomcat (ver 9.0.80). I inspected heap dumps and found the instances of org.postgresql.util.LazyCleaner retained in Metaspace i.e. with path to GC roots with excluded phantom/weak/soft/etc references. Regression seem to have happened with release of 42.6.0 version. Downgraded to 42.5.4 - after excessive testing - no leaks in Metaspace and everything was released correctly. Perhaps this has to do something with [fix 2817](https://github.com/pgjdbc/pgjdbc/pull/2817) and/or [fix 2847 ](https://github.com/pgjdbc/pgjdbc/pull/2847)? I didn't have time to investigate more but it should be hopefully enough to point the experts in right direction.    

**Driver Version?** 
42.6.0
**Java Version?**
17
**OS Version?**
Ubuntu 22.04
**PostgreSQL Version?**
14
**To Reproduce**
Steps to reproduce the behaviour:
Re-deploy .war files in Apache Tomcat. 
**Expected behaviour**
GC should be able to clean Metaspace when classes are unloaded (as with deploy).
And what actually happens:
Old Classloader is bound with reference from LazyCleaner thus preventing GC to remove it. 

**Logs**
If possible PostgreSQL logs surrounding the occurrence of the issue (N/A)

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Tue, 09 Jan 2024 14:08:14 +0000

Thank you for the report. It would help if you could provide a reproducer or a heapdump.

`LazyCleaner` thread should terminate after (30000ms of inactivity, `pgjdbc.config.cleanup.thread.ttl` system property), so it should not generate a long-lasting leak.

Do you think you redeploy multiple times within 30sec window?

As you are using Java 17, we could probably make a patch that uses Java's `Cleaner` for Java 9+, so it is more robust.

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 14:22:08 +0000

No problem, however I am not at liberty to provide heap dumps (policy). I tried re-deploys in various configurations - as you suggested rather fast, with multiple apps shorter span than 30 seconds but even extremes on the other end such as waiting for 2h (default tcp_idle timeout) but as long as Tomcat is not restarted it self, the reference seems to survive.

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Tue, 09 Jan 2024 14:30:14 +0000

I wonder if https://github.com/pgjdbc/pgjdbc/blob/aa235e14c2213d1386100e62206006d83082d914/pgjdbc/src/main/java/org/postgresql/util/LazyCleaner.java#L97 ever returns true ?

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Tue, 09 Jan 2024 14:54:59 +0000

@embuc are there any error messages from tomcat suggesting there will be a leak ?

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 15:16:03 +0000

@davecramer  Unfortunately none whatsoever. It took me a while to track it down to this driver - I had more probable suspects to check first to say at least. Tomcat crashes eventually (CI/CD dev servers with somewhat limited RAM) - OOM is thrown and deploys fail.

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Tue, 09 Jan 2024 15:28:35 +0000

@embuc Not sure this https://cwiki.apache.org/confluence/display/tomcat/MemoryLeakProtection is still relevant. Apparently there is a FindLeaks button. Does that exist ?

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 15:35:32 +0000

Oh, such a blast from the past, haven't used that for more than a decade - lo and behold -it's still there and I had to try it. It does confirm that: '...classes from previous runs are still loaded in memory, thus causing a memory leak (use a profiler to confirm)...' Which corresponds with my findings (it simply states which contexts contain leaks).

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Tue, 09 Jan 2024 15:47:37 +0000

Ok, so nothing useful.

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 15:59:10 +0000

Unfortunately, no. I can not share heap dumps but here is an excerpt from GC path traversal of what is retained (after phantom/weak.. are pruned):  
![image](https://github.com/pgjdbc/pgjdbc/assets/1480007/ac3cbf0a-f3bd-4776-96c6-ab905198db92)

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Tue, 09 Jan 2024 16:02:00 +0000

@vlsi probably the best thing to do is use Java's Cleaner for anything above 8.

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Tue, 09 Jan 2024 16:26:58 +0000

@embuc , can you find `org.postgresql.util.LazyCleaner` instance in the heap dump and show its values?
The question is whether `LazyCleaner#first` value is `non-null` and if it is `non-null`, then what keeps the value in the heap (show paths to GC roots for the `LazyCleaner.first.referent`)

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 18:14:53 +0000

@vlsi sure, `LazyCleaner#first` seems to be `non-null` indeed, however referent is sparse on info, i can only see it as an `Object`: 
![image](https://github.com/pgjdbc/pgjdbc/assets/1480007/878240c9-4d43-4aac-a4ea-3c3f805768d3)

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Tue, 09 Jan 2024 18:20:31 +0000

Can you show "paths to GC roots" for the referent anyway?

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 18:29:57 +0000

It doesn't seems to be available, I can see a lot of threads like this however: 
![image](https://github.com/pgjdbc/pgjdbc/assets/1480007/537e6f6e-bf5c-4d75-b356-f151ced5415a)

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Tue, 09 Jan 2024 18:43:00 +0000

So the remove is not completing during a reload.

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Tue, 09 Jan 2024 18:45:56 +0000

> It doesn't seems to be availabl

Can you elaborate? Can you open the dump in Eclipse Memory Analyzer?

---

From: "embuc (@embuc)" <[email-masked]>
Date: Tue, 09 Jan 2024 19:17:01 +0000

Yes, it was easier in MAT - I couldn't find it in Yourkit. Here it is (_all_ references to GC root): 
![image](https://github.com/pgjdbc/pgjdbc/assets/1480007/f7f732ec-7897-437c-a388-6ad22d825b27)

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Wed, 10 Jan 2024 10:35:35 +0000

It appears that Object.wait is not returning during a reload.

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Wed, 10 Jan 2024 10:40:02 +0000

We need a reproducible case. If the object is not referenced, then the phantom reference should be cleared at some point and it should enqueue the reference so the thread should terminate.

if the reference does not clear, it might be caused by **lack of GC activity**. In other words, if GC does not happen, then `PhantomReferences` do not clear

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Wed, 10 Jan 2024 11:31:14 +0000

Do we know what tomcat does when it reloads? I'm guessing it's quite complicated.

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Wed, 10 Jan 2024 11:35:11 +0000

It would be interesting to see if Java's Cleaner has the same issue. Although looking at the implementation it does exactly the same thing ` Cleanable ref = (Cleanable) queue.remove(60 * 1000L);`

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Wed, 10 Jan 2024 19:20:19 +0000

> It doesn't seems to be available, I can see a lot of threads like this however: ![image](https://private-user-images.githubusercontent.com/1480007/295302840-537e6f6e-bf5c-4d75-b356-f151ced5415a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ5MTI5NjYsIm5iZiI6MTcwNDkxMjY2NiwicGF0aCI6Ii8xNDgwMDA3LzI5NTMwMjg0MC01MzdlNmY2ZS1iZjVjLTRkNzUtYjM1Ni1mMTUxY2VkNTQxNWEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDExMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAxMTBUMTg1MTA2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Y2Q2MjdiZDljYzg0YmZlODRiZGIxZWE2YmU0YjgzNWNiNTg2ODBjYmJkYzc4NDg1YzJkMGVhMGFiMDQ1N2E0NSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.3rzdhz9TPQzFYjhjYk8_IUaEdwhqObioRTbMz3R6nrU)

One thing that would be interesting to know is size of the `queue`

---

From: "embuc (@embuc)" <[email-masked]>
Date: Thu, 11 Jan 2024 10:16:25 +0000

> One thing that would be interesting to know is size of the `queue`

I'm not 100% sure which of these retained instances one should look at but on random if I inspect them in MAT i see that queue is not null but it has length 0?

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 11 Jan 2024 10:19:28 +0000

So that is consistent with @vlsi suggestion that gc() didn't run. There's nothing in the queue for us to remove. Which explains they leak

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 11 Jan 2024 10:20:51 +0000

you could try building the jar using https://github.com/pgjdbc/pgjdbc/pull/3090 but if gc() isn't running I fail to see how using Cleaner will work

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Thu, 11 Jan 2024 12:07:51 +0000

@embuc, just for reference, if the `LazyCleaner` queue is not empty, then it means there are unclosed connections.
In other words, it means the application leaks connections (or it still uses the connections).

If the application closed connections properly, there would be nothing to clean in the first place.

You might debug that by adding `logUnclosedConnections=true` connection parameters, and then check logs or `PgConnectionCleaningAction.openStackTrace` field for the stacktrace which opened the connection that was eventually left unclosed.

---

From: "embuc (@embuc)" <[email-masked]>
Date: Thu, 11 Jan 2024 12:46:43 +0000

> you could try building the jar using #3090 but if gc() isn't running I fail to see how using Cleaner will work

@davecramer I built the snapshot with #3090 and tried but I get same thing :/ GC is definitely running but I can see that it frees up about ~15MB of Metaspace but thats not enough (when close the limit) and the OOM is thrown. 
@vlsi  the queue seems to be empty on all instances I checked.

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Thu, 11 Jan 2024 12:58:06 +0000

> queue seems to be empty on all instances I checked

If the queue is empty the thread should terminate, and it should no longer hold the resources.

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 11 Jan 2024 13:02:55 +0000

> @davecramer I built the snapshot with #3090 and tried but I get same thing :/ GC is definitely running but I can see that it frees up about ~15MB of Metaspace but thats not enough (when close the limit) and the OOM is thrown. @vlsi the queue seems to be empty on all instances I checked.

What this tells us is that the issue is not with our code as this uses java 9+ Cleaner.

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 11 Jan 2024 13:23:18 +0000

> If the queue is empty the thread should terminate, and it should no longer hold the resources.

I think you are right about an application leak. Something is holding a reference to the connections so gc() is not collecting them and they aren't in the queue.

---

From: "bokken (@bokken)" <[email-masked]>
Date: Thu, 11 Jan 2024 13:59:25 +0000

If there is a leak, the heap dump should show the connection objects with
references to the problematic class loader. Perhaps there are other details
on the connection objects which could provide a hint of where they are
leaked from.

On Thu, Jan 11, 2024 at 7:23 AM Dave Cramer ***@***.***>
wrote:

> I think you are right about an application leak. Something is holding a
> reference to the connections so gc() is not collecting them and they aren't
> in the queue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pgjdbc/pgjdbc/issues/3089#issuecomment-1887155340>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAW3U3OGL52XVVRVOGJCCYLYN7RVLAVCNFSM6AAAAABBTEMBACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBXGE2TKMZUGA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

---

From: "embuc (@embuc)" <[email-masked]>
Date: Thu, 11 Jan 2024 15:15:04 +0000

I looked around a bit now that i have cloned the code and omitting the Cleaner (Java's and/or LazyCleaner) and calling the close directly on the queryExecutor works. Or should I say does not block the GC during the hot reload.  I was thinking that this would be interesting to explore: "The cleaning action could be a lambda but all too easily will capture the object reference, by referring to fields of the object being cleaned, preventing the object from becoming phantom reachable. Using a static nested class, as above, will avoid accidentally retaining the object reference.

Cleaning actions should be prepared to be invoked concurrently with other cleaning actions. Typically the cleaning actions should be very quick to execute and not block. If the cleaning action blocks, it may delay processing other cleaning actions registered to the same cleaner" <- This is from the API docs.

Edit: Just to be clear after this wall of text:  42.7.2-SNAPSHOT with calling close on queryExecutor (in PgConnection#close()) works as expected regarding the memory leak.

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Thu, 11 Jan 2024 16:02:45 +0000

If I read the above correctly, then pgjdbc does everything right, and it was application that holded the reference to the connection.

In that case, I am leaning towards closing the issue

---

From: "embuc (@embuc)" <[email-masked]>
Date: Thu, 11 Jan 2024 16:16:38 +0000

Perhaps i dont follow but i recompiled pgjdbc with omitting the code that seems to block the GC and it works correctly. I can't see how that has to do with application holding reference?

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 11 Jan 2024 16:20:50 +0000

So here is what should happen:
gc() is called and adds any references that can be cleaned up to the PhantomReferenceQueue. For whatever reason this is not occurring since the queue is empty. However we have added the connections to the linked list so the thread keeps running waiting for either connection.close to be called or gc() to add them to the queue. By. removing that code you are bypassing the second step. I'd be curious how many connections are still open after a reload

---

From: "embuc (@embuc)" <[email-masked]>
Date: Thu, 11 Jan 2024 16:54:33 +0000

No connections seem to survive re-deploy if that tells you anything - after the new applications are deployed there are a few opened connections (this is completely normal for our environment due to batch jobs etc.).

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Thu, 11 Jan 2024 17:22:17 +0000

I have no systems that reproduce the issue.
There is no reproducer, and the reporter does not provide information for the analysis.
I have no desire to spend time trying to reproduce it.

So I unsubscribe.

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 18 Jan 2024 13:16:01 +0000

unreproducible

---

From: "bataroland (@bataroland)" <[email-masked]>
Date: Mon, 02 Dec 2024 11:13:57 +0000

This problem appears on: JBoss EAP 7.4, but I think on ALL JBoss?

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Mon, 02 Dec 2024 11:56:45 +0000

> This problem appears on: JBoss EAP 7.0.9, but I think on ALL JBoss?

Still occurring on the latest version of the driver ?

---

From: "bataroland (@bataroland)" <[email-masked]>
Date: Mon, 02 Dec 2024 16:09:52 +0000

@davecramer We are using 42.7.3

---

From: "bataroland (@bataroland)" <[email-masked]>
Date: Thu, 12 Dec 2024 06:49:12 +0000

@davecramer At early of 2025, we are try to investigate the problem, and If needs any information about to identify the problem, then we will share any information as we can.

FYI: EDB JDBC drivers impacted too, cause EDB forks from the pgjdbc driver (we are using that).

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 12 Dec 2024 10:15:53 +0000

> FYI: EDB JDBC drivers impacted too, cause EDB forks from the pgjdbc driver (we are using that).

Interesting,

Thanks look forward to it.

Dave

---

From: "bataroland (@bataroland)" <[email-masked]>
Date: Wed, 29 Jan 2025 15:20:26 +0000

### Analysis of the PostgreSQL JDBC Cleaner Thread Issue in PostgreSQL JDBC on Application Servers  

After extensive analysis, we identified a classloader leak issue related to the `PostgreSQL-JDBC-Cleaner` thread in PostgreSQL JDBC when used with application servers. This issue is similar to the one described in [this StackOverflow answer](https://stackoverflow.com/a/78171021/9004863), but in our case, it occurs on a different application server (JBoss).  

#### **Root Cause of the Issue**  

The core problem lies in how the `PostgreSQL-JDBC-Cleaner` thread inherits its execution context from the thread that creates it. Specifically, when a new thread is started in Java, it inherits an `AccessControlContext` from its parent thread. This means that the `PostgreSQL-JDBC-Cleaner` thread may retain a reference to the `ProtectionDomain` of the web application that initially triggered its creation. As a result the web application’s classloader remains referenced, preventing garbage collection (GC) from reclaiming it.  

In PostgreSQL JDBC, the `org.postgresql.util.LazyCleaner` class manages a background cleaner thread named **"PostgreSQL-JDBC-Cleaner"**. When this thread is created, the following method is invoked:  
```java
# JDK8
java.lang.Thread#init(java.lang.ThreadGroup, java.lang.Runnable, java.lang.String, long, java.security.AccessControlContext, boolean)
```  
This method sets an internal field called `inheritedAccessControlContext`, which holds an instance of `java.security.AccessControlContext` inherited from the creator thread. If the parent thread belongs to a web application, its classloader will be retained in memory through this field. Consequently, the classloader cannot be garbage collected until all `org.postgresql.util.LazyCleaner.Node` instances have been removed and cleaned.  

#### **How the Issue Manifests in Application Servers**  

The impact of this issue depends on when the `PostgreSQL-JDBC-Cleaner` thread is initialized:  

- **Scenario 1 (Worst Case): The Cleaner Thread is created after a Web Application gets a connection**  
  - If a web application is deployed and establishes a database connection via JNDI, the `PostgreSQL-JDBC-Cleaner` thread is initialized within the application's context.  
  - Since the `PostgreSQL-JDBC-Cleaner` thread holds a reference to the web application’s classloader, undeploying the application leaves a dangling reference, causing a classloader leak.  
  - Over time, repeated (re)deployments can accumulate these leaked classloaders, leading to increased memory usage and after some time OOM.  

- **Scenario 2 (Better Case): The Cleaner Thread is created before any Web Application is deployed**  
  - If a test connection is made **immediately after the application server starts** (before any web application is deployed), the `PostgreSQL-JDBC-Cleaner` thread is initialized within the application server’s context instead of a specific web application’s classloader.  
  - In this case, subsequent web applications will not be referenced by the cleaner thread, and their classloaders can be garbage collected.  
  - However, connection pools automatically removes idle connections after **30 minutes**. If this happens and the application gets a connection from the pool, then a new `PostgreSQL-JDBC-Cleaner` thread may be created under the web application’s classloader, reintroducing the classloader leak.

#### **Why the Problem Persists**  

- The PostgreSQL JDBC driver is registered **globally** in the application server.
- Connection pooling exacerbates the problem:  
  - If the application server manages database connections through a connection pool, the `PostgreSQL-JDBC-Cleaner` thread may outlive the applications using it.  
  - If the application itself manages the connections instead of relying on the server’s connection pool, the problem would not occur.  
- In a production application server, it is unlikely that **all** pooled connections would be closed. As a result, the leaked classloader remains in memory indefinitely unless the server is restarted.  

#### **Steps to Reproduce the Issue**  

1. Configure a **PostgreSQL JDBC datasource** in the application server.  
2. Deploy a web application that retrieves the datasource via **JNDI**.  
3. Execute a simple SQL statement within the application (e.g., `SELECT version();`).  
4. Undeploy the web application.  
5. Create a **heap dump** of the application server’s memory.  
    1. If a `PostgreSQL-JDBC-Cleaner` thread was created from a thread owned by the web application’s classloader, the web application’s classloader will remain in memory and **cannot be garbage collected** unless:  
        - The server is completely shut down.  
        - All database connections are explicitly closed and cleaned.  
    2. If a `PostgreSQL-JDBC-Cleaner` thread was created **before any web application deployment**, then until the `pgjdbc.config.cleanup.thread.ttl` expires, only the application server’s classloader is referenced in `inheritedAccessControlContext`, preventing web application classloader leaks.  

#### **Conclusion**  

The `PostgreSQL-JDBC-Cleaner` thread in PostgreSQL JDBC introduces a potential **classloader leak** when used with application servers that manage connection pooling. The issue arises because the cleaner thread inherits the `AccessControlContext` of the thread that creates it.  

- If the cleaner thread is initialized **after a web application gets a connection**, as a result, the web application’s classloader remains referenced indefinitely, leading to **memory leaks**.  
- If the cleaner thread is initialized **before any web application deploys**, the issue can be avoided, but idle connection removal mechanisms can reintroduce it over time.  

To mitigate this issue, possible workarounds include:  
- Ensuring that the `PostgreSQL-JDBC-Cleaner` thread is **initialized only once at the driver init** before any applications deploy. 

#### Used and tested JDBC drivers
- Postgres JDBC 42.7.5
- EDB JDBC 42.7.3.2

Collab: @bodzso

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Wed, 29 Jan 2025 20:39:57 +0000

wow!, nice thanks for the detective work and detailed report

---

From: "bataroland (@bataroland)" <[email-masked]>
Date: Thu, 27 Feb 2025 05:57:43 +0000

@davecramer any news about that? Do you need any further information?

---

From: "davecramer (@davecramer)" <[email-masked]>
Date: Thu, 27 Feb 2025 12:57:39 +0000

No news, just need more round tuit's. Any chance you could provide a PR ?

---

From: "vlsi (@vlsi)" <[email-masked]>
Date: Tue, 31 Mar 2026 06:06:58 +0000

`Thread.inheritedAccessControlContext` leak was fixed in https://github.com/pgjdbc/pgjdbc/pull/3886 (pgjdbc 42.7.10+)