public inbox for [email protected]help / color / mirror / Atom feed
Re: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed 4+ messages / 3 participants [nested] [flat]
* Re: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed @ 2026-02-17 18:47 Tom Lane <[email protected]> 0 siblings, 2 replies; 4+ messages in thread From: Tom Lane @ 2026-02-17 18:47 UTC (permalink / raw) To: Matt Carter <[email protected]>; +Cc: [email protected] <[email protected]> Matt Carter <[email protected]> writes: > Tom Lane <[email protected]> wrote: >> I think there are way too many moving parts here, and too few configuration details, >> to allow assigning blame confidently. > Thank you for taking the time to test this and for the feedback. Your C test showing no leak suggests the issue is specific to how psycopg2 uses libpq, not libpq itself. I apologize for not including enough environmental details. I used Kerberos/GSSAPI with SSL (TLS 1.2 connections). My connection string was: "postgresql://hostname/database" (no password, Kerberos auth). > Your mention of "years ago libpq did leak memory while using GSSAPI encryption" is interesting because we ARE using GSSAPI/Kerberos authentication. Interesting. I wondered about GSSAPI, but spinning up such an environment is more work than I wanted to do on speculation. > I can test with non-GSSAPI authentication to try to isolate that variable. I can also create a pure psycopg2 reproducer (without SQLAlchemy). I can also test whether disabling GSSAPI encryption (but keeping GSSAPI auth) changes the behavior. Would testing with GSSAPI authentication help narrow this down? I can also report this to the psycopg2 project if you think it's their issue. Please try varying the connection type and encryption. I do suspect this may be psycopg2's fault, but we lack enough data to pin blame as yet. regards, tom lane ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed @ 2026-02-17 19:45 Andres Freund <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 4+ messages in thread From: Andres Freund @ 2026-02-17 19:45 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Matt Carter <[email protected]>; [email protected] <[email protected]> Hi, On 2026-02-17 13:47:49 -0500, Tom Lane wrote: > Matt Carter <[email protected]> writes: > > Thank you for taking the time to test this and for the feedback. Your C test showing no leak suggests the issue is specific to how psycopg2 uses libpq, not libpq itself. I apologize for not including enough environmental details. I used Kerberos/GSSAPI with SSL (TLS 1.2 connections). My connection string was: "postgresql://hostname/database" (no password, Kerberos auth). > > Your mention of "years ago libpq did leak memory while using GSSAPI encryption" is interesting because we ARE using GSSAPI/Kerberos authentication. > > Interesting. I wondered about GSSAPI, but spinning up such an > environment is more work than I wanted to do on speculation. Heh, understandable... > > I can test with non-GSSAPI authentication to try to isolate that variable. I can also create a pure psycopg2 reproducer (without SQLAlchemy). I can also test whether disabling GSSAPI encryption (but keeping GSSAPI auth) changes the behavior. Would testing with GSSAPI authentication help narrow this down? I can also report this to the psycopg2 project if you think it's their issue. > > Please try varying the connection type and encryption. I do suspect > this may be psycopg2's fault, but we lack enough data to pin blame > as yet. Matt, you could try analyzing the memory usage with heaptrack, it tends to be pretty good at finding them even in uninstrumented builds, as long as enough debug symbols for a backtrace are available. Often enough it'll pinpoint where the leak is coming from quite easily (but note that it will report some constant-sized leaks that are "intentional"). Greetings, Andres ^ permalink raw reply [nested|flat] 4+ messages in thread
* RE: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed @ 2026-02-17 20:54 Matt Carter <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 4+ messages in thread From: Matt Carter @ 2026-02-17 20:54 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> Tom Lane <[email protected]> wrote: > Please try varying the connection type and encryption. Good idea. Here's what I found: Test 1: Configuration: Original (SQLAlchemy + NullPool) libpq: 16.0.7 SQLAlchemy: Yes Leak Rate: 803 KB/min Test 2: Configuration: Pure psycopg2 (default) libpq: 16.0.7 SQLAlchemy: NO Leak Rate: 801 KB/min Test 3: Configuration: Pure psycopg2 + gssencmode=disable libpq: 16.0.7 SQLAlchemy: NO Leak Rate: 858 KB/min Test 4: Configuration: Pure psycopg2 + SSL only libpq: 16.0.7 SQLAlchemy: NO Leak Rate: 861 KB/min Test 5: Configuration: Pure psycopg2 libpq: 13.0.11 SQLAlchemy: NO Leak Rate: 17 KB/min So, it seems none of these changes avoids the leak: - Removing SQLAlchemy - Disabling GSSAPI encryption - Using SSL-only - Changing PostgreSQL DB Server version from 16 to 13 The only changes that I found that avoid the leak are: - Changing libpq version from 16 to 13, or - Changing psycopg version from 2 to 3. Tom, since your C test showed no leak, the issue is likely in how psycopg2 calls libpq, not pure libpq itself. I guess I should report this to the psycopg2 project. ^ permalink raw reply [nested|flat] 4+ messages in thread
* RE: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed @ 2026-02-17 23:12 Matt Carter <[email protected]> parent: Matt Carter <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Matt Carter @ 2026-02-17 23:12 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> Matt Carter wrote: > Tom, since your C test showed no leak, the issue is likely in how psycopg2 calls libpq, not pure libpq itself. FYI, I have reported the issue to the psycopg project: https://github.com/psycopg/psycopg2/issues/1827 Thanks you for your help, Tom. Your test results helped show that this is likely a psycopg2 issue. Cheers, Matt ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-02-17 23:12 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-02-17 18:47 Re: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed Tom Lane <[email protected]> 2026-02-17 19:45 ` Andres Freund <[email protected]> 2026-02-17 20:54 ` Matt Carter <[email protected]> 2026-02-17 23:12 ` Matt Carter <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox