public inbox for [email protected]
help / color / mirror / Atom feedUse-after-free crash
5+ messages / 2 participants
[nested] [flat]
* Use-after-free crash
@ 2026-06-11 07:59 Emond Papegaaij <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Emond Papegaaij @ 2026-06-11 07:59 UTC (permalink / raw)
To: [email protected]
Hi,
Yesterday one of our tests detected a segmentation fault in pgpool,
caused by a use-after-free. It is triggered during the shutdown of the
process, so it probably isn't too harmful, but still I think it should
be fixed. I've worked with Claude Code to build a patch (against 4.7)
and a reproducer of the problem. The patch also contains a detailed
explanation of how this crashes in the commit message. To run the
reproducer:
tar xzf pgpool-uaf-repro.tar.gz
cd pgpool-uaf-repro
PGPOOL=/path/to/their/pgpool ./reproduce.sh
Best regards,
Emond
Attachments:
[text/x-patch] 0001-Fix-use-after-free-of-query-context-after-a-backend-.patch (4.4K, ../../CAGXsc+aFfv2_PUrEdCt8TousYP9dPmty3i+0mkJpWwVg=fVN+Q@mail.gmail.com/2-0001-Fix-use-after-free-of-query-context-after-a-backend-.patch)
download | inline diff:
From 9b4f7d5f4f3a8ebadcabda6f1993790441a75b80 Mon Sep 17 00:00:00 2001
From: Emond Papegaaij <[email protected]>
Date: Thu, 11 Jun 2026 09:37:44 +0200
Subject: [PATCH] Fix use-after-free of query context after a backend node
shutdown
A POOL_QUERY_CONTEXT is referenced from the session scoped sent message
list (session_context->message_list) and pending message list, which
live in session_context->memory_context and survive for the whole
client session. However pool_init_query_context() allocated the query
context in the global QueryContext memory context.
do_child() resets QueryContext (MemoryContextResetAndDeleteChildren)
at the top of every iteration of its query processing loop. In normal
operation pool_process_query() runs the whole session in a single
iteration, so QueryContext is effectively session lived and this is
harmless. But when pool_process_query() returns POOL_CONTINUE in the
middle of a session, the loop iterates and QueryContext is reset while
the sent/pending message lists still reference query contexts that were
allocated in it. The freed query contexts are later dereferenced (for
example from pool_clear_sent_message_list() in reset_connection(), or
from pool_remove_sent_message() while binding a reused statement name),
causing a use-after-free and a SIGSEGV.
This is reachable when a backend node is shut down by an administrative
command (e.g. "fast" shutdown of PostgreSQL) while a client session that
has named prepared statements is open. read_packets_and_process()
detects the admin shutdown, sets was_error and returns POOL_CONTINUE
(cont = false), so pool_process_query() returns to do_child() mid
session and the next loop iteration frees the still referenced query
contexts.
Fix this by allocating the query context under the session memory
context rather than the per-iteration QueryContext, so its lifetime
matches the session scoped lists that reference it. When there is no
session context (internal queries issued before a session exists) fall
back to QueryContext, preserving the previous behaviour. Query contexts
are still freed explicitly by pool_query_context_destroy(), and at the
latest when the session memory context is destroyed, so this does not
change memory reclamation for the normal single-iteration case (in which
QueryContext was likewise only freed at session end).
---
src/context/pool_query_context.c | 34 +++++++++++++++++++++++++-------
1 file changed, 27 insertions(+), 7 deletions(-)
diff --git a/src/context/pool_query_context.c b/src/context/pool_query_context.c
index a056ac596..7eef39367 100644
--- a/src/context/pool_query_context.c
+++ b/src/context/pool_query_context.c
@@ -78,15 +78,35 @@ static char *get_associated_object_from_dml_adaptive_relations
POOL_QUERY_CONTEXT *
pool_init_query_context(void)
{
- MemoryContext memory_context = AllocSetContextCreate(QueryContext,
- "QueryContextMemoryContext",
- ALLOCSET_SMALL_MINSIZE,
- ALLOCSET_SMALL_INITSIZE,
- ALLOCSET_SMALL_MAXSIZE);
-
- MemoryContext oldcontext = MemoryContextSwitchTo(memory_context);
+ POOL_SESSION_CONTEXT *session_context;
+ MemoryContext parent;
+ MemoryContext memory_context;
+ MemoryContext oldcontext;
POOL_QUERY_CONTEXT *qc;
+ /*
+ * Parent the query context under the session memory context rather than
+ * the per-iteration QueryContext. A query context is referenced from the
+ * session scoped sent message list and pending message list. These lists
+ * outlive the QueryContext, which do_child() resets between the iterations
+ * of its query processing loop (for instance when pool_process_query()
+ * returns after a backend node was shut down). If the query context lived
+ * in QueryContext, such a reset would free it while the message lists
+ * still reference it, resulting in a use-after-free. When there is no
+ * session context (e.g. internal queries issued before a session exists),
+ * fall back to QueryContext.
+ */
+ session_context = pool_get_session_context(true);
+ parent = session_context ? session_context->memory_context : QueryContext;
+
+ memory_context = AllocSetContextCreate(parent,
+ "QueryContextMemoryContext",
+ ALLOCSET_SMALL_MINSIZE,
+ ALLOCSET_SMALL_INITSIZE,
+ ALLOCSET_SMALL_MAXSIZE);
+
+ oldcontext = MemoryContextSwitchTo(memory_context);
+
qc = palloc0(sizeof(*qc));
qc->memory_context = memory_context;
MemoryContextSwitchTo(oldcontext);
--
2.53.0
[application/gzip] pgpool-uaf-repro.tar.gz (9.6K, ../../CAGXsc+aFfv2_PUrEdCt8TousYP9dPmty3i+0mkJpWwVg=fVN+Q@mail.gmail.com/3-pgpool-uaf-repro.tar.gz)
download
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Use-after-free crash
@ 2026-06-11 23:34 Tatsuo Ishii <[email protected]>
parent: Emond Papegaaij <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Tatsuo Ishii @ 2026-06-11 23:34 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Hi Emond,
> Hi,
>
> Yesterday one of our tests detected a segmentation fault in pgpool,
> caused by a use-after-free. It is triggered during the shutdown of the
> process, so it probably isn't too harmful, but still I think it should
> be fixed. I've worked with Claude Code to build a patch (against 4.7)
> and a reproducer of the problem. The patch also contains a detailed
> explanation of how this crashes in the commit message. To run the
> reproducer:
> tar xzf pgpool-uaf-repro.tar.gz
> cd pgpool-uaf-repro
> PGPOOL=/path/to/their/pgpool ./reproduce.sh
>
> Best regards,
> Emond
Thanks for the report. I will look into this.
Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Use-after-free crash
@ 2026-06-15 10:07 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Tatsuo Ishii @ 2026-06-15 10:07 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Hi Emond,
>> Hi,
>>
>> Yesterday one of our tests detected a segmentation fault in pgpool,
>> caused by a use-after-free. It is triggered during the shutdown of the
>> process, so it probably isn't too harmful, but still I think it should
>> be fixed. I've worked with Claude Code to build a patch (against 4.7)
>> and a reproducer of the problem. The patch also contains a detailed
>> explanation of how this crashes in the commit message. To run the
>> reproducer:
>> tar xzf pgpool-uaf-repro.tar.gz
>> cd pgpool-uaf-repro
>> PGPOOL=/path/to/their/pgpool ./reproduce.sh
>>
>> Best regards,
>> Emond
>
> Thanks for the report. I will look into this.
After reading README.md, I tried to build ASan enabled pgpool but failed:
LANG=C make -C src CFLAGS="-g -O1 -fsanitize=address -fno-omit-frame-pointer \
-Wno-format-truncation -Wno-stringop-truncation -fno-strict-aliasing" \
LIBS="-lcrypt -lm" pgpool
:
:
gcc -DHAVE_CONFIG_H -DDEFAULT_CONFIGDIR=\"/usr/local/etc\" -DPGSQL_BIN_DIR=\"/usr/local/pgsql/bin\" -I. -I../src/include -D_GNU_SOURCE -I /usr/local/pgsql/include -g -O1 -fsanitize=address -fno-omit-frame-pointer -Wno-format-truncation -Wno-stringop-truncation -fno-strict-aliasing -c -o utils/pg_prng.o utils/pg_prng.c
make: *** No rule to make target 'parser/libsql-parser.a', needed by 'pgpool'. Stop.
make: Leaving directory '/home/t-ishii/work/Pgpool-II/current/pgpool2/src'
So I modifed the src/Makefile
#CFLAGS = -g -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -Wno-format-truncation -Wno-stringop-truncation -fno-strict-aliasing
CFLAGS = -g -O1 -fsanitize=address -fno-omit-frame-pointer \
-Wmissing-prototypes -Wmissing-declarations \
-Wno-format-truncation -Wno-stringop-truncation -fno-strict-aliasing
and succeeded in building ASan enabled pgpool.
Then run the reproducer (pgpool was installed as /usr/local/bin/pgpool)
$ ./reproduce.sh
pgpool : /usr/local/bin/pgpool
pgpool-II version 4.8devel (mitsukakeboshi)
pgbin : /usr/local/pgsql/bin
postgres (PostgreSQL) 18.4
workdir: /tmp/pgpool-uaf-kYI6hy
--- running client ---
session ready; firing trigger
client done
--- result ---
RESULT: CRASH REPRODUCED (use-after-free / signal)
268:==434672==ERROR: AddressSanitizer: heap-use-after-free on address 0x51d0000215a0 at pc 0x5cdb1e84b6ab bp 0x7ffc54297120 sp 0x7ffc54297110
270: #0 0x5cdb1e84b6aa in pool_query_context_destroy context/pool_query_context.c:106
274: #4 0x5cdb1e8447e8 in pool_clear_sent_message_list context/pool_session_context.c:500
275: #5 0x5cdb1e7fa679 in reset_connection protocol/pool_process_query.c:1041
300:SUMMARY: AddressSanitizer: heap-use-after-free context/pool_query_context.c:106 in pool_query_context_destroy
334:2026-06-15 10:21:53.476: main pid 434668: DEBUG: child process with pid: 434672 exits with status 134 by signal 6
ASan deteced heap-use-after-free as expected.
After the patch, no crash occured.
./reproduce.sh
pgpool : /usr/local/bin/pgpool
pgpool-II version 4.8devel (mitsukakeboshi)
pgbin : /usr/local/pgsql/bin
postgres (PostgreSQL) 18.4
workdir: /tmp/pgpool-uaf-XvNAWK
--- running client ---
session ready; firing trigger
client done
--- result ---
RESULT: no crash (this is the expected outcome on a FIXED pgpool)
253:2026-06-15 11:59:49.821: [unknown] pid 530619: LOG: reading and processing packets
254:2026-06-15 11:59:49.821: [unknown] pid 530619: DETAIL: postmaster on DB node 0 was shutdown by administrative command
262:2026-06-15 11:59:51.319: [unknown] pid 530619: WARNING: write on backend 0 failed with error :"Connection reset by peer"
268:2026-06-15 11:59:51.319: [unknown] pid 530619: WARNING: write on backend 0 failed with error :"Broken pipe"
270:2026-06-15 11:59:51.319: [unknown] pid 530619: WARNING: write on backend 0 failed with error :"Broken pipe"
Also I ran our regression test with the patch and all the tests
passed. I read README.md and the patch and the explanation and the
patch look correct to me. I am going to push the patch.
Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Use-after-free crash
@ 2026-06-15 18:51 Emond Papegaaij <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Emond Papegaaij @ 2026-06-15 18:51 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]
Hi Tatsuo,
Thanks for including this. I think Claude made a mistake cleaning up
the build files to produce a clean tarball that was not linked to my
local system. I'm glad you got it to work. I must admit Claude Code is
really impressive with its ability to troubleshoot issues like these.
In this case, all it had was a single core dump and the test scenario.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Use-after-free crash
@ 2026-06-18 09:39 Tatsuo Ishii <[email protected]>
parent: Emond Papegaaij <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Tatsuo Ishii @ 2026-06-18 09:39 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Hi Emond,
> Hi Tatsuo,
>
> Thanks for including this. I think Claude made a mistake cleaning up
> the build files to produce a clean tarball that was not linked to my
> local system. I'm glad you got it to work. I must admit Claude Code is
> really impressive with its ability to troubleshoot issues like these.
> In this case, all it had was a single core dump and the test scenario.
> From there, it managed to find the issue, create a fix and build the
> reproducer.
I have pushed the patch to all supported branches. Thanks!
Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2026-06-18 09:39 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-06-11 07:59 Use-after-free crash Emond Papegaaij <[email protected]>
2026-06-11 23:34 ` Tatsuo Ishii <[email protected]>
2026-06-15 10:07 ` Tatsuo Ishii <[email protected]>
2026-06-15 18:51 ` Emond Papegaaij <[email protected]>
2026-06-18 09:39 ` Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox