agora inbox for pgsql-bugs@postgresql.org
help / color / mirror / Atom feedBUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
5+ messages / 5 participants
[nested] [flat]
* BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
@ 2026-08-04 12:27 PG Bug reporting form <noreply@postgresql.org>
2026-08-05 11:20 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x Артём Зарубин <a.prototype7@gmail.com>
2026-08-05 22:43 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x John Naylor <johncnaylorls@gmail.com>
0 siblings, 2 replies; 5+ messages in thread
From: PG Bug reporting form @ 2026-08-04 12:27 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: a.prototype7@gmail.com
The following bug has been logged on the website:
Bug reference: 19609
Logged by: Artem Zarubin
Email address: a.prototype7@gmail.com
PostgreSQL version: 19beta2
Operating system: Ubuntu 24.04.4 LTS (s390x)
Description:
Hello, I found a reproducible server crash when an SQL-language function is
executed with LLVM JIT enabled on s390x.
Tested commit:
bad: 0392fb900eb89f52988cccd33046443c39c70d1c, PostgreSQL 19devel
The server was built with:
./configure \
--prefix=/home/test/pginstall-vanilla18 \
--enable-debug \
--enable-cassert \
--with-llvm \
LLVM_CONFIG=/usr/bin/llvm-config-18 \
CLANG=/usr/bin/clang-18
Environment:
Architecture: s390x
OS: Ubuntu 24.04.4 LTS
Kernel: Linux 6.8.0-136-generic
GCC: 13.3.0
LLVM: 18.1.3
The following parameters are used by the attached TAP test:
restart_after_crash = on
jit = on
Minimal SQL script to reproduce:
CREATE FUNCTION type_text(oid) RETURNS text
LANGUAGE sql STABLE
AS $$
SELECT typname::text
FROM pg_catalog.pg_type
WHERE oid = $1
$$;
SET jit = on;
SET jit_above_cost = 0;
SET jit_inline_above_cost = -1;
SET jit_optimize_above_cost = -1;
SET jit_expressions = on;
SET jit_tuple_deforming = off;
SELECT count(*)
FROM (
SELECT oid
FROM pg_catalog.pg_type
ORDER BY oid
LIMIT 7
) AS t
WHERE type_text(t.oid) = 'int2vector';
Expected result:
count
-------
1
(1 row)
Actual result:
server closed the connection unexpectedly
This probably means the server terminated abnormally before or while
processing the request.
connection to server was lost
The server log contains:
LOG: client backend (PID ...) was terminated by signal 11: Segmentation
fault
DETAIL: Failed process was running: SELECT count(*) ...
Complete backtrace available from the core dump produced by the minimal
reproducer:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000002aa3d0f29bc in cstring_to_text (s=0x0) at varlena.c:186
#1 name_text (fcinfo=<optimized out>) at varlena.c:2709
#2 0x000003ff9ae36126 in ?? ()
Backtrace stopped: frame did not save the PC
si_signo = 11 (SIGSEGV)
si_code = 1 (SEGV_MAPERR)
si_addr = 0x0
r1 = 0x0
r11 = 0x0
pc = 0x2aa3d0f29bc <name_text+60>
The frame above name_text() contains JIT-generated code without unwind
information. name_text() receives a NULL C-string pointer and crashes in
cstring_to_text().
With jit=off, the same query completes successfully and returns 1. LLVM
inlining and PostgreSQL's expensive-query JIT optimization tier are not
required: the crash is also reproduced with jit_inline_above_cost and
jit_optimize_above_cost set to -1.
The attached patch adds the reproducer as
src/test/modules/test_misc/t/014_jit_s390x.pl. From the PostgreSQL source
tree, I ran it against the installed build with:
PG_INSTALL=/home/test/pginstall-vanilla18
PATH="$PG_INSTALL/bin:$PATH" \
PERL5LIB="$PWD/src/test/perl" \
PG_REGRESS="$PWD/src/test/regress/pg_regress" \
prove -v src/test/modules/test_misc/t/014_jit_s390x.pl
The TAP test fails as follows because the backend crashes:
not ok 1 - JIT-compiled SQL function expression does not crash the backend
got: '2'
expected: '0'
not ok 2 - JIT-compiled SQL function expression returns expected row
got: ''
expected: '1'
The crash reproduced on every run of the final minimal test on this machine,
including runs against newly initialized test clusters.
---
Best regards,
Artem Zarubin
Postgres Professional: https://postgrespro.com/
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
2026-08-04 12:27 BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x PG Bug reporting form <noreply@postgresql.org>
@ 2026-08-05 11:20 ` Артём Зарубин <a.prototype7@gmail.com>
2026-08-05 14:15 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 5+ messages in thread
From: Артём Зарубин @ 2026-08-05 11:20 UTC (permalink / raw)
To: a.prototype7@gmail.com; pgsql-bugs@lists.postgresql.org
Hello,
I am attaching the TAP test mentioned in the bug report.
The patch adds the self-contained reproducer as
src/test/modules/test_misc/t/014_jit_s390x.pl and registers it in the Meson
test list.
Best regards,
Artem Zarubin
Postgres Professional: https://postgrespro.com/
вт, 4 авг. 2026 г. в 19:28, PG Bug reporting form <noreply@postgresql.org>:
> The following bug has been logged on the website:
>
> Bug reference: 19609
> Logged by: Artem Zarubin
> Email address: a.prototype7@gmail.com
> PostgreSQL version: 19beta2
> Operating system: Ubuntu 24.04.4 LTS (s390x)
> Description:
>
> Hello, I found a reproducible server crash when an SQL-language function is
> executed with LLVM JIT enabled on s390x.
>
> Tested commit:
>
> bad: 0392fb900eb89f52988cccd33046443c39c70d1c, PostgreSQL 19devel
>
> The server was built with:
>
> ./configure \
> --prefix=/home/test/pginstall-vanilla18 \
> --enable-debug \
> --enable-cassert \
> --with-llvm \
> LLVM_CONFIG=/usr/bin/llvm-config-18 \
> CLANG=/usr/bin/clang-18
>
> Environment:
>
> Architecture: s390x
> OS: Ubuntu 24.04.4 LTS
> Kernel: Linux 6.8.0-136-generic
> GCC: 13.3.0
> LLVM: 18.1.3
>
> The following parameters are used by the attached TAP test:
>
> restart_after_crash = on
> jit = on
>
> Minimal SQL script to reproduce:
>
> CREATE FUNCTION type_text(oid) RETURNS text
> LANGUAGE sql STABLE
> AS $$
> SELECT typname::text
> FROM pg_catalog.pg_type
> WHERE oid = $1
> $$;
>
> SET jit = on;
> SET jit_above_cost = 0;
> SET jit_inline_above_cost = -1;
> SET jit_optimize_above_cost = -1;
> SET jit_expressions = on;
> SET jit_tuple_deforming = off;
>
> SELECT count(*)
> FROM (
> SELECT oid
> FROM pg_catalog.pg_type
> ORDER BY oid
> LIMIT 7
> ) AS t
> WHERE type_text(t.oid) = 'int2vector';
>
> Expected result:
>
> count
> -------
> 1
> (1 row)
>
> Actual result:
>
> server closed the connection unexpectedly
> This probably means the server terminated abnormally before or while
> processing the request.
> connection to server was lost
>
> The server log contains:
>
> LOG: client backend (PID ...) was terminated by signal 11: Segmentation
> fault
> DETAIL: Failed process was running: SELECT count(*) ...
>
> Complete backtrace available from the core dump produced by the minimal
> reproducer:
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x000002aa3d0f29bc in cstring_to_text (s=0x0) at varlena.c:186
> #1 name_text (fcinfo=<optimized out>) at varlena.c:2709
> #2 0x000003ff9ae36126 in ?? ()
> Backtrace stopped: frame did not save the PC
>
> si_signo = 11 (SIGSEGV)
> si_code = 1 (SEGV_MAPERR)
> si_addr = 0x0
>
> r1 = 0x0
> r11 = 0x0
> pc = 0x2aa3d0f29bc <name_text+60>
>
> The frame above name_text() contains JIT-generated code without unwind
> information. name_text() receives a NULL C-string pointer and crashes in
> cstring_to_text().
>
> With jit=off, the same query completes successfully and returns 1. LLVM
> inlining and PostgreSQL's expensive-query JIT optimization tier are not
> required: the crash is also reproduced with jit_inline_above_cost and
> jit_optimize_above_cost set to -1.
>
> The attached patch adds the reproducer as
> src/test/modules/test_misc/t/014_jit_s390x.pl. From the PostgreSQL source
> tree, I ran it against the installed build with:
>
> PG_INSTALL=/home/test/pginstall-vanilla18
> PATH="$PG_INSTALL/bin:$PATH" \
> PERL5LIB="$PWD/src/test/perl" \
> PG_REGRESS="$PWD/src/test/regress/pg_regress" \
> prove -v src/test/modules/test_misc/t/014_jit_s390x.pl
>
> The TAP test fails as follows because the backend crashes:
>
> not ok 1 - JIT-compiled SQL function expression does not crash the
> backend
> got: '2'
> expected: '0'
> not ok 2 - JIT-compiled SQL function expression returns expected row
> got: ''
> expected: '1'
>
> The crash reproduced on every run of the final minimal test on this
> machine,
> including runs against newly initialized test clusters.
>
> ---
> Best regards,
> Artem Zarubin
> Postgres Professional: https://postgrespro.com/
>
>
>
Attachments:
[application/octet-stream] pg_jit_s390x_minimal_tap.patch (2.1K, ../../CA+9qmHrrL5_PEhj8U4YtrmUhVM5o4_T+QiTsGh7=jb4qU4j=tg@mail.gmail.com/3-pg_jit_s390x_minimal_tap.patch)
download | inline diff:
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 969e90b396d..e1f5f915084 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -22,6 +22,7 @@ tests += {
't/011_lock_stats.pl',
't/012_ddlutils.pl',
't/013_temp_obj_multisession.pl',
+ 't/014_jit_s390x.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/014_jit_s390x.pl b/src/test/modules/test_misc/t/014_jit_s390x.pl
new file mode 100644
index 00000000000..e80bc29fdb5
--- /dev/null
+++ b/src/test/modules/test_misc/t/014_jit_s390x.pl
@@ -0,0 +1,63 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('jit_s390x');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf', qq{
+restart_after_crash = on
+jit = on
+});
+$node->start;
+
+my $version = $node->safe_psql('postgres', 'SELECT version()');
+plan skip_all => 's390x-specific LLVM JIT test'
+ unless $version =~ /s390x/;
+
+my $jit_available =
+ $node->safe_psql('postgres', 'SELECT pg_catalog.pg_jit_available()');
+plan skip_all => 'LLVM JIT is not available'
+ unless $jit_available eq 't';
+
+$node->safe_psql(
+ 'postgres',
+ q{
+CREATE FUNCTION type_text(oid) RETURNS text
+LANGUAGE sql STABLE
+AS $$
+ SELECT typname::text
+ FROM pg_catalog.pg_type
+ WHERE oid = $1
+$$;
+});
+
+my ($ret, $stdout, $stderr) = $node->psql(
+ 'postgres',
+ q{
+SET jit = on;
+SET jit_above_cost = 0;
+SET jit_inline_above_cost = -1;
+SET jit_optimize_above_cost = -1;
+SET jit_expressions = on;
+SET jit_tuple_deforming = off;
+
+SELECT count(*)
+FROM (
+ SELECT oid
+ FROM pg_catalog.pg_type
+ ORDER BY oid
+ LIMIT 7
+) AS t
+WHERE type_text(t.oid) = 'int2vector';
+});
+
+is($ret, 0, 'JIT-compiled SQL function expression does not crash the backend');
+is($stdout, "1\n", 'JIT-compiled SQL function expression returns expected row');
+diag($stderr) if $ret != 0 && $stderr ne '';
+
+done_testing();
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
2026-08-04 12:27 BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x PG Bug reporting form <noreply@postgresql.org>
2026-08-05 11:20 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x Артём Зарубин <a.prototype7@gmail.com>
@ 2026-08-05 14:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2026-08-05 20:04 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x Andrey Rachitskiy <pl0h0yp1@gmail.com>
0 siblings, 1 reply; 5+ messages in thread
From: Tom Lane @ 2026-08-05 14:15 UTC (permalink / raw)
To: Артём Зарубин <a.prototype7@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
=?UTF-8?B?0JDRgNGC0ZHQvCDQl9Cw0YDRg9Cx0LjQvQ==?= <a.prototype7@gmail.com> writes:
> I am attaching the TAP test mentioned in the bug report.
If the same test doesn't fail on other arches (x86, arm, etc) then
this is almost certainly an LLVM bug, and you should be reporting
it to them.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
2026-08-04 12:27 BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x PG Bug reporting form <noreply@postgresql.org>
2026-08-05 11:20 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x Артём Зарубин <a.prototype7@gmail.com>
2026-08-05 14:15 ` Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x Tom Lane <tgl@sss.pgh.pa.us>
@ 2026-08-05 20:04 ` Andrey Rachitskiy <pl0h0yp1@gmail.com>
0 siblings, 0 replies; 5+ messages in thread
From: Andrey Rachitskiy @ 2026-08-05 20:04 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Артём Зарубин <a.prototype7@gmail.com>; pgsql-bugs@lists.postgresql.org
Hi, Tom and Artem!
I looked at this on qemu-system-s390x (TCG),
Ubuntu 24.04, with an unpatched PG JIT emitter.
Arch test
---------------
Same PG sources and JIT GUCs (jit=on, jit_above_cost=0,
jit_inline_above_cost=-1, …, jit_dump_bitcode=off):
s390x TCG, LLVM 16.0.6 / 18.1.3 / 19.1.1 — crash
("server closed the connection" on clean builds)
x86_64, LLVM 18.1.3 and 20.1.8 — OK (8/8; EXPLAIN shows
JIT Functions: 3, Inlining false)
So the failure is arch-specific in the sense Tom meant. Caveat:
my s390x crashes are under TCG, not a real Z box (the original
report was on hardware). qemu-user did not reproduce, full-system
TCG did.
Repro
-----
CREATE FUNCTION type_text(oid) RETURNS text
LANGUAGE sql STABLE
AS $$ SELECT typname::text FROM pg_catalog.pg_type
WHERE oid = $1 $$;
SELECT type_text(22) FROM generate_series(1,7) g;
-- TAP form with LIMIT 7 / count(*) fails the same way
-- generate_series(1,6) / LIMIT 6 is fine; jit=off and
-- allowing inlining are fine
With a diagnostic build, name_text sees a NULL byref Datum with
isnull=false on the 7th call. jit_dump_bitcode=on often hides
the bug.
Where it breaks (short)
-----------------------
Rough IR:
store i64 %d, ptr inttoptr (i64 <op->resvalue>)
call i64 @name_text(ptr inttoptr (i64 <fcinfo>))
Slot content on call #7 is still good (C read). Reloading
tts_values in IR does not help. A C helper that writes through
the resvalue pointer can make name_text see a good Datum. The IR
store via baked inttoptr does not. On call #7 I also saw live
ExprState/fcinfo addresses differ from earlier calls — so the
JIT is using a stronger "freeze absolute addresses at first
compile" contract than the interpreter.
I tried a tiny standalone clang test on the same s390x TCG guest
(store / store+call through compile-time inttoptr to a stable
heap cell, 8 times). That did *not* fail. So I do not yet have
a reduced LLVM testcase, a naive inttoptr store is not enough.
As an academic check only: if the JIT loads resvalue / resnull /
fcinfo from the live ExprState each step (IR GEP/load from
state->steps[opno]…) instead of baking them as l_ptr_const /
inttoptr, the repro stopped failing on my harness. I am not
proposing that as a patch — just a data point that the failure
is tied to the baked-address model on this target.
-------
Please treat QEMU TCG as a possible confounder
until someone confirms on real Z (original reporter: hardware,
LLVM 18.1.3).
I am filing this mainly as investigation notes. If any of it
helps with the LLVM report or with later PG work, good.
ср, 5 авг. 2026 г. в 19:15, Tom Lane <tgl@sss.pgh.pa.us>:
> =?UTF-8?B?0JDRgNGC0ZHQvCDQl9Cw0YDRg9Cx0LjQvQ==?= <a.prototype7@gmail.com>
> writes:
> > I am attaching the TAP test mentioned in the bug report.
>
> If the same test doesn't fail on other arches (x86, arm, etc) then
> this is almost certainly an LLVM bug, and you should be reporting
> it to them.
>
> regards, tom lane
>
>
>
--
Regards,
Rachitskiy Andrey
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
2026-08-04 12:27 BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x PG Bug reporting form <noreply@postgresql.org>
@ 2026-08-05 22:43 ` John Naylor <johncnaylorls@gmail.com>
1 sibling, 0 replies; 5+ messages in thread
From: John Naylor @ 2026-08-05 22:43 UTC (permalink / raw)
To: a.prototype7@gmail.com; pgsql-bugs@lists.postgresql.org
On Wed, Aug 5, 2026 at 3:50 PM PG Bug reporting form
<noreply@postgresql.org> wrote:
> Architecture: s390x
> OS: Ubuntu 24.04.4 LTS
> Kernel: Linux 6.8.0-136-generic
> GCC: 13.3.0
> LLVM: 18.1.3
FWIW, I've found that clang 18 and early minors of clang 19 have other
kinds of broken code generation on this architecture.
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2026-08-05 22:43 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-08-04 12:27 BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x PG Bug reporting form <noreply@postgresql.org>
2026-08-05 11:20 ` Артём Зарубин <a.prototype7@gmail.com>
2026-08-05 14:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2026-08-05 20:04 ` Andrey Rachitskiy <pl0h0yp1@gmail.com>
2026-08-05 22:43 ` John Naylor <johncnaylorls@gmail.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox