Date: Tue, 8 Mar 2022 21:15:59 +0100
From: hubert depesz lubaczewski <depesz@depesz.com>
Sender: depesz@depesz.com
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>,
	PostgreSQL General <pgsql-general@lists.postgresql.org>
Subject: Re: Interesting fail when migrating Pg from Ubuntu Bionic to Focal
Message-ID: <20220308201559.GA27977@depesz.com>
Reply-To: depesz@depesz.com
References: <20220303151156.GB26036@depesz.com>
 <20220303163921.GA10776@depesz.com>
 <20220303164401.GA13056@depesz.com>
 <3969790.1646420992@sss.pgh.pa.us>
 <20220304204510.GA21340@depesz.com>
 <3984384.1646431394@sss.pgh.pa.us>
 <20220306140402.GA6360@depesz.com>
 <2045.1646583000@sss.pgh.pa.us>
 <20220307151128.GA22074@depesz.com>
 <20220307175422.3iuxiiiypfowx3ch@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20220307175422.3iuxiiiypfowx3ch@alap3.anarazel.de>
User-Agent: Mutt/1.10.1 (2018-07-13)
Archived-At: 
 <https://www.postgresql.org/message-id/20220308201559.GA27977%40depesz.com>
Precedence: bulk

On Mon, Mar 07, 2022 at 09:54:22AM -0800, Andres Freund wrote:
> > Initially select didn't break anything, but when I tuned down
> > jit_above_cost so that it will kick in - got fails immediately.
> Could you set jit_debugging_support=on and show a backtrace with that?

Here you go:
Program received signal SIGSEGV, Segmentation fault.
0x0000fffe7587b6c8 in deform_0_8 ()
(gdb) bt
#0  0x0000fffe7587b6c8 in deform_0_8 ()
#1  0x0000fffe7587b5f4 in evalexpr_0_7 ()
#2  0x0000aaaab67e6f7c in ExecEvalExprSwitchContext (isNull=0xfffff7956217, econtext=0xaaaaf6014608, state=0xaaaaf6014ab0) at ./build/../src/include/executor/executor.h:316
#3  ExecProject (projInfo=0xaaaaf6014aa8) at ./build/../src/include/executor/executor.h:350
#4  ExecScan (node=<optimized out>, accessMtd=0xaaaab68051b8 <SeqNext>, recheckMtd=0xaaaab6805248 <SeqRecheck>) at ./build/../src/backend/executor/execScan.c:239
#5  0x0000aaaab67ded5c in ExecProcNode (node=0xaaaaf60143b8) at ./build/../src/include/executor/executor.h:242
#6  ExecutePlan (execute_once=<optimized out>, dest=0xaaaaf5fc15e8, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0xaaaaf60143b8, estate=0xaaaaf5e01360)
    at ./build/../src/backend/executor/execMain.c:1632
#7  standard_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=<optimized out>, count=0, execute_once=<optimized out>) at ./build/../src/backend/executor/execMain.c:350
#8  0x0000ffffa87751fc in pgss_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=ForwardScanDirection, count=0, execute_once=true) at ./build/../contrib/pg_stat_statements/pg_stat_statements.c:893
#9  0x0000ffffa875f688 in explain_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=ForwardScanDirection, count=0, execute_once=true) at ./build/../contrib/auto_explain/auto_explain.c:320
#10 0x0000aaaab69354c8 in PortalRunSelect (portal=portal@entry=0xaaaaf5e83ef0, forward=forward@entry=true, count=0, count@entry=9223372036854775807, dest=dest@entry=0xaaaaf5fc15e8) at ./build/../src/backend/tcop/pquery.c:938
#11 0x0000aaaab6936a64 in PortalRun (portal=portal@entry=0xaaaaf5e83ef0, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true, dest=dest@entry=0xaaaaf5fc15e8, altdest=altdest@entry=0xaaaaf5fc15e8, 
    completionTag=completionTag@entry=0xfffff7956848 "") at ./build/../src/backend/tcop/pquery.c:779
#12 0x0000aaaab6932a18 in exec_simple_query (query_string=query_string@entry=0xaaaaf5d9af90 "select p.proname, (SELECT rolname from pg_catalog.pg_roles where oid = p.proowner) from pg_proc p;") at ./build/../src/backend/tcop/postgres.c:1215
#13 0x0000aaaab69336ac in PostgresMain (argc=<optimized out>, argv=argv@entry=0xaaaaf5df6f18, dbname=<optimized out>, username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4271
#14 0x0000aaaab68ba768 in BackendRun (port=0xaaaaf5df0910, port=0xaaaaf5df0910) at ./build/../src/backend/postmaster/postmaster.c:4510
#15 BackendStartup (port=0xaaaaf5df0910) at ./build/../src/backend/postmaster/postmaster.c:4193
#16 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1725
#17 0x0000aaaab68bb74c in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at ./build/../src/backend/postmaster/postmaster.c:1398
#18 0x0000aaaab664c8a8 in main (argc=5, argv=0xaaaaf5d94ea0) at ./build/../src/backend/main/main.c:228

> > > FWIW, I went to the trouble of installing Ubuntu Focal on my
> > > raspberry pi to see if I could duplicate this, and I couldn't.
> > > However, what you get from a fresh install now is
> > > $ dpkg -l | grep libllvm                  
> > > ii  libllvm10:arm64                1:10.0.0-4ubuntu1                     arm64        Modular compiler and toolchain technologies, runtime library
> > > not 9.0.1.  I also found that Fedora 31/aarch64 is still downloadable, and
> > > that does contain LLVM 9 ... and I could not reproduce it there either.
> > 
> > Soo... plot thickens.
> > 
> > Looks that pg 12 supplied by pgdg required libllvm9:
> What are the libllvm package versions on which you can reproduce the crash?

Hmm .. not sure I fully understand. Here are all packages that I think are relevant:

root@c999-pg1302:~# dpkg -l | grep -iE 'postgresql-12|llvm|clang'
ii  libllvm9:arm64                     1:9.0.1-12                            arm64        Modular compiler and toolchain technologies, runtime library
ii  postgresql-12                      12.9-2.pgdg20.04+1                    arm64        The World's Most Advanced Open Source Relational Database
ii  postgresql-12-dbgsym               12.9-2.pgdg20.04+1                    arm64        debug symbols for postgresql-12
ii  postgresql-12-pg-collkey           0.5.1-1insops1                        arm64        ICU collation function wrapper for PostgreSQL 12
ii  postgresql-12-postgis-2.5          2.5.5+dfsg-1.pgdg20.04+2              arm64        Geographic objects support for PostgreSQL 12
ii  postgresql-12-postgis-2.5-scripts  2.5.5+dfsg-1.pgdg20.04+2              all          Geographic objects support for PostgreSQL 12 -- SQL scripts
ii  postgresql-12-repack               1.4.7-2.pgdg20.04+1                   arm64        reorganize tables in PostgreSQL databases with minimal locks
ii  postgresql-12-system-stats         1.0.0-1                               arm64        A packaged version of EnterpriseDB's system_stats postgres extension

Hope it helps.

Best regards,

depesz