Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1s9hoa-004SVI-Jh for pgsql-bugs@arkaria.postgresql.org; Wed, 22 May 2024 08:59:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1s9hoa-00F7od-N6 for pgsql-bugs@arkaria.postgresql.org; Wed, 22 May 2024 08:59:16 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1s9fe8-00EMJE-DT for pgsql-bugs@lists.postgresql.org; Wed, 22 May 2024 06:40:20 +0000 Received: from mr3.vodafonemail.de ([145.253.228.163]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1s9fe0-001POd-Nk for pgsql-bugs@lists.postgresql.org; Wed, 22 May 2024 06:40:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arcor.de; s=vfde-mb-mr2-23sep; t=1716360009; bh=VMGtaj5GUQc+30BE0dyQ6Ai/eaxEVUzx6lK4OUHCP/0=; h=In-Reply-To:From:To:Message-ID:Date:Subject:Content-type:From; b=V5T6qUcVFYpSFz2PipgHxeDoqUDywshtL8Me+WDeKMXmfAXflgRt/GjOFuhkYAkTL 0JZzpLkJYOEHsmLRqoDhaI6+cAzWpR+PORaT21NvBG2svpcG8fBUt2+318PmOQ5n+2 VQeLSzWatF9ZCde+oGkGSWaBpwbLmFzhj2KJyUMs= Received: from mp102.fra-mediabeam.com (unknown [10.0.0.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mr3.vodafonemail.de (Postfix) with ESMTPS id 4VkhWD5KdXz2Frb; Wed, 22 May 2024 06:40:08 +0000 (UTC) Received: from mailbackend01 (unknown [10.0.0.2]) by mp102.fra-mediabeam.com (Postfix) with ESMTP id 4VkhWD3Xy9z8t8Q; Wed, 22 May 2024 06:40:08 +0000 (UTC) In-Reply-To: Received: from [46.142.162.141] by mail.vodafone.de via HTTP; Wed, 22 May 2024 08:40:07 +0200 From: joachim.haecker-becker@arcor.de To: "Dmitry Dolgov" <9erthalion6@gmail.com>, pgsql-bugs@lists.postgresql.org Message-ID: <673ce6ea298c4f989a34d63e1d13a8fa@arcor.de> Date: Wed, 22 May 2024 08:40:07 +0200 X-Priority: 3 Subject: AW: Re: BUG #18471: Possible JIT memory leak resulting in signal 11:Segmentation fault on ARM MIME-Version: 1.0 Content-type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-purgate-type: clean X-purgate: clean X-purgate-size: 7534 X-purgate-ID: 155817::1716360008-50B6E11D-9A87199D/0/0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk
Hi Dmitry,

thanks for = looking into this=2E 

Maybe it is a combination of JIT and = some other postgres config changes we have in our environment?
I wil= l try to reproduce with a blank config and only change the JIT settings= =2E

This is where the source is <> default:
 namesettingunitautovacuum_analyze_scale_factor0= =2E03 autov= acuum_max_workers6 autovacuum_naptime300sautovac= uum_vacuum_insert_scale_factor0=2E05&= nbsp;autovacuum_vacuum_sc= ale_factor0=2E03 autovacuum_vacuum_threshold1000 client_connection_check_interval30000msdefault_text_searc= h_configpg_catalog=2Eenglish dynamic_shared_memory_typeposix effective_cach= e_size10485768kBenable_partitionwise_aggregateon enable_partitionw= ise_joinon hash_mem_multiplier1=2E5&nbs= p;jiton=  jit_above_cost= 1 jit_inline_above_cost1&nbs= p;jit_optimize_above_cost= 1 listen_addresses* log_destinationjsonlog log_file_mode= 640 log_lock_waitson log_rotation_size1= 02400kBlog_timez= oneEtc/UTC logging_collectoron maintenance_work_mem1048576kBmax_co= nnections150 max_locks_per_transaction1024 = max_parallel_workers8 = max_parallel_workers_per_gather2 max_wal_size2048MBmin_wal_size80MBrandom_= page_cost1 shared_buffers7864328kBTimeZoneEtc/UTC wo= rk_mem512000kB

The docker container has a 6gb shm_size=2E

Let me kno= w if there is anything else I can provide to get this resolved=2E
 
Von: Dmitry Dolgov <9erthalion6@gmail=2Eco= m>
Ge= sendet: 21=2E05=2E2024 18:08
An: <joachim=2Ehaecker-becker@arcor=2Ed= e>,<pgsql-bugs@lists=2Epostgresql=2Eorg>
Betreff: Re: BUG #18471: = Possible JIT memory leak resulting in signal 11:Segmentation fault on A= RM
 
> On Fri, May 17, 2024 at 01:13:06PM +0000, PG Bu= g reporting form wrote:
> The following bug has been logged on th= e website:
>
> Bug reference: 18471
> Logged by: Joac= him Haecker-Becker
> Email address: joachim=2Ehaecker-becker@arcor=2Ede
&g= t; PostgreSQL version: 16=2E3
> Operating system: Debian Bookworm=
> Description:
>
> We have a reproducible way to for= ce a postgres process to consume more and
> more RAM until it cra= shes on ARM=2E
> The same works on X86 without any issue=2E
&g= t; With jit=3Doff it runs on ARM as well=2E
>
> We run into= this situation in a real-life database situation with a lot of
>= joins and aggregate functions=2E
> The following code is just a = mock to reproduce a similar situation without
> needing access to= our real data=2E
> This issue blocks us from upgrading or ARM-ho= sted databases into something
> newer than 14=2E7=2E

I thi= nk it would be useful to know how much memory difference are we
talk= ing about and, just to make everything clear, how exactly postgres
c= rashes (OOM kill I assume)? It's important to differentiate between the=
case "ARM with jit crashes, ARM without jit doesn't" and "ARM with = jit
crashes, ARM without jit crashes with even more columns" (the sa= me goes
for x86)=2E

I've tried to reproduce it on an arm64 VM= (16=2E3 build with llvm 17), and
although I could observe some diff= erence in memory consumption between
JIT on/off, but it wasn't huge = (around 10% or so)=2E Running it under
valgrind shows only complains= about memory allocated for bitcode
modules, which is expected -- as= far as I recall postgres is somewhat
wasteful when it comes to allo= cating memory for those modules, even more
so for parallel workers=2E= This is the case here, where there is growing
number of parallel ha= sh workers=2E This would not explain any difference
from x86 of cour= se, but there might be different baseline memory
consumption for dif= ferent architectures=2E