Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sHgs3-00A7gl-3F for pgsql-bugs@arkaria.postgresql.org; Thu, 13 Jun 2024 09:35:51 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sHgs0-00GPne-Ku for pgsql-bugs@arkaria.postgresql.org; Thu, 13 Jun 2024 09:35:49 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sHdno-00DxOv-VG for pgsql-bugs@lists.postgresql.org; Thu, 13 Jun 2024 06:19:18 +0000 Received: from mo4-p00-ob.smtp.rzone.de ([81.169.146.161]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sHdnm-001Q7z-6U for pgsql-bugs@lists.postgresql.org; Thu, 13 Jun 2024 06:19:17 +0000 ARC-Seal: i=1; a=rsa-sha256; t=1718259553; cv=none; d=strato.com; s=strato-dkim-0002; b=NHLrfQkv1FyJ75r59oC5V9lp5iZQJMnJj3ARgel4fWEOVpEhhLFXyiaRJNusd020r0 cgs/XGysDBqbia+kf+ryHEGSci94xqyoeyaTIp+kD8Z2D7Rq9uxBrZ8MyBkqPQgIo+Wi qIklFrlE8ngwi2cd+neseFGzh6/NixMShC0o1eWiM2gtfgfwhMPA3t8+bw7/SZ3RzyaO +KjHuXqYcGuGpJGgG0MNh929WZd/PfyGbA4JVZwhqbGBy5tSySlJQoky1xN0GnVTsadx qHjEt3bY8qJdGhn8m4WokCA1g5gIT4Sr3oDPWbAqxHKuWRq3DokJ5h6Fi9VQxm7PdhkH iZ3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1718259553; s=strato-dkim-0002; d=strato.com; h=In-Reply-To:From:References:Cc:To:Subject:Date:Message-ID:Cc:Date: From:Subject:Sender; bh=k472rindhxODPmalGkrGsmrxp6DiHX6yWNZrkBJ2eJE=; b=IX2NTDg1BGvDKnkdy2VkTSIQT+vtLIgz4oTOr8KhsiU3f9PJwREqNt24pd1M23zc7+ VGwR5KoMG+JYCT2ewnjchq/rmV3wqajOJxmy3T9Mf66d/Brzj1/yu+EFQNNZQQh6KPuv UdNnVGd/RMMB0zOEuoI2frn9uMKCAjGMkQunx1Y2xx7vQPeHZn3eRT/tSSNfAMpBv5Qe L7zaw0Xd1hb6PZ3BcJ5vKt/+FaKpvgo3MnavFfKpZMP08/VHQ7fWe85VjiCE9gOo3FfO Dw2KLN1VeIg7WeZ5VcXoCQisOfhLt4yxCu0qsqSwh689g4zsrNgiIBwmCjPv7XwtuP7z uqLw== ARC-Authentication-Results: i=1; strato.com; arc=none; dkim=none X-RZG-CLASS-ID: mo00 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1718259553; s=strato-dkim-0002; d=heine7.de; h=In-Reply-To:From:References:Cc:To:Subject:Date:Message-ID:Cc:Date: From:Subject:Sender; bh=k472rindhxODPmalGkrGsmrxp6DiHX6yWNZrkBJ2eJE=; b=soShrJxPG52bTrlzrt4PPD5NTcjAOYrun4Iyvfe9uaHJVoWW7kohX3ydAe1bvvq1AX 0kRFs3EPyGfYKJ0Obrf0RDVyp/bkD9xmPr5WQEdeIuF7nM2EQjGH0Edfa+nRW0ZGfHPq LpYtf++cK0qJzt+gtSv0lgUk5RH92OxZWJGEdhlLa2wbf6+Lw0oSa76NGiJhzTWInVvj 9JWeAPw2RdUqgSMGmF/R0xaUVIyydFQEQueqD4NdtXTWzWw9j7ZLvd3jbxeCPdeW+4es Hk4RW+ks1pKTX38vRhkctWjbf1jGzZc73TOQEhMpnWJ0jrwi6DcxfYL3PvJGvLoa6ct1 VVCw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1718259553; s=strato-dkim-0003; d=heine7.de; h=In-Reply-To:From:References:Cc:To:Subject:Date:Message-ID:Cc:Date: From:Subject:Sender; bh=k472rindhxODPmalGkrGsmrxp6DiHX6yWNZrkBJ2eJE=; b=8K1DJ9DIdTfb1+K9u62UOXnpsdcomou4RUv1OXm135Ex007Mb0FtyjkplOX/CWVj1j Yzy9QCUfbi6qbYoZMrBw== X-RZG-AUTH: ":K2UQfVWqNexjutYFLWF4K6qbrznTNjZhogOPhnqUGmgNZnxZB9kOFZ0wXqH/Zmz0Mm6b7MShLk1BKg==" Received: from [192.168.6.17] by smtp.strato.de (RZmta 50.5.0 DYNA|AUTH) with ESMTPSA id Y05c3e05D6JC7KM (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate); Thu, 13 Jun 2024 08:19:12 +0200 (CEST) Content-Type: multipart/alternative; boundary="------------a4NX3Sv6NLC1c0nUVjzQWPow" Message-ID: Date: Thu, 13 Jun 2024 08:19:12 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: BUG #18503: Reproducible 'Segmentation fault' in 16.3 on ARM64 To: Thomas Munro Cc: pgsql-bugs@lists.postgresql.org References: <18503-6e0f5ab2f9c319c1@postgresql.org> Content-Language: en-US, de-DE From: Stefan Heine In-Reply-To: Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------a4NX3Sv6NLC1c0nUVjzQWPow Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2024-06-12 23:41, Thomas Munro wrote: > On Thu, Jun 13, 2024 at 1:47 AM Stefan Heine wrote: >> do you have instructions for the gdb backtrace? > gdb /path/to/executable -c /path/to/core > ... loads stuff ... > (gdb) bt > ... prints out function call stack ... > > It will probably just show some library names and addresses, but so > far we don't even know if this is crashing in LLVM or in PostgreSQL > code so that'd be a clue. Maybe function names would appear if you > set up DEBUGINFOD_URLS, depending on where you got your packages from: > > https://wiki.debian.org/HowToGetABacktrace # gdb /usr/lib/postgresql/16/bin/postgres -c core.19 GNU gdb (Debian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at:    . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/lib/postgresql/16/bin/postgres... (No debugging symbols found in /usr/lib/postgresql/16/bin/postgres) warning: Can't open file /dev/shm/PostgreSQL.384567174 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.2343312096 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.1247406204 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.50860586 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.4136010652 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.2304500154 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.817475720 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.526004662 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.1223723046 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.4190931822 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.3836724180 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.1707942452 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.4107375064 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.2885303254 during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.4136268764 during file-backed mapping note processing warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing warning: Can't open file /dev/shm/PostgreSQL.3153232120 during file-backed mapping note processing warning: Can't open file /SYSV03e40001 (deleted) during file-backed mapping note processing [New LWP 19] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1". Core was generated by `postgres: sa postgres 164.99.242.100(57456) EXPLAIN                           '. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000fffe0fb635b8in ??() (gdb) bt #0 0x0000fffe0fb635b8in ??() #1 0x0000aaaaefd84330in ??() Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) quit > Hoping to find time to repro this later on a cloud host. If this is a > cloud host, can you tell me which cloud, instance type, memory size > etc? I had already been trying on some local ARM hardware with no > luck (same versions but diferrent OS, so going to try making more > things match you case)... > (I realise that the email from the other thread gives an AWS instance > type that I can try, but that report is about memory usage and yours > has a segfault so I'm curious to know what conditions are different > for you..) it's running on AWS, t4g.large, 8GB RAM. this server is running Ubuntu 22.04.3 LTS and hosting docker. inside docker, there is a container running postgres, based on the official postgres:16.3 (Based on Debian Bookwork) from https://hub.docker.com/_/postgres . --------------a4NX3Sv6NLC1c0nUVjzQWPow Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 2024-06-12 23:41, Thomas Munro wrote:
On Thu, Jun 13, 2024 at 1:47 AM Stefan Heine <github.stheine@heine7.de> wrote:

do you have instructions for the gdb backtrace?

gdb /path/to/executable -c /path/to/core
... loads stuff ...
(gdb) bt
... prints out function call stack ...

It will probably just show some library names and addresses, but so
far we don't even know if this is crashing in LLVM or in PostgreSQL
code so that'd be a clue.  Maybe function names would appear if you
set up DEBUGINFOD_URLS, depending on where you got your packages from:

https://wiki.debian.org/HowToGetABacktrace

# gdb /usr/lib/postgresql/16/bin/postgres -c core.19  
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/postgresql/16/bin/postgres...
(No debugging symbols found in /usr/lib/postgresql/16/bin/postgres)
warning: Can't open file /dev/shm/PostgreSQL.384567174 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.2343312096 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.1247406204 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.50860586 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.4136010652 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.2304500154 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.817475720 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.526004662 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.1223723046 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.4190931822 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.3836724180 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.1707942452 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.4107375064 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.2885303254 during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.4136268764 during file-backed mapping note processing
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.3153232120 during file-backed mapping note processing
warning: Can't open file /SYSV03e40001 (deleted) during file-backed mapping note processing
[New LWP 19]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: sa postgres 164.99.242.100(57456) EXPLAIN                           '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000fffe0fb635b8 in ?? ()
(gdb) bt
#0  0x0000fffe0fb635b8 in ?? ()
#1  0x0000aaaaefd84330 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) quit



 
Hoping to find time to repro this later on a cloud host.  If this is a
cloud host, can you tell me which cloud, instance type, memory size
etc?  I had already been trying on some local ARM hardware with no
luck (same versions but diferrent OS, so going to try making more
things match you case)...

(I realise that the email from the other thread gives an AWS instance
type that I can try, but that report is about memory usage and yours
has a segfault so I'm curious to know what conditions are different
for you..)

it's running on AWS, t4g.large, 8GB RAM. this server is running Ubuntu 22.04.3 LTS and hosting docker.
inside docker, there is a container running postgres, based on the official postgres:16.3 (Based on Debian Bookwork) from https://hub.docker.com/_/postgres .


--------------a4NX3Sv6NLC1c0nUVjzQWPow--