Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k97nq-00086C-1E for pgsql-admin@arkaria.postgresql.org; Fri, 21 Aug 2020 14:13:58 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1k97nm-00040P-MR for pgsql-admin@arkaria.postgresql.org; Fri, 21 Aug 2020 14:13:54 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k97nm-0003zz-CK for pgsql-admin@lists.postgresql.org; Fri, 21 Aug 2020 14:13:54 +0000 Received: from mail4.neteven.com ([5.135.111.119]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k97ni-0005ve-Pb for pgsql-admin@lists.postgresql.org; Fri, 21 Aug 2020 14:13:53 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by mail4.neteven.com (Postfix) with ESMTP id E8557190CF; Fri, 21 Aug 2020 16:13:47 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 mail4.neteven.com E8557190CF DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=neteven.com; s=dkim; t=1598019227; bh=wP9wCe7SMrxtKXDrF7NWIwII8Gql++erhJVyak06NjU=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=VR88sFNq+dz4BGqOKTAatQtfYBFAgIRuKSekFcoU1e25Hvc65VWCmOdh01gQGOKuA 6o9wGrzhH40QfanNTOQv7mN2BLE+WJ3fbuw49C3ol6/zyyzVwAjoqrxXZgI5xUU13t CquqDR4noC/n+QCXKfC/UAGi+V66hOSGJHKFq5aA= X-Virus-Scanned: Debian amavisd-new at mail4.neteven.com Received: from mail4.neteven.com ([127.0.0.1]) by localhost (mail4.neteven.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sB_FC7xr6ORF; Fri, 21 Aug 2020 16:13:47 +0200 (CEST) Subject: Re: Segfault on postgresql 12.3 To: Julien Rouhaud Cc: pgsql-admin@lists.postgresql.org References: From: Thomas SIMON Message-ID: <42a8b12b-622c-0510-a36d-ad3b927f149d@neteven.com> Date: Fri, 21 Aug 2020 16:13:45 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: fr List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Hi Julien, thanks for answering me. The only extension I use is repmgr. I've tried to use gdb to see something (I don't know if i use it correctly) , below the backtarce : [16:03:13]root@db13:/tmp$ gdb -q -c /tmp/core /usr/lib/postgresql/12/bin/postgres Reading symbols from /usr/lib/postgresql/12/bin/postgres...(no debugging symbols found)...done. [New LWP 177990] warning: Unexpected size of section `.reg-xstate/177990' in core file. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `postgres: 12/main: supervision neteven2 localhost(34868) SELECT               '. Program terminated with signal SIGSEGV, Segmentation fault. warning: Unexpected size of section `.reg-xstate/177990' in core file. #0  0x00005636d2d844f1 in equalTupleDescs () (gdb) (gdb) bt #0  0x00005636d2d844f1 in equalTupleDescs () #1  0x00005636d31a65cf in ?? () #2  0x00005636d31b5fd3 in hash_search_with_hash_value () #3  0x00005636d31a83b1 in assign_record_type_typmod () #4  0x00005636d31b4855 in ?? () #5  0x00005636d31b4b43 in get_expr_result_type () #6  0x00005636d31b4b7b in get_expr_result_tupdesc () #7  0x00005636d2e8bcce in get_rte_attribute_is_dropped () #8  0x00005636d303fc7a in AcquireRewriteLocks () #9  0x00005636d304039e in ?? () #10 0x00005636d3043aa2 in QueryRewrite () #11 0x00005636d307de90 in ?? () #12 0x00005636d307df70 in pg_analyze_and_rewrite () #13 0x00005636d307e68f in ?? () #14 0x00005636d30804ad in PostgresMain () #15 0x00005636d2d73f00 in ?? () #16 0x00005636d3006f89 in PostmasterMain () #17 0x00005636d2d75128 in main () (gdb) cont The program is not being run. thomas Le 21/08/2020 à 14:34, Julien Rouhaud a écrit : > On Fri, Aug 21, 2020 at 2:25 PM Thomas SIMON wrote: >> Hi all, >> >> I just had strange behavior on my postgresql instance, with postgresql >> auto restart >> >> Looking for logs, I've found a segfault in kern.log >> >> [12:24:09]root@db12:~# cat /var/log/kern.log >> 2020-08-21T12:00:01.436378+02:00 db12 kernel: postgres[177990]: segfault >> at 0 ip 00005636d2d844f1 sp 00007fff4fa69910 error 4 in >> postgres[5636d2cb7000+775000] >> >> I've also enabled core dump, file output is : >> >> [12:24:13]root@db12:~# file /data/postgresql/12/main/core >> /data/postgresql/12/main/core: ELF 64-bit LSB core file x86-64, version >> 1 (SYSV), SVR4-style, from 'postgres: 12/main: supervision neteven2 >> localhost(34868) SELECT', real uid: 110, effective uid: 110, real gid: >> 114, effective gid: 114, execfn: '/usr/lib/postgresql/12/bin/postgres', >> platform: 'x86_64' >> >> >> In logs , I have these messages >> >> 2020-08-21 12:00:01.451 CEST [274137]: [299-1] user=,db=,app=,client= >> LOG: server process (PID 177990) was terminated by signal 11: >> Segmentation fault >> 2020-08-21 12:00:01.451 CEST [274137]: [300-1] user=,db=,app=,client= >> DETAIL: Failed process was running: SELECT usename,count(*) FROM >> pg_stat_activity WHERE pid != pg_backend_pid() GROUP BY usename ORDER BY 1 >> >> .. >> 2020-08-21 12:00:02.776 CEST [274137]: [302-1] user=,db=,app=,client= >> LOG: archiver process (PID 274215) exited with exit code 1 >> 2020-08-21 12:00:02.774 CEST [274214]: [1-1] user=,db=,app=,client= >> WARNING: terminating connection because of crash of another server process >> 2020-08-21 12:00:02.774 CEST [274214]: [2-1] user=,db=,app=,client= >> DETAIL: The postmaster has commanded this server process to roll back >> the current transaction and exit, because another s >> erver process exited abnormally and possibly corrupted shared memory. >> 2020-08-21 12:00:02.774 CEST [274214]: [3-1] user=,db=,app=,client= >> HINT: In a moment you should be able to reconnect to the database and >> repeat your command. >> (many times until full restart) >> >> >> I'm on 12.3 version, on a dedicated host on prem. > Note that version 12.4 is now available, however I don't see any relevant fix. > >> root@db12:~# dpkg -l | grep postgresql >> ii pgdg-keyring 2018.2 all >> keyring for apt.postgresql.org >> ii postgresql-12 12.3-1.pgdg90+1 amd64 >> object-relational SQL database, version 12 server >> ii postgresql-12-repmgr 5.1.0-1.stretch+1 >> amd64 replication manager for PostgreSQL 12 >> ii postgresql-client-12 12.3-1.pgdg90+1 >> amd64 front-end programs for PostgreSQL 12 >> ii postgresql-client-common 215.pgdg90+1 >> all manager for multiple PostgreSQL client versions >> ii postgresql-common 215.pgdg90+1 >> all PostgreSQL database-cluster manager >> ii postgresql-server-dev-12 12.3-1.pgdg90+1 >> amd64 development files for PostgreSQL 12 server-side programming >> >> >> Could you please help me to find what is the root cause ? for > This is unfortunately not enough information to find the root issue. > > Do you have any custom extension? Is there any chance you can get a > backtrace of the generated coredump? See > https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD#Getting_a_trace_from_a_randomly_crashing_backend > for more details on how to do that.