Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1v9f3B-004qVp-Dn for pgsql-admin@arkaria.postgresql.org; Fri, 17 Oct 2025 07:38:57 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1v9f3A-000YBF-Dr for pgsql-admin@arkaria.postgresql.org; Fri, 17 Oct 2025 07:38:55 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1v9YiM-00GW6h-7H for pgsql-admin@lists.postgresql.org; Fri, 17 Oct 2025 00:53:01 +0000 Received: from mx0a-007a3a01.pphosted.com ([205.220.169.168]) by makus.postgresql.org with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1v9YiI-002BtD-1Y for pgsql-admin@postgresql.org; Fri, 17 Oct 2025 00:53:00 +0000 Received: from pps.filterd (m0312997.ppops.net [127.0.0.1]) by mx0a-007a3a01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 59GKsp0H027935 for ; Thu, 16 Oct 2025 17:52:57 -0700 Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by mx0a-007a3a01.pphosted.com (PPS) with ESMTPS id 49tfx1a8at-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 16 Oct 2025 17:52:56 -0700 (PDT) Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-3f030846a41so834880f8f.2 for ; Thu, 16 Oct 2025 17:52:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=platformscience.com; s=google; t=1760662375; x=1761267175; darn=postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LCZDXmAmmMgr19tg1zqAFj4VYumEts8xzDDBsC3jMMo=; b=YFoi34CW/iDv8tCIsL66Igrn1C8yHYbzb8Ak921+jLGgLay2dpY0T/drr1odztaapL brbLqBH5UjoJaQxOFxWU6j8s6hXstdj/eITBgMOnT4YFXMpRHLdyJyLe8lrXBUY/IDdh hCO+LPMnxsQDmZU18oz9+leVlV30V5BPIrcF0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760662375; x=1761267175; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LCZDXmAmmMgr19tg1zqAFj4VYumEts8xzDDBsC3jMMo=; b=KXfZ7FFBrXkEZV54F4laF5izZADZ1Z30gucsKK23rZpFVT9bPUwFQxxuTIsiZ/uzCs +VvO1bIbDJmNuShzjBIe7qbgB0u9BLnxzglA5rnkylSuQJyDOUkJhPjwiX4oPXdBvei2 DL4sPjioqXRFk3rDphDm2k7KtsjQiRijKhOymzUUC1ktuvGtQfcrJkvMy5aS7z448It9 FyjX93R9RHAyQFENEZbt2GWD3ba8dZKp4zYKMm4HRjtOzyG0ecp6JkpL8z22eOYK27kQ XIoZfvY1zjF/JaaY59NToqeQFIL/ehOnrpJ7vp3dpRlW74RyLs+vzuzrQrt3iOtbkRrd rBSg== X-Gm-Message-State: AOJu0YwFjtcWD1B58FgOD49W9XOQ1UvyCk57Jk5/Ix3E0SUW6tqqHNnF PITjonLktVqc9da2IaRwYkV6uR3k8FIn0wWOKpBRWOzyGRd5FsUAMN79pXXO253FaJtAPClHtZG lvk5VOfl+eUcUDfk3Z70QLiB6+86B3dKzTW/cZgaHPnk/ZWdeElX/Yhagfy9ATUe7pF2oJhQYtF 1hj8mQ+lKAywspY+5ogtYwB5wzEqNY03Oh23WJinL9Iq80vbjvO2bPdeKBMw== X-Gm-Gg: ASbGncu12bOpqRMVuom2JOYcWBFfGzMcEvOdPa4p0clZ1X/LVKwgJ6IzcWJln6JcQYe IxnBxIaHx0p7jh+IgF/6VuedTUuu9gZArPs6wFCtCxR9XRH/d9jcL4KIRQK4tzMFUbk5DX2KV2i ++VaUp2q+M5D8i9ntjK+8OjwXa7TccR4nF6SL6rQ+uQHy3b+Ac/fPRGWXmxBz2OnSuxB7aE8xGR 1hxClTxhSIBiD9u X-Received: by 2002:a5d:5c89:0:b0:3e8:ee5d:f31e with SMTP id ffacd0b85a97d-42704d7eb32mr1470217f8f.25.1760662374852; Thu, 16 Oct 2025 17:52:54 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFrhInZd8CYQG5MtM3JJN1l6JrJinpQbwQQPH+FM1/WE7FzPQKcTKN/FofoGKLiIs7hgeBhxxlxScGpeYZOlt4= X-Received: by 2002:a5d:5c89:0:b0:3e8:ee5d:f31e with SMTP id ffacd0b85a97d-42704d7eb32mr1470206f8f.25.1760662374347; Thu, 16 Oct 2025 17:52:54 -0700 (PDT) MIME-Version: 1.0 References: <1bef483a9e134bb595d3b25a35ac0e16@netcracker.com> In-Reply-To: <1bef483a9e134bb595d3b25a35ac0e16@netcracker.com> From: Willian Colognesi Date: Thu, 16 Oct 2025 21:52:43 -0300 X-Gm-Features: AS18NWDJFLcMTp6FFXLTkD_spgT2L62TZAhJHGHxUmYLRsBke8pKqKH2Zo4qfbc Message-ID: Subject: Re: postgres server crash with "Segmentation fault" To: Ishan Arunkumar Joshi Cc: pgsql-admin@postgresql.org Content-Type: multipart/alternative; boundary="00000000000064a0ee06415027cc" X-Proofpoint-ORIG-GUID: kPMQ1bA2GMgAFaP-H-j9VMiDgmUuhjLE X-Proofpoint-GUID: kPMQ1bA2GMgAFaP-H-j9VMiDgmUuhjLE X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUxMDE1MDEyNiBTYWx0ZWRfX/eZu1oqupkLx OVuk/c6EWJBTTBPA1Himg2pfxYHVKBW/GvCNbGfDLbUBjj8TxY0Mu8Xt+hydsosWy1hEn/OgHN6 6wYU+t8WlzEMS4OgkE/tp5YhPx9TJqVhzQynXHZA2wyka3WYP95fr08dgKK5QESLJnpdr/aSjHG Vg0Cc7OR6lpEAAm758+jHzTXW7xTTRiEdXFoXS2lvaN48a6YmdawfNvak4KYjUAXzV+BX/xdccr p3Dvlqe3wawR4+3S/QH8/WzDc+OVh/JNVaqc9Qc4Z07mQuabEpBBXtbCmrsmj2TJC3WQo3v+6mN ZCSaT8mc1D/UK/m5YgdGS8IqsmxdmcLYSmcP0xbCG6vSVzRgvEv2A3GdIaIL/AOCPNTsjA24xh2 EL5Lxnx7KxYrUJssFwCk5kRV1mo1zg== X-Authority-Analysis: v=2.4 cv=Wssm8Nfv c=1 sm=1 tr=0 ts=68f19368 cx=c_pps a=CsXZvLRfiTx/ye2xXAwb9g==:117 a=x6icFKpwvdMA:10 a=SKi_fFMLSnQA:10 a=VkNPw1HP01LnGYTKEx00:22 a=uMN424e-AAAA:8 a=0z6FynTJAAAA:8 a=lQvPdeR4ehGubD8Tk3oA:9 a=QEXdDO2ut3YA:10 a=QMG5EL1NAAAA:8 a=URH0RuqZmekgQkxNv_IA:9 a=wNiwnkKOUZ0Uz8Ns:21 a=lqcHg5cX4UMA:10 a=F7q00xkr9EfWfQvbdVXI:22 a=zJTtOugoHXhIODWjBLSu:22 a=AKfdhyqiTic-xPh-5ZLI:22 a=El-nWJp-vY32lyFZDeps:22 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --00000000000064a0ee06415027cc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Not sure if it's the same problem but I saw segmentation faults in the past when jit was enabled in postgres. After disabled it never happened again. Willian Colognesi | Principal Engineer willian.colognesi@platformscience.com R. Adhemar Pereira de Barros, 1500, 14=C2=B0 floor, Londrina/PR | Brazil On Thu, Oct 16, 2025, 18:54 Ishan Arunkumar Joshi < ishan.joshi@netcracker.com> wrote: > Hi team, > > > > We are using PG16.9 in Patroni Postgres setup in production. Last night w= e > have face an issue where postgres server got crash with =E2=80=9CSegmenta= tion > fault=E2=80=9D on table with auto vacuum task. Interestingly in standby n= ode while > we perform vacuum on same table standby node was also got crash. The tabl= e > was not able to get query on the table as while executing select stateme= nt > also crash the database. > > we have observed few error prior to crash for same table. (Table name and > function details change for purpose) > > > "ERROR : Error occurred at function get_details page 117 of relation > ""impacted_table"" should be empty but is not" > > > > During the same time the other table also getting below error. However > once the database restart, we were not getting any issue for table oid=3D > 1108029 > > > =E2=80=9CERROR : Error occurred at function get_details unexpected data b= eyond EOF > in block 16276 of relation base/33195/1108029" > > > > At last it got failed as follows > > 2025-10-15 02:50:52.428 [432443]LOG: terminating any other active server > processes" > > 2025-10-15 02:50:52.428 [432443]DETAIL: Failed process was running: > autovacuum: VACUUM ANALYZE schema.impacted_table" > > 2025-10-15 02:50:52.428 [432443]LOG: server process (PID 390906) was > terminated by *signal 11: Segmentation fault"* > > 2025-10-15 02:50:55.475 [432443]LOG: all server processes terminated; > reinitializing" > > 2025-10-15 02:51:32.575 [432443]LOG: received immediate shutdown request= " > > 2025-10-15 02:51:32.629 [432443]LOG: database system is shut down" > > > > The function which was executing having truncate table and insert/update > statement executing on this table. As this is normally functionality but > we are suspecting it during the execution runtime it corrupt the shared > memory and data inflight which got replicated to Replica node/DR site and > corrupt the same table. > > > > We had to drop the table =E2=80=9Cimpacted_table=E2=80=9D from database o= nce the database > is up. > > however we are not able to identify exact root cause behind =E2=80=9Csegm= entation > fault=E2=80=9D error for this table and need expert advice to find the ro= ot case > and also need suggestions to prevention steps. > > > > *Thanks & Regards,* > > ------------------------- > Ishan Joshi > > > > > > ------------------------------ > The information transmitted herein is intended only for the person or > entity to which it is addressed and may contain confidential, proprietary > and/or privileged material. Any review, retransmission, dissemination or > other use of, or taking of any action in reliance upon, this information = by > persons or entities other than the intended recipient is prohibited. If y= ou > received this in error, please contact the sender and delete the material > from any computer. > > --00000000000064a0ee06415027cc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Not sure if it's the same problem but I saw= segmentation faults in the past when jit was enabled in postgres. After di= sabled it never happened again.


<= /div>

Willian Colognesi |= Principal Engineer

willian.colognesi@pla= tformscience.com

R. Adhemar Pereira de Barros, 1500, 14=C2=B0 floor,= Londrina/PR | Brazil


On Thu, O= ct 16, 2025, 18:54 Ishan Arunkumar Joshi <ishan.joshi@netcracker.com> wrote:

Hi team,

=C2=A0

We are using PG16.9 in Patroni Postgres setup in pro= duction. Last night we have face an issue where postgres server got crash w= ith =E2=80=9CSegmentation fault=E2=80=9D on table with auto vacuum task. In= terestingly in standby node while we perform vacuum on same table standby node was also got crash. The table was not able to g= et query =C2=A0on the table as while executing select statement also crash = the database.

we have observed few error prior to crash for same table. (Table name and f= unction details change for purpose) =C2=A0


"ERROR : Error occurred at function get_details page 117 of rel= ation ""impacted_table"" should be empty but is not&quo= t;

=C2=A0

During the same time the other table also getting be= low error. However once the database restart, we were not getting any issue= for table oid=3D 1108029


=E2=80=9CERROR : Error occurred at function get_details unexpected data bey= ond EOF in block 16276 of relation base/33195/1108029"

=C2=A0

At last it got failed as follows

2025-10-15 02:50:52.428 [432443]LOG:=C2=A0 terminating any other active ser= ver processes"

2025-10-15 02:50:52.428 [432443]DETAIL:=C2=A0 Failed= process was running: autovacuum: VACUUM ANALYZE schema.impacted_table"= ;

2025-10-15 02:50:52.428 [432443]LOG:=C2=A0 server pr= ocess (PID 390906) was terminated by signal 11: Segmentation fault"

2025-10-15 02:50:55.475 [432443]LOG:=C2=A0 all serve= r processes terminated; reinitializing"

2025-10-15 02:51:32.575 [432443]LOG:=C2=A0 received = immediate shutdown request"

2025-10-15 02:51:32.629 [432443]LOG:=C2=A0 database = system is shut down"

=C2=A0

The function which was executing having truncate tab= le and insert/update statement executing on this table.=C2=A0 As this is no= rmally functionality but we are suspecting it during the execution runtime = it corrupt the shared memory and data inflight which got replicated to Replica node/DR site and corrupt the same table.

=C2=A0

We had to drop the table =E2=80=9Cimpacted_table=E2= =80=9D from database once the database is up.

however we are not able to identify exact root cause behind =E2=80=9Csegmen= tation fault=E2=80=9D error for this table and need expert advice to find t= he root case and also need suggestions to prevention steps.

=C2=A0

Thanks & Regards,

-----------= --------------
Ishan Joshi

=C2=A0

= =C2=A0

=

The information transmitted herein is intended only for the person or entit= y to which it is addressed and may contain confidential, proprietary and/or= privileged material. Any review, retransmission, dissemination or other us= e of, or taking of any action in reliance upon, this information by persons or entities other than the inte= nded recipient is prohibited. If you received this in error, please contact= the sender and delete the material from any computer. =C2=A0

--00000000000064a0ee06415027cc--