Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1vAtNS-0070cY-N4 for pgsql-bugs@arkaria.postgresql.org; Mon, 20 Oct 2025 17:08:58 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1vAtMT-002VOj-Kl for pgsql-bugs@arkaria.postgresql.org; Mon, 20 Oct 2025 17:07:56 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1vAspx-002Hlf-B1 for pgsql-bugs@lists.postgresql.org; Mon, 20 Oct 2025 16:34:20 +0000 Received: from mail-lf1-x12b.google.com ([2a00:1450:4864:20::12b]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1vAspu-003GTF-0B for pgsql-bugs@lists.postgresql.org; Mon, 20 Oct 2025 16:34:20 +0000 Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-57bb7ee3142so5289538e87.0 for ; Mon, 20 Oct 2025 09:34:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1760978057; x=1761582857; darn=lists.postgresql.org; h=in-reply-to:content-language:references:to:from:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=0Bb7x+h5b4oYFpav9h7RVTWL/iWeFx4cMqwOWc+MOUY=; b=abFiIycIqQy4ACQDjK8hcddnTyHY7PcSFYA4kcKMYQdlF0s/rXiI7xv3B5RfsQg4MO uRLQvdidvGoaagm5espdqGnD/2fVl4xqbCE6MjivR8/EGw042EgSKMB5CcAp6gBB2AaY +4Iv0lZCEmkrDAD735CVBT1pGJvHt2sMukf9WDlbaQuRZ8zN7umyXSrqKU84MGQr8wiq gVkoCDdSrfxWPv9tgFKpbw8aF6Pa/IgCDFSwAbCbrSvG8fAtVbL8ja0qaKOqy5hJi1Pi NSzPGG8Doq0pqUBl4EBOA+PpFt5Vjcs9yviwRwXzC6nZMpMGRQpYRzWX3InIuTZ1rbwh jjUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760978057; x=1761582857; h=in-reply-to:content-language:references:to:from:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=0Bb7x+h5b4oYFpav9h7RVTWL/iWeFx4cMqwOWc+MOUY=; b=KJ3pEoJw+RTjM65dODyx3NNJ3S5trOb1oxPSFL8FTl9asNasZXJwrZrvcxN1YHiJiL lFcNaSrp4sG+o9gM3WUbYjNzblKzmHQo0/apNqI8SMqLxQ3lpUaa87vkzCBlfkCvjxde ubkgGnB0+DgWbsXA2LyPhTeikmdcCEJCX+L7/Uqvc3OnamZ9n/Obs/i5PlfPAu3b5VW6 +8lWHVorlvRYd6vo3+48ggf6fn5dcyPyTILcZEQkNle6NA2lvhXptWQ+m+rQ+WfjvAZZ niZ5yH4gDl2IZJMmC0XgVgOA9rb+z6VgycNkgoMI5EtmMLcmgt3mxc4kDYNn+EVaxR5K 1zbA== X-Gm-Message-State: AOJu0YxlH5g4Dy7Guz2yTza9XzVoQxFdr8RkCePdqKwnllPYVBLLmK1m oLl0ji7LYXllEO5TucqEdsCpguHkOKbFlkLHrwT5gIkAOH2Rgy03o1mV7qCtacve X-Gm-Gg: ASbGncu9Y1gaGgsc45DfzLQXW+VdMZBWgQJXcELWlRX4pJW6m7I93xMIHsHpZXllHsw 9LmKYiMA3fa/EFT4bjdMdgjmTXOLhemu+KE3yk4hV0Xyp8UDH6/4vOxpp1RsDWxd4L3nNYh4M5v IIHkjmvCVPamdoOkpH7VQy3dapZ5Fw1FkrUV59bR4omzhc4Z/A4hY06OzVONpdGENGyJeFy+Y8n 2m2uGkEsEkiKlg8snc1Gc1hitCWebKEh9ARwlx+ExUs+PbY3y0d8ocmajSPGJh45COQzdCmPg6m aFb4wTuEEAg8byW+Dh8pPYB1FsaisbziZJKZBZO38fLWLkJl2LPvFUiyllpcRcy6VY3M98KYZQ6 j+B8FZii/qDIumdyLFEc+TwFULe3teZDecWt71GkCUtOrX+yWiSkJjaoanNLW948CptRog6fMWp bDvStomeR6FGV0+jcNCMqfVD4xCNcJmLD2EA== X-Google-Smtp-Source: AGHT+IFzN39jIuTJfUqygEwt/+AW+ay8/jftJLitYsEgLvLgLomIc3n9zotP1IjmXWoM2/jyBDCOQw== X-Received: by 2002:a05:6512:39c2:b0:58b:211:3323 with SMTP id 2adb3069b0e04-591d857b737mr4665546e87.56.1760978056492; Mon, 20 Oct 2025 09:34:16 -0700 (PDT) Received: from [172.22.22.22] (smtp.shulev.su. [146.66.171.44]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-591def16b31sm2643537e87.60.2025.10.20.09.34.14 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 20 Oct 2025 09:34:14 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------CGDVw5eKG0n27KxnYY0hc6zr" Message-ID: Date: Mon, 20 Oct 2025 19:34:14 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: PG17.6 wal apply bug (SIGSEGV) From: "badfilez@gmail.com" To: pgsql-bugs@lists.postgresql.org References: Content-Language: en-US In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------CGDVw5eKG0n27KxnYY0hc6zr Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit backrtace Program terminated with signal SIGSEGV, Segmentation fault. #0  0x000000000057eff2 in _bt_restore_page (page=0x7f6f48fd1000 "", from=0x7f6fe2eccd80 "", len=) at nbtxlog.c:63 63            itemsz = MAXALIGN(itemsz); (gdb) bt full #0  0x000000000057eff2 in _bt_restore_page (page=0x7f6f48fd1000 "", from=0x7f6fe2eccd80 "", len=) at nbtxlog.c:63         itupdata =         itemsz = 0         end = 0x7f6fe2ecd8c0 "(\265/\375`\260\005\205\023"         items = {0x0 , 0x7f6f00000000 "\211\243\362hw\366\371\003\b", 0x7f6fe2eccd80 "" }         itemsizes = {24 , 0 }         i = 1318         nitems =         __func__ = "_bt_restore_page"         __errno_location = On 20/10/2025 13:58, badfilez@gmail.com wrote: > Hello, > > Postgres 17 cluster from official repo on RHEL8 (master and 2 replicas) > > on both replicas, I get > > 2025-10-18 15:40:50.843 MSK [1448] LOG:  entering standby mode > 2025-10-18 15:40:50.865 MSK [1448] LOG:  redo starts at 1F35/D08DE298 > 2025-10-18 15:41:14.553 MSK [1381] LOG:  startup process (PID 1448) was terminated by signal 11: Segmentation fault > 2025-10-18 15:41:14.553 MSK [1381] LOG:  terminating any other active server processes > 2025-10-18 15:41:14.555 MSK [1381] LOG:  shutting down due to startup process failure > 2025-10-18 15:41:14.677 MSK [1381] LOG:  database system is shut down > > After debugging, > > replica recovery creates corrupted index file from wal, > waldump does not show any wal corruption, no prior io errors in logs > master has not crashed and working ok, no errors in log > > the operation on which segfault happens is (if i stop recovery on previous operation it does not trigger segfault) > > rmgr: Btree len (rec/tot): 3758/ 5774, tx: 1711720455, lsn: 1F36/30E3C7B8, prev 1F36/30E3C760, desc: SPLIT_L level: 0, firstrightoff: 140, > newitemoff: 140, postingoff: 0, blkref #0: rel 1663/16385/151181595blk 63203FPW, blkref #1: rel 1663/16385/151181595blk 112208, blkref #2: rel > 1663/16385/151181595blk 108144FPW > > the wal segment containing the instruction attached > > --------------CGDVw5eKG0n27KxnYY0hc6zr Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
backrtace

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000057eff2 in _bt_restore_page (page=0x7f6f48fd1000 "", from=0x7f6fe2eccd80 "", len=<optimized out>) at nbtxlog.c:63
63            itemsz = MAXALIGN(itemsz);
(gdb) bt full
#0  0x000000000057eff2 in _bt_restore_page (page=0x7f6f48fd1000 "", from=0x7f6fe2eccd80 "", len=<optimized out>) at nbtxlog.c:63
        itupdata = <optimized out>
        itemsz = 0
        end = 0x7f6fe2ecd8c0 "(\265/\375`\260\005\205\023"
        items = {0x0 <repeats 227 times>, 0x7f6f00000000 "\211\243\362hw\366\371\003\b", 0x7f6fe2eccd80 "" <repeats 180 times>}
        itemsizes = {24 <repeats 33 times>, 0 <repeats 375 times>}
        i = 1318
        nitems = <optimized out>
        __func__ = "_bt_restore_page"
        __errno_location = <optimized out>


On 20/10/2025 13:58, badfilez@gmail.com wrote:
Hello,

Postgres 17 cluster from official repo on RHEL8 (master and 2 replicas)

on both replicas, I get

2025-10-18 15:40:50.843 MSK [1448] LOG:  entering standby mode
2025-10-18 15:40:50.865 MSK [1448] LOG:  redo starts at 1F35/D08DE298
2025-10-18 15:41:14.553 MSK [1381] LOG:  startup process (PID 1448) was terminated by signal 11: Segmentation fault
2025-10-18 15:41:14.553 MSK [1381] LOG:  terminating any other active server processes
2025-10-18 15:41:14.555 MSK [1381] LOG:  shutting down due to startup process failure
2025-10-18 15:41:14.677 MSK [1381] LOG:  database system is shut down

After debugging,

replica recovery creates corrupted index file from wal,
waldump does not show any wal corruption, no prior io errors in logs
master has not crashed and working ok, no errors in log

the operation on which segfault happens is (if i stop recovery on previous operation it does not trigger segfault)

rmgr: Btree len (rec/tot): 3758/ 5774, tx: 1711720455, lsn: 1F36/30E3C7B8, prev 1F36/30E3C760, desc: SPLIT_L level: 0, firstrightoff: 140, newitemoff: 140, postingoff: 0, blkref #0: rel 1663/16385/151181595 blk 63203 FPW, blkref #1: rel 1663/16385/151181595 blk 112208, blkref #2: rel 1663/16385/151181595 blk 108144 FPW

the wal segment containing the instruction attached



--------------CGDVw5eKG0n27KxnYY0hc6zr--