Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1vBWTy-00057j-Q7 for pgsql-bugs@arkaria.postgresql.org; Wed, 22 Oct 2025 10:54:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1vBWTx-00FM5f-OO for pgsql-bugs@arkaria.postgresql.org; Wed, 22 Oct 2025 10:54:16 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1vBUSa-00EWQi-NW for pgsql-bugs@lists.postgresql.org; Wed, 22 Oct 2025 08:44:44 +0000 Received: from mail-lj1-x22c.google.com ([2a00:1450:4864:20::22c]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1vBUSX-00382K-2X for pgsql-bugs@lists.postgresql.org; Wed, 22 Oct 2025 08:44:42 +0000 Received: by mail-lj1-x22c.google.com with SMTP id 38308e7fff4ca-3737d0920e6so72050031fa.1 for ; Wed, 22 Oct 2025 01:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1761122680; x=1761727480; darn=lists.postgresql.org; h=in-reply-to:content-language:references:cc:to:from:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=z38FXOy/Uzcrlk9VgslZ9Qfb8w6wcs5HFZIRC02YJ1A=; b=AipUBerpdNJbEnTJlmsHKM8BAUrpW2aIiLpOWBW+65SAAAXeZBnmid4CO9zGWJSKNG Qr7asx5+nwkFTlQxgrAxXfYQ5NcCGIclwV0i05V2S5raFvlhkPMo9/m5q4oUTmZi0HeS zQoLgTOL2Zo76UnStFNnESFQKz4DobKUCyJH4224CrRvzMEMzwU7z9Betpbym7jx9iLW kHSsdK/o/86mJJTJKfxzjhaTka+L0biixH+BXXOHTf5UzRsd02FyWKDd4AtQFJp6mS/H wMBts/hsSGBK1CdCY5E4HGQdhbaGj9eWVQfjY/pgVPFR+KguaG11sRObKPNRjBT4BwGg QxmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761122680; x=1761727480; h=in-reply-to:content-language:references:cc:to:from:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=z38FXOy/Uzcrlk9VgslZ9Qfb8w6wcs5HFZIRC02YJ1A=; b=r3RpGVTsavR3B92nl6r/IENt6UPdmF4b4P5Mzf2k8x+P4v/DJweiI3s6uNO77Vulix YG+rht2CN7b+zbLrCRC0geIOT3ATQyoYJFuC0g599ln0TrRJZLdyy0bqnxh+vd5do+7Y tDZ84U1ENnkWUJunHL1p3HRWj/TzW6JGIDTTZ4MCa6byiu9lVLwkzdLvzBQSx0oyE34r 3ByenkHHBipgXKHOHQ1e7y0zZMxj7sC/GLmmEqMfsKp9pqomVzg1aDWtQw3GFxL6/s1X qyQEa/YEuuBAqTV7MZ0o7knSNsFGnNProilWEGungvNAEypsB87+aHb4Sv72phdAL9ua gx7g== X-Gm-Message-State: AOJu0Yx1KOEiUyFDag6ygyZNGuYBNz7RFohBC4Pn8UJ3Ng1HVBeqYWNZ +/+GF7TFttwTnxD5fPECe6T4Z9R6olgQzW+CXHGNmx4w/iWSBG0clHEwCir5Qocv X-Gm-Gg: ASbGncumLAbUsjpVWPMmYYs5E2HB4+JYfhw6qgjfPqFJCcCuwdgKpQLAIhg88EHxPkO jaP5qa3eR7+dvu661ScTkZl8Z3fCGShanZQ1f8Bve4Rm3Cm9Aoo3+ESt6GQomdgs8GHjIuiAeCm 5gZuPI9giyXY8rn+mU2RjyBUlTA7eMjic/olNjdHMKHlMzY14RKaK03B2Iwl4QI/9cTiTwj3CiE wxWN+3rQuJ6DpVUTxtxFsvpd6bG6ZtVxGOAQiK/39lQ1hOcpN3JgtD3G6M7EMekyJJJFFHViY8X B0/AA6uzsnwK+JBxDQBnFXUooBOwPDaU7coBHllMjm9BKB2aCcyPwHJYySkWdl0YrUmSgCkSS65 ZoOBRzMPL0P+Uei5IfhOWg1W5KvLRKPqBD/8bjAqdL7W8OMFb5IwQOi/MeJF+1U9Snv9H20alEe RxmgCQ/KhQJOynTS1yYGt92gE= X-Google-Smtp-Source: AGHT+IFpaVeTjFmRHoa6ru/vZYcsxCai5zrKU+XsFPKvJ8CkWMGby4gxEvU/4S5Jd88P69sQ3/RBuQ== X-Received: by 2002:a05:651c:992:b0:36b:2bde:a016 with SMTP id 38308e7fff4ca-3779781e35amr55965951fa.2.1761122679893; Wed, 22 Oct 2025 01:44:39 -0700 (PDT) Received: from [172.22.22.22] (smtp.shulev.su. [146.66.171.44]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-377a951a7a6sm35515051fa.28.2025.10.22.01.44.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 22 Oct 2025 01:44:39 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------SDGjaKHY8PfTF1d30Jt34nsv" Message-ID: <105a1561-9403-4ad4-8754-2520bf5c9bd7@gmail.com> Date: Wed, 22 Oct 2025 11:44:38 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: PG17.6 wal apply bug (SIGSEGV) From: "badfilez@gmail.com" To: Peter Geoghegan Cc: pgsql-bugs@lists.postgresql.org References: <630d3d31-40b2-40c2-b1bf-c857ba88322d@gmail.com> Content-Language: en-US In-Reply-To: <630d3d31-40b2-40c2-b1bf-c857ba88322d@gmail.com> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------SDGjaKHY8PfTF1d30Jt34nsv Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Follow up: the error detected on master is: ERROR: XX002: item orderinvariant violated forindex"2722_1401_trends_uint_pkey" DETAIL: Lower indextid=(63203,46) (points toheap tid=(96487,80)) higher indextid=(63203,47) (points toheap tid=(0,0)) page lsn=1F5B/CB8F8098. LOCATION: bt_target_page_check, verify_nbtree.c:1773query was: SELECT"public".bt_index_check(index:= c.oid, heapallindexed := false) FROMpg_catalog.pg_class c, pg_catalog.pg_index i WHEREc.oid = 151181595ANDc.oid = i.indexrelid ANDc.relpersistence != 't'ANDi.indisready ANDi.indisvalid ANDi.indislivebtree index"zabbix._timescaledb_internal._hyper_9_2722_chunk_trends_uint_clock_idx": Do I get in right, this corruption was somehow transferred to replicas first, and then wal was tried to apply over corrupted index? Why it did not crash the master then? On 22/10/2025 09:19, badfilez@gmail.com wrote: > Hi, > > Thank you, > there still are 2 broken indexes in master DB, > one of them exactly matches the said relation 151181595. > > still, > is it proper wal apply procedure, to segfault in such a case? > > > On 20/10/2025 20:18, Peter Geoghegan wrote: >> On Mon, Oct 20, 2025 at 1:07 PM badfilez@gmail.com wrote: >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> #0  0x000000000057eff2 in _bt_restore_page (page=0x7f6f48fd1000 "", from=0x7f6fe2eccd80 "", len=) at nbtxlog.c:63 >>> 63            itemsz = MAXALIGN(itemsz); >>> (gdb) bt full >> "itemsz = 0" suggests that the index was already corrupt, before the >> WAL record is applied. >> >> I suggest that you use contrib/amcheck (or the pg_amcheck frontend >> program) to ascertain the extent of any index corruption on this >> database. >> > --------------SDGjaKHY8PfTF1d30Jt34nsv Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
Follow up:

the error detected on master is:

ERROR: XX002: item order invariant violated for index "2722_1401_trends_uint_pkey" DETAIL: Lower index tid=(63203,46) (points to heap tid=(96487,80)) higher index tid=(63203,47) (points to heap tid=(0,0)) page lsn=1F5B/CB8F8098. LOCATION: bt_target_page_check, verify_nbtree.c:1773 query was: SELECT "public".bt_index_check(index := c.oid, heapallindexed := false ) FROM pg_catalog.pg_class c, pg_catalog.pg_index i WHERE c.oid = 151181595 AND c.oid = i.indexrelid AND c.relpersistence != 't' AND i.indisready AND i.indisvalid AND i.indislivebtree index "zabbix._timescaledb_internal._hyper_9_2722_chunk_trends_uint_clock_idx":

Do I get in right, 
this corruption was somehow transferred to replicas first, and then wal was tried to apply over corrupted index?

Why it did not crash the master then?


On 22/10/2025 09:19, badfilez@gmail.com wrote:
Hi,

Thank you,
there still are 2 broken indexes in master DB,
one of them exactly matches the said relation 151181595.

still,
is it proper wal apply procedure, to segfault in such a case?


On 20/10/2025 20:18, Peter Geoghegan wrote:
On Mon, Oct 20, 2025 at 1:07 PM badfilez@gmail.com <badfilez@gmail.com> wrote:
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000057eff2 in _bt_restore_page (page=0x7f6f48fd1000 "", from=0x7f6fe2eccd80 "", len=<optimized out>) at nbtxlog.c:63
63            itemsz = MAXALIGN(itemsz);
(gdb) bt full
"itemsz = 0" suggests that the index was already corrupt, before the
WAL record is applied.

I suggest that you use contrib/amcheck (or the pg_amcheck frontend
program) to ascertain the extent of any index corruption on this
database.



--------------SDGjaKHY8PfTF1d30Jt34nsv--