Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pnF0v-0005L7-Eb for pgsql-general@arkaria.postgresql.org; Fri, 14 Apr 2023 08:42:37 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pnF0s-0003Zx-WD for pgsql-general@arkaria.postgresql.org; Fri, 14 Apr 2023 08:42:35 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pnF0s-0003Zo-Jb for pgsql-general@lists.postgresql.org; Fri, 14 Apr 2023 08:42:34 +0000 Received: from mail-ej1-x62c.google.com ([2a00:1450:4864:20::62c]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1pnF0n-002oil-VN for pgsql-general@postgresql.org; Fri, 14 Apr 2023 08:42:33 +0000 Received: by mail-ej1-x62c.google.com with SMTP id dm2so43931363ejc.8 for ; Fri, 14 Apr 2023 01:42:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681461748; x=1684053748; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=AGqnouLEO1JNUtz0MIqJ+raw+hizFNAEMW7u7tnjG4o=; b=YiA9nLxA48EJHYJMMJVhfixvGQ8SDQ6RibGw4vnJ0Qh9wzv8Ik0lH3HBY/0TabgKzw QChZOgaIKdT458xVB+RHq16M06igSmCnf4eAZ5k1wQJW8qAyT1w8RmMppiBlzdJ6WuCG 41MVXHbx4ZiL92dc8gbOJmYhNBRXMCnrFiaFJ1kS6zdhE+5e4exzp7/4Fc522R+DM5xH MZkclducy9DCxNExyB5Zql4tERCnPGO3Hr7vsigXPBS0pBT1ecCnHukCdHC5OeE1wqIZ pd7WcZL80+ILtRPydxW7f4LNLIc5/+fEjimAY/CvOZTE0Zjf4zmDfxZ668FKneueHHlH 8Xnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681461748; x=1684053748; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AGqnouLEO1JNUtz0MIqJ+raw+hizFNAEMW7u7tnjG4o=; b=FWXcuJ0KPUWj7tErOPaujWoROAllqhc2Gjc/jc7vRWwuc1f+D5VgGFYBm+tXCw6LP4 UDm8yudUw03AVPIp73EQkSqK5F82N4isCHUPb6ues5YNr/4sc8z/4vc3fNUZkEqNlOCR Spn6XUjOnmpNv+7tUdQJR+cdVJ0KvQ0SHgVYprnJgACwIks+g/mTW1NlogtsM8mVJOX1 nwKXF3XqnNdQJp/TtnN/8ecSkvM/WdGZgZMFIfd+B4XjvVu6/WQ6ayd2tHaDLypUR5da qdQnNXD3kiHbg7g2wwvCk0Vwl367pQ/mAkVz6SnXxLn1SEwjMWVlL8ylDQuPlqgxQFA8 q3GQ== X-Gm-Message-State: AAQBX9cOOftNyl/+sFNCGWV4eTr2LEAcD8Mwd41YIYliP3fqiPSiJo6W cnPGMmkXPwC4o63djbA1OjI= X-Google-Smtp-Source: AKy350bz+dctaNWf7J0lAYs/1Z/QIvDZQUihRIrBUj9FrCGY93Os2F+uO2Er9+22S3qXAncsisP+Pw== X-Received: by 2002:a17:907:3187:b0:94e:e5fe:b54f with SMTP id xe7-20020a170907318700b0094ee5feb54fmr1244053ejb.23.1681461747991; Fri, 14 Apr 2023 01:42:27 -0700 (PDT) Received: from hollewijn.internal (217-19-30-105.dsl.cambrium.nl. [217.19.30.105]) by smtp.gmail.com with ESMTPSA id y12-20020a17090614cc00b0094ec3271be5sm1367955ejc.53.2023.04.14.01.42.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Apr 2023 01:42:27 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: "PANIC: could not open critical system index 2662" - twice From: Alban Hertroys In-Reply-To: <010201877eb3c125-d9aff979-cad6-407b-8b4e-012487666088-000000@eu-west-1.amazonses.com> Date: Fri, 14 Apr 2023 10:42:26 +0200 Cc: Laurenz Albe , PostgreSQL General Content-Transfer-Encoding: quoted-printable Message-Id: References: <01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000@eu-west-1.amazonses.com> <4f4ab16320d7b3a4e9950eaabde752dce204dc77.camel@cybertec.at> <010201877134be1c-fb837249-04a1-4cb0-a13f-c542425b50a0-000000@eu-west-1.amazonses.com> <010201877966fc0d-e09f5a0b-e846-417b-b30a-b8d57ac61fa3-000000@eu-west-1.amazonses.com> <010201877c0423c8-0a35b03c-d274-4857-ba99-ff7ea51d676b-000000@eu-west-1.amazonses.com> <8486f71f9c11e98af787e15e2d2ed0ecfdbf6ee7.camel@cybertec.at> <010201877eb3c125-d9aff979-cad6-407b-8b4e-012487666088-000000@eu-west-1.amazonses.com> To: Evgeny Morozov X-Mailer: Apple Mail (2.3608.120.23.2.7) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk > On 14 Apr 2023, at 9:38, Evgeny Morozov = wrote: (=E2=80=A6) > I don't know whether ZFS zero-fills blocks on disk errors. As I > understood, ZFS should have been able to recover from disk errors = (that > were "unrecoverable" at the hardware level) using the data on the = other > two disks (which did not report any errors). Thus, PG should not have > seen any corrupted data (if ZFS was working correctly). > = https://unix.stackexchange.com/questions/341614/understanding-the-error-re= porting-of-zfs-on-linux > seems to confirm this. Am I misunderstanding something? Your problem coincides with a thread at freebsd-current with very = similar data corruption after a recent OpenZFS import: blocks of all = zeroes, but also missing files. So, perhaps these problems are related? Apparently, there was a recent fix for a data corruption issue with the = block_cloning feature enabled, but people are still seeing corruption = even when they never enabled that feature. I couldn=E2=80=99t really find the start of the thread in the archives, = so this one kind of jumps into the middle of the thread at a = relevant-looking point: = https://lists.freebsd.org/archives/freebsd-current/2023-April/003446.html Regards, Alban Hertroys -- There is always an exception to always.