Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pkjtc-00037v-NC for pgsql-general@arkaria.postgresql.org; Fri, 07 Apr 2023 11:04:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pkjta-0002Ki-H3 for pgsql-general@arkaria.postgresql.org; Fri, 07 Apr 2023 11:04:42 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pkjta-0002KX-1X for pgsql-general@lists.postgresql.org; Fri, 07 Apr 2023 11:04:42 +0000 Received: from mail-ej1-x62c.google.com ([2a00:1450:4864:20::62c]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1pkjtX-0003uO-Fy for pgsql-general@postgresql.org; Fri, 07 Apr 2023 11:04:40 +0000 Received: by mail-ej1-x62c.google.com with SMTP id 11so7623748ejw.0 for ; Fri, 07 Apr 2023 04:04:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20210112.gappssmtp.com; s=20210112; t=1680865477; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date :message-id:reply-to; bh=p0riIXAfCBlCWZvI//M5ML/zhmtcBgyMJrHIzVhxVGY=; b=EP4/RSlRL5bgBbLPtUqP9xHVxAfAjHHjMc/tFrkCdj12IDYi3mXle+rf+Wz8tSBvML R1IYaG8Kz6UL+a04vs/A5FJYOgj1gTrfoy7WMwUlZyUjTxhxKatX+E1GIeft1/UKPqJE NUk7G7hXssIneR+Bfgn2qfSnW72pF5oJWu0I0By8k3ns33vvPbpacbL1dEW3QBTukXer 2/K6x+/FOiQP+yb5FB92Jy6TCN7UsPoHTwPxDhHLbhdEK7gLucfWUKRO7w6bFKVxfVF/ JA9Y7WIrpg3AFz78c4HoRftdTkdpI8jkP3rkmvqSDwRlwnIxX97123CztaBTJPU/O25t NyBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680865477; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=p0riIXAfCBlCWZvI//M5ML/zhmtcBgyMJrHIzVhxVGY=; b=2oO0fomw77m0K5L6EkpN4NSk0XVO+31kUf8QeeGHC0ThclMPSm2Mb8PVH0Qyu0Dtiy zYMjjwi4OIsSncb1BiaApzCWphmuWiqpW0Rq695CHgCPPfOUlsjXkbc8deWxzaPkuULk hgmjuD/qWCnOd2qTw7y0hU1RwIpH95fzPOQOkTWLMxbk62yeaBJEcsNtH6gmGhLO5Ltz 0Eh2pdtGeecMoD/tejuW/1ceA1Xp9x8vn/Wo/OkGYlqk6xQtR+Y24kuu78Joqfofu6Cp FH2ixoTtA+xCIjb7s6d2A3MmY13u+JEdyoj9YafcWGlTMfGtBcRF3B63vw0xVpYULHy6 MwWQ== X-Gm-Message-State: AAQBX9cYrN5r3JVa7P+vxq4E190ejKuKwa+QDsJT0kpTiC2BqRNbhiMn V2J4WZmugudXGyuyazmazPx01c755km4I8gVlzE= X-Google-Smtp-Source: AKy350Y9wZIMY1ZloTbvyZX3DfOgYkTxRCXU0fS96IhIxPgGHT9F7GohA4/TDMoXGNtrZXOsd5+0TQ== X-Received: by 2002:a17:907:a056:b0:944:43e:7983 with SMTP id gz22-20020a170907a05600b00944043e7983mr1655554ejc.67.1680865476708; Fri, 07 Apr 2023 04:04:36 -0700 (PDT) Received: from localhost.localdomain ([2001:871:5e:4ed6:35c3:7f0f:3d7e:bd63]) by smtp.gmail.com with ESMTPSA id n16-20020a170906379000b008b17de9d1f2sm1984139ejc.15.2023.04.07.04.04.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Apr 2023 04:04:36 -0700 (PDT) Message-ID: <4f4ab16320d7b3a4e9950eaabde752dce204dc77.camel@cybertec.at> Subject: Re: "PANIC: could not open critical system index 2662" - twice From: Laurenz Albe To: Evgeny Morozov , PostgreSQL General Date: Fri, 07 Apr 2023 13:04:34 +0200 In-Reply-To: <01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000@eu-west-1.amazonses.com> References: <01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000@eu-west-1.amazonses.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.46.4 (3.46.4-1.fc37) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Thu, 2023-04-06 at 16:41 +0000, Evgeny Morozov wrote: > =C2=A0Our PostgreSQL 15.2 instance running on Ubuntu 18.04 has crashed wi= th this error:=20 >=20 > 2023-04-05 09:24:03.448 UTC [15227] ERROR: =C2=A0index "pg_class_oid_inde= x" contains unexpected zero page at block 0 > [...] >=20 > We had the same thing happened about a month ago on a different database = on the same cluster. > For a while PG actually ran OK as long as you didn't access that specific= DB, but when trying > to back up that DB with pg_dump it would crash every time. At that time o= ne of the disks > hosting the ZFS dataset with the PG data directory on it was reporting er= rors, so we thought > it was likely due to that. >=20 > Unfortunately, before we could replace the disks, PG crashed completely a= nd would not start > again at all, so I had to rebuild the cluster from scratch and restore fr= om pg_dump backups > (still onto the old, bad disks). Once the disks were replaced (all of the= m) I just copied > the data to them using zfs send | zfs receive and didn't bother restoring= pg_dump backups > again - which was perhaps foolish in hindsight. >=20 > Well, yesterday it happened again. The server still restarted OK, so I to= ok fresh pg_dump > backups of the databases we care about (which ran fine), rebuilt the clus= ter and restored > the pg_dump backups again - now onto the new disks, which are not reporti= ng any problems. >=20 > So while everything is up and running now this error has me rather concer= ned. Could the > error we're seeing now have been caused by some corruption in the PG data= that's been there > for a month (so it could still be attributed to the bad disk), which shou= ld now be fixed by > having restored from backups onto good disks? Yes, that is entirely possible. > Could this be a PG bug? It could be, but data corruption caused by bad hardware is much more likely= . > What can I do to figure out why this is happening and prevent it from hap= pening again? No idea about the former, but bad hardware is a good enough explanation. As to keeping it from happening: use good hardware. Yours, Laurenz Albe