Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uyYkn-009e4x-JB for pgsql-general@arkaria.postgresql.org; Tue, 16 Sep 2025 16:42:05 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uyYkd-001Tkg-Os for pgsql-general@arkaria.postgresql.org; Tue, 16 Sep 2025 16:41:56 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uyYkd-001TkY-CV for pgsql-general@lists.postgresql.org; Tue, 16 Sep 2025 16:41:56 +0000 Received: from mail-lf1-x12d.google.com ([2a00:1450:4864:20::12d]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1uyYka-001A0x-24 for pgsql-general@lists.postgresql.org; Tue, 16 Sep 2025 16:41:55 +0000 Received: by mail-lf1-x12d.google.com with SMTP id 2adb3069b0e04-5679dbcf9d8so5014582e87.0 for ; Tue, 16 Sep 2025 09:41:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1758040912; x=1758645712; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ginFwg7INBwsCDapl9jswkHLY9U92iOl5o40UJpybDo=; b=RB/dTDMyOj6J5C9WISnEqqWb2pUb+UMDeeztjmSeGUvdGlZokdKQ9Y3FK/ogP1LFCT skjWHXDloZ/KYlgDvKgcSrktrMAnRWvZLeKdDfkpqZiR5xfC0RyBnx+qK9UJmPmnTTKU ejLFkoA0GDVnKRS8dAaMwjZmnbXkmupYdnoUMSbaKZ8YRwWXZwym5+ceDNw6D0M8Ce9r 8YbSgUybCj9pYJUKpWV6FUlpz41seD3KSk5aWr2tCAZWW/mJyFzPqkp2/lLyBSsnp2u+ u7wzA/jGK/EP+VAKzK5VtVhBvUm2qTnhfPltPFBh3OyGvMmptBJrsaJl6Bb1+hGIyLJB JdSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758040912; x=1758645712; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ginFwg7INBwsCDapl9jswkHLY9U92iOl5o40UJpybDo=; b=kZCjl8rOOpR9Ti6lCLIpQM3DVUKDhgpSF4viyuKvBUbdDNiwY0FFfRVmIPzRmjeLqH HrmzVVSygDIZYcdb17F6Br/80JsM2WWybgQ6yOf7sCZBYlxKerVQNMrGQu1/B0JXt/5D i4B64REFZppH3PnNbKVISbT12UK6zDMVYhhPX26+ZdvZxAl5b7eqvealGvbp9IBN/SIl ILCtevgCGwFI53RfJZ7ailM2an/xj3Ro9ywOVHHmVhmr7C6sOuRoAkB1CkgndVkuja1o aZZRa73b9RfjZCf5g/pr0coXh0pYLamepo4p5etsPHgBl0Z45b7fxJgf/Fb39qaOAqCr KgfA== X-Forwarded-Encrypted: i=1; AJvYcCXoVBLriYIsP9XIkcK3CTa5Clr1Na9wfTnTszxR5k7mPxjiFmK3JLr0D7A1rrIbFEjN2CMPAFtQAJAQWhkd@lists.postgresql.org X-Gm-Message-State: AOJu0YzACwuXE665gwkDgAKAIAaqleg3Bp73ZRA5frlij1+S8/tkMoog nCR8TTM8KaJpG4kOUDOjycsD/anQfgRyD7cWde4v0zVgihJXO99FA/LZubxH8t2gxFnHwbXdViO 8mpYrIVeLuCIlXGITDI5v67HwH5Wx61s7tw== X-Gm-Gg: ASbGncvndZEXfWFzEJO0QArpNMG5s4+tGmWtjOIwjPWXt8gt3d7fNiJQmlscBUhIP/H e08sSWmIL2rIe3yJxICDBzRIuU5pR2POlZ5Dty2QkQkcHiuv+eJa7IszcW3QU5kowIw1c6NQgaB 3TWesBnWEdscCr5/9WSfRxWEywul6w55FVkeCG9k7QbUKbiijOQNurAYOCU66nhAfnPHDkYOIgf NzZZ8Pwmt5SnmZf85e7GjO1DUYJWyxqFiOjQgsk10g+XztD8b8O6QWe8EZs3ljHyIoQINTFCFj2 0qoGGOd4 X-Google-Smtp-Source: AGHT+IFQGj1VAXldn/H5BOC4VIpOcnqAWa/42qPeeL5CLPJC6PWimsFe7WGi/qye0z5Tpkd4acuoTGrFoA88IXdD094= X-Received: by 2002:a05:6512:3caa:b0:576:d5df:fe1b with SMTP id 2adb3069b0e04-576d5e00195mr583431e87.10.1758040911997; Tue, 16 Sep 2025 09:41:51 -0700 (PDT) MIME-Version: 1.0 References: <20e5800983ca176d696f2f90ecc0a830@postgrespro.ru> In-Reply-To: <20e5800983ca176d696f2f90ecc0a830@postgrespro.ru> From: Merlin Moncure Date: Tue, 16 Sep 2025 10:41:38 -0600 X-Gm-Features: AS18NWDqWKhuc1pKkTXwKRbpMIRp5F0f4jq9aURSsAqm0HwPJSF44agYGkcht24 Message-ID: Subject: Re: Silent data corruption in PostgreSQL 17 - how to detect it proactively? To: Vladlen Popolitov Cc: Pawel Kudzia , pgsql-general@lists.postgresql.org Content-Type: multipart/alternative; boundary="0000000000000f73b7063eedcc25" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000000f73b7063eedcc25 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Sep 16, 2025 at 7:25=E2=80=AFAM Vladlen Popolitov < v.popolitov@postgrespro.ru> wrote: > Checksum calculation takes ~0.5% of query time, it is not bottleneck > in PostgreSQL. I consider checksums=3Don to be a mandatory setting. Often, these types of things are not bugs in postgres itself, but bugs in storage, the underlying operating system, or extensions. Checksums can and will protect you, and may even bring you close to the thing causing the corruption. Given that your replica is ok, this very much smells like a similar type of issue. In a prior case, I was using pl/sh to load data to the database with 'copy', and for what I believe to be o/s issues, was getting corruption. Enabling checksums completely addressed the source of the damage. Turn them on! merlin --0000000000000f73b7063eedcc25 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Sep 16, 2025 at = 7:25=E2=80=AFAM Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:
<= div class=3D"gmail_quote">
Checksum calculation takes ~0.5% of query time, it is not bottleneck
in PostgreSQL.

I consider checksums=3Don to= be a mandatory setting.=C2=A0 Often, these types of things are not bugs in= postgres itself, but bugs in storage, the underlying operating system, or = extensions.=C2=A0 Checksums can and will protect you, and may even bring yo= u close to the thing causing the corruption.=C2=A0 Given that your replica = is ok, this very much smells like a similar type of issue.

In a prior case, I was using pl/sh to load data to the database wi= th 'copy', and for what I believe to be o/s issues, was getting cor= ruption.=C2=A0 Enabling checksums completely addressed the source of the da= mage.=C2=A0 Turn them on!

merlin
--0000000000000f73b7063eedcc25--