Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1gAK7R-000460-5u for pgsql-admin@arkaria.postgresql.org; Wed, 10 Oct 2018 19:26:05 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gAK7P-00018j-AN for pgsql-admin@arkaria.postgresql.org; Wed, 10 Oct 2018 19:26:03 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1gAK7P-00018c-34 for pgsql-admin@lists.postgresql.org; Wed, 10 Oct 2018 19:26:03 +0000 Received: from out3-smtp.messagingengine.com ([66.111.4.27]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1gAK7J-0006GK-2t for pgsql-admin@postgresql.org; Wed, 10 Oct 2018 19:26:02 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id C7AD721F82; Wed, 10 Oct 2018 15:25:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 10 Oct 2018 15:25:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=KuLvFUzUSZ/V6or/XkJMVyHmqDY gn6hRhR++43X0RsA=; b=YT1YphDUIde52NgY6j41dOMpuallG9R+OTm+p8DJ8xN f+JU5AlPV54oe+BajGE7qFp6WjGM2eIQEj/cGVk4eHWAD8ppUXEe213Yu7gAQRZZ 4ift6Fo+MlpdCSoZSEHo8wkub2ABY3YUxM7JyrXi1x70iJkhGhHlDfX3XG2CQuTP zWJfqQgpbDm127xpOCO/eJGnLFFQ/Yu89lWGlkAp/F+kRdS3KDqYLMoBPm3EAX1S 15jraOHeKmRg9MMz6QjvCk8QcqgXExCw2dSeqKq3kyplhkaAsBj8JQfr+nTiTLB9 juqUI3fsvT5huoATJlFde2ib8/hb80/YoqD/He3GiNw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=KuLvFU zUSZ/V6or/XkJMVyHmqDYgn6hRhR++43X0RsA=; b=ZL0aLOzWVIH+taSGsvKGsl MVpj7zrAdTuMGc68B2xqLFpcgfb90rVSLIgf95/BDF+ISQvmi6Mkc/MfoH5k2Yew sHcYLIuN6sGubRbbdv6UF13QPgTVl6d0zDuDnjQavYtjp4H1oJaE6i7IyFXsZjNE PmYQKA/6U0anw94IuE14H0eA0OPPa8oUjJw6W/lS8Ho+mVuWgH/qZ92mozsSSbYA CEoRzNEQUUCXhV/LLfxq39Rk5yJyq36pTsENYwSZqbea/Eux3GYPlW0ONIk7jaJb YJoijr3OFkRqwOrf5gN5UsrBhmbIXDKQRnrSzkyL5XX9C7ZHRcVmthgCoFvBp8Ew == X-ME-Sender: X-ME-Proxy: Received: from intern.anarazel.de (104-2-144-147.lightspeed.sntcca.sbcglobal.net [104.2.144.147]) by mail.messagingengine.com (Postfix) with ESMTPA id 35F40E4623; Wed, 10 Oct 2018 15:25:55 -0400 (EDT) Date: Wed, 10 Oct 2018 12:25:53 -0700 From: Andres Freund To: Mike Broers Cc: pgsql-admin@postgresql.org Subject: Re: root cause of corruption in hot standby Message-ID: <20181010192553.yithduf3yrxasilz@alap3.anarazel.de> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Hi, On 2018-09-06 12:28:18 -0500, Mike Broers wrote: > So I have discovered corruption in a postgres 9.5.12 read replica, yay > checksums! > > 2018-09-06 12:00:53 CDT [1563]: [4-1] user=postgres,db=production WARNING: > page verification failed, calculated checksum 3482 but expected 32232 > > 2018-09-06 12:00:53 CDT [1563]: [5-1] user=postgres,db=production ERROR: > invalid page in block 15962 of relation base/16384/464832386 Have you compared whether that block looks significantly different on the primary from the standby with a hex editor? Greetings, Andres Freund