Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1krCD9-00060P-NL for pgsql-admin@arkaria.postgresql.org; Mon, 21 Dec 2020 03:50:16 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1krCD7-0000lO-9D for pgsql-admin@arkaria.postgresql.org; Mon, 21 Dec 2020 03:50:13 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1krCD6-0000lH-Ux for pgsql-admin@lists.postgresql.org; Mon, 21 Dec 2020 03:50:13 +0000 Received: from mail-ed1-x529.google.com ([2a00:1450:4864:20::529]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1krCCz-0008HP-It for pgsql-admin@lists.postgresql.org; Mon, 21 Dec 2020 03:50:12 +0000 Received: by mail-ed1-x529.google.com with SMTP id c7so8264634edv.6 for ; Sun, 20 Dec 2020 19:50:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=AyEdfkAufXQIWLJlRWZtAXGl8snvEaZvL+6EOlbjNeE=; b=RKi8ejZqLms4tQgU1QvNXQsOy0sACa34vhedHbi5ptwJ0hbQ11TEYCEIhZkGVkQNqd 1NnbSXv8wYdr5N2sH/esc0p+mMLj9oGASqXhjC/yefxjb7uEnjjcci504HwJjQxS7HDH W3PHtVmkf9VYC+w5SaO0oYTCy6jSpjmpQ8gVrj0P9Ma5H4ev6FTLKrFmljw9jz8JyPLi IvzLwdgSzH8ybPuejVzF6vGwBjBtcTtBo3HiD2lfbsck5qfyW0ntRaiKEi3ifxYSbp+V lYksSEojsBADyjTQu55WWENpjxtL7lFTuISUdmgSW/PLwDuOFlZupqItVDR5UP6fLbL3 xhDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=AyEdfkAufXQIWLJlRWZtAXGl8snvEaZvL+6EOlbjNeE=; b=LbR/HF+Gi0hrDPj53T8sX2DtXTB7UExJ32kyEcZcLpjlsU02ZqZAZ9CQLkurnMYdPT hf1BnS//N83FI6ESHRwL8ofNYCYkNCeRr8pUKmiEHPFxc24MEBugqCuHitbDUZXyvvSS 5zIaSXsE86UHpxXbVn4I0GODS9379Jdk40ExUzudw/dGReQCGKi2iR5gm9YhtaGRftHK uGddz01qZlCu6152MKN16a214kHDtW9qlUF6DmV/SuA1Xu1Xizt1SEpzvYWFaVYNhdss YbS5ghVt19cFO8MmzsUkYqsF2Qm0PYjY4N8z4b8HL5r5eeqkcVyf/OCHNPDDSUHCaGhl NkCA== X-Gm-Message-State: AOAM530Q2m61qA+ckhIlqxJ8aGw1uycZZ0EKr/n3MgGPinIM3YJFU1gE XT1dNWBHrxtoCLmvP0q/bqZVxw== X-Google-Smtp-Source: ABdhPJy1nhnDQA3BI9DE9GyK5I+TJFs2QcDgWmfjhGYqG+MhYgQ67zmVJ6lO4smkIKRxTcLJ0rUBSw== X-Received: by 2002:a05:6402:1386:: with SMTP id b6mr14384386edv.42.1608522603735; Sun, 20 Dec 2020 19:50:03 -0800 (PST) Received: from localhost.localdomain (62-46-249-101.adsl.highway.telekom.at. [62.46.249.101]) by smtp.gmail.com with ESMTPSA id r7sm21241830edh.86.2020.12.20.19.50.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Dec 2020 19:50:03 -0800 (PST) Message-ID: Subject: Re: Replication failed From: Laurenz Albe To: Rambabu V Cc: Sreejith P , pgsql-admin@lists.postgresql.org Date: Mon, 21 Dec 2020 04:50:02 +0100 In-Reply-To: References: <9492f3e4fff86b6fcb1d064d680920e8579d9c37.camel@cybertec.at> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Mon, 2020-12-21 at 07:46 +0530, Rambabu V wrote: > On Fri, Dec 18, 2020 at 2:13 PM Laurenz Albe wrote: > > > On Fri, 2020-12-18 at 00:56 +0530, Sreejith P wrote: > > > We had 1 M x 4 slave server streaming replication using Postgres 10. Was working successfully for long time. > > > > > > Suddenly all replication servers got failed and getting following message. > > > > > > 2020-12-17 22:24:32 +04 [1587]: [357-1] db=,user=LOG: invalid contrecord length 2722 at AEA/A1FFF9E0. > > > > > > Requesting help for identifying root cause. > > > > Looks like this problem: > > https://postgr.es/m/77734732-44A4-4209-8C2F-3AF36C9D4D18%40amazon.com > > > > Was there a crash on the primary sever? > > > > There is a patch under development at this thread: > > https://postgr.es/m/CBDDFA01-6E40-46BB-9F98-9340F4379505%40amazon.com > > > > please let us know how to apply the patch, we are able to see below attachments , once downloaded how to apply this patch. > > Attachment Content-Type Size > repro_helper.patch application/octet-stream 1.3 KB > ready_file_fix.patch application/octet-stream This is still under development, so you should not consider using the patch in a production system. The bug is only triggered by a rare coincidence (crash on while a WAL record is written that spans more than one segment). But if you want to review the patch (see https://wiki.postgresql.org/wiki/Reviewing_a_Patch) to help getting it applied soon, that would be great. You'd have to read the documentation about building PostgreSQL from source. The path is applied with the program "patch", typically with patch -p1 <../ready_file_fix.patch while you are inside the git repository. You need some familiarity with building C programs from source. Yours, Laurenz Albe -- Cybertec | https://www.cybertec-postgresql.com