Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1khaoz-0005d2-PQ for pgsql-admin@arkaria.postgresql.org; Tue, 24 Nov 2020 16:05:37 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1khaox-0002xw-DR for pgsql-admin@arkaria.postgresql.org; Tue, 24 Nov 2020 16:05:35 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1khaox-0002xm-2t for pgsql-admin@lists.postgresql.org; Tue, 24 Nov 2020 16:05:35 +0000 Received: from mail-ed1-x530.google.com ([2a00:1450:4864:20::530]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1khaot-0004dp-TA for pgsql-admin@postgresql.org; Tue, 24 Nov 2020 16:05:34 +0000 Received: by mail-ed1-x530.google.com with SMTP id v22so21432399edt.9 for ; Tue, 24 Nov 2020 08:05:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:date:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=26yMiDIWVtrnvccVfZjGjGPZ9y/7Jv3ObGU6aPZCxzc=; b=xmeicVsc7ax93qRaDc48XT/V0Xp+soR7w+WogOsDrbWRVSoQEEUj+/pLGLFVRSF/pG 0SVzNEpzxlIILMFqh1R6Ga8/Z23erATfLbvATvTF0AeErYvrWtE5gVeptEicWDUt4DWC eXTtWsRYGTi1RbUMpgSRO8ca40PZF8LcuKRSwx+lR9pA2Peul03FHnXRuh4V9qfcnnVo 65hqa8rdKBCsibj+OvfR0UWazNvX3N477nEPDWbM84PSOgtz6L2sTlg2V76+N+sC9MK/ Lu6pt8t1cl/V1iWHJT52cZSbPC8QnW8EL0Jh2fo07SlqD/n+WCPnUxNTo3xTCi59shsm mWVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=26yMiDIWVtrnvccVfZjGjGPZ9y/7Jv3ObGU6aPZCxzc=; b=ryOAauhXVf7ZLOWZbxizQyZs5W0iVya/VlKg1srLgWucupmWnyEAGO8wBolvCyLxtx upuEFKI3Bye2PR5lsKY/vujEeapQ8mgwNtTzWz0x76I27dqenOtmQZ8rQnV2n8ATVUX8 I+itGV6Smw3iN0E3+Up+XCsAtAz7VlBq360kbnBYn2GTK4Bb/+URGuTF4el5/02p2kGX 2wwqxCU3cl8TW2Z/YamErg80SqMH9GapQHd4HPl02eCospx1E/0WfKNlLTVT9oP/QNSO gVUjEkQvrKiphMFEtmGE3S/6vRR0zvnXreL7bbpFztaaeJynlqL6JRsunZSzsz/3l16Q hs9A== X-Gm-Message-State: AOAM533thMky8IDI3dNnx7SwNZxc3ozp8nvktOezU29XxryP3o/uAY5H OjuPRh/63iFpcbs01UpOU4bPgA== X-Google-Smtp-Source: ABdhPJxlsEUDpv9mw7XhufH/+7Eo+bMaQnudLDLtMRrqyLyAskQuFWOUXJOzGwr3toEgj6zD7wcv3Q== X-Received: by 2002:a05:6402:206f:: with SMTP id bd15mr4459942edb.122.1606233931146; Tue, 24 Nov 2020 08:05:31 -0800 (PST) Received: from localhost.localdomain (217-149-174-92.nat.highway.telekom.at. [217.149.174.92]) by smtp.gmail.com with ESMTPSA id w15sm7083518edx.37.2020.11.24.08.05.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Nov 2020 08:05:30 -0800 (PST) Message-ID: <3c462aa5a2da6b00f52fa90b69b7a542011cd924.camel@cybertec.at> Subject: Re: Streaming Replication replay lag From: Laurenz Albe To: Nikhil Shetty , pgsql-admin@postgresql.org Date: Tue, 24 Nov 2020 17:05:29 +0100 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-1.fc32) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Tue, 2020-11-24 at 19:54 +0530, Nikhil Shetty wrote: > We have one Primary and three Standby Postgresql setup. Two standby in DC and one standby in DR. > > We have configured streaming replication.When there are a lot of write transactions > in the master, we observed that DR lags behind the master for 5-6 hours. We initially > thought this could be because of the slow network between DC and DR and this is normal > but when we saw the replay_lsn and receive_lsn , they had a huge difference. > > DR standby was receiving the WAL's in time and there was no delay, the delay was in > replaying the lag. Can anybody point me to where I can start with the investigation? > DC (Master and 2 Standby) and DR(Standby) have the same setup in terms of storage > and some database configuration parameters but DR has a little less CPU and RAM. If there is a delay replaying WAL, the cause is probably a replication conflict. Set "hot_standby = off" on the standby to avoid the problem totally. If you want queries on the standby, set "max_standby_streaming_delay" to 0 to keep replication from falling behind. Yours, Laurenz Albe -- Cybertec | https://www.cybertec-postgresql.com