Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p4ZcQ-0005fY-MD for pgsql-admin@arkaria.postgresql.org; Mon, 12 Dec 2022 03:36:42 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1p4ZcO-0003yz-OY for pgsql-admin@arkaria.postgresql.org; Mon, 12 Dec 2022 03:36:40 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p4ZcO-0003yp-Ce for pgsql-admin@lists.postgresql.org; Mon, 12 Dec 2022 03:36:40 +0000 Received: from mail-qt1-x832.google.com ([2607:f8b0:4864:20::832]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1p4ZcH-0001Z5-Ih for pgsql-admin@lists.postgresql.org; Mon, 12 Dec 2022 03:36:39 +0000 Received: by mail-qt1-x832.google.com with SMTP id g7so8054719qts.1 for ; Sun, 11 Dec 2022 19:36:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:from:content-language:references:cc:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=cX98zHGFREvkpLeoThqwPGhDhCpLXz47KA/dJgrnpw0=; b=pANI5VdJoktK8oat9caFwMaiSRVVLoMo2RuZ5dfNmJVRAcsti9RampUdg1EoF9cOhq TSDpo1XvcrCc0BVdaP/jLJHs7Y81unTlNEalyJaRpcLwsmeUt0DaqdDIjmUXCq70hDiD rAVotfXKaAfhzsFT2p79R8KVfKaStsoh6rc7qO4tUiTuVhhPH9cC00b75t0ch0P/pv4l 79eyzXD0i7FWRgC/2Ar0v8bo5BD8Wc5yPu1a58V+H47l9OVJ/QOvFBo3SOMR2kICZ/hO 8Dp4dGSdkxWsHBGN33ms67TErnoIRrUEQEJydWrp6dFUUJS2j7EXCgYgv83cz1WclS3Y z5Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:from:content-language:references:cc:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=cX98zHGFREvkpLeoThqwPGhDhCpLXz47KA/dJgrnpw0=; b=GBwpUNpChlJRw39B+xD10OLyaHxTpl2fjWVPrBiCL1+3oedav/6uliFXTyA3M0QkGZ j9yXl4MqfOBY6fmNZk+c4c3b0noXOl19apsChk5xqiSpYWiHgjrRfaxp+x9FhBLxPEF5 sUl2aw9XEbYdJOZn0ePrFFql1/SXglvWo5gw0BXi8GQ/rvoD3T4KgYqw6jlpODT0UkuF Dw9yOqnpAY6wbh9RQqKNDLX4Io0pRrdEMdxb4v4XlrZ9hCiDQL04z0Day3ly2lD4QYw6 yESr086Xxy3cvnDAvtAleBtCy+XKy0g1GmXsT7RiDgowVElR4E4R0HuZkntrRDbr8Jm1 2xGA== X-Gm-Message-State: ANoB5pm3oaw4REQwKU1d03QGwbbwLrV3LIfeJcOc+WRHRbafKCxzBmgK 98dLEjxjWgIIDeq2dFCIjQE= X-Google-Smtp-Source: AA0mqf51QwoaS9WtobU/CEwNI3VQPblPiR+2sVBQdaxCFshJT1BJF5IqYjeRUtJkdLqInT9jd+sMpg== X-Received: by 2002:a05:622a:5144:b0:3a8:1677:bc46 with SMTP id ew4-20020a05622a514400b003a81677bc46mr3247806qtb.49.1670816191107; Sun, 11 Dec 2022 19:36:31 -0800 (PST) Received: from [192.168.2.100] (pool-72-79-30-4.nwrknj.east.verizon.net. [72.79.30.4]) by smtp.gmail.com with ESMTPSA id bp6-20020a05620a458600b006feba101f85sm5006885qkb.13.2022.12.11.19.36.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 11 Dec 2022 19:36:30 -0800 (PST) Content-Type: multipart/alternative; boundary="------------RAQcmPOeO9JacnYfdzvju6EQ" Message-ID: <771c8913-f10e-892d-7b00-d71ca84ce7b5@gmail.com> Date: Sun, 11 Dec 2022 22:36:29 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.5.1 Subject: Re: How to speed WAL apply in destination To: srinivas oguri Cc: pgsql-admin@lists.postgresql.org References: <364439211.2445507.1670562179127.ref@mail.yahoo.com> <364439211.2445507.1670562179127@mail.yahoo.com> <1E6A35F6-6866-4FDB-8CE8-38C328FF1865@kain.org> <44581ceb3c9c96f48e131ed3de33389db07317e9.camel@cybertec.at> <203270855.2602687.1670583423055@mail.yahoo.com> <31247466-cdde-71d4-2beb-ffcf7d7fe4db@gmail.com> Content-Language: en-US From: Mladen Gogala In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------RAQcmPOeO9JacnYfdzvju6EQ Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 12/11/22 22:02, srinivas oguri wrote: > Hi, > > We are facing similar problems but the lag is at replay on standby. > How can I troubleshoot the replay lag. > > Thanks > Srinivas Well, replay uses CPU and disk I/O. You should monitor both using the OS tools. On Linux, Slowaris and AIX, that would be top, atop, iotop, sar, iostat, nmon, dstat and alike. Also perf tools "perf top" are useful. On Windows, that would be the performance monitor. You first have to figure out where exactly is the bottleneck. Then I would go about looking for newer version that has improved on WAL replay. For instance, PostgreSQL 15 can do parallel replay, which should speed things up, at least a bit. Then, you should test the new version. After that you should decide whether to beef up the hardware with or without upgrade. At any rate, you should treat that like a project. Create a project plan, define milestones, define how you test and what you test and give it a go. You didn't specify the Postgres version or the OS, so a generic answer like this is the best I can come up with. Second best would be the answer to the question of life, universe and everything. Good luck! -- Mladen Gogala Database Consultant Tel: (347) 321-1217 https://dbwhisperer.wordpress.com --------------RAQcmPOeO9JacnYfdzvju6EQ Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 12/11/22 22:02, srinivas oguri wrote:
Hi, 

We are facing similar problems but the lag is at replay on standby. How can I troubleshoot the replay lag.

Thanks
Srinivas

Well, replay uses CPU and disk I/O. You should monitor both using the OS tools. On Linux, Slowaris and AIX, that would be top, atop, iotop, sar, iostat, nmon, dstat and alike. Also perf tools "perf top" are useful. On Windows, that would be the performance monitor. You first have to figure out where exactly is the bottleneck. Then I would go about looking for newer version that has improved on WAL replay. For instance, PostgreSQL 15 can do parallel replay, which should speed things up, at least a bit. Then, you should test the new version. After that you should decide whether to beef up the hardware with or without upgrade. At any rate, you should treat that like a project. Create a project plan, define milestones, define how you test and what you test and give it a go. You didn't specify the Postgres version or the OS, so a generic answer like this is the best I can come up with. Second best would be the answer to the question of life, universe and everything. Good luck!

-- 
Mladen Gogala
Database Consultant
Tel: (347) 321-1217
https://dbwhisperer.wordpress.com
--------------RAQcmPOeO9JacnYfdzvju6EQ--