Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r6a1k-001U85-5Y for pgsql-general@arkaria.postgresql.org; Fri, 24 Nov 2023 17:31:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r6a1i-00DFXW-Le for pgsql-general@arkaria.postgresql.org; Fri, 24 Nov 2023 17:31:38 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r6a1i-00DFXO-2j for pgsql-general@lists.postgresql.org; Fri, 24 Nov 2023 17:31:38 +0000 Received: from mail-ej1-x636.google.com ([2a00:1450:4864:20::636]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1r6a1c-007Ag0-0F for pgsql-general@lists.postgresql.org; Fri, 24 Nov 2023 17:31:36 +0000 Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a00a9d677fcso300194166b.0 for ; Fri, 24 Nov 2023 09:31:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20230601.gappssmtp.com; s=20230601; t=1700847090; x=1701451890; darn=lists.postgresql.org; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id:from:to :cc:subject:date:message-id:reply-to; bh=nI3M6X7chtEgvhj7FwSflb+0WoWawjriyhBXkg/3fSI=; b=TC5d6AOpcTW8bvmF51/VPYTyIlCg2wecrQSyLAMgWNC6nFkdMu83eIkLzXqKhwvUPS U5F5XN2seKyvoz2MY7G7bOgmUjcCgOahzE1N0todzH+K2pteUSo1zOuz84I6OVMqGygT is/P8X8nxjtmpBY1Q5VC1HQmEurwrkJBRf6HikDbcyB/xPIBYxzs9yS5YW0vqcWka4Y4 YCfSufLvnFdeKJu0vAz6tnlLJR9uEFUV6yBh0ajc+coDtE12w4mVtIJnVnkpW7ST/jGK d3rimwZ1VSWPmlCXm3o74v4ExjBreNOG+p/C7VO8suaRbqM8w8Q2k/ub5qLXSAOGmR3g LXIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700847090; x=1701451890; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nI3M6X7chtEgvhj7FwSflb+0WoWawjriyhBXkg/3fSI=; b=OUAXOaFdUxD+ISBbjGOf0Ip1sVbdOrg+eH1Cn21baBxhz3dJ+7GP7pwWB3b/h8wJfL GfhG1BCbhuGbvcf9HykIvLJgmVKpkj/cP1XXmLV1nnQop37ZCL9Da+6n2BxJg9odwbv4 1BlQ3pYHtWOPAJFO9yPD8W5yawcKzvpmpU9uSORlGm532+Mm4EQyVBHFzZaHaqgttStC 97vV9fvkbcwHN3KjfqzP1In7P92aSxk4ggGANvAXzK1gqhETnSCyq6G68ViesXRRofgB RC5Uz3nyYgrWcNX3Di5qPZn8QkvcgqUuJ8hyAAnJ+7t6gukSfIOsUNQjGtAY1e9+C6qo YZTg== X-Gm-Message-State: AOJu0Yzn55sjwYHph7/iOaV7JsGU6nrt9j3asVlJsAfmZifSSODyOeYn I576ITF326FSgxH/KOEQOf0H5jEZrg9UBWy1Fz0= X-Google-Smtp-Source: AGHT+IGeigKD4/mdCXkAySURsMnHn/83u93aI86TGerYmDtqvgf7ixCG+4ZJm8GJdkTwrKO6ux6lxw== X-Received: by 2002:a17:906:6ad9:b0:9e6:4923:680a with SMTP id q25-20020a1709066ad900b009e64923680amr2847243ejs.3.1700847089988; Fri, 24 Nov 2023 09:31:29 -0800 (PST) Received: from localhost.localdomain ([2001:871:260:9b97:159d:8e84:8b31:67d7]) by smtp.gmail.com with ESMTPSA id x22-20020a1709060a5600b009c3827134e5sm2290567ejf.117.2023.11.24.09.31.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 09:31:29 -0800 (PST) Message-ID: <6097c54582a91b40eaab2d3d24152d98fcaf998f.camel@cybertec.at> Subject: Re: replication primary writting infinite number of WAL files From: Laurenz Albe To: Les Cc: pgsql-general@lists.postgresql.org Date: Fri, 24 Nov 2023 18:31:29 +0100 In-Reply-To: References: Autocrypt: addr=laurenz.albe@cybertec.at; prefer-encrypt=mutual; keydata=mQINBGGDwAQBEADgbWy5cKXQld3N2mF+DFyiNFbi2oBl2T+XgxpPF8wTRw2D/u4bBKXP0SYSE/lA86jIVNWWU0gf1KODIkVvgJm2w4vH2VBV1b7ddVViGl1Iu+9zaRnv9wulhnH42KefepXnoean6UT1EzLM0opF/Ik0j+40TxdRtobkBprkQUyHDXWlHc2ffPs3SipyFEP9AVLf7ejRC46CXWDnsqjOBSMEW8Z4HiK/8RrPZBsKLts8dJxKF4pygOdJb0CWk8k/X1jbcfdxo+zOLjOMvJcSJ2pFdJmQHU+JufB3rePziqQ2S9Ur6sccr9XnTC1GVBWN4Lf5VHq+vf+bFJjVwg+2hrySZnAVfcOrxoqFLErr7ug1zN2nM1kcpgA4VWn4gxlJtYNYYq+9WxX5dtvnNANlG3ZCrRKQzl8lxtzoF6Zo7LUhEqPaHDwn7Rvs+IdbOn41lF5UDTJGqmC4gS/bZydW2Fy3YWm4aSaN9fgFf8D+PVkrlKAZB7gBLz1TyHjbcRf85cYF+GKKrDld5SzMB/V60VX3oP/Eo8ikFpyWaqiz1f9X7MBot3/PjJkY+wDzp3nmb19QEcOBuQiSQ4xds2r0HewbuHTAR68u8jNNMGmpm2j4x+g09Jd/WQDjqlTBZ/jEltH41fYCCPWMfljXTOOXu2eLNGdfi7ETZogtwjM9oTtSPQARAQABtCdMYXVyZW56IEFsYmUgPGxhdXJlbnouYWxiZUBjeWJlcnRlYy5hdD6JAk4EEwEIADgWIQR0CqhbZGGABqoaSbdi8bhXA2EdmAUCYYPABAIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBi8bhXA2EdmM/6EADK232JCwmBzhlj8h7U9CjG6kx0JHP3uJGv+XfsHtHAlmY/RCwF1BHMEsRlk bT5UrLvJ2jb99bA9QARzhFaxzyn0F/BUKzuIjRGNs/n6d5dNUFA0kOt8sX+TacmC GEyjEBCrVCm4ranBiUyePn9NhHNWnaex7pJyqvMLLdwW9BEMJx0Fqo+DN8ukbXmYRsmhEtd3ue+x/luYmOmJnaGtzInaY5aOJYbW9XqoRIZkZvOCgbi1FfvNmoqWa+3oVxTOgw9RafjJDyW0lTHzKGjbGI5ofMU98l+/hKJFYJqWUF6VpFJY5YIcN/1lf4ZICMwDl+MPIVo/tpq8L10seJL28nLlvw3K+cI+TVW8IW/qL/LyVoDofI3USeOORuYmhpWRhik8JXX6xf3v6GrRilJIPWNFIJbxm1ZblQiQnOw3IOW7T+8nAmPin1HKqM3VrOrJQ2VtShsefNBibNAsr1oFaqcDBkn3yGG8i6CTW+FyO4PZ+/EwNxMVgktxbYdy5AT1/lpXr5tB+phhLIyVfiBvrWs5EThxYMQ/L8Y85c3GMsAy1l/x4h3jqySIYy3SCU9+jc5UVuNnXljbvkEzJ+NLWJ6C1rACFWrMszgPdh5tCrlRY9PpmYll4JbCgb8BtxEIUmR+xr50/ZElEK5iml7Q00KUekCcDt+36PsyGFTXBzNOrkCDQRhg8AEARAAzOZ2tLHlI4rrhG411h6cdCFjBZxuljaFCxFyHn3m6wbGLqwBUWC5k8UrRqjHMz88KcTSaNO7XGAmCqPdWd2SeflPZRnNTbjsVpw7mLdffsBm4JX7kki2Pvk5h0NtYeidXT1PSpc2ri4DutYXuT9uD8RAm1wUDCE5HQNUihT/WH6opt+hskHW21uHao0+y822tG0QQcGMqdQR5Vxdxj89wiEPdqW+HpU/oOZIhrf2E7prduAppxixjHy/o1rcnoznnJvc8D3+YgI9O0LrBMij89dM55pRGbLovTR1oGR3U74sX774+0xmSzeIKwZfiMUz7Atlvfk5SHOsRUFPN2Ux9kaXiiBibQpHFxt7b lDrT4wxdLJ/XCdbPPAyl+lZtOLsaHEEZvYNyTXwZc35dVf3R4/oz20HoG6s7ct8e1 AQygj43XAERzty9SkWgxs8+grp1PrGx6FHVSYRqBM8dS/ZR6yRVwOwJXPyaSSqfIF21DkE4j1y4n+ItSewPGoRp8K/yWCikt6qlkVkO2ASNIiX04fAbtzwVOaNn8ZMRNqyvLc1fED4sr49onE4cAIcBLjcC3KL+w9DUGRQCdziROj5H2Yl/sXGPdMciUHo/Uz2rggc+2th3bQiMhrHWSsBpUkDQp0yWewemstPpPgBL3h2fHKaX8B9oH5Qu/H1IgrOuX8AEQEAAYkCNgQYAQgAIBYhBHQKqFtkYYAGqhpJt2LxuFcDYR2YBQJhg8AEAhsMAAoJEGLxuFcDYR2YuPwQAMkpGtR80pQ1gVsONhdkqj0H2eU66efP/gO3CoyaoIcvrpKYj7C2HipVSmkt1gpByL0X4AMQ/vKuknUz3wd28Ba+G1dCfbVs/Xiusq+SmpUj5rTwmYqdSjWMuCo1R6oS5hdJMdUUJYGMT0QkVlm1KnW8jkmCTl9GzjDxOAsN9O6/6lPzaGFtk9XF+34Bry/N4HKiJkqpC4+UTd0AprPfzJ2jdT64e1F0+W88X8y1bTTgNrHwK4mDiLnlE4SKRuEm54lNhJz//ar86Or5BErzNpM6TL7lk44QS06hwsMrEdKIy8J/SYJPjfzR8tIUnKscclVpOgjKaBqC+0iFiVaRqAgfOlIEiezX6kMh5Q2FIUfqs46qWhhXjRrdKOEoStYAaikdLu5ZXr7vfb0ZaDh+ZwTQtbSMFolyOkecwI81MCdbMfT/1TqIGTOdAj5as9fAakk0jb2pXgUYQ8X1DVTR8ahSDVEaw9VTmWiSvTxvguVJ1Mb7gG4Gmh6aviDTJhfXtH4rPUNXhDLqrTH8JkJjyKROOMakIF68Hjse5vUfUxreBEOtb5r1Coa2Fe7ncJayaSE7ryrDbFqpZ 36UMAx4ulWMyqJajLNGY0DdG8qIsR5nxRhrnK/mrCidZ8F9/D3bWAl4rjtHlsztN59 +AnW5l0HsQcY9ntFL/zEBOaonjdJf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.50.1 (3.50.1-1.fc39) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Fri, 2023-11-24 at 16:59 +0100, Les wrote: >=20 >=20 > Laurenz Albe =C2=A0 (2023. nov. 24., P, 16:00): > > On Fri, 2023-11-24 at 12:39 +0100, Les wrote: > > > Under normal circumstances, the number of write operations is relativ= ely low, with an > > > average of 4-5 MB/sec total write speed on the disk associated with t= he data directory. > > > Yesterday, the primary server suddenly started writing to the pg_wal = directory at a > > > crazy pace, 1.5GB/sec, but sometimes it went up to over 3GB/sec. > > > [...] > > > Upon further analysis of the database, we found that we did not see a= ny mass data > > > changes in any of the tables. The only exception is a sequence value = that was moved > > > millions of steps within a single minute. > >=20 > > That looks like some application went crazy and inserted millions of ro= ws, but the > > inserts were rolled back.=C2=A0 But it is hard to be certain with the c= lues given. >=20 > Writing of WAL files continued after we shut down all clients, and restar= ted the primary PostgreSQL server. >=20 > How can the=C2=A0primary server generate more and more WAL files (writes)= after all clients have > been shut down and the server was restarted? My only bet was the autovacu= um. But I ruled > that out, because removing a replication slot has no effect on the autova= cuum (am I wrong?). It must have been autovacuum. Removing a replication slot has an influence= , since then autovacuum can do more work. If the problem stopped when you dropped the r= eplication slot, it could be a coincidence. > Now you are saying that this looks like a huge rollback. It could have been many small rollbacks. > Does=C2=A0rolling back changes require even more data to be written to th= e WAL after server > restart? No. My assumption would be that something generated lots of INSERTs that w= ere all rolled back. That creates WAL, even though you see no change in the table = data. > Does removing a replication slot lessen the amount of data needed to be w= ritten for > a rollback (or for anything else)? No: the WAL is generated by whatever precedes the ROLLBACK, and the ROLLBAC= K does not create a lot of WAL. > It is a fact that the primary stopped writing=C2=A0at 1.5GB/sec the momen= t we removed the slot. I have no explanation for that, except a coincidence. Replication slots don't generate WAL. Yours, Laurenz Albe