Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r9z8n-003rmc-I2 for pgsql-general@arkaria.postgresql.org; Mon, 04 Dec 2023 02:57:01 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r9z8m-00EKI8-7D for pgsql-general@arkaria.postgresql.org; Mon, 04 Dec 2023 02:57:00 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r9z8l-00EKHz-J0 for pgsql-general@lists.postgresql.org; Mon, 04 Dec 2023 02:56:59 +0000 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1r9z8d-008g8w-Ov for pgsql-general@lists.postgresql.org; Mon, 04 Dec 2023 02:56:57 +0000 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-3333224c7b9so2065845f8f.1 for ; Sun, 03 Dec 2023 18:56:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20230601.gappssmtp.com; s=20230601; t=1701658610; x=1702263410; darn=lists.postgresql.org; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:to:from:subject:message-id:from:to:cc :subject:date:message-id:reply-to; bh=VteJovk+LxVXzbzYo60mcwJj+s8ddlovVIr/C2cCp3c=; b=bBiNXXcLHWunNYBVpYawz/a5ar3gW83enk+cpmhQCzTlddr4q0ACiNEyRxAid0mVZD N/6AOBz570IHBgXLc3IS2AhUR5MiZHNlVU6zWjkgocPxshlpT3vhHlkHiqtwfZBIcIxc W3CV9R01Yu1KsdmT/6J9dzpmzg+kXxPonQ3D1uprIFbAtdPEIZ5W8cspHCfyEO//IDOU 9A1NhwjiKvcUB5IhE+Qc6TiPBONlCVWE2qXipe40zKL3BqwuQIq4lWUGO24SOUvTvp6S 1PK4otLcuyBz5rNLInnyE6SN9W9x3/357lKMt4KzUajMnb3NwmkzTwFQFdI5W9lHBuMk UTJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701658610; x=1702263410; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:to:from:subject:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VteJovk+LxVXzbzYo60mcwJj+s8ddlovVIr/C2cCp3c=; b=UNBXx5EGDjXoGpwCVQvbJ7rfN6h7n7C09m833sccbn+f3OYTCs/qUydRP+GSs6Zz3g 5yN9sEFDo5IuHFx5eP6BshcdTKL4EKGHXwQD8BVrUfg1fqrr76TrgRY/91evj15fFCc5 yoDpJNN1cwHnb+J9CEUqIUTsXoA3z6ItQUO6o7bTwdyG1DKqa1tnSb+8laXvyGGLU02u pdklsQdS4xxSABSDiZPXnKJcB7eZIHgCOHga2UxWe0xHTH1pnzYMGwxUPSzBiMXBF5xw WODyS+5HSuxWl1eJa01t6v2xOpdli8tJV8sBN2EMPJpKOnMcfLovIH4rAYm9zrQLtnpf B/Tw== X-Gm-Message-State: AOJu0Yz2LCgt484Eup0mztLwDz5M8oD2zW/3uCWBBy551s2q+nNBRTLg Ap+eMcIFCZmlUFtqeu6PThjsQA== X-Google-Smtp-Source: AGHT+IE1ngkhGaJ2n1lcVjYTzfMLWnw/75EU8qNCxLeTenv8+4VPVWcDGVi1hnabiA9vNWgtZrb5WQ== X-Received: by 2002:a5d:4c4c:0:b0:333:48d7:f56 with SMTP id n12-20020a5d4c4c000000b0033348d70f56mr674331wrt.49.1701658610012; Sun, 03 Dec 2023 18:56:50 -0800 (PST) Received: from localhost.localdomain ([2001:871:260:6:266c:1078:a93c:8aed]) by smtp.gmail.com with ESMTPSA id l5-20020a5d5605000000b0033341165b0csm3249973wrv.36.2023.12.03.18.56.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Dec 2023 18:56:49 -0800 (PST) Message-ID: <28bb8938984b0d97f25f413d794efa782286a440.camel@cybertec.at> Subject: Re: vacuum visibility relevance From: Laurenz Albe To: senor , "pgsql-general@lists.postgresql.org" Date: Mon, 04 Dec 2023 03:56:49 +0100 In-Reply-To: References: Autocrypt: addr=laurenz.albe@cybertec.at; prefer-encrypt=mutual; keydata=mQINBGGDwAQBEADgbWy5cKXQld3N2mF+DFyiNFbi2oBl2T+XgxpPF8wTRw2D/u4bBKXP0SYSE/lA86jIVNWWU0gf1KODIkVvgJm2w4vH2VBV1b7ddVViGl1Iu+9zaRnv9wulhnH42KefepXnoean6UT1EzLM0opF/Ik0j+40TxdRtobkBprkQUyHDXWlHc2ffPs3SipyFEP9AVLf7ejRC46CXWDnsqjOBSMEW8Z4HiK/8RrPZBsKLts8dJxKF4pygOdJb0CWk8k/X1jbcfdxo+zOLjOMvJcSJ2pFdJmQHU+JufB3rePziqQ2S9Ur6sccr9XnTC1GVBWN4Lf5VHq+vf+bFJjVwg+2hrySZnAVfcOrxoqFLErr7ug1zN2nM1kcpgA4VWn4gxlJtYNYYq+9WxX5dtvnNANlG3ZCrRKQzl8lxtzoF6Zo7LUhEqPaHDwn7Rvs+IdbOn41lF5UDTJGqmC4gS/bZydW2Fy3YWm4aSaN9fgFf8D+PVkrlKAZB7gBLz1TyHjbcRf85cYF+GKKrDld5SzMB/V60VX3oP/Eo8ikFpyWaqiz1f9X7MBot3/PjJkY+wDzp3nmb19QEcOBuQiSQ4xds2r0HewbuHTAR68u8jNNMGmpm2j4x+g09Jd/WQDjqlTBZ/jEltH41fYCCPWMfljXTOOXu2eLNGdfi7ETZogtwjM9oTtSPQARAQABtCdMYXVyZW56IEFsYmUgPGxhdXJlbnouYWxiZUBjeWJlcnRlYy5hdD6JAk4EEwEIADgWIQR0CqhbZGGABqoaSbdi8bhXA2EdmAUCYYPABAIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBi8bhXA2EdmM/6EADK232JCwmBzhlj8h7U9CjG6kx0JHP3uJGv+XfsHtHAlmY/RCwF1BHMEsRlk bT5UrLvJ2jb99bA9QARzhFaxzyn0F/BUKzuIjRGNs/n6d5dNUFA0kOt8sX+TacmC GEyjEBCrVCm4ranBiUyePn9NhHNWnaex7pJyqvMLLdwW9BEMJx0Fqo+DN8ukbXmYRsmhEtd3ue+x/luYmOmJnaGtzInaY5aOJYbW9XqoRIZkZvOCgbi1FfvNmoqWa+3oVxTOgw9RafjJDyW0lTHzKGjbGI5ofMU98l+/hKJFYJqWUF6VpFJY5YIcN/1lf4ZICMwDl+MPIVo/tpq8L10seJL28nLlvw3K+cI+TVW8IW/qL/LyVoDofI3USeOORuYmhpWRhik8JXX6xf3v6GrRilJIPWNFIJbxm1ZblQiQnOw3IOW7T+8nAmPin1HKqM3VrOrJQ2VtShsefNBibNAsr1oFaqcDBkn3yGG8i6CTW+FyO4PZ+/EwNxMVgktxbYdy5AT1/lpXr5tB+phhLIyVfiBvrWs5EThxYMQ/L8Y85c3GMsAy1l/x4h3jqySIYy3SCU9+jc5UVuNnXljbvkEzJ+NLWJ6C1rACFWrMszgPdh5tCrlRY9PpmYll4JbCgb8BtxEIUmR+xr50/ZElEK5iml7Q00KUekCcDt+36PsyGFTXBzNOrkCDQRhg8AEARAAzOZ2tLHlI4rrhG411h6cdCFjBZxuljaFCxFyHn3m6wbGLqwBUWC5k8UrRqjHMz88KcTSaNO7XGAmCqPdWd2SeflPZRnNTbjsVpw7mLdffsBm4JX7kki2Pvk5h0NtYeidXT1PSpc2ri4DutYXuT9uD8RAm1wUDCE5HQNUihT/WH6opt+hskHW21uHao0+y822tG0QQcGMqdQR5Vxdxj89wiEPdqW+HpU/oOZIhrf2E7prduAppxixjHy/o1rcnoznnJvc8D3+YgI9O0LrBMij89dM55pRGbLovTR1oGR3U74sX774+0xmSzeIKwZfiMUz7Atlvfk5SHOsRUFPN2Ux9kaXiiBibQpHFxt7b lDrT4wxdLJ/XCdbPPAyl+lZtOLsaHEEZvYNyTXwZc35dVf3R4/oz20HoG6s7ct8e1 AQygj43XAERzty9SkWgxs8+grp1PrGx6FHVSYRqBM8dS/ZR6yRVwOwJXPyaSSqfIF21DkE4j1y4n+ItSewPGoRp8K/yWCikt6qlkVkO2ASNIiX04fAbtzwVOaNn8ZMRNqyvLc1fED4sr49onE4cAIcBLjcC3KL+w9DUGRQCdziROj5H2Yl/sXGPdMciUHo/Uz2rggc+2th3bQiMhrHWSsBpUkDQp0yWewemstPpPgBL3h2fHKaX8B9oH5Qu/H1IgrOuX8AEQEAAYkCNgQYAQgAIBYhBHQKqFtkYYAGqhpJt2LxuFcDYR2YBQJhg8AEAhsMAAoJEGLxuFcDYR2YuPwQAMkpGtR80pQ1gVsONhdkqj0H2eU66efP/gO3CoyaoIcvrpKYj7C2HipVSmkt1gpByL0X4AMQ/vKuknUz3wd28Ba+G1dCfbVs/Xiusq+SmpUj5rTwmYqdSjWMuCo1R6oS5hdJMdUUJYGMT0QkVlm1KnW8jkmCTl9GzjDxOAsN9O6/6lPzaGFtk9XF+34Bry/N4HKiJkqpC4+UTd0AprPfzJ2jdT64e1F0+W88X8y1bTTgNrHwK4mDiLnlE4SKRuEm54lNhJz//ar86Or5BErzNpM6TL7lk44QS06hwsMrEdKIy8J/SYJPjfzR8tIUnKscclVpOgjKaBqC+0iFiVaRqAgfOlIEiezX6kMh5Q2FIUfqs46qWhhXjRrdKOEoStYAaikdLu5ZXr7vfb0ZaDh+ZwTQtbSMFolyOkecwI81MCdbMfT/1TqIGTOdAj5as9fAakk0jb2pXgUYQ8X1DVTR8ahSDVEaw9VTmWiSvTxvguVJ1Mb7gG4Gmh6aviDTJhfXtH4rPUNXhDLqrTH8JkJjyKROOMakIF68Hjse5vUfUxreBEOtb5r1Coa2Fe7ncJayaSE7ryrDbFqpZ 36UMAx4ulWMyqJajLNGY0DdG8qIsR5nxRhrnK/mrCidZ8F9/D3bWAl4rjtHlsztN59 +AnW5l0HsQcY9ntFL/zEBOaonjdJf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.50.1 (3.50.1-1.fc39) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Mon, 2023-12-04 at 02:08 +0000, senor wrote: > Simplified Scenario: > 40+ gig table with 20+ indexes receiving log event type records in an app= end only fashion. Log tables typically receive tons of INSERTs. Over twenty indexes are too = many. > One of the columns is a sequence ID. > PG version 11.4 >=20 >=20 > If I start a vacuum on this table when sequence ID is 1 million, and the = table continues > to accumulate new events, when vacuum completes, should I be thinking abo= ut the state of > the table and indexes as being vacuumed up to 1 million? Or is vacuum als= o operating on > records inserted after vacuum started? VACUUM processes the whole table, but it determines an "xmin horizon" when = it starts, that is, it determines which tuples are safe to remove. Rows added after V= ACUUM started won't qualify, so they won't be removed. > Is there any reason to think about this differently when the vacuum is ma= nual as opposed to auto? No, the behavior is the same. > I attempted to deal with the append only issue corrected in PG13 by setti= ng > autovacuum_freeze_max_age low enough to trigger vacuum but that has been = such a wild card > I abandoned it. I'm now trying to find a formula to trigger a manual vacu= um. There are > stats gathering processes pulling from the most recently added data all t= he time so my > priority is to keep performance high for those. Secondly, there's a ton o= f memory allotted > so running vacuum freeze before table pages age out of memory seems like = a good idea. You can monitor the "n_tup_ins" column in "pg_stat_user_tables" and run VAC= UUM whebever it has increased enough. But the simple solution is to upgrade. Any version older than v13 is due f= or an upgrade soon. > I am having trouble with autovacuum keeping up and I suspect there's a co= mmunication > problem with the stats collector but have put off looking into that becau= se updating > to PG15 was "right around the corner". Meanwhile, I have had to run multi= ple-thread > scripts to keep up vacuum. Manual vacuums always work and updates stats c= orrectly but > autovacuums hang and occupies workers doing nothing. Autovacuum deliberately runs slower than a manual VACUUM. If you are running a version older than v12, the default value for "autovacuum_vacuum_cost_delay" is 20ms, which makes autovacuum really slow.= You should change that parameter to 2ms, and if that is still too slow, use ALTER TABL= E to set it to 0 on the affected table. Then autovacuum will run as fast as manual VAC= UUM. In addition, see that "maintenance_work_mem" is set high to make autovacuum= fast. Yours, Laurenz Albe --=20 Cybertec | https://www.cybertec-postgresql.com