Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rVp3R-001Wmx-Dd for pgsql-general@arkaria.postgresql.org; Fri, 02 Feb 2024 08:37:46 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rVp3Q-00BQ1a-K4 for pgsql-general@arkaria.postgresql.org; Fri, 02 Feb 2024 08:37:44 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rVp3Q-00BPzT-1E for pgsql-general@lists.postgresql.org; Fri, 02 Feb 2024 08:37:44 +0000 Received: from mail-ej1-x633.google.com ([2a00:1450:4864:20::633]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1rVp3N-004fWh-8Y for pgsql-general@lists.postgresql.org; Fri, 02 Feb 2024 08:37:42 +0000 Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-a354408e6bfso500496066b.1 for ; Fri, 02 Feb 2024 00:37:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20230601.gappssmtp.com; s=20230601; t=1706863059; x=1707467859; darn=lists.postgresql.org; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id:from:to :cc:subject:date:message-id:reply-to; bh=GBtU5zDS8L5vALkte7SX31Afo53H+CDULGZRr7DoYdE=; b=eG7biUU+5QLhSfdjyWYSIiOQ8ejoA0icF+RejhxqzP8lo8Uf+P2HkYI8DXzxhtDQti BCydf6Mvemui016h4tOKI8NuZuNAdnJcWhl9/b/VzJ1iXJy9MG2D+lsdmAei4cqSRSg7 ahvR/CUs8hH6WvlUlX/GPyCSuT4xmXACkhVbSsv+E5+DQZ4jc9ioDFZGt18RhYV5qEQo PwqvbMrRlgttkNNKpN3flMXIlkSnH54EdxyoNpMrshQBIn3x1PLtI0wAAzIWf4AL/hJ1 i0G+6QUykHHytqfEjgASHGpXhIufI070tWnIRdzoY8gJRok8XD5PvbSe5ydr6H7fLHJE j3jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706863059; x=1707467859; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GBtU5zDS8L5vALkte7SX31Afo53H+CDULGZRr7DoYdE=; b=sXQTkRFZ2uDSK74xjiTO679JflTnCZb7OCnOFeaJPO5VLUMVk4yDXlhcw7xEOA/9nk sAxmdA5QWPrwsLd8w9Ee0gSPuDxVslWNbY2/WSulJ2dwGA3/VuG5WC1YDU66VHOspV4e UL2Se0XShAvXT8ueoniiJdESpA56BOPQLZTBC1x4C+b/QIHoE7Q81EYrogdFVwp3zuPm QtIT8ZsQpysgpvihfDh8IiPV84R/QPr53wWXPP6SM7GG99hg/9EO4xTYHHkBSWAGXeED 9U3LjUISCovAdw3OtMt6c6A9X8kYuzCuzgASZCnbJXH/vogalGZIw0SmIY6rjW4+bjn8 JnNA== X-Gm-Message-State: AOJu0YzlHXwjDc2e9TJYOcTpyYp41Q3k5vUK70tAw+rOaSNjxUmxyKAL RBwcjQGHBJlp46vbBYWf7USDH7QGRofMLEcSU5U0s7PTL9c64KmTdVYZDH1UvyQ= X-Google-Smtp-Source: AGHT+IFLmLT40GRFJMpO2OPlaiITrs+KWJfaFB3Xq7VlI0EMKHIfCADoyqxdDS1DxcmV9YI72dtWKA== X-Received: by 2002:a17:906:4bd3:b0:a31:70cc:8a0f with SMTP id x19-20020a1709064bd300b00a3170cc8a0fmr1167720ejv.12.1706863059242; Fri, 02 Feb 2024 00:37:39 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCV7qUFtUHM1wOD2HOk3pVEP/HgICZl1UGEPj4Tlzr+ymf+ZjRuyzVO6x4BlkGSMnOU6R7VD8RI7YaX65cvoa9XcvVsz634UgA== Received: from localhost.localdomain ([2001:871:5e:3408:1f15:fed6:ee4e:7d04]) by smtp.gmail.com with ESMTPSA id ot8-20020a170906ccc800b00a35d6482fdbsm648512ejb.77.2024.02.02.00.37.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:37:39 -0800 (PST) Message-ID: <60efac4f2e23719e71638ca1be5d678e1a02cc6d.camel@cybertec.at> Subject: Re: Query running longer From: Laurenz Albe To: veem v , Ron Cc: pgsql-general Date: Fri, 02 Feb 2024 09:37:38 +0100 In-Reply-To: References: <64de715ce18ab55159468cfdd8b8958f55eb8808.camel@cybertec.at> Autocrypt: addr=laurenz.albe@cybertec.at; prefer-encrypt=mutual; keydata=mQINBGGDwAQBEADgbWy5cKXQld3N2mF+DFyiNFbi2oBl2T+XgxpPF8wTRw2D/u4bBKXP0SYSE/lA86jIVNWWU0gf1KODIkVvgJm2w4vH2VBV1b7ddVViGl1Iu+9zaRnv9wulhnH42KefepXnoean6UT1EzLM0opF/Ik0j+40TxdRtobkBprkQUyHDXWlHc2ffPs3SipyFEP9AVLf7ejRC46CXWDnsqjOBSMEW8Z4HiK/8RrPZBsKLts8dJxKF4pygOdJb0CWk8k/X1jbcfdxo+zOLjOMvJcSJ2pFdJmQHU+JufB3rePziqQ2S9Ur6sccr9XnTC1GVBWN4Lf5VHq+vf+bFJjVwg+2hrySZnAVfcOrxoqFLErr7ug1zN2nM1kcpgA4VWn4gxlJtYNYYq+9WxX5dtvnNANlG3ZCrRKQzl8lxtzoF6Zo7LUhEqPaHDwn7Rvs+IdbOn41lF5UDTJGqmC4gS/bZydW2Fy3YWm4aSaN9fgFf8D+PVkrlKAZB7gBLz1TyHjbcRf85cYF+GKKrDld5SzMB/V60VX3oP/Eo8ikFpyWaqiz1f9X7MBot3/PjJkY+wDzp3nmb19QEcOBuQiSQ4xds2r0HewbuHTAR68u8jNNMGmpm2j4x+g09Jd/WQDjqlTBZ/jEltH41fYCCPWMfljXTOOXu2eLNGdfi7ETZogtwjM9oTtSPQARAQABtCdMYXVyZW56IEFsYmUgPGxhdXJlbnouYWxiZUBjeWJlcnRlYy5hdD6JAk4EEwEIADgWIQR0CqhbZGGABqoaSbdi8bhXA2EdmAUCYYPABAIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBi8bhXA2EdmM/6EADK232JCwmBzhlj8h7U9CjG6kx0JHP3uJGv+XfsHtHAlmY/RCwF1BHMEsRlk bT5UrLvJ2jb99bA9QARzhFaxzyn0F/BUKzuIjRGNs/n6d5dNUFA0kOt8sX+TacmC GEyjEBCrVCm4ranBiUyePn9NhHNWnaex7pJyqvMLLdwW9BEMJx0Fqo+DN8ukbXmYRsmhEtd3ue+x/luYmOmJnaGtzInaY5aOJYbW9XqoRIZkZvOCgbi1FfvNmoqWa+3oVxTOgw9RafjJDyW0lTHzKGjbGI5ofMU98l+/hKJFYJqWUF6VpFJY5YIcN/1lf4ZICMwDl+MPIVo/tpq8L10seJL28nLlvw3K+cI+TVW8IW/qL/LyVoDofI3USeOORuYmhpWRhik8JXX6xf3v6GrRilJIPWNFIJbxm1ZblQiQnOw3IOW7T+8nAmPin1HKqM3VrOrJQ2VtShsefNBibNAsr1oFaqcDBkn3yGG8i6CTW+FyO4PZ+/EwNxMVgktxbYdy5AT1/lpXr5tB+phhLIyVfiBvrWs5EThxYMQ/L8Y85c3GMsAy1l/x4h3jqySIYy3SCU9+jc5UVuNnXljbvkEzJ+NLWJ6C1rACFWrMszgPdh5tCrlRY9PpmYll4JbCgb8BtxEIUmR+xr50/ZElEK5iml7Q00KUekCcDt+36PsyGFTXBzNOrkCDQRhg8AEARAAzOZ2tLHlI4rrhG411h6cdCFjBZxuljaFCxFyHn3m6wbGLqwBUWC5k8UrRqjHMz88KcTSaNO7XGAmCqPdWd2SeflPZRnNTbjsVpw7mLdffsBm4JX7kki2Pvk5h0NtYeidXT1PSpc2ri4DutYXuT9uD8RAm1wUDCE5HQNUihT/WH6opt+hskHW21uHao0+y822tG0QQcGMqdQR5Vxdxj89wiEPdqW+HpU/oOZIhrf2E7prduAppxixjHy/o1rcnoznnJvc8D3+YgI9O0LrBMij89dM55pRGbLovTR1oGR3U74sX774+0xmSzeIKwZfiMUz7Atlvfk5SHOsRUFPN2Ux9kaXiiBibQpHFxt7b lDrT4wxdLJ/XCdbPPAyl+lZtOLsaHEEZvYNyTXwZc35dVf3R4/oz20HoG6s7ct8e1 AQygj43XAERzty9SkWgxs8+grp1PrGx6FHVSYRqBM8dS/ZR6yRVwOwJXPyaSSqfIF21DkE4j1y4n+ItSewPGoRp8K/yWCikt6qlkVkO2ASNIiX04fAbtzwVOaNn8ZMRNqyvLc1fED4sr49onE4cAIcBLjcC3KL+w9DUGRQCdziROj5H2Yl/sXGPdMciUHo/Uz2rggc+2th3bQiMhrHWSsBpUkDQp0yWewemstPpPgBL3h2fHKaX8B9oH5Qu/H1IgrOuX8AEQEAAYkCNgQYAQgAIBYhBHQKqFtkYYAGqhpJt2LxuFcDYR2YBQJhg8AEAhsMAAoJEGLxuFcDYR2YuPwQAMkpGtR80pQ1gVsONhdkqj0H2eU66efP/gO3CoyaoIcvrpKYj7C2HipVSmkt1gpByL0X4AMQ/vKuknUz3wd28Ba+G1dCfbVs/Xiusq+SmpUj5rTwmYqdSjWMuCo1R6oS5hdJMdUUJYGMT0QkVlm1KnW8jkmCTl9GzjDxOAsN9O6/6lPzaGFtk9XF+34Bry/N4HKiJkqpC4+UTd0AprPfzJ2jdT64e1F0+W88X8y1bTTgNrHwK4mDiLnlE4SKRuEm54lNhJz//ar86Or5BErzNpM6TL7lk44QS06hwsMrEdKIy8J/SYJPjfzR8tIUnKscclVpOgjKaBqC+0iFiVaRqAgfOlIEiezX6kMh5Q2FIUfqs46qWhhXjRrdKOEoStYAaikdLu5ZXr7vfb0ZaDh+ZwTQtbSMFolyOkecwI81MCdbMfT/1TqIGTOdAj5as9fAakk0jb2pXgUYQ8X1DVTR8ahSDVEaw9VTmWiSvTxvguVJ1Mb7gG4Gmh6aviDTJhfXtH4rPUNXhDLqrTH8JkJjyKROOMakIF68Hjse5vUfUxreBEOtb5r1Coa2Fe7ncJayaSE7ryrDbFqpZ 36UMAx4ulWMyqJajLNGY0DdG8qIsR5nxRhrnK/mrCidZ8F9/D3bWAl4rjtHlsztN59 +AnW5l0HsQcY9ntFL/zEBOaonjdJf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.50.3 (3.50.3-1.fc39) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Fri, 2024-02-02 at 10:14 +0530, veem v wrote: > On Fri, 2 Feb 2024 at 02:43, Laurenz Albe wrot= e: > > On Fri, 2024-02-02 at 02:27 +0530, veem v wrote: > > > We have the below query which is running for ~45 seconds on postgres = aurora reader instance. > > > I have captured the explain analyze. Want to understand, where exactl= y the resources are > > > getting spent and if we can be able to optimize it further. > >=20 > > Aurora <> PostgreSQL, but here is what I can see: > >=20 > > - The index scan on SCHEMA1."TAB2" has to check 2 million extra > > =C2=A0 rows because "work_mem" is too small.=C2=A0 Almost the complete = time > > =C2=A0 is spent there. > >=20 > > - You may be getting a bad plan, because the statistics on > > =C2=A0 SCHEMA1.TAB4 are either out of date or not detailed enough, > > =C2=A0 which makes PostgreSQL underestimate the result size. >=20 > As you mentioned below, So wondering how you got to know, if this is the = step where majority of the DB resources get spent. And as total time the qu= ery ran was ~45 seconds, and out of that how much time it spent in this ste= p, how can i get that? And to fix this one line with regards to TAB2, shoul= d we increase "work_mem" here? >=20 > - The index scan on SCHEMA1."TAB2" has to check 2 million extra > =C2=A0 rows because "work_mem" is too small.=C2=A0 Almost the complete ti= me > =C2=A0 is spent there. >=20 > =C2=A0-> =C2=A0Parallel Bitmap Heap Scan on SCHEMA1.""TAB2"" TAB2 =C2=A0(= cost=3D84860.50..13040301.00 rows=3D1175611 width=3D80) (actual time=3D713.= 054..26942.082 rows=3D956249 loops=3D5)" > " =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 O= utput: TAB2.TAB2_time, TAB2.PRI, TAB2.AID" > =C2=A0=C2=A0Recheck Cond: (TAB2.MID =3D 'XXXXX'::numeric) > =C2=A0=C2=A0Rows Removed by Index Recheck: 2137395 > =C2=A0=C2=A0Filter: ((TAB2.TAB2_time >=3D '2024-01-01 00:00:00+00'::times= tamp with time zone) AND (TAB2.TAB2_time <=3D '2024-01-31 00:00:00+00'::tim= estamp with time zone)) > =C2=A0=C2=A0Heap Blocks: exact=3D5300 lossy=3D782577 > =C2=A0=C2=A0Buffers: shared hit=3D1651569 read=3D2245157 > =C2=A0=C2=A0I/O Timings: shared/local read=3D29063.286=C2=A0=C2=A0 The bitmap index scan builds a bitmap, the size of which is limited by "wor= k_mem". If that is big enough, the bitmap will contain a bit for each table row, if= not, it only contains a bit per table row for some table blocks (the "exact" one= s), then it degrades to a bit per block (whe "lossy" ones). For lossy blocks, *all* rows in the block have to be rechecked, which is overhead. Given that only a small part of the time (26942.082 - 713.054) *= 5 is spent doing I/O (29063.286), I guess that the rest is spent processing table rows. > Another point you mentioned as below , for this , should we run vacuum an= alyze on the table TAB4? >=20 > - You may be getting a bad plan, because the statistics on > =C2=A0 SCHEMA1.TAB4 are either out of date or not detailed enough, > =C2=A0 which makes PostgreSQL underestimate the result size. > =C2=A0=C2=A0 > =C2=A0=C2=A0-> =C2=A0Parallel Bitmap Heap Scan on SCHEMA1.TAB4 TAB4 =C2= =A0(cost=3D26.39..7042.63 rows=3D1049 width=3D37) (actual time=3D23.650..20= 1.606 rows=3D27613 loops=3D5) > =C2=A0-> =C2=A0Bitmap Index Scan on TAB4_idx1 =C2=A0(cost=3D0.00..2= 5.95 rows=3D1784 width=3D0) (actual time=3D23.938..23.938 rows=3D138067 loo= ps=3D1) > =C2=A0=C2=A0Index Cond: ((TAB4.TAB4_code)::text =3D 'XX'::text) > =C2=A0=C2=A0Buffers: shared hit=3D72 An ANALYZE might be enough. If not, you can try to collect more detailed statistics for the column: ALTER TABLE SCHEMA1.TAB4 ALTER TAB4_code SET STATISTICS 1000; ANALYZE SCHEMA1.TAB4; Yours, Laurenz Albe