Message-ID: <40701055b4405ad3bf64467517401c8a44e41973.camel@cybertec.at>
Subject: Re: Slow query in table where many rows were deleted. VACUUM FULL
 fixes it
From: Laurenz Albe <laurenz.albe@cybertec.at>
To: Pavlos Kallis <pkallis@yourhero.com>, pgsql-performance@postgresql.org
Date: Tue, 30 Jan 2024 16:37:09 +0100
In-Reply-To: 
 <CABzFZgHm0=9ibE-6-85+Fy+BxcTLDOob4VrzJ0+eOqQXSze7gw@mail.gmail.com>
References: 
	<CABzFZgHm0=9ibE-6-85+Fy+BxcTLDOob4VrzJ0+eOqQXSze7gw@mail.gmail.com>
Autocrypt: addr=laurenz.albe@cybertec.at; prefer-encrypt=mutual;
 keydata=mQINBGGDwAQBEADgbWy5cKXQld3N2mF+DFyiNFbi2oBl2T+XgxpPF8wTRw2D/u4bBKXP0SYSE/lA86jIVNWWU0gf1KODIkVvgJm2w4vH2VBV1b7ddVViGl1Iu+9zaRnv9wulhnH42KefepXnoean6UT1EzLM0opF/Ik0j+40TxdRtobkBprkQUyHDXWlHc2ffPs3SipyFEP9AVLf7ejRC46CXWDnsqjOBSMEW8Z4HiK/8RrPZBsKLts8dJxKF4pygOdJb0CWk8k/X1jbcfdxo+zOLjOMvJcSJ2pFdJmQHU+JufB3rePziqQ2S9Ur6sccr9XnTC1GVBWN4Lf5VHq+vf+bFJjVwg+2hrySZnAVfcOrxoqFLErr7ug1zN2nM1kcpgA4VWn4gxlJtYNYYq+9WxX5dtvnNANlG3ZCrRKQzl8lxtzoF6Zo7LUhEqPaHDwn7Rvs+IdbOn41lF5UDTJGqmC4gS/bZydW2Fy3YWm4aSaN9fgFf8D+PVkrlKAZB7gBLz1TyHjbcRf85cYF+GKKrDld5SzMB/V60VX3oP/Eo8ikFpyWaqiz1f9X7MBot3/PjJkY+wDzp3nmb19QEcOBuQiSQ4xds2r0HewbuHTAR68u8jNNMGmpm2j4x+g09Jd/WQDjqlTBZ/jEltH41fYCCPWMfljXTOOXu2eLNGdfi7ETZogtwjM9oTtSPQARAQABtCdMYXVyZW56IEFsYmUgPGxhdXJlbnouYWxiZUBjeWJlcnRlYy5hdD6JAk4EEwEIADgWIQR0CqhbZGGABqoaSbdi8bhXA2EdmAUCYYPABAIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBi8bhXA2EdmM/6EADK232JCwmBzhlj8h7U9CjG6kx0JHP3uJGv+XfsHtHAlmY/RCwF1BHMEsRlk
 bT5UrLvJ2jb99bA9QARzhFaxzyn0F/BUKzuIjRGNs/n6d5dNUFA0kOt8sX+TacmC
 GEyjEBCrVCm4ranBiUyePn9NhHNWnaex7pJyqvMLLdwW9BEMJx0Fqo+DN8ukbXmYRsmhEtd3ue+x/luYmOmJnaGtzInaY5aOJYbW9XqoRIZkZvOCgbi1FfvNmoqWa+3oVxTOgw9RafjJDyW0lTHzKGjbGI5ofMU98l+/hKJFYJqWUF6VpFJY5YIcN/1lf4ZICMwDl+MPIVo/tpq8L10seJL28nLlvw3K+cI+TVW8IW/qL/LyVoDofI3USeOORuYmhpWRhik8JXX6xf3v6GrRilJIPWNFIJbxm1ZblQiQnOw3IOW7T+8nAmPin1HKqM3VrOrJQ2VtShsefNBibNAsr1oFaqcDBkn3yGG8i6CTW+FyO4PZ+/EwNxMVgktxbYdy5AT1/lpXr5tB+phhLIyVfiBvrWs5EThxYMQ/L8Y85c3GMsAy1l/x4h3jqySIYy3SCU9+jc5UVuNnXljbvkEzJ+NLWJ6C1rACFWrMszgPdh5tCrlRY9PpmYll4JbCgb8BtxEIUmR+xr50/ZElEK5iml7Q00KUekCcDt+36PsyGFTXBzNOrkCDQRhg8AEARAAzOZ2tLHlI4rrhG411h6cdCFjBZxuljaFCxFyHn3m6wbGLqwBUWC5k8UrRqjHMz88KcTSaNO7XGAmCqPdWd2SeflPZRnNTbjsVpw7mLdffsBm4JX7kki2Pvk5h0NtYeidXT1PSpc2ri4DutYXuT9uD8RAm1wUDCE5HQNUihT/WH6opt+hskHW21uHao0+y822tG0QQcGMqdQR5Vxdxj89wiEPdqW+HpU/oOZIhrf2E7prduAppxixjHy/o1rcnoznnJvc8D3+YgI9O0LrBMij89dM55pRGbLovTR1oGR3U74sX774+0xmSzeIKwZfiMUz7Atlvfk5SHOsRUFPN2Ux9kaXiiBibQpHFxt7b
 lDrT4wxdLJ/XCdbPPAyl+lZtOLsaHEEZvYNyTXwZc35dVf3R4/oz20HoG6s7ct8e1
 AQygj43XAERzty9SkWgxs8+grp1PrGx6FHVSYRqBM8dS/ZR6yRVwOwJXPyaSSqfIF21DkE4j1y4n+ItSewPGoRp8K/yWCikt6qlkVkO2ASNIiX04fAbtzwVOaNn8ZMRNqyvLc1fED4sr49onE4cAIcBLjcC3KL+w9DUGRQCdziROj5H2Yl/sXGPdMciUHo/Uz2rggc+2th3bQiMhrHWSsBpUkDQp0yWewemstPpPgBL3h2fHKaX8B9oH5Qu/H1IgrOuX8AEQEAAYkCNgQYAQgAIBYhBHQKqFtkYYAGqhpJt2LxuFcDYR2YBQJhg8AEAhsMAAoJEGLxuFcDYR2YuPwQAMkpGtR80pQ1gVsONhdkqj0H2eU66efP/gO3CoyaoIcvrpKYj7C2HipVSmkt1gpByL0X4AMQ/vKuknUz3wd28Ba+G1dCfbVs/Xiusq+SmpUj5rTwmYqdSjWMuCo1R6oS5hdJMdUUJYGMT0QkVlm1KnW8jkmCTl9GzjDxOAsN9O6/6lPzaGFtk9XF+34Bry/N4HKiJkqpC4+UTd0AprPfzJ2jdT64e1F0+W88X8y1bTTgNrHwK4mDiLnlE4SKRuEm54lNhJz//ar86Or5BErzNpM6TL7lk44QS06hwsMrEdKIy8J/SYJPjfzR8tIUnKscclVpOgjKaBqC+0iFiVaRqAgfOlIEiezX6kMh5Q2FIUfqs46qWhhXjRrdKOEoStYAaikdLu5ZXr7vfb0ZaDh+ZwTQtbSMFolyOkecwI81MCdbMfT/1TqIGTOdAj5as9fAakk0jb2pXgUYQ8X1DVTR8ahSDVEaw9VTmWiSvTxvguVJ1Mb7gG4Gmh6aviDTJhfXtH4rPUNXhDLqrTH8JkJjyKROOMakIF68Hjse5vUfUxreBEOtb5r1Coa2Fe7ncJayaSE7ryrDbFqpZ
 36UMAx4ulWMyqJajLNGY0DdG8qIsR5nxRhrnK/mrCidZ8F9/D3bWAl4rjtHlsztN59
 +AnW5l0HsQcY9ntFL/zEBOaonjdJf
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.50.3 (3.50.3-1.fc39) 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/40701055b4405ad3bf64467517401c8a44e41973.camel%40cybertec.at>
Precedence: bulk

On Tue, 2024-01-30 at 11:40 +0200, Pavlos Kallis wrote:
> I have the following table:
>=20
> CREATE TABLE IF NOT EXISTS public.shortened_url
> (
> =C2=A0 =C2=A0 id character varying(12) COLLATE pg_catalog."default" NOT N=
ULL,
> =C2=A0 =C2=A0 created_at timestamp without time zone,
> =C2=A0 =C2=A0 expires_at timestamp without time zone,
> =C2=A0 =C2=A0 url text COLLATE pg_catalog."default" NOT NULL,
> =C2=A0 =C2=A0 CONSTRAINT shortened_url_pkey PRIMARY KEY (id)
> )
>=20
> The table contains only the following index on PRIMARY KEY:
>=20
> CREATE UNIQUE INDEX IF NOT EXISTS shortened_url_pkey
> =C2=A0 =C2=A0 ON public.shortened_url USING btree
> =C2=A0 =C2=A0 (id COLLATE pg_catalog."default" ASC NULLS LAST)
> =C2=A0 =C2=A0 TABLESPACE pg_default;
>=20
> This table has approximately 5 million rows of expired URLs (expires_at <=
 now()), and 5 thousand rows of non-expired URLs (expires_at > now())
>=20
> I deleted all expired URLs with this query:
>=20
> DELETE FROM shortened_url WHERE expires_at < now().
>=20
> Then, I tried to query the table for expired URLs:
>=20
> SELECT * FROM shortened_url WHERE expires_at < now();
>=20
> This query was very slow. It took around 1-2 minutes to run, while it had=
 to fetch only 5000 rows (the non-expired URLs, since the other ones were d=
eleted).
>=20
> After that, I tried to run VACUUM ANALYZE and REINDEX to the table.
> The query was still slow.
>=20
> Finally, I ran VACUUM FULL and re-executed the query. Only then, it start=
ed running fast (1-2 seconds).
>=20
> Do you have observed a similar behavior with VACUUM ANALYZE / VACUUM FULL=
 and why this can happen?=C2=A0
> Is this because data is compacted after VACUUM FULL and sequential disk r=
eads are faster?=C2=A0
> Shouldn't VACUUM ANALYZE reclaim the disk space and make the query run fa=
st?
> Is this because RDS might do some magic? Is it something I am missing?

There are too many unknowns here.  Please enable "track_io_timing" and send=
 us
the output of EXPLAIN (ANALYZE, BUFFERS) for the slow statements.

One theory could be that there was a long running transaction or something =
else
that prevented VACUUM from cleaning up.  For that, the output of
"VACUUM (VERBOSE) shortened_url" would be interesting.

> Additional details
> PostgreSQL version: 14.7 on db.t3.micro RDS
> PG configuration: Default of RDS

We can only speak about real PostgreSQL...

Yours,
Laurenz Albe