Feedback-ID: i0fe9450f:Fastmail
Date: Tue, 6 Feb 2024 15:21:48 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Maxim Orlov <orlovmg@gmail.com>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Add Index-level REINDEX with multiple jobs
Message-ID: <ZcHP_Km25qoDcc5P@paquier.xyz>
References: <CACG=ezZU_VwDi-1PN8RUSE6mcYG+Yx1NH_rJO4+Ke-mKqLp=Nw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="Gx0Q9ibEqBH0HdFK"
Content-Disposition: inline
In-Reply-To: <CACG=ezZU_VwDi-1PN8RUSE6mcYG+Yx1NH_rJO4+Ke-mKqLp=Nw@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/ZcHP_Km25qoDcc5P%40paquier.xyz>
Precedence: bulk


--Gx0Q9ibEqBH0HdFK
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Fri, Dec 29, 2023 at 04:15:35PM +0300, Maxim Orlov wrote:
> Recently, one of our customers came to us with the question: why do reindex
> utility does not support multiple jobs for indices (-i opt)?
> And, of course, it is because we cannot control the concurrent processing
> of multiple indexes on the same relation.  This was
> discussed somewhere in [0], I believe.  So, customer have to make a shell
> script to do his business and so on.

Yep, that should be the correct thread.  As far as I recall, one major
reason was code simplicity because dealing with parallel jobs at table
level is a no-brainer on the client side (see 0003): we know that
relations with physical storage will never interact with each other.

> But. This seems to be not that complicated to split indices by parent
> tables and do reindex in multiple jobs?  Or I miss something?
> PFA patch implementing this.

+   appendPQExpBufferStr(&catalog_query,
+                       "WITH idx as (\n"
+                       "  SELECT c.relname, ns.nspname\n"
+                       "  FROM pg_catalog.pg_class c,\n"
+                       "       pg_catalog.pg_namespace ns\n"
+                       "  WHERE c.relnamespace OPERATOR(pg_catalog.=) ns.oid AND\n"
+                       "        c.oid OPERATOR(pg_catalog.=) ANY(ARRAY['\n");

The problem may be actually trickier than that, no?  Could there be
other factors to take into account for their classification, like
their sizes (typically, we'd want to process the biggest one first, I
guess)?
--
Michael

--Gx0Q9ibEqBH0HdFK
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmXBz/sACgkQnvQgOdby
QH2hLA//QVIA7iWQo6iV+R0OoJVwHp7aiupYsKF/2VXEBqQKTsyNrvQWEPIImDU0
gI/WQZgtU5hay60toQIEa/CbF/F1Ya2YU5AxHteb1j5ysYPZixn79iOU1Q7JJQeZ
tKGV5ySA8ZyZflFGLX/HcmOUKlp094Lg4MuIqtPRLtt3pOdUIYztktFwCtMin6EN
98k5c/kdRyfXeA7aPUG7VHN1ULymKxDeiEb5a2YbS9jqMkOX2dG4mMwtK19tRDJ1
yi+7AXDogRcZcyyeRVfASsKZyVAlS76LzemuCQ4Exbv8hO7MH9K16lmUaj0mA5yB
G+FOfyHyDkYs7mo6Twueb0gk5Eyby1wyhN6TwB5W/5YdPQlrb6bVQC+y52nbGwKP
iLnT0FJKmBJHGHc9RMb3ROq/NhMFTb4vvnXDKSFVBqrJVc+9GmPJxr/5w//eOLP8
ewWEhIwx7uUaVrC10iKpcnbmwU/Kh5f0cM+/FQMYDSvZUH8WhId94Rdvpo5tHSbu
VQbrMSFfaQtclpnTozm4fDkSWu3qS86lbQ/wm65IqEd1Sm+OfHJMBIGTzzpsnZ8g
8kroxEE5kkBiGAn+FvtO5XSroIEEtBXU4bvPsVexkvLo0ptiR2U/oBw8COkhhHPu
IjgNWFkKyLKPIT84su+j7TeUD+Xto/7qZGTOAvG1gR1osrLJRmM=
=N0KS
-----END PGP SIGNATURE-----

--Gx0Q9ibEqBH0HdFK--