Feedback-ID: i0fe9450f:Fastmail
Date: Wed, 6 Nov 2024 08:39:07 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>,
	Alvaro Herrera <alvherre@alvh.no-ip.org>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: per backend I/O statistics
Message-ID: <Zyqsm6ilLLADAGw2@paquier.xyz>
References: <Zul++79EOPsFB+Gd@ip-10-97-1-34.eu-west-3.compute.internal>
 <CAN55FZ0xEC6K=EqTwboHJtzZnENDoEwvWOerKkHbZwFT-cU6XQ@mail.gmail.com>
 <ZumKktat4RtHmxL/@ip-10-97-1-34.eu-west-3.compute.internal>
 <Zuz5iQ4AjcuOMx_w@paquier.xyz>
 <ZwOvzX219kDhEEMu@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZwS5H5Fu0LnSev90@paquier.xyz>
 <ZwVdt8/0tTu5pv8K@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZyMRJIbUpNPoCXUe@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZyibjiLgoLx+kS4n@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZypXy3fH/EyALaTR@ip-10-97-1-34.eu-west-3.compute.internal>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="SQ0ywFv/uUZpKN9n"
Content-Disposition: inline
In-Reply-To: <ZypXy3fH/EyALaTR@ip-10-97-1-34.eu-west-3.compute.internal>
Archived-At: <https://www.postgresql.org/message-id/Zyqsm6ilLLADAGw2%40paquier.xyz>
Precedence: bulk


--SQ0ywFv/uUZpKN9n
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Nov 05, 2024 at 05:37:15PM +0000, Bertrand Drouvot wrote:
> I'm starting working on option 2, I think it will be easier to discuss wi=
th
> a patch proposal to look at.
>=20
> If in the meantime, one strongly disagree with option 2 (means implement =
a brand
> new PGSTAT_KIND_BACKEND and keep PGSTAT_KIND_IO), please let me know.

Sorry for the late reply, catching up a bit.

As you are quoting in [1], you do not expect the backend-io stats and
the more global pg_stat_io to achieve the same level of consistency as
the backend stats would be gone at restart, and wiped out when a
backend shuts down.  So, splitting them with a different stats kind
feels more natural because it would be possible to control how each
stat kind behaves depending on the code shutdown and reset paths
within their own callbacks rather than making the callbacks of
PGSTAT_KIND_IO more complex than they already are.  And pg_stat_io is
a fixed-numbered stats kind because of the way it aggregates its stats
with a number states defined at compile-time.

Is the structure you have in mind different than PgStat_BktypeIO?
Perhaps a split is better anyway with that in mind.

The amount of memory required to store the snapshots of backend-IO
does not worry me much, TBH, but you are worried about a high turnover
of connections that could cause a lot of bloat in the backend-IO
snapshots because of the persistency that these stats would have,
right?  Actually, we already have cases with other stats kinds where
it is possible for a backend to hold references to some stats on an
object that has been dropped concurrently.  At least, that's possible
when a backend shuts down.  If possible, supporting snapshots would be
more consistent with the other stats.

Just to be clear, I am not in favor of making PgStat_HashKey larger
than it already is.  With 8 bytes allocated for the object ID, the
chances of conflicts in the dshash for a single variable-numbered
stats kind play in our favor already even with a couple of million
entries.

[1]: https://www.postgresql.org/message-id/ZyMRJIbUpNPoCXUe@ip-10-97-1-34.e=
u-west-3.compute.internal
--
Michael

--SQ0ywFv/uUZpKN9n
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmcqrJsACgkQnvQgOdby
QH1CAA//dqWiZo9g+GZZLO5HeJM6QqWWLuxwHjpQ67D9LYUH2ATx5bIFIc0wJQI9
hvz5erDu520zOIm4k5sHAzGFr3DfidUE5mK6mbVCsi/PPJXoI8qqe+ypoAY5vp8k
BqOqmur1G6k8KPUP6APvmM/LvvwdR4RBbDXaDz+t64k1wHDR7Qf0jW01LO/05SPw
XtU2fGYG58NBXhORY4cT8LeyhOp7nTJU6mDIN9cVRK3TEhuj8GuJqQpYe0mLA0Ld
lu/M70RUaZh6iX06SEUASNuL49cR/67FPxnPvXvPN2tT5TvFMoLSyLfKRgY0wOog
2/0UjeESPGI5QZ/dq92QJZCq8qcLF/2rKZWAbLp+E0iPpq347oR4yji4nA8hu65j
IQBLXsN/HXgVuvpi5s0qsIo562xRLYJYg0sD0fOGe0CepxB/tbpU53Zv1g1MvN8a
XpfaUKWAiBlSY93HmQdGgNZoFZOEDDUwVa8T89ZKA96tGDkr4D1RkaUj5tH+sZ71
kgIjh1gcdaYstNyFiFNPWLOIjtYIxFY0n3HShK+CvVtopCbuzTSXTIozox5JokiC
ss7W8sFd5PB10ScCh7JME4pl1egIq5CY2o51ZkB6rjtxikgCwe/bKu6FHt3Giusx
NP1s+eRNQs3j5bvYV7i3tDUhbVQWi7qhL0tKcF+5tZbESilgRK0=
=WQb+
-----END PGP SIGNATURE-----

--SQ0ywFv/uUZpKN9n--