MIME-Version: 1.0
References: <fe99e0ca-e564-480e-b865-5f0cee30bc60@tantorlabs.com>
 <cebfee9c-a894-46f1-b6bc-5bf4a0753b41@tantorlabs.com> <7AD765EF-14DE-4B3E-8282-E82A9B44DBAA@yandex-team.ru>
 <e738eac5-ad4d-454c-a079-38766e32ce52@tantorlabs.com> <CAA5RZ0s+khQ_u84qS_cAHzxVQQHK_UmWXxRBWTTex7zW1250tQ@mail.gmail.com>
 <0c7770bd-9750-4200-a472-bdb262583c2f@tantorlabs.com> <CAA5RZ0vxn_UiUkUK5SdngObLZzw40RhafB7SHydxzBw2_xjjiA@mail.gmail.com>
 <cc1beb52-3e4e-4866-8c32-a5967b98c977@tantorlabs.com> <18631d46-1741-4edc-b116-8d9631cdf919@tantorlabs.com>
 <4f98ec33-c627-4eb4-84f3-9994a0982277@tantorlabs.com> <CAA5RZ0tJjrX6YiQVDZCOA_BDMuhO-po_T8a3PmjxEEH_iL-qMg@mail.gmail.com>
 <623db69f-b40a-4d50-b1bd-bb073fcc4336@tantorlabs.com> <3546e2a7-5831-4c0a-996b-358d4ec483a6@tantorlabs.com>
 <6bf3059f-7ebe-4640-8870-1220f83b88ed@tantorlabs.com> <7b9df88f-3015-464b-881e-4c6a7b3a5c03@tantorlabs.com>
 <CAA5RZ0vz+Abyck0dn0fHJ5nTfJU7wdLRxfUj481Ga-BGNY0Txg@mail.gmail.com>
 <6f305fd1-44c3-4189-81ef-238777277e5e@tantorlabs.com> <CAA5RZ0trJoVqGmELHOc=G2SxHQHKdTb1aAvzf3vHdfVP8UH+Xg@mail.gmail.com>
In-Reply-To: <CAA5RZ0trJoVqGmELHOc=G2SxHQHKdTb1aAvzf3vHdfVP8UH+Xg@mail.gmail.com>
From: Sami Imseih <samimseih@gmail.com>
Date: Wed, 19 Feb 2025 19:04:41 -0600
Message-ID: <CAA5RZ0vOjQDC=OvUHuKMPYc8UT_peKt_yWpVaEqiP-JG2yqB9w@mail.gmail.com>
Subject: Re: Sample rate added to pg_stat_statements
To: Ilia Evdokimov <ilya.evdokimov@tantorlabs.com>
Cc: Alexander Korotkov <aekorotkov@gmail.com>, Michael Paquier <michael@paquier.xyz>, 
	Greg Sabino Mullane <htamfids@gmail.com>, Andrey Borodin <x4mmm@yandex-team.ru>, 
	Alena Rybakina <a.rybakina@postgrespro.ru>, pgsql-hackers <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://www.postgresql.org/message-id/CAA5RZ0vOjQDC%3DOvUHuKMPYc8UT_peKt_yWpVaEqiP-JG2yqB9w%40mail.gmail.com>
Precedence: bulk

> But instead of blindly reducing the frequency via PRNG, we can take a more thoughtful approach with threshold by execute time:

> Find the most frequent query by column 'calls' in pg_stat_statements;
> In this query look at info about execution time: min_exec_time, max_exec_time, etc;
> Gradually increase the threshold from min_exec_time to max_exec_time, limiting the tracking of this query.
> Monitor performance: once the bottleneck is resolved, stop at the current threshold value.

This approach allows us to:

> Eliminate the spin-lock bottleneck;
> Preserve data about slow queries, which may be critical for performance analysis;
> Reduce the load on the most frequent queries causing contention, instead of uniformly reducing the frequency for all queries.

In my opinion, sample rate is a better fit for pg_stat_statements,
since the queries that
you care about the most are usually the most frequently executed. Sampling them
will still provide enough good data without the risk of not capturing
statistics about
them at all.

Longer running queries will also likely be the least frequent, so they
are already not likely
contributing to the spinlock contention. Also, the least frequent
queries will likely be aged
out faster, so pg_stat_statements was never really a good candidate to
track those anyways;
slow query logging with log_min_duration_statement is a better way to
ensure you capture
the data.

Maybe others may have a different opinion?

--

Sami