From: Benoit Lobréau <benoit.lobreau@dalibo.com>
To: Sami Imseih <samimseih@gmail.com>
To: Tomas Vondra <tomas@vondra.me>
Cc: Guillaume Lelarge <guillaume@lelarge.info>
Cc: Alena Rybakina <a.rybakina@postgrespro.ru>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Imseih (AWS), Sami <simseih@amazon.com>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>
Cc: Andrey M. Borodin <x4mmm@yandex-team.ru>
Subject: Re: Logging parallel worker draught
Date: Fri, 3 Jan 2025 11:25:35 +0100
Message-ID: <82ffc7a2-b77d-437b-a684-4a79a32fd87b@dalibo.com> (raw)
In-Reply-To: <CAA5RZ0tS=Fmgku9=qCp-U4EBC-dqVtj+v_EGrKV_NA=THnUDsQ@mail.gmail.com>
References: <10D5C31A-A303-4440-BCA2-921440A9E1CC@amazon.com>
<0d08b978-10cb-4512-bcef-ca1f192049ab@enterprisedb.com>
<0b7ef57e-bf30-440b-9478-d8f587f2637c@dalibo.com>
<2cccd814-d7b3-4c9a-a66a-6df4c45fd95e@enterprisedb.com>
<17f9faf1-4edb-46d0-80ed-91bba16423a3@dalibo.com>
<7ECC97AE-2670-4792-BC6F-FD91B23F8E8C@yandex-team.ru>
<39ac8951-6b99-469b-b9f1-2791efe43fbd@dalibo.com>
<8123423a-f041-4f4c-a771-bfd96ab235b0@dalibo.com>
<2a8e9e1c-0f05-4ebd-9a4d-d023d26c0def@dalibo.com>
<04e6d7e2-f67a-438c-bc11-9354ab64ef1f@dalibo.com>
<fbf19680-ce74-4f27-a890-e427ab330411@dalibo.com>
<d4825c9e-9fd0-440e-aa3a-411f0b92bfc9@postgrespro.ru>
<c14381b1-1f7a-4677-8b94-baa4dead9bc5@postgrespro.ru>
<63d182f8-1a65-4e34-a335-403f53b14ba7@dalibo.com>
<fd9b82bc-c853-4143-ba8f-35a693ee439c@dalibo.com>
<CAECtzeX+tqQ0gLAiK558MtqLHvBEQM60i2UGSNJ1-QjMRFNYZg@mail.gmail.com>
<31dc1548-30dc-4423-b939-2842c469f129@dalibo.com>
<CAA5RZ0tS=Fmgku9=qCp-U4EBC-dqVtj+v_EGrKV_NA=THnUDsQ@mail.gmail.com>
Hi, thank you for the review and sorry for the delayed answer.
On 12/28/24 05:17, Sami Imseih wrote:> Thinking about this further, it
seems to me this logging only makes sense
> for parallel query and not maintenance commands. This is because
> for the latter case, the commands are executed manually and a user
can issue
> VACUUM VERBOSE ( for the case of vacuum ). For CREATE INDEX, one
> can enable DEBUG1. For example:
>
> postgres=# CREATE INDEX tbl_c1 ON tbl(c1);
> DEBUG: building index "tbl_c1" on table "tbl" with request for 1
> parallel workers
> DEBUG: index "tbl_c1" can safely use deduplication
> CREATE INDEX
In my opinion, it's useful:
* Centralization of logs: The maintenance operation can be planned in
cron jobs. In that context, it could certainly be logged separately via
the script. However, when I am doing an audit on a client database, I
think it's useful to have all the relevant information in PostgreSQL logs.
* Centralization of the configuration: If I am auditing the usage of
parallelism, I would like to have a single GUC to enable the logging
for all relevant information.
* Context: Maintenance workers are part of the global pool of workers.
To know why there is a shortage after the fact, having all the information
on what was using workers could be useful. I tied this argument when we
added the parallel worker info to pg_stat_database and wasn't successful.
It would be nice to have the info somewhere.
The logging for CREATE INDEX and VACUUM is in separate patches. If you
don't mind, I would like to keep it as long as possible.
> 2/ For logging parallel query workers, the log line could be simpler.
>
> Instead of:
>
> + elog(LOG, "%i parallel nodes planned (%i obtained all their workers,
> %i obtained none), "
> + "%i workers planned to be launched (%i workers launched)",
>
> what about something like:
>
> "launched %d parallel workers (planned: %)"
>
> For example, if I have 2 gather nodes and they each plan and launch
> 2 workers, the log line should be as simple as
>
> "launched 4 parallel workers (planned: 4)"
>
> This behavior in which workers planned and launched are aggregated in the
> log can be described in the documentation.
"%i parallel nodes planned (%i obtained all their workers, %i obtained
none),"
I added this part during my first reorganization. I was trying to extract
as much information as possible and thought that the special case where a
parallel node was planned but no workers were obtained was the worst-case
scenario, which might be interesting to log.
If you think it doesn't bring much value, we can strip it.
> 3/ Also, log_parallel_query_workers as the name of the GUC will make
more sense
> if we go logging parallel query only.
I agree that log_parallel_query_workers would make more sense if we focus
on logging parallel queries only. Since the maintenance-related logging
was added later, I missed this point. Thank you for bringging this up.
If agreeable, I will rename the GUC to log_parallel_workers as long as
the argument regarding maintenance workers is unresolved. If we decide
to strip them, I will revert it to log_parallel_query_workers.
>> I don't think "failure" is a good name for the setting as it's
>> a bit too harsh. What we really have is a "shortage" of
>> workers.
I agree that "failure" is too harsh. The term "shortage" better describes
the situation.
Note: I added Tomas Vondra to the recipients of the email since he advised
me on this topic initially and might have an opinion on this.
--
Benoit Lobréau
Consultant
http://dalibo.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: benoit.lobreau@dalibo.com, samimseih@gmail.com, tomas@vondra.me, guillaume@lelarge.info, a.rybakina@postgrespro.ru, simseih@amazon.com, alvherre@alvh.no-ip.org, robertmhaas@gmail.com, amit.kapila16@gmail.com, melanieplageman@gmail.com, x4mmm@yandex-team.ru
Subject: Re: Logging parallel worker draught
In-Reply-To: <82ffc7a2-b77d-437b-a684-4a79a32fd87b@dalibo.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox