From: Benoit Lobréau <benoit.lobreau@dalibo.com>
To: Imseih (AWS), Sami <simseih@amazon.com>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Melanie Plageman <melanieplageman@gmail.com>
Subject: Re: Logging parallel worker draught
Date: Thu, 12 Oct 2023 12:01:46 +0200
Message-ID: <11e34b80-b0a6-e2e4-1606-1f5077379a34@dalibo.com> (raw)
In-Reply-To: <9E9E69BD-BABB-49BB-8B69-61939179F20D@amazon.com>
References: <10D5C31A-A303-4440-BCA2-921440A9E1CC@amazon.com>
<202310111011.lyo66rk7k7pu@alvherre.pgsql>
<9E9E69BD-BABB-49BB-8B69-61939179F20D@amazon.com>
On 10/11/23 17:26, Imseih (AWS), Sami wrote:
Thank you for resurrecting this thread.
>> Well, if you read Benoit's earlier proposal at [1] you'll see that he
>> does propose to have some cumulative stats; this LOG line he proposes
>> here is not a substitute for stats, but rather a complement. I don't
>> see any reason to reject this patch even if we do get stats.
I believe both cumulative statistics and logs are needed. Logs excel in
pinpointing specific queries at precise times, while statistics provide
a broader overview of the situation. Additionally, I often encounter
situations where clients lack pg_stat_statements and can't restart their
production promptly.
> Regarding the current patch, the latest version removes the separate GUC,
> but the user should be able to control this behavior.
I created this patch in response to Amit Kapila's proposal to keep the
discussion ongoing. However, I still favor the initial version with the
GUCs.
> Query text is logged when log_min_error_statement > default level of "error".
>
> This could be especially problematic when there is a query running more than 1 Parallel
> Gather node that is in draught. In those cases each node will end up
> generating a log with the statement text. So, a single query execution could end up
> having multiple log lines with the statement text.
> ...
> I wonder if it will be better to accumulate the total # of workers planned and # of workers launched and
> logging this information at the end of execution?
log_temp_files exhibits similar behavior when a query involves multiple
on-disk sorts. I'm uncertain whether this is something we should or need
to address. I'll explore whether the error message can be made more
informative.
[local]:5437 postgres@postgres=# SET work_mem to '125kB';
[local]:5437 postgres@postgres=# SET log_temp_files TO 0;
[local]:5437 postgres@postgres=# SET client_min_messages TO log;
[local]:5437 postgres@postgres=# WITH a AS ( SELECT x FROM
generate_series(1,10000) AS F(x) ORDER BY 1 ) , b AS (SELECT x FROM
generate_series(1,10000) AS F(x) ORDER BY 1 ) SELECT * FROM a,b;
LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp138850.20", size
122880 => First sort
LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp138850.19", size 140000
LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp138850.23", size 140000
LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp138850.22", size
122880 => Second sort
LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp138850.21", size 140000
--
Benoit Lobréau
Consultant
http://dalibo.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: benoit.lobreau@dalibo.com, simseih@amazon.com, alvherre@alvh.no-ip.org, robertmhaas@gmail.com, amit.kapila16@gmail.com, melanieplageman@gmail.com
Subject: Re: Logging parallel worker draught
In-Reply-To: <11e34b80-b0a6-e2e4-1606-1f5077379a34@dalibo.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox