public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 5/9] doc: speculative: Perform apply of large transactions by parallel workers.
6+ messages / 4 participants
[nested] [flat]
* [PATCH 5/9] doc: speculative: Perform apply of large transactions by parallel workers.
@ 2023-01-10 04:38 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Justin Pryzby @ 2023-01-10 04:38 UTC (permalink / raw)
f745739697e49a38a31a755d7d36d6c97008dd9f
---
doc/src/sgml/system-views.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 5b1c73269b9..7c8fc3f654b 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1580,7 +1580,7 @@
</para>
<para>
- A specualtive insertion lock consists of a transaction ID and a speculative
+ A speculative insertion lock consists of a transaction ID and a speculative
insertion token. The speculative insertion token is displayed in the
<structfield>objid</structfield> column.
</para>
--
2.25.1
--KFztAG8eRSV9hGtP
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0006-typos.patch"
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size
@ 2024-11-29 20:57 Kirill Gavrilov <[email protected]>
2024-11-30 06:58 ` Re: Truncate logs by max_log_size Kirill Reshke <[email protected]>
2024-12-02 10:08 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
0 siblings, 2 replies; 6+ messages in thread
From: Kirill Gavrilov @ 2024-11-29 20:57 UTC (permalink / raw)
To: Jim Jones <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected]
On Fri, Nov 29, 2024 at 10:12 PM Jim Jones <[email protected]>
wrote:
>
>
> On 29.11.24 12:05, Kirill Gavrilov wrote:
> > Setting max_log_size to 0 already disables truncation.
> > I changed guc units to bytes, just to make it more human-friendly.
> > Did some refactoring and fixed all warnings. Set default value in
> > postgresql.conf.
>
> Nice. The patch applies cleanly and the warnings are gone.
>
> A few other things:
>
> IMHO the documentation can be a bit clearer. It says to input the values
> in bytes, but the default value is in megabytes.
>
> + If greater than zero, each query logged is truncated to this
> many bytes.
> + Zero disables the setting.
> + Default value is 5MB.
>
> Perhaps something like "If this value is specified without units, it is
> taken as bytes." (see wal_keep_size)
>
Added to documentation. I hope it's clear now.
>
> And a pedantic note: This hint suggests that you can use TB as unit ..
>
> postgres=# set max_log_size = '1foo';
> ERROR: invalid value for parameter "max_log_size": "1foo"
> HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB".
>
> .. but it can never be used, as it exceeds the integer range:
>
> postgres=# SET max_log_size = '1TB';
> ERROR: invalid value for parameter "max_log_size": "1TB"
> HINT: Value exceeds integer range.
>
> Thanks
>
> --
> Jim
>
Same thing applies to log_parameter_max_length, for example.
postgres=# set log_parameter_max_length = '1foo';
ERROR: invalid value for parameter "log_parameter_max_length": "1foo"
HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB".
postgres=# set log_parameter_max_length = '1TB';
ERROR: invalid value for parameter "log_parameter_max_length": "1TB"
HINT: Value exceeds integer range.
I think we can leave it as is.
Attachments:
[application/octet-stream] V5-0001-parameter-max_log_size-to-truncate-logs.patch (9.0K, ../../CA+E0NR6MJfE8Hkk2Q-MgWUkk=Yy_4fWdiiaAKW_Ux_hcAAtAFQ@mail.gmail.com/3-V5-0001-parameter-max_log_size-to-truncate-logs.patch)
download | inline diff:
From 15aa94e0cf2066e3135e6739d6fc957510568a58 Mon Sep 17 00:00:00 2001
From: diphantxm <[email protected]>
Date: Fri, 29 Nov 2024 23:52:11 +0300
Subject: [PATCH] parameter max_log_size to truncate logs
There is no need to log the entire query, because it may be large and take lots of space on disk. Parameter max_log_size set the maximum length for logged query. Everything beyond that length is truncated. Value 0 disables the parameter.
---
doc/src/sgml/config.sgml | 16 +++++++
src/backend/tcop/postgres.c | 18 ++++++-
src/backend/utils/error/elog.c | 47 +++++++++++++++++++
src/backend/utils/misc/guc_tables.c | 12 +++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_ctl/t/004_logrotate.pl | 15 ++++++
src/include/utils/elog.h | 4 ++
7 files changed, 112 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 76ab72db96..83b1b40b0f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7913,6 +7913,22 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-log-size" xreflabel="max_log_size">
+ <term><varname>max_log_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_log_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ If greater than zero, each query logged is truncated to this many bytes.
+ Zero disables the setting.
+ If this value is specified without units, it is taken as bytes.
+ This feature is disabled by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
<sect2 id="runtime-config-logging-csvlog">
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 4b985bd056..033de5862a 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -70,6 +70,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/elog.h"
#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/lsyscache.h"
@@ -1029,11 +1030,22 @@ exec_simple_query(const char *query_string)
bool was_logged = false;
bool use_implicit_block;
char msec_str[32];
+ char* truncated_query = NULL;
+ const char* query_log;
/*
* Report query to various monitoring facilities.
*/
debug_query_string = query_string;
+ if (need_truncate_query_log(query_string))
+ {
+ truncated_query = truncate_query_log(query_string);
+ query_log = truncated_query;
+ }
+ else
+ {
+ query_log = query_string;
+ }
pgstat_report_activity(STATE_RUNNING, query_string);
@@ -1078,7 +1090,7 @@ exec_simple_query(const char *query_string)
if (check_log_statement(parsetree_list))
{
ereport(LOG,
- (errmsg("statement: %s", query_string),
+ (errmsg("statement: %s", query_log),
errhidestmt(true),
errdetail_execute(parsetree_list)));
was_logged = true;
@@ -1377,7 +1389,7 @@ exec_simple_query(const char *query_string)
case 2:
ereport(LOG,
(errmsg("duration: %s ms statement: %s",
- msec_str, query_string),
+ msec_str, query_log),
errhidestmt(true),
errdetail_execute(parsetree_list)));
break;
@@ -1388,6 +1400,8 @@ exec_simple_query(const char *query_string)
TRACE_POSTGRESQL_QUERY_DONE(query_string);
+ if (truncated_query)
+ pfree(truncated_query);
debug_query_string = NULL;
}
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 289059435a..8a1071677b 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -111,6 +111,7 @@ int Log_destination = LOG_DESTINATION_STDERR;
char *Log_destination_string = NULL;
bool syslog_sequence_numbers = true;
bool syslog_split_messages = true;
+int max_log_size = 0;
/* Processed form of backtrace_functions GUC */
static char *backtrace_function_list;
@@ -1688,11 +1689,18 @@ EmitErrorReport(void)
{
ErrorData *edata = &errordata[errordata_stack_depth];
MemoryContext oldcontext;
+ char* truncated_query = NULL;
recursion_depth++;
CHECK_STACK_DEPTH();
oldcontext = MemoryContextSwitchTo(edata->assoc_context);
+ if (need_truncate_query_log(debug_query_string))
+ {
+ truncated_query = truncate_query_log(debug_query_string);
+ debug_query_string = truncated_query;
+ }
+
/*
* Reset the formatted timestamp fields before emitting any logs. This
* includes all the log destinations and emit_log_hook, as the latter
@@ -1733,6 +1741,9 @@ EmitErrorReport(void)
MemoryContextSwitchTo(oldcontext);
recursion_depth--;
+
+ if (truncated_query)
+ pfree(truncated_query);
}
/*
@@ -3773,3 +3784,39 @@ write_stderr(const char *fmt,...)
#endif
va_end(ap);
}
+
+/*
+ * Apply truncation to build query that will be logged.
+ *
+ * If query needs to be truncated, copied will be set to true
+ * and returned string must be freed
+*/
+char*
+truncate_query_log(const char* query)
+{
+ size_t truncated_query_len;
+ char* truncatd_query;
+ size_t query_len;
+
+ if (!query)
+ return NULL;
+
+ query_len = strlen(query);
+ truncated_query_len = pg_mbcliplen(query, query_len, max_log_size);
+ truncatd_query = (char *) palloc(truncated_query_len+1);
+ memcpy(truncatd_query, query, truncated_query_len);
+ truncatd_query[truncated_query_len] = '\0';
+ return truncatd_query;
+}
+
+/*
+ * Checks if query should be truncated
+ * according to max_log_size
+*/
+bool
+need_truncate_query_log(const char* query)
+{
+ if (!query)
+ return false;
+ return !(max_log_size == 0 || strlen(query) < max_log_size);
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9845abd693..8ad18f7a47 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3724,6 +3724,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_log_size", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Sets max size in bytes of logged statement."),
+ NULL,
+ GUC_UNIT_BYTE
+ },
+ &max_log_size,
+ 0,
+ 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 407cd1e08c..de695e257d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -617,6 +617,8 @@
# bind-parameter values to N bytes;
# -1 means print in full, 0 disables
#log_statement = 'none' # none, ddl, mod, all
+#max_log_size = 0 # max size of logged statement
+ # 0 disables the feature
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/bin/pg_ctl/t/004_logrotate.pl b/src/bin/pg_ctl/t/004_logrotate.pl
index eacca1a652..1c1f89cb0a 100644
--- a/src/bin/pg_ctl/t/004_logrotate.pl
+++ b/src/bin/pg_ctl/t/004_logrotate.pl
@@ -69,6 +69,7 @@ log_destination = 'stderr, csvlog, jsonlog'
# these ensure stability of test results:
log_rotation_age = 0
lc_messages = 'C'
+max_log_size = 32
));
$node->start();
@@ -135,6 +136,20 @@ check_log_pattern('stderr', $new_current_logfiles, 'syntax error', $node);
check_log_pattern('csvlog', $new_current_logfiles, 'syntax error', $node);
check_log_pattern('jsonlog', $new_current_logfiles, 'syntax error', $node);
+$node->psql('postgres', 'INSERT INTO SOME_NON_EXISTANT_TABLE VALUES (TEST)');
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ eval {
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
+ };
+ last unless $@;
+ usleep(100_000);
+}
+die $@ if $@;
+check_log_pattern('stderr', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node);
+check_log_pattern('csvlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node);
+check_log_pattern('jsonlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node);
+
$node->stop();
done_testing();
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index e54eca5b48..cafa8de3c0 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -502,6 +502,7 @@ extern PGDLLIMPORT int Log_destination;
extern PGDLLIMPORT char *Log_destination_string;
extern PGDLLIMPORT bool syslog_sequence_numbers;
extern PGDLLIMPORT bool syslog_split_messages;
+extern PGDLLIMPORT int max_log_size;
/* Log destination bitmap */
#define LOG_DESTINATION_STDERR 1
@@ -517,6 +518,9 @@ extern void DebugFileOpen(void);
extern char *unpack_sql_state(int sql_state);
extern bool in_error_recursion_trouble(void);
+extern bool need_truncate_query_log(const char* query);
+extern char* truncate_query_log(const char* query);
+
/* Common functions shared across destinations */
extern void reset_formatted_start_time(void);
extern char *get_formatted_start_time(void);
--
2.39.2 (Apple Git-143)
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size
2024-11-29 20:57 Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
@ 2024-11-30 06:58 ` Kirill Reshke <[email protected]>
1 sibling, 0 replies; 6+ messages in thread
From: Kirill Reshke @ 2024-11-30 06:58 UTC (permalink / raw)
To: Kirill Gavrilov <[email protected]>; +Cc: Jim Jones <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected]
On Sat, 30 Nov 2024 at 01:57, Kirill Gavrilov <[email protected]> wrote:
>
> On Fri, Nov 29, 2024 at 10:12 PM Jim Jones <[email protected]> wrote:
>>
>>
>>
>> On 29.11.24 12:05, Kirill Gavrilov wrote:
>> > Setting max_log_size to 0 already disables truncation.
>> > I changed guc units to bytes, just to make it more human-friendly.
>> > Did some refactoring and fixed all warnings. Set default value in
>> > postgresql.conf.
>>
>> Nice. The patch applies cleanly and the warnings are gone.
>>
>> A few other things:
>>
>> IMHO the documentation can be a bit clearer. It says to input the values
>> in bytes, but the default value is in megabytes.
>>
>> + If greater than zero, each query logged is truncated to this
>> many bytes.
>> + Zero disables the setting.
>> + Default value is 5MB.
>>
>> Perhaps something like "If this value is specified without units, it is
>> taken as bytes." (see wal_keep_size)
>
>
> Added to documentation. I hope it's clear now.
>
>>
>>
>> And a pedantic note: This hint suggests that you can use TB as unit ..
>>
>> postgres=# set max_log_size = '1foo';
>> ERROR: invalid value for parameter "max_log_size": "1foo"
>> HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB".
>>
>> .. but it can never be used, as it exceeds the integer range:
>>
>> postgres=# SET max_log_size = '1TB';
>> ERROR: invalid value for parameter "max_log_size": "1TB"
>> HINT: Value exceeds integer range.
>>
>> Thanks
>>
>> --
>> Jim
>
>
> Same thing applies to log_parameter_max_length, for example.
>
> postgres=# set log_parameter_max_length = '1foo';
> ERROR: invalid value for parameter "log_parameter_max_length": "1foo"
> HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB".
> postgres=# set log_parameter_max_length = '1TB';
> ERROR: invalid value for parameter "log_parameter_max_length": "1TB"
> HINT: Value exceeds integer range.
>
> I think we can leave it as is.
Hi
> +for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
> +{
> + eval {
> + $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
> + };
> + last unless $@;
> + usleep(100_000);
> +}
`usleep` in tap tests is usually a bad pattern. Do we have a chance to
test this using `wait_for_log` or similar?
--
Best regards,
Kirill Reshke
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size
2024-11-29 20:57 Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
@ 2024-12-02 10:08 ` Jim Jones <[email protected]>
2025-01-31 10:46 ` Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
1 sibling, 1 reply; 6+ messages in thread
From: Jim Jones @ 2024-12-02 10:08 UTC (permalink / raw)
To: Kirill Gavrilov <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected]
On 29.11.24 21:57, Kirill Gavrilov wrote:
> Same thing applies to log_parameter_max_length, for example.
>
> postgres=# set log_parameter_max_length = '1foo';
> ERROR: invalid value for parameter "log_parameter_max_length": "1foo"
> HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB".
> postgres=# set log_parameter_max_length = '1TB';
> ERROR: invalid value for parameter "log_parameter_max_length": "1TB"
> HINT: Value exceeds integer range.
>
> I think we can leave it as is.
I see. So I guess it is out of scope to change this message here.
Small nitpicks:
1) The indentation of the comment at postgresql.conf.sample is a little
bit off
#max_log_size = 0 # max size of logged statement
# 0 disables the feature
IMHO it looks better like this:
#max_log_size = 0 # max size of logged statement
# 0 disables the feature
2) You introduced a trailing whitespace at L34 (Not critical :))
+ Zero disables the setting.
It happens to me all the time, so I usually try to apply my patches in a
clean branch just to make sure I didn't miss anything.
Other than that, I have nothing more to add at this point.
Thanks
--
Jim
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size
2024-11-29 20:57 Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
2024-12-02 10:08 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
@ 2025-01-31 10:46 ` Kirill Gavrilov <[email protected]>
2025-02-03 10:31 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Kirill Gavrilov @ 2025-01-31 10:46 UTC (permalink / raw)
To: Jim Jones <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected]
On Mon, Dec 2, 2024 at 1:09 PM Jim Jones <[email protected]> wrote:
>
>
> On 29.11.24 21:57, Kirill Gavrilov wrote:
> > Same thing applies to log_parameter_max_length, for example.
> >
> > postgres=# set log_parameter_max_length = '1foo';
> > ERROR: invalid value for parameter "log_parameter_max_length": "1foo"
> > HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and
> "TB".
> > postgres=# set log_parameter_max_length = '1TB';
> > ERROR: invalid value for parameter "log_parameter_max_length": "1TB"
> > HINT: Value exceeds integer range.
> >
> > I think we can leave it as is.
>
> I see. So I guess it is out of scope to change this message here.
>
> Small nitpicks:
>
> 1) The indentation of the comment at postgresql.conf.sample is a little
> bit off
>
> #max_log_size = 0 # max size of logged statement
> # 0 disables the feature
>
> IMHO it looks better like this:
>
> #max_log_size = 0 # max size of logged statement
> # 0 disables the feature
>
>
> 2) You introduced a trailing whitespace at L34 (Not critical :))
>
> + Zero disables the setting.
>
> It happens to me all the time, so I usually try to apply my patches in a
> clean branch just to make sure I didn't miss anything.
>
> Other than that, I have nothing more to add at this point.
>
> Thanks
>
> --
> Jim
>
>
Sorry for the long silence. I fixed the indentation and a trailing
whitespace. Should look fine now.
Attachments:
[application/octet-stream] V6-0001-parameter-max_log_size-to-truncate-logs.patch (9.0K, ../../CA+E0NR7XD3wpts2GEFpZpO-M8kj9f8pTrhnAKvq5n4xjtLgLKg@mail.gmail.com/3-V6-0001-parameter-max_log_size-to-truncate-logs.patch)
download | inline diff:
From a020529b6ac33393e9992e2118762f2f39762d9b Mon Sep 17 00:00:00 2001
From: diphantxm <[email protected]>
Date: Fri, 31 Jan 2025 13:31:50 +0300
Subject: [PATCH] parameter max_log_size to truncate logs
There is no need to log the entire query, because it may be large and take lots of space on disk. Parameter max_log_size set the maximum length for logged query. Everything beyond that length is truncated. Value 0 disables the parameter.
---
doc/src/sgml/config.sgml | 16 +++++++
src/backend/tcop/postgres.c | 18 ++++++-
src/backend/utils/error/elog.c | 47 +++++++++++++++++++
src/backend/utils/misc/guc_tables.c | 12 +++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_ctl/t/004_logrotate.pl | 15 ++++++
src/include/utils/elog.h | 4 ++
7 files changed, 112 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 76ab72db96..86dcebfe36 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7913,6 +7913,22 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-log-size" xreflabel="max_log_size">
+ <term><varname>max_log_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_log_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ If greater than zero, each query logged is truncated to this many bytes.
+ Zero disables the setting.
+ If this value is specified without units, it is taken as bytes.
+ This feature is disabled by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
<sect2 id="runtime-config-logging-csvlog">
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 4b985bd056..033de5862a 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -70,6 +70,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/elog.h"
#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/lsyscache.h"
@@ -1029,11 +1030,22 @@ exec_simple_query(const char *query_string)
bool was_logged = false;
bool use_implicit_block;
char msec_str[32];
+ char* truncated_query = NULL;
+ const char* query_log;
/*
* Report query to various monitoring facilities.
*/
debug_query_string = query_string;
+ if (need_truncate_query_log(query_string))
+ {
+ truncated_query = truncate_query_log(query_string);
+ query_log = truncated_query;
+ }
+ else
+ {
+ query_log = query_string;
+ }
pgstat_report_activity(STATE_RUNNING, query_string);
@@ -1078,7 +1090,7 @@ exec_simple_query(const char *query_string)
if (check_log_statement(parsetree_list))
{
ereport(LOG,
- (errmsg("statement: %s", query_string),
+ (errmsg("statement: %s", query_log),
errhidestmt(true),
errdetail_execute(parsetree_list)));
was_logged = true;
@@ -1377,7 +1389,7 @@ exec_simple_query(const char *query_string)
case 2:
ereport(LOG,
(errmsg("duration: %s ms statement: %s",
- msec_str, query_string),
+ msec_str, query_log),
errhidestmt(true),
errdetail_execute(parsetree_list)));
break;
@@ -1388,6 +1400,8 @@ exec_simple_query(const char *query_string)
TRACE_POSTGRESQL_QUERY_DONE(query_string);
+ if (truncated_query)
+ pfree(truncated_query);
debug_query_string = NULL;
}
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 289059435a..8a1071677b 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -111,6 +111,7 @@ int Log_destination = LOG_DESTINATION_STDERR;
char *Log_destination_string = NULL;
bool syslog_sequence_numbers = true;
bool syslog_split_messages = true;
+int max_log_size = 0;
/* Processed form of backtrace_functions GUC */
static char *backtrace_function_list;
@@ -1688,11 +1689,18 @@ EmitErrorReport(void)
{
ErrorData *edata = &errordata[errordata_stack_depth];
MemoryContext oldcontext;
+ char* truncated_query = NULL;
recursion_depth++;
CHECK_STACK_DEPTH();
oldcontext = MemoryContextSwitchTo(edata->assoc_context);
+ if (need_truncate_query_log(debug_query_string))
+ {
+ truncated_query = truncate_query_log(debug_query_string);
+ debug_query_string = truncated_query;
+ }
+
/*
* Reset the formatted timestamp fields before emitting any logs. This
* includes all the log destinations and emit_log_hook, as the latter
@@ -1733,6 +1741,9 @@ EmitErrorReport(void)
MemoryContextSwitchTo(oldcontext);
recursion_depth--;
+
+ if (truncated_query)
+ pfree(truncated_query);
}
/*
@@ -3773,3 +3784,39 @@ write_stderr(const char *fmt,...)
#endif
va_end(ap);
}
+
+/*
+ * Apply truncation to build query that will be logged.
+ *
+ * If query needs to be truncated, copied will be set to true
+ * and returned string must be freed
+*/
+char*
+truncate_query_log(const char* query)
+{
+ size_t truncated_query_len;
+ char* truncatd_query;
+ size_t query_len;
+
+ if (!query)
+ return NULL;
+
+ query_len = strlen(query);
+ truncated_query_len = pg_mbcliplen(query, query_len, max_log_size);
+ truncatd_query = (char *) palloc(truncated_query_len+1);
+ memcpy(truncatd_query, query, truncated_query_len);
+ truncatd_query[truncated_query_len] = '\0';
+ return truncatd_query;
+}
+
+/*
+ * Checks if query should be truncated
+ * according to max_log_size
+*/
+bool
+need_truncate_query_log(const char* query)
+{
+ if (!query)
+ return false;
+ return !(max_log_size == 0 || strlen(query) < max_log_size);
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9845abd693..8ad18f7a47 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3724,6 +3724,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_log_size", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Sets max size in bytes of logged statement."),
+ NULL,
+ GUC_UNIT_BYTE
+ },
+ &max_log_size,
+ 0,
+ 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 407cd1e08c..80a45341df 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -617,6 +617,8 @@
# bind-parameter values to N bytes;
# -1 means print in full, 0 disables
#log_statement = 'none' # none, ddl, mod, all
+#max_log_size = 0 # max size of logged statement
+ # 0 disables the feature
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/bin/pg_ctl/t/004_logrotate.pl b/src/bin/pg_ctl/t/004_logrotate.pl
index eacca1a652..1c1f89cb0a 100644
--- a/src/bin/pg_ctl/t/004_logrotate.pl
+++ b/src/bin/pg_ctl/t/004_logrotate.pl
@@ -69,6 +69,7 @@ log_destination = 'stderr, csvlog, jsonlog'
# these ensure stability of test results:
log_rotation_age = 0
lc_messages = 'C'
+max_log_size = 32
));
$node->start();
@@ -135,6 +136,20 @@ check_log_pattern('stderr', $new_current_logfiles, 'syntax error', $node);
check_log_pattern('csvlog', $new_current_logfiles, 'syntax error', $node);
check_log_pattern('jsonlog', $new_current_logfiles, 'syntax error', $node);
+$node->psql('postgres', 'INSERT INTO SOME_NON_EXISTANT_TABLE VALUES (TEST)');
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ eval {
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
+ };
+ last unless $@;
+ usleep(100_000);
+}
+die $@ if $@;
+check_log_pattern('stderr', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node);
+check_log_pattern('csvlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node);
+check_log_pattern('jsonlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node);
+
$node->stop();
done_testing();
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index e54eca5b48..cafa8de3c0 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -502,6 +502,7 @@ extern PGDLLIMPORT int Log_destination;
extern PGDLLIMPORT char *Log_destination_string;
extern PGDLLIMPORT bool syslog_sequence_numbers;
extern PGDLLIMPORT bool syslog_split_messages;
+extern PGDLLIMPORT int max_log_size;
/* Log destination bitmap */
#define LOG_DESTINATION_STDERR 1
@@ -517,6 +518,9 @@ extern void DebugFileOpen(void);
extern char *unpack_sql_state(int sql_state);
extern bool in_error_recursion_trouble(void);
+extern bool need_truncate_query_log(const char* query);
+extern char* truncate_query_log(const char* query);
+
/* Common functions shared across destinations */
extern void reset_formatted_start_time(void);
extern char *get_formatted_start_time(void);
--
2.39.2 (Apple Git-143)
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size
2024-11-29 20:57 Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
2024-12-02 10:08 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
2025-01-31 10:46 ` Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
@ 2025-02-03 10:31 ` Jim Jones <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Jim Jones @ 2025-02-03 10:31 UTC (permalink / raw)
To: Kirill Gavrilov <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected]
Hi Kirill
On 31.01.25 11:46, Kirill Gavrilov wrote:
> Sorry for the long silence. I fixed the indentation and a trailing
> whitespace. Should look fine now.
The patch applies cleanly, the documentation is clear, and all tests pass.
It is possible to change this new parameter session-wise, which is nice!
postgres=# SET max_log_size TO 7;
SET
postgres=# SHOW max_log_size;
max_log_size
--------------
7B
(1 row)
The default value now is clear and it corresponds to the value set on
postgresql.conf:
#max_log_size = 0 # max size of logged statement
postgres=# SHOW max_log_size;
max_log_size
--------------
0
(1 row)
Logs are truncated as expected:
postgres=# SET max_log_size TO 6;
SET
postgres=# SELECT length('CALL xyz;');
length
--------
9
(1 row)
postgres=# CALL xyz;
ERROR: syntax error at or near ";"
LINE 1: CALL xyz;
log entry:
2025-02-03 10:58:19.975 CET [123945] ERROR: syntax error at or near ";"
at character 9
2025-02-03 10:58:19.975 CET [123945] STATEMENT: CALL x
The issue with log entry sizes for queries containing special characters
was resolved by setting the unit to bytes.
Overall, everythingLGTM.
The new status of this patch is: Ready for Committer
Jim
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2025-02-03 10:31 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-01-10 04:38 [PATCH 5/9] doc: speculative: Perform apply of large transactions by parallel workers. Justin Pryzby <[email protected]>
2024-11-29 20:57 Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
2024-11-30 06:58 ` Re: Truncate logs by max_log_size Kirill Reshke <[email protected]>
2024-12-02 10:08 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
2025-01-31 10:46 ` Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]>
2025-02-03 10:31 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox