public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 5/9] doc: speculative: Perform apply of large transactions by parallel workers. 6+ messages / 4 participants [nested] [flat]
* [PATCH 5/9] doc: speculative: Perform apply of large transactions by parallel workers. @ 2023-01-10 04:38 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 6+ messages in thread From: Justin Pryzby @ 2023-01-10 04:38 UTC (permalink / raw) f745739697e49a38a31a755d7d36d6c97008dd9f --- doc/src/sgml/system-views.sgml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 5b1c73269b9..7c8fc3f654b 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1580,7 +1580,7 @@ </para> <para> - A specualtive insertion lock consists of a transaction ID and a speculative + A speculative insertion lock consists of a transaction ID and a speculative insertion token. The speculative insertion token is displayed in the <structfield>objid</structfield> column. </para> -- 2.25.1 --KFztAG8eRSV9hGtP Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-typos.patch" ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size @ 2024-11-29 20:57 Kirill Gavrilov <[email protected]> 0 siblings, 2 replies; 6+ messages in thread From: Kirill Gavrilov @ 2024-11-29 20:57 UTC (permalink / raw) To: Jim Jones <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected] On Fri, Nov 29, 2024 at 10:12 PM Jim Jones <[email protected]> wrote: > > > On 29.11.24 12:05, Kirill Gavrilov wrote: > > Setting max_log_size to 0 already disables truncation. > > I changed guc units to bytes, just to make it more human-friendly. > > Did some refactoring and fixed all warnings. Set default value in > > postgresql.conf. > > Nice. The patch applies cleanly and the warnings are gone. > > A few other things: > > IMHO the documentation can be a bit clearer. It says to input the values > in bytes, but the default value is in megabytes. > > + If greater than zero, each query logged is truncated to this > many bytes. > + Zero disables the setting. > + Default value is 5MB. > > Perhaps something like "If this value is specified without units, it is > taken as bytes." (see wal_keep_size) > Added to documentation. I hope it's clear now. > > And a pedantic note: This hint suggests that you can use TB as unit .. > > postgres=# set max_log_size = '1foo'; > ERROR: invalid value for parameter "max_log_size": "1foo" > HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB". > > .. but it can never be used, as it exceeds the integer range: > > postgres=# SET max_log_size = '1TB'; > ERROR: invalid value for parameter "max_log_size": "1TB" > HINT: Value exceeds integer range. > > Thanks > > -- > Jim > Same thing applies to log_parameter_max_length, for example. postgres=# set log_parameter_max_length = '1foo'; ERROR: invalid value for parameter "log_parameter_max_length": "1foo" HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB". postgres=# set log_parameter_max_length = '1TB'; ERROR: invalid value for parameter "log_parameter_max_length": "1TB" HINT: Value exceeds integer range. I think we can leave it as is. Attachments: [application/octet-stream] V5-0001-parameter-max_log_size-to-truncate-logs.patch (9.0K, ../../CA+E0NR6MJfE8Hkk2Q-MgWUkk=Yy_4fWdiiaAKW_Ux_hcAAtAFQ@mail.gmail.com/3-V5-0001-parameter-max_log_size-to-truncate-logs.patch) download | inline diff: From 15aa94e0cf2066e3135e6739d6fc957510568a58 Mon Sep 17 00:00:00 2001 From: diphantxm <[email protected]> Date: Fri, 29 Nov 2024 23:52:11 +0300 Subject: [PATCH] parameter max_log_size to truncate logs There is no need to log the entire query, because it may be large and take lots of space on disk. Parameter max_log_size set the maximum length for logged query. Everything beyond that length is truncated. Value 0 disables the parameter. --- doc/src/sgml/config.sgml | 16 +++++++ src/backend/tcop/postgres.c | 18 ++++++- src/backend/utils/error/elog.c | 47 +++++++++++++++++++ src/backend/utils/misc/guc_tables.c | 12 +++++ src/backend/utils/misc/postgresql.conf.sample | 2 + src/bin/pg_ctl/t/004_logrotate.pl | 15 ++++++ src/include/utils/elog.h | 4 ++ 7 files changed, 112 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 76ab72db96..83b1b40b0f 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7913,6 +7913,22 @@ log_line_prefix = '%m [%p] %q%u@%d/%a ' </listitem> </varlistentry> + <varlistentry id="guc-max-log-size" xreflabel="max_log_size"> + <term><varname>max_log_size</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_log_size</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + If greater than zero, each query logged is truncated to this many bytes. + Zero disables the setting. + If this value is specified without units, it is taken as bytes. + This feature is disabled by default. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> <sect2 id="runtime-config-logging-csvlog"> diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 4b985bd056..033de5862a 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -70,6 +70,7 @@ #include "tcop/pquery.h" #include "tcop/tcopprot.h" #include "tcop/utility.h" +#include "utils/elog.h" #include "utils/guc_hooks.h" #include "utils/injection_point.h" #include "utils/lsyscache.h" @@ -1029,11 +1030,22 @@ exec_simple_query(const char *query_string) bool was_logged = false; bool use_implicit_block; char msec_str[32]; + char* truncated_query = NULL; + const char* query_log; /* * Report query to various monitoring facilities. */ debug_query_string = query_string; + if (need_truncate_query_log(query_string)) + { + truncated_query = truncate_query_log(query_string); + query_log = truncated_query; + } + else + { + query_log = query_string; + } pgstat_report_activity(STATE_RUNNING, query_string); @@ -1078,7 +1090,7 @@ exec_simple_query(const char *query_string) if (check_log_statement(parsetree_list)) { ereport(LOG, - (errmsg("statement: %s", query_string), + (errmsg("statement: %s", query_log), errhidestmt(true), errdetail_execute(parsetree_list))); was_logged = true; @@ -1377,7 +1389,7 @@ exec_simple_query(const char *query_string) case 2: ereport(LOG, (errmsg("duration: %s ms statement: %s", - msec_str, query_string), + msec_str, query_log), errhidestmt(true), errdetail_execute(parsetree_list))); break; @@ -1388,6 +1400,8 @@ exec_simple_query(const char *query_string) TRACE_POSTGRESQL_QUERY_DONE(query_string); + if (truncated_query) + pfree(truncated_query); debug_query_string = NULL; } diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c index 289059435a..8a1071677b 100644 --- a/src/backend/utils/error/elog.c +++ b/src/backend/utils/error/elog.c @@ -111,6 +111,7 @@ int Log_destination = LOG_DESTINATION_STDERR; char *Log_destination_string = NULL; bool syslog_sequence_numbers = true; bool syslog_split_messages = true; +int max_log_size = 0; /* Processed form of backtrace_functions GUC */ static char *backtrace_function_list; @@ -1688,11 +1689,18 @@ EmitErrorReport(void) { ErrorData *edata = &errordata[errordata_stack_depth]; MemoryContext oldcontext; + char* truncated_query = NULL; recursion_depth++; CHECK_STACK_DEPTH(); oldcontext = MemoryContextSwitchTo(edata->assoc_context); + if (need_truncate_query_log(debug_query_string)) + { + truncated_query = truncate_query_log(debug_query_string); + debug_query_string = truncated_query; + } + /* * Reset the formatted timestamp fields before emitting any logs. This * includes all the log destinations and emit_log_hook, as the latter @@ -1733,6 +1741,9 @@ EmitErrorReport(void) MemoryContextSwitchTo(oldcontext); recursion_depth--; + + if (truncated_query) + pfree(truncated_query); } /* @@ -3773,3 +3784,39 @@ write_stderr(const char *fmt,...) #endif va_end(ap); } + +/* + * Apply truncation to build query that will be logged. + * + * If query needs to be truncated, copied will be set to true + * and returned string must be freed +*/ +char* +truncate_query_log(const char* query) +{ + size_t truncated_query_len; + char* truncatd_query; + size_t query_len; + + if (!query) + return NULL; + + query_len = strlen(query); + truncated_query_len = pg_mbcliplen(query, query_len, max_log_size); + truncatd_query = (char *) palloc(truncated_query_len+1); + memcpy(truncatd_query, query, truncated_query_len); + truncatd_query[truncated_query_len] = '\0'; + return truncatd_query; +} + +/* + * Checks if query should be truncated + * according to max_log_size +*/ +bool +need_truncate_query_log(const char* query) +{ + if (!query) + return false; + return !(max_log_size == 0 || strlen(query) < max_log_size); +} diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 9845abd693..8ad18f7a47 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -3724,6 +3724,18 @@ struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_log_size", PGC_SUSET, LOGGING_WHAT, + gettext_noop("Sets max size in bytes of logged statement."), + NULL, + GUC_UNIT_BYTE + }, + &max_log_size, + 0, + 0, INT_MAX, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 407cd1e08c..de695e257d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -617,6 +617,8 @@ # bind-parameter values to N bytes; # -1 means print in full, 0 disables #log_statement = 'none' # none, ddl, mod, all +#max_log_size = 0 # max size of logged statement + # 0 disables the feature #log_replication_commands = off #log_temp_files = -1 # log temporary files equal or larger # than the specified size in kilobytes; diff --git a/src/bin/pg_ctl/t/004_logrotate.pl b/src/bin/pg_ctl/t/004_logrotate.pl index eacca1a652..1c1f89cb0a 100644 --- a/src/bin/pg_ctl/t/004_logrotate.pl +++ b/src/bin/pg_ctl/t/004_logrotate.pl @@ -69,6 +69,7 @@ log_destination = 'stderr, csvlog, jsonlog' # these ensure stability of test results: log_rotation_age = 0 lc_messages = 'C' +max_log_size = 32 )); $node->start(); @@ -135,6 +136,20 @@ check_log_pattern('stderr', $new_current_logfiles, 'syntax error', $node); check_log_pattern('csvlog', $new_current_logfiles, 'syntax error', $node); check_log_pattern('jsonlog', $new_current_logfiles, 'syntax error', $node); +$node->psql('postgres', 'INSERT INTO SOME_NON_EXISTANT_TABLE VALUES (TEST)'); +for (my $attempts = 0; $attempts < $max_attempts; $attempts++) +{ + eval { + $current_logfiles = slurp_file($node->data_dir . '/current_logfiles'); + }; + last unless $@; + usleep(100_000); +} +die $@ if $@; +check_log_pattern('stderr', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node); +check_log_pattern('csvlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node); +check_log_pattern('jsonlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node); + $node->stop(); done_testing(); diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h index e54eca5b48..cafa8de3c0 100644 --- a/src/include/utils/elog.h +++ b/src/include/utils/elog.h @@ -502,6 +502,7 @@ extern PGDLLIMPORT int Log_destination; extern PGDLLIMPORT char *Log_destination_string; extern PGDLLIMPORT bool syslog_sequence_numbers; extern PGDLLIMPORT bool syslog_split_messages; +extern PGDLLIMPORT int max_log_size; /* Log destination bitmap */ #define LOG_DESTINATION_STDERR 1 @@ -517,6 +518,9 @@ extern void DebugFileOpen(void); extern char *unpack_sql_state(int sql_state); extern bool in_error_recursion_trouble(void); +extern bool need_truncate_query_log(const char* query); +extern char* truncate_query_log(const char* query); + /* Common functions shared across destinations */ extern void reset_formatted_start_time(void); extern char *get_formatted_start_time(void); -- 2.39.2 (Apple Git-143) ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size @ 2024-11-30 06:58 Kirill Reshke <[email protected]> parent: Kirill Gavrilov <[email protected]> 1 sibling, 0 replies; 6+ messages in thread From: Kirill Reshke @ 2024-11-30 06:58 UTC (permalink / raw) To: Kirill Gavrilov <[email protected]>; +Cc: Jim Jones <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected] On Sat, 30 Nov 2024 at 01:57, Kirill Gavrilov <[email protected]> wrote: > > On Fri, Nov 29, 2024 at 10:12 PM Jim Jones <[email protected]> wrote: >> >> >> >> On 29.11.24 12:05, Kirill Gavrilov wrote: >> > Setting max_log_size to 0 already disables truncation. >> > I changed guc units to bytes, just to make it more human-friendly. >> > Did some refactoring and fixed all warnings. Set default value in >> > postgresql.conf. >> >> Nice. The patch applies cleanly and the warnings are gone. >> >> A few other things: >> >> IMHO the documentation can be a bit clearer. It says to input the values >> in bytes, but the default value is in megabytes. >> >> + If greater than zero, each query logged is truncated to this >> many bytes. >> + Zero disables the setting. >> + Default value is 5MB. >> >> Perhaps something like "If this value is specified without units, it is >> taken as bytes." (see wal_keep_size) > > > Added to documentation. I hope it's clear now. > >> >> >> And a pedantic note: This hint suggests that you can use TB as unit .. >> >> postgres=# set max_log_size = '1foo'; >> ERROR: invalid value for parameter "max_log_size": "1foo" >> HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB". >> >> .. but it can never be used, as it exceeds the integer range: >> >> postgres=# SET max_log_size = '1TB'; >> ERROR: invalid value for parameter "max_log_size": "1TB" >> HINT: Value exceeds integer range. >> >> Thanks >> >> -- >> Jim > > > Same thing applies to log_parameter_max_length, for example. > > postgres=# set log_parameter_max_length = '1foo'; > ERROR: invalid value for parameter "log_parameter_max_length": "1foo" > HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB". > postgres=# set log_parameter_max_length = '1TB'; > ERROR: invalid value for parameter "log_parameter_max_length": "1TB" > HINT: Value exceeds integer range. > > I think we can leave it as is. Hi > +for (my $attempts = 0; $attempts < $max_attempts; $attempts++) > +{ > + eval { > + $current_logfiles = slurp_file($node->data_dir . '/current_logfiles'); > + }; > + last unless $@; > + usleep(100_000); > +} `usleep` in tap tests is usually a bad pattern. Do we have a chance to test this using `wait_for_log` or similar? -- Best regards, Kirill Reshke ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size @ 2024-12-02 10:08 Jim Jones <[email protected]> parent: Kirill Gavrilov <[email protected]> 1 sibling, 1 reply; 6+ messages in thread From: Jim Jones @ 2024-12-02 10:08 UTC (permalink / raw) To: Kirill Gavrilov <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected] On 29.11.24 21:57, Kirill Gavrilov wrote: > Same thing applies to log_parameter_max_length, for example. > > postgres=# set log_parameter_max_length = '1foo'; > ERROR: invalid value for parameter "log_parameter_max_length": "1foo" > HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and "TB". > postgres=# set log_parameter_max_length = '1TB'; > ERROR: invalid value for parameter "log_parameter_max_length": "1TB" > HINT: Value exceeds integer range. > > I think we can leave it as is. I see. So I guess it is out of scope to change this message here. Small nitpicks: 1) The indentation of the comment at postgresql.conf.sample is a little bit off #max_log_size = 0 # max size of logged statement # 0 disables the feature IMHO it looks better like this: #max_log_size = 0 # max size of logged statement # 0 disables the feature 2) You introduced a trailing whitespace at L34 (Not critical :)) + Zero disables the setting. It happens to me all the time, so I usually try to apply my patches in a clean branch just to make sure I didn't miss anything. Other than that, I have nothing more to add at this point. Thanks -- Jim ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size @ 2025-01-31 10:46 Kirill Gavrilov <[email protected]> parent: Jim Jones <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: Kirill Gavrilov @ 2025-01-31 10:46 UTC (permalink / raw) To: Jim Jones <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected] On Mon, Dec 2, 2024 at 1:09 PM Jim Jones <[email protected]> wrote: > > > On 29.11.24 21:57, Kirill Gavrilov wrote: > > Same thing applies to log_parameter_max_length, for example. > > > > postgres=# set log_parameter_max_length = '1foo'; > > ERROR: invalid value for parameter "log_parameter_max_length": "1foo" > > HINT: Valid units for this parameter are "B", "kB", "MB", "GB", and > "TB". > > postgres=# set log_parameter_max_length = '1TB'; > > ERROR: invalid value for parameter "log_parameter_max_length": "1TB" > > HINT: Value exceeds integer range. > > > > I think we can leave it as is. > > I see. So I guess it is out of scope to change this message here. > > Small nitpicks: > > 1) The indentation of the comment at postgresql.conf.sample is a little > bit off > > #max_log_size = 0 # max size of logged statement > # 0 disables the feature > > IMHO it looks better like this: > > #max_log_size = 0 # max size of logged statement > # 0 disables the feature > > > 2) You introduced a trailing whitespace at L34 (Not critical :)) > > + Zero disables the setting. > > It happens to me all the time, so I usually try to apply my patches in a > clean branch just to make sure I didn't miss anything. > > Other than that, I have nothing more to add at this point. > > Thanks > > -- > Jim > > Sorry for the long silence. I fixed the indentation and a trailing whitespace. Should look fine now. Attachments: [application/octet-stream] V6-0001-parameter-max_log_size-to-truncate-logs.patch (9.0K, ../../CA+E0NR7XD3wpts2GEFpZpO-M8kj9f8pTrhnAKvq5n4xjtLgLKg@mail.gmail.com/3-V6-0001-parameter-max_log_size-to-truncate-logs.patch) download | inline diff: From a020529b6ac33393e9992e2118762f2f39762d9b Mon Sep 17 00:00:00 2001 From: diphantxm <[email protected]> Date: Fri, 31 Jan 2025 13:31:50 +0300 Subject: [PATCH] parameter max_log_size to truncate logs There is no need to log the entire query, because it may be large and take lots of space on disk. Parameter max_log_size set the maximum length for logged query. Everything beyond that length is truncated. Value 0 disables the parameter. --- doc/src/sgml/config.sgml | 16 +++++++ src/backend/tcop/postgres.c | 18 ++++++- src/backend/utils/error/elog.c | 47 +++++++++++++++++++ src/backend/utils/misc/guc_tables.c | 12 +++++ src/backend/utils/misc/postgresql.conf.sample | 2 + src/bin/pg_ctl/t/004_logrotate.pl | 15 ++++++ src/include/utils/elog.h | 4 ++ 7 files changed, 112 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 76ab72db96..86dcebfe36 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7913,6 +7913,22 @@ log_line_prefix = '%m [%p] %q%u@%d/%a ' </listitem> </varlistentry> + <varlistentry id="guc-max-log-size" xreflabel="max_log_size"> + <term><varname>max_log_size</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_log_size</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + If greater than zero, each query logged is truncated to this many bytes. + Zero disables the setting. + If this value is specified without units, it is taken as bytes. + This feature is disabled by default. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> <sect2 id="runtime-config-logging-csvlog"> diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 4b985bd056..033de5862a 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -70,6 +70,7 @@ #include "tcop/pquery.h" #include "tcop/tcopprot.h" #include "tcop/utility.h" +#include "utils/elog.h" #include "utils/guc_hooks.h" #include "utils/injection_point.h" #include "utils/lsyscache.h" @@ -1029,11 +1030,22 @@ exec_simple_query(const char *query_string) bool was_logged = false; bool use_implicit_block; char msec_str[32]; + char* truncated_query = NULL; + const char* query_log; /* * Report query to various monitoring facilities. */ debug_query_string = query_string; + if (need_truncate_query_log(query_string)) + { + truncated_query = truncate_query_log(query_string); + query_log = truncated_query; + } + else + { + query_log = query_string; + } pgstat_report_activity(STATE_RUNNING, query_string); @@ -1078,7 +1090,7 @@ exec_simple_query(const char *query_string) if (check_log_statement(parsetree_list)) { ereport(LOG, - (errmsg("statement: %s", query_string), + (errmsg("statement: %s", query_log), errhidestmt(true), errdetail_execute(parsetree_list))); was_logged = true; @@ -1377,7 +1389,7 @@ exec_simple_query(const char *query_string) case 2: ereport(LOG, (errmsg("duration: %s ms statement: %s", - msec_str, query_string), + msec_str, query_log), errhidestmt(true), errdetail_execute(parsetree_list))); break; @@ -1388,6 +1400,8 @@ exec_simple_query(const char *query_string) TRACE_POSTGRESQL_QUERY_DONE(query_string); + if (truncated_query) + pfree(truncated_query); debug_query_string = NULL; } diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c index 289059435a..8a1071677b 100644 --- a/src/backend/utils/error/elog.c +++ b/src/backend/utils/error/elog.c @@ -111,6 +111,7 @@ int Log_destination = LOG_DESTINATION_STDERR; char *Log_destination_string = NULL; bool syslog_sequence_numbers = true; bool syslog_split_messages = true; +int max_log_size = 0; /* Processed form of backtrace_functions GUC */ static char *backtrace_function_list; @@ -1688,11 +1689,18 @@ EmitErrorReport(void) { ErrorData *edata = &errordata[errordata_stack_depth]; MemoryContext oldcontext; + char* truncated_query = NULL; recursion_depth++; CHECK_STACK_DEPTH(); oldcontext = MemoryContextSwitchTo(edata->assoc_context); + if (need_truncate_query_log(debug_query_string)) + { + truncated_query = truncate_query_log(debug_query_string); + debug_query_string = truncated_query; + } + /* * Reset the formatted timestamp fields before emitting any logs. This * includes all the log destinations and emit_log_hook, as the latter @@ -1733,6 +1741,9 @@ EmitErrorReport(void) MemoryContextSwitchTo(oldcontext); recursion_depth--; + + if (truncated_query) + pfree(truncated_query); } /* @@ -3773,3 +3784,39 @@ write_stderr(const char *fmt,...) #endif va_end(ap); } + +/* + * Apply truncation to build query that will be logged. + * + * If query needs to be truncated, copied will be set to true + * and returned string must be freed +*/ +char* +truncate_query_log(const char* query) +{ + size_t truncated_query_len; + char* truncatd_query; + size_t query_len; + + if (!query) + return NULL; + + query_len = strlen(query); + truncated_query_len = pg_mbcliplen(query, query_len, max_log_size); + truncatd_query = (char *) palloc(truncated_query_len+1); + memcpy(truncatd_query, query, truncated_query_len); + truncatd_query[truncated_query_len] = '\0'; + return truncatd_query; +} + +/* + * Checks if query should be truncated + * according to max_log_size +*/ +bool +need_truncate_query_log(const char* query) +{ + if (!query) + return false; + return !(max_log_size == 0 || strlen(query) < max_log_size); +} diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 9845abd693..8ad18f7a47 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -3724,6 +3724,18 @@ struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_log_size", PGC_SUSET, LOGGING_WHAT, + gettext_noop("Sets max size in bytes of logged statement."), + NULL, + GUC_UNIT_BYTE + }, + &max_log_size, + 0, + 0, INT_MAX, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 407cd1e08c..80a45341df 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -617,6 +617,8 @@ # bind-parameter values to N bytes; # -1 means print in full, 0 disables #log_statement = 'none' # none, ddl, mod, all +#max_log_size = 0 # max size of logged statement + # 0 disables the feature #log_replication_commands = off #log_temp_files = -1 # log temporary files equal or larger # than the specified size in kilobytes; diff --git a/src/bin/pg_ctl/t/004_logrotate.pl b/src/bin/pg_ctl/t/004_logrotate.pl index eacca1a652..1c1f89cb0a 100644 --- a/src/bin/pg_ctl/t/004_logrotate.pl +++ b/src/bin/pg_ctl/t/004_logrotate.pl @@ -69,6 +69,7 @@ log_destination = 'stderr, csvlog, jsonlog' # these ensure stability of test results: log_rotation_age = 0 lc_messages = 'C' +max_log_size = 32 )); $node->start(); @@ -135,6 +136,20 @@ check_log_pattern('stderr', $new_current_logfiles, 'syntax error', $node); check_log_pattern('csvlog', $new_current_logfiles, 'syntax error', $node); check_log_pattern('jsonlog', $new_current_logfiles, 'syntax error', $node); +$node->psql('postgres', 'INSERT INTO SOME_NON_EXISTANT_TABLE VALUES (TEST)'); +for (my $attempts = 0; $attempts < $max_attempts; $attempts++) +{ + eval { + $current_logfiles = slurp_file($node->data_dir . '/current_logfiles'); + }; + last unless $@; + usleep(100_000); +} +die $@ if $@; +check_log_pattern('stderr', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node); +check_log_pattern('csvlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node); +check_log_pattern('jsonlog', $current_logfiles, 'INSERT INTO SOME_NON_EXISTANT_TA(?!(BLE VALUES \(TEST\)))', $node); + $node->stop(); done_testing(); diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h index e54eca5b48..cafa8de3c0 100644 --- a/src/include/utils/elog.h +++ b/src/include/utils/elog.h @@ -502,6 +502,7 @@ extern PGDLLIMPORT int Log_destination; extern PGDLLIMPORT char *Log_destination_string; extern PGDLLIMPORT bool syslog_sequence_numbers; extern PGDLLIMPORT bool syslog_split_messages; +extern PGDLLIMPORT int max_log_size; /* Log destination bitmap */ #define LOG_DESTINATION_STDERR 1 @@ -517,6 +518,9 @@ extern void DebugFileOpen(void); extern char *unpack_sql_state(int sql_state); extern bool in_error_recursion_trouble(void); +extern bool need_truncate_query_log(const char* query); +extern char* truncate_query_log(const char* query); + /* Common functions shared across destinations */ extern void reset_formatted_start_time(void); extern char *get_formatted_start_time(void); -- 2.39.2 (Apple Git-143) ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Truncate logs by max_log_size @ 2025-02-03 10:31 Jim Jones <[email protected]> parent: Kirill Gavrilov <[email protected]> 0 siblings, 0 replies; 6+ messages in thread From: Jim Jones @ 2025-02-03 10:31 UTC (permalink / raw) To: Kirill Gavrilov <[email protected]>; +Cc: Kirill Reshke <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected] Hi Kirill On 31.01.25 11:46, Kirill Gavrilov wrote: > Sorry for the long silence. I fixed the indentation and a trailing > whitespace. Should look fine now. The patch applies cleanly, the documentation is clear, and all tests pass. It is possible to change this new parameter session-wise, which is nice! postgres=# SET max_log_size TO 7; SET postgres=# SHOW max_log_size; max_log_size -------------- 7B (1 row) The default value now is clear and it corresponds to the value set on postgresql.conf: #max_log_size = 0 # max size of logged statement postgres=# SHOW max_log_size; max_log_size -------------- 0 (1 row) Logs are truncated as expected: postgres=# SET max_log_size TO 6; SET postgres=# SELECT length('CALL xyz;'); length -------- 9 (1 row) postgres=# CALL xyz; ERROR: syntax error at or near ";" LINE 1: CALL xyz; log entry: 2025-02-03 10:58:19.975 CET [123945] ERROR: syntax error at or near ";" at character 9 2025-02-03 10:58:19.975 CET [123945] STATEMENT: CALL x The issue with log entry sizes for queries containing special characters was resolved by setting the unit to bytes. Overall, everythingLGTM. The new status of this patch is: Ready for Committer Jim ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2025-02-03 10:31 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-01-10 04:38 [PATCH 5/9] doc: speculative: Perform apply of large transactions by parallel workers. Justin Pryzby <[email protected]> 2024-11-29 20:57 Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]> 2024-11-30 06:58 ` Re: Truncate logs by max_log_size Kirill Reshke <[email protected]> 2024-12-02 10:08 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]> 2025-01-31 10:46 ` Re: Truncate logs by max_log_size Kirill Gavrilov <[email protected]> 2025-02-03 10:31 ` Re: Truncate logs by max_log_size Jim Jones <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox