public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v2 1/5] pg_rewind: Move syncTargetDirectory() to file_ops.c 2+ messages / 2 participants [nested] [flat]
* [PATCH v2 1/5] pg_rewind: Move syncTargetDirectory() to file_ops.c @ 2020-08-19 12:34 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 2+ messages in thread From: Heikki Linnakangas @ 2020-08-19 12:34 UTC (permalink / raw) For consistency. All the other low-level functions that operate on the target directory are in file_ops.c. --- src/bin/pg_rewind/file_ops.c | 19 +++++++++++++++++++ src/bin/pg_rewind/file_ops.h | 1 + src/bin/pg_rewind/pg_rewind.c | 22 +--------------------- src/bin/pg_rewind/pg_rewind.h | 1 + 4 files changed, 22 insertions(+), 21 deletions(-) diff --git a/src/bin/pg_rewind/file_ops.c b/src/bin/pg_rewind/file_ops.c index b3bf091c546..55439db20ba 100644 --- a/src/bin/pg_rewind/file_ops.c +++ b/src/bin/pg_rewind/file_ops.c @@ -19,6 +19,7 @@ #include <unistd.h> #include "common/file_perm.h" +#include "common/file_utils.h" #include "file_ops.h" #include "filemap.h" #include "pg_rewind.h" @@ -266,6 +267,24 @@ remove_target_symlink(const char *path) dstpath); } +/* + * Sync target data directory to ensure that modifications are safely on disk. + * + * We do this once, for the whole data directory, for performance reasons. At + * the end of pg_rewind's run, the kernel is likely to already have flushed + * most dirty buffers to disk. Additionally fsync_pgdata uses a two-pass + * approach (only initiating writeback in the first pass), which often reduces + * the overall amount of IO noticeably. + */ +void +sync_target_dir(void) +{ + if (!do_sync || dry_run) + return; + + fsync_pgdata(datadir_target, PG_VERSION_NUM); +} + /* * Read a file into memory. The file to be read is <datadir>/<path>. diff --git a/src/bin/pg_rewind/file_ops.h b/src/bin/pg_rewind/file_ops.h index 025f24141c9..d8466385cf5 100644 --- a/src/bin/pg_rewind/file_ops.h +++ b/src/bin/pg_rewind/file_ops.h @@ -19,6 +19,7 @@ extern void remove_target_file(const char *path, bool missing_ok); extern void truncate_target_file(const char *path, off_t newsize); extern void create_target(file_entry_t *t); extern void remove_target(file_entry_t *t); +extern void sync_target_dir(void); extern char *slurpFile(const char *datadir, const char *path, size_t *filesize); diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 23fc749e445..c9b9e480c0f 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -20,7 +20,6 @@ #include "catalog/pg_control.h" #include "common/controldata_utils.h" #include "common/file_perm.h" -#include "common/file_utils.h" #include "common/restricted_token.h" #include "common/string.h" #include "fe_utils/recovery_gen.h" @@ -38,7 +37,6 @@ static void createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, static void digestControlFile(ControlFileData *ControlFile, char *source, size_t size); -static void syncTargetDirectory(void); static void getRestoreCommand(const char *argv0); static void sanityChecks(void); static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex); @@ -455,7 +453,7 @@ main(int argc, char **argv) if (showprogress) pg_log_info("syncing target data directory"); - syncTargetDirectory(); + sync_target_dir(); if (writerecoveryconf && !dry_run) WriteRecoveryConfig(conn, datadir_target, @@ -803,24 +801,6 @@ digestControlFile(ControlFileData *ControlFile, char *src, size_t size) checkControlFile(ControlFile); } -/* - * Sync target data directory to ensure that modifications are safely on disk. - * - * We do this once, for the whole data directory, for performance reasons. At - * the end of pg_rewind's run, the kernel is likely to already have flushed - * most dirty buffers to disk. Additionally fsync_pgdata uses a two-pass - * approach (only initiating writeback in the first pass), which often reduces - * the overall amount of IO noticeably. - */ -static void -syncTargetDirectory(void) -{ - if (!do_sync || dry_run) - return; - - fsync_pgdata(datadir_target, PG_VERSION_NUM); -} - /* * Get value of GUC parameter restore_command from the target cluster. * diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h index 8a9319ed675..67f90c2a38c 100644 --- a/src/bin/pg_rewind/pg_rewind.h +++ b/src/bin/pg_rewind/pg_rewind.h @@ -24,6 +24,7 @@ extern char *datadir_source; extern char *connstr_source; extern bool showprogress; extern bool dry_run; +extern bool do_sync; extern int WalSegSz; /* Target history */ -- 2.20.1 --------------BF34D0120055DC3839060F92 Content-Type: text/x-patch; charset=UTF-8; name="v2-0002-Refactor-pg_rewind-for-more-clear-decision-making.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="v2-0002-Refactor-pg_rewind-for-more-clear-decision-making.pa"; filename*1="tch" ^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Truncate logs by max_log_size @ 2024-11-25 16:52 Kirill Reshke <[email protected]> 0 siblings, 0 replies; 2+ messages in thread From: Kirill Reshke @ 2024-11-25 16:52 UTC (permalink / raw) To: Kirill Gavrilov <[email protected]>; +Cc: Jim Jones <[email protected]>; Andrey M. Borodin <[email protected]>; Euler Taveira <[email protected]>; [email protected] On Tue, 1 Oct 2024 at 20:46, Kirill Gavrilov <[email protected]> wrote: > > On Tue, Oct 1, 2024 at 2:46 PM Jim Jones <[email protected]> wrote: >> >> >> On 27.09.24 12:36, Andrey M. Borodin wrote: >> > Consider max_log_size = 10Mb. The perspective might look very different. >> > It’s not about WHERE anymore. It's a guard against heavy abuse. >> > >> > The feature looks very important for me. >> I have the same opinion. As a fail safe it sounds very useful to me. >> >> Unfortunately, the patch does not apply: >> >> $ git apply >> ~/patches/max_log_query/0001-parameter-max_log_size-to-truncate-logs.patch >> -v >> Checking patch src/backend/utils/error/elog.c... >> error: while searching for: >> char *Log_destination_string = NULL; >> bool syslog_sequence_numbers = true; >> bool syslog_split_messages = true; >> >> /* Processed form of backtrace_symbols GUC */ >> static char *backtrace_symbol_list; >> >> error: patch failed: src/backend/utils/error/elog.c:114 >> error: src/backend/utils/error/elog.c: patch does not apply >> Checking patch src/backend/utils/misc/guc_tables.c... >> Hunk #1 succeeded at 3714 (offset 247 lines). >> Checking patch src/backend/utils/misc/postgresql.conf.sample... >> Hunk #1 succeeded at 615 (offset 23 lines). >> Checking patch src/bin/pg_ctl/t/004_logrotate.pl... >> error: while searching for: >> check_log_pattern('csvlog', $new_current_logfiles, 'syntax error', $node); >> check_log_pattern('jsonlog', $new_current_logfiles, 'syntax error', $node); >> >> $node->stop(); >> >> done_testing(); >> >> error: patch failed: src/bin/pg_ctl/t/004_logrotate.pl:135 >> error: src/bin/pg_ctl/t/004_logrotate.pl: patch does not apply >> Checking patch src/include/utils/elog.h... >> Hunk #1 succeeded at 502 (offset 1 line). >> >> Does it truncate only single queries or also a transaction with many >> statements? >> >> Thanks! >> >> Best, Jim > > > My apologies, attached patch should work on master branch. > It truncates single queries and a statement in transaction that occurred an error. > Hello! Please deliver the most recent patch version and fix the issues Jim identified [0] as the current commitfest draws to a close. Do not forget to include both parts of this patch (This was actually developed off-list, and we are now using this on our cloud PostgreSQL distribution on versions 12–17). [0] https://www.postgresql.org/message-id/35096a36-04d4-480b-a7cd-a2d8151fb737%40uni-muenster.de -- Best regards, Kirill Reshke ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2024-11-25 16:52 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-08-19 12:34 [PATCH v2 1/5] pg_rewind: Move syncTargetDirectory() to file_ops.c Heikki Linnakangas <[email protected]> 2024-11-25 16:52 Re: Truncate logs by max_log_size Kirill Reshke <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox