public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v16 01/10] Document historic behavior of links to directories.. 8+ messages / 5 participants [nested] [flat]
* [PATCH v16 01/10] Document historic behavior of links to directories.. @ 2020-03-16 19:12 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 8+ messages in thread From: Justin Pryzby @ 2020-03-16 19:12 UTC (permalink / raw) Backpatch to 9.5: pg_stat_file --- doc/src/sgml/func.sgml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 96ea57eedd..9b885102da 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -25486,7 +25486,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); size, last accessed time stamp, last modified time stamp, last file status change time stamp (Unix platforms only), file creation time stamp (Windows only), and a <type>boolean</type> - indicating if it is a directory. Typical usages include: + indicating if it is a directory (or a symbolic link to a directory). + Typical usages include: <programlisting> SELECT * FROM pg_stat_file('filename'); SELECT (pg_stat_file('filename')).modification; -- 2.17.0 --2FkSFaIQeDFoAt0B Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v16-0002-pg_stat_file-and-pg_ls_dir_-to-use-lstat.patch" ^ permalink raw reply [nested|flat] 8+ messages in thread
* avoid multiple hard links to same WAL file after a crash @ 2022-04-07 18:29 Nathan Bossart <[email protected]> 0 siblings, 2 replies; 8+ messages in thread From: Nathan Bossart @ 2022-04-07 18:29 UTC (permalink / raw) To: pgsql-hackers Hi hackers, I am splitting this off of a previous thread aimed at reducing archiving overhead [0], as I believe this fix might deserve back-patching. Presently, WAL recycling uses durable_rename_excl(), which notes that a crash at an unfortunate moment can result in two links to the same file. My testing [1] demonstrated that it was possible to end up with two links to the same file in pg_wal after a crash just before unlink() during WAL recycling. Specifically, the test produced links to the same file for the current WAL file and the next one because the half-recycled WAL file was re-recycled upon restarting. This seems likely to lead to WAL corruption. The attached patch prevents this problem by using durable_rename() instead of durable_rename_excl() for WAL recycling. This removes the protection against accidentally overwriting an existing WAL file, but there shouldn't be one. This patch also sets the stage for reducing archiving overhead (as discussed in the other thread [0]). The proposed change to reduce archiving overhead will make it more likely that the server will attempt to re-archive segments after a crash. This might lead to archive corruption if the server concurrently writes to the same file via the aforementioned bug. [0] https://www.postgresql.org/message-id/20220222011948.GA3850532%40nathanxps13 [1] https://www.postgresql.org/message-id/20220222173711.GA3852671%40nathanxps13 -- Nathan Bossart Amazon Web Services: https://aws.amazon.com Attachments: [text/x-diff] v1-0001-Avoid-multiple-hard-links-to-same-WAL-file-after-.patch (2.1K, ../../20220407182954.GA1231544@nathanxps13/2-v1-0001-Avoid-multiple-hard-links-to-same-WAL-file-after-.patch) download | inline diff: From 244726f6a78aca52c2fe6e70cef966f152057191 Mon Sep 17 00:00:00 2001 From: Nathan Bossart <[email protected]> Date: Thu, 7 Apr 2022 10:07:42 -0700 Subject: [PATCH v1 1/1] Avoid multiple hard links to same WAL file after a crash. Presently, WAL recycling uses durable_rename_excl(), which notes that a crash at an unfortunate moment can result in two links to the same file. My testing demonstrated that it was possible to end up with two links to the same file in pg_wal after a crash just before unlink() during WAL recycling. Specifically, the test produced links to the same file for the current WAL file and the next one because the half-recycled WAL file was re-recycled upon restarting. This seems likely to lead to WAL corruption. This change prevents this problem by using durable_rename() instead of durable_rename_excl() for WAL recycling. This removes the protection against accidentally overwriting an existing WAL file, but there shouldn't be one. Back-patch to all supported versions. Author: Nathan Bossart --- src/backend/access/transam/xlog.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 6770c3ddba..6ab5b2a622 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -3324,13 +3324,15 @@ InstallXLogFileSegment(XLogSegNo *segno, char *tmppath, } /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). + * Perform the rename. Ideally, we'd use link() and unlink() to avoid + * overwriting an existing file (there shouldn't be one). However, that + * approach opens up the possibility that pg_wal will contain multiple hard + * links to the same WAL file after a crash. */ - if (durable_rename_excl(tmppath, path, LOG) != 0) + if (durable_rename(tmppath, path, LOG) != 0) { LWLockRelease(ControlFileLock); - /* durable_rename_excl already emitted log message */ + /* durable_rename already emitted log message */ return false; } -- 2.25.1 ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: avoid multiple hard links to same WAL file after a crash @ 2022-04-08 14:38 Robert Haas <[email protected]> parent: Nathan Bossart <[email protected]> 1 sibling, 1 reply; 8+ messages in thread From: Robert Haas @ 2022-04-08 14:38 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers On Thu, Apr 7, 2022 at 2:30 PM Nathan Bossart <[email protected]> wrote: > Presently, WAL recycling uses durable_rename_excl(), which notes that a > crash at an unfortunate moment can result in two links to the same file. > My testing [1] demonstrated that it was possible to end up with two links > to the same file in pg_wal after a crash just before unlink() during WAL > recycling. Specifically, the test produced links to the same file for the > current WAL file and the next one because the half-recycled WAL file was > re-recycled upon restarting. This seems likely to lead to WAL corruption. Wow, that's bad. > The attached patch prevents this problem by using durable_rename() instead > of durable_rename_excl() for WAL recycling. This removes the protection > against accidentally overwriting an existing WAL file, but there shouldn't > be one. I see that durable_rename_excl() has the following comment: "Similar to durable_rename(), except that this routine tries (but does not guarantee) not to overwrite the target file." If those are the desired semantics, we could achieve them more simply and more safely by just trying to stat() the target file and then, if it's not found, call durable_rename(). I think that would be a heck of a lot safer than what this function is doing right now. I'd actually be in favor of nuking durable_rename_excl() from orbit and putting the file-exists tests in the callers. Otherwise, someone might assume that it actually has the semantics that its name suggests, which could be pretty disastrous. If we don't want to do that, then I'd changing to do the stat-then-durable-rename thing internally, so we don't leave hard links lying around in *any* code path. Perhaps that's the right answer for the back-branches in any case, since there could be third-party code calling this function. Your proposed fix is OK if we don't want to do any of that stuff, but personally I'm much more inclined to blame durable_rename_excl() for being horrible than I am to blame the calling code for using it improvidently. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: avoid multiple hard links to same WAL file after a crash @ 2022-04-08 19:43 Nathan Bossart <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2022-04-08 19:43 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: pgsql-hackers On Fri, Apr 08, 2022 at 10:38:03AM -0400, Robert Haas wrote: > I'd actually be in favor of nuking durable_rename_excl() from orbit > and putting the file-exists tests in the callers. Otherwise, someone > might assume that it actually has the semantics that its name > suggests, which could be pretty disastrous. If we don't want to do > that, then I'd changing to do the stat-then-durable-rename thing > internally, so we don't leave hard links lying around in *any* code > path. Perhaps that's the right answer for the back-branches in any > case, since there could be third-party code calling this function. I've attached a patch that simply removes durable_rename_excl() and replaces existing calls with durable_rename(). I noticed that Andres expressed similar misgivings about durable_rename_excl() last year [0] [1]. I can create a stat-then-durable-rename version of this for back-patching if that is still the route we want to go. [0] https://postgr.es/me/20210318014812.ds2iz4jz5h7la6un%40alap3.anarazel.de [1] https://postgr.es/m/20210318023004.gz2aejhze2kkkqr2%40alap3.anarazel.de -- Nathan Bossart Amazon Web Services: https://aws.amazon.com Attachments: [text/x-diff] v2-0001-Remove-durable_rename_excl.patch (7.7K, ../../20220408194345.GA1541826@nathanxps13/2-v2-0001-Remove-durable_rename_excl.patch) download | inline diff: From d3c633e19555dc0cf98207ad5e7c08ab9ce85dc0 Mon Sep 17 00:00:00 2001 From: Nathan Bossart <[email protected]> Date: Fri, 8 Apr 2022 11:48:17 -0700 Subject: [PATCH v2 1/1] Remove durable_rename_excl(). durable_rename_excl() attempts to avoid overwriting any existing files by using link() and unlink(), but it falls back to rename() on some platforms (e.g., Windows), which offers no such ovewrite protection. Most callers use durable_rename_excl() just in case there is an existing file, but in practice there shouldn't be one. basic_archive uses it to avoid overwriting an archive concurrently created by another server, but as mentioned above, it will still overwrite files on some platforms. Furthermore, failures during durable_rename_excl() can result in multiple hard links to the same file. My testing demonstrated that it was possible to end up with two links to the same file in pg_wal after a crash just before unlink() during WAL recycling. Specifically, the test produced links to the same file for the current WAL file and the next one because the half-recycled WAL file was re-recycled upon restarting. This seems likely to lead to WAL corruption. This change removes durable_rename_excl() and replaces all existing calls with durable_rename(). This removes the protection against accidentally overwriting an existing file, but some platforms are already living without it, and ordinarily there shouldn't be one. Author: Nathan Bossart Reviewed-by: Robert Haas Discussion: https://postgr.es/m/20220407182954.GA1231544%40nathanxps13 --- contrib/basic_archive/basic_archive.c | 5 ++- src/backend/access/transam/timeline.c | 14 +----- src/backend/access/transam/xlog.c | 8 +--- src/backend/storage/file/fd.c | 63 --------------------------- src/include/pg_config_manual.h | 7 --- src/include/storage/fd.h | 1 - 6 files changed, 7 insertions(+), 91 deletions(-) diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c index e7efbfb9c3..ed33854c57 100644 --- a/contrib/basic_archive/basic_archive.c +++ b/contrib/basic_archive/basic_archive.c @@ -281,9 +281,10 @@ basic_archive_file_internal(const char *file, const char *path) /* * Sync the temporary file to disk and move it to its final destination. - * This will fail if destination already exists. + * Note that this will overwrite any existing file, but this is only + * possible if someone else created the file since the stat() above. */ - (void) durable_rename_excl(temp, destination, ERROR); + (void) durable_rename(temp, destination, ERROR); ereport(DEBUG1, (errmsg("archived \"%s\" via basic_archive", file))); diff --git a/src/backend/access/transam/timeline.c b/src/backend/access/transam/timeline.c index be21968293..128f754e87 100644 --- a/src/backend/access/transam/timeline.c +++ b/src/backend/access/transam/timeline.c @@ -441,12 +441,7 @@ writeTimeLineHistory(TimeLineID newTLI, TimeLineID parentTLI, * Now move the completed history file into place with its final name. */ TLHistoryFilePath(path, newTLI); - - /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). - */ - durable_rename_excl(tmppath, path, ERROR); + durable_rename(tmppath, path, ERROR); /* The history file can be archived immediately. */ if (XLogArchivingActive()) @@ -519,12 +514,7 @@ writeTimeLineHistoryFile(TimeLineID tli, char *content, int size) * Now move the completed history file into place with its final name. */ TLHistoryFilePath(path, tli); - - /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). - */ - durable_rename_excl(tmppath, path, ERROR); + durable_rename(tmppath, path, ERROR); } /* diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index a7814d4019..d19215ab24 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -3323,14 +3323,10 @@ InstallXLogFileSegment(XLogSegNo *segno, char *tmppath, } } - /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). - */ - if (durable_rename_excl(tmppath, path, LOG) != 0) + if (durable_rename(tmppath, path, LOG) != 0) { LWLockRelease(ControlFileLock); - /* durable_rename_excl already emitted log message */ + /* durable_rename already emitted log message */ return false; } diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 14b77f2861..88645ed83d 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -807,69 +807,6 @@ durable_unlink(const char *fname, int elevel) return 0; } -/* - * durable_rename_excl -- rename a file in a durable manner. - * - * Similar to durable_rename(), except that this routine tries (but does not - * guarantee) not to overwrite the target file. - * - * Note that a crash in an unfortunate moment can leave you with two links to - * the target file. - * - * Log errors with the caller specified severity. - * - * On Windows, using a hard link followed by unlink() causes concurrency - * issues, while a simple rename() does not cause that, so be careful when - * changing the logic of this routine. - * - * Returns 0 if the operation succeeded, -1 otherwise. Note that errno is not - * valid upon return. - */ -int -durable_rename_excl(const char *oldfile, const char *newfile, int elevel) -{ - /* - * Ensure that, if we crash directly after the rename/link, a file with - * valid contents is moved into place. - */ - if (fsync_fname_ext(oldfile, false, false, elevel) != 0) - return -1; - -#ifdef HAVE_WORKING_LINK - if (link(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not link file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } - unlink(oldfile); -#else - if (rename(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } -#endif - - /* - * Make change persistent in case of an OS crash, both the new entry and - * its parent directory need to be flushed. - */ - if (fsync_fname_ext(newfile, false, false, elevel) != 0) - return -1; - - /* Same for parent directory */ - if (fsync_parent_path(newfile, elevel) != 0) - return -1; - - return 0; -} - /* * InitFileAccess --- initialize this module during backend startup * diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 84ce5a4a5d..830804fdfb 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -163,13 +163,6 @@ #define USE_BARRIER_SMGRRELEASE #endif -/* - * Define this if your operating system supports link() - */ -#if !defined(WIN32) && !defined(__CYGWIN__) -#define HAVE_WORKING_LINK 1 -#endif - /* * USE_POSIX_FADVISE controls whether Postgres will attempt to use the * posix_fadvise() kernel call. Usually the automatic configure tests are diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index 69549b000f..2b4a8e0ffe 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -187,7 +187,6 @@ extern void fsync_fname(const char *fname, bool isdir); extern int fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel); extern int durable_rename(const char *oldfile, const char *newfile, int loglevel); extern int durable_unlink(const char *fname, int loglevel); -extern int durable_rename_excl(const char *oldfile, const char *newfile, int loglevel); extern void SyncDataDirectory(void); extern int data_sync_elevel(int elevel); -- 2.25.1 ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: avoid multiple hard links to same WAL file after a crash @ 2022-04-11 09:12 Kyotaro Horiguchi <[email protected]> parent: Nathan Bossart <[email protected]> 1 sibling, 1 reply; 8+ messages in thread From: Kyotaro Horiguchi @ 2022-04-11 09:12 UTC (permalink / raw) To: [email protected]; +Cc: pgsql-hackers At Thu, 7 Apr 2022 11:29:54 -0700, Nathan Bossart <[email protected]> wrote in > The attached patch prevents this problem by using durable_rename() instead > of durable_rename_excl() for WAL recycling. This removes the protection > against accidentally overwriting an existing WAL file, but there shouldn't > be one. From another direction, if the new segment was the currently active one, we just mustn't install it. Otherwise we don't care. So, the only thing we need to care is segment switch. Without it, the segment that InstallXLogFileSegment found by the stat loop is known to be safe to overwrite even if exists. When segment switch finds an existing file, it's no problem since the segment switch doesn't create a new segment. Otherwise segment switch always calls InstallXLogFileSegment. The section from searching for an empty segmetn slot until calling durable_rename_excl() is protected by ControlFileLock. Thus if a process is in the section, no other process can switch to a newly-created segment. If this diagnosis is correct, the comment is proved to be paranoid. > * Perform the rename using link if available, paranoidly trying to avoid > * overwriting an existing file (there shouldn't be one). As the result, I think Nathan's fix is correct that we can safely use durable_rename() instead. And I propose to use renameat2 on Linux so that we can detect the contradicting case by the regression tests even though only on Linux. regards. -- Kyotaro Horiguchi NTT Open Source Software Center ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: avoid multiple hard links to same WAL file after a crash @ 2022-04-11 16:15 Robert Haas <[email protected]> parent: Kyotaro Horiguchi <[email protected]> 0 siblings, 1 reply; 8+ messages in thread From: Robert Haas @ 2022-04-11 16:15 UTC (permalink / raw) To: Kyotaro Horiguchi <[email protected]>; +Cc: Nathan Bossart <[email protected]>; pgsql-hackers On Mon, Apr 11, 2022 at 5:12 AM Kyotaro Horiguchi <[email protected]> wrote: > So, the only thing we need to care is segment switch. Without it, the > segment that InstallXLogFileSegment found by the stat loop is known to > be safe to overwrite even if exists. > > When segment switch finds an existing file, it's no problem since the > segment switch doesn't create a new segment. Otherwise segment switch > always calls InstallXLogFileSegment. The section from searching for > an empty segmetn slot until calling durable_rename_excl() is protected > by ControlFileLock. Thus if a process is in the section, no other > process can switch to a newly-created segment. > > If this diagnosis is correct, the comment is proved to be paranoid. It's sometimes difficult to understand what problems really old code comments are worrying about. For example, could they have been worrying about bugs in the code? Could they have been worrying about manual interference with the pg_wal directory? It's hard to know. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: avoid multiple hard links to same WAL file after a crash @ 2022-04-11 16:28 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 8+ messages in thread From: Tom Lane @ 2022-04-11 16:28 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Nathan Bossart <[email protected]>; pgsql-hackers Robert Haas <[email protected]> writes: > On Mon, Apr 11, 2022 at 5:12 AM Kyotaro Horiguchi > <[email protected]> wrote: >> If this diagnosis is correct, the comment is proved to be paranoid. > It's sometimes difficult to understand what problems really old code > comments are worrying about. For example, could they have been > worrying about bugs in the code? Could they have been worrying about > manual interference with the pg_wal directory? It's hard to know. "git blame" can be helpful here, if you trace back to when the comment was written and then try to find the associated mailing-list discussion. (That leap can be difficult for commits pre-dating our current convention of including links in the commit message, but it's usually not *that* hard to locate contemporaneous discussion.) regards, tom lane ^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: avoid multiple hard links to same WAL file after a crash @ 2022-04-11 16:52 Nathan Bossart <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2022-04-11 16:52 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Kyotaro Horiguchi <[email protected]>; pgsql-hackers On Mon, Apr 11, 2022 at 12:28:47PM -0400, Tom Lane wrote: > Robert Haas <[email protected]> writes: >> On Mon, Apr 11, 2022 at 5:12 AM Kyotaro Horiguchi >> <[email protected]> wrote: >>> If this diagnosis is correct, the comment is proved to be paranoid. > >> It's sometimes difficult to understand what problems really old code >> comments are worrying about. For example, could they have been >> worrying about bugs in the code? Could they have been worrying about >> manual interference with the pg_wal directory? It's hard to know. > > "git blame" can be helpful here, if you trace back to when the comment > was written and then try to find the associated mailing-list discussion. > (That leap can be difficult for commits pre-dating our current > convention of including links in the commit message, but it's usually > not *that* hard to locate contemporaneous discussion.) I traced this back a while ago. I believe the link() was first added in November 2000 as part of f0e37a8. This even predates WAL recycling, which was added in July 2001 as part of 7d4d5c0. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2022-04-11 16:52 UTC | newest] Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-03-16 19:12 [PATCH v16 01/10] Document historic behavior of links to directories.. Justin Pryzby <[email protected]> 2022-04-07 18:29 avoid multiple hard links to same WAL file after a crash Nathan Bossart <[email protected]> 2022-04-08 14:38 ` Re: avoid multiple hard links to same WAL file after a crash Robert Haas <[email protected]> 2022-04-08 19:43 ` Re: avoid multiple hard links to same WAL file after a crash Nathan Bossart <[email protected]> 2022-04-11 09:12 ` Re: avoid multiple hard links to same WAL file after a crash Kyotaro Horiguchi <[email protected]> 2022-04-11 16:15 ` Re: avoid multiple hard links to same WAL file after a crash Robert Haas <[email protected]> 2022-04-11 16:28 ` Re: avoid multiple hard links to same WAL file after a crash Tom Lane <[email protected]> 2022-04-11 16:52 ` Re: avoid multiple hard links to same WAL file after a crash Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox