agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH v2 1/1] Remove durable_rename_excl(). 6+ messages / 2 participants [nested] [flat]
* [PATCH v2 1/1] Remove durable_rename_excl(). @ 2022-04-08 18:48 Nathan Bossart <nathandbossart@gmail.com> 0 siblings, 0 replies; 6+ messages in thread From: Nathan Bossart @ 2022-04-08 18:48 UTC (permalink / raw) durable_rename_excl() attempts to avoid overwriting any existing files by using link() and unlink(), but it falls back to rename() on some platforms (e.g., Windows), which offers no such ovewrite protection. Most callers use durable_rename_excl() just in case there is an existing file, but in practice there shouldn't be one. basic_archive uses it to avoid overwriting an archive concurrently created by another server, but as mentioned above, it will still overwrite files on some platforms. Furthermore, failures during durable_rename_excl() can result in multiple hard links to the same file. My testing demonstrated that it was possible to end up with two links to the same file in pg_wal after a crash just before unlink() during WAL recycling. Specifically, the test produced links to the same file for the current WAL file and the next one because the half-recycled WAL file was re-recycled upon restarting. This seems likely to lead to WAL corruption. This change removes durable_rename_excl() and replaces all existing calls with durable_rename(). This removes the protection against accidentally overwriting an existing file, but some platforms are already living without it, and ordinarily there shouldn't be one. Author: Nathan Bossart Reviewed-by: Robert Haas Discussion: https://postgr.es/m/20220407182954.GA1231544%40nathanxps13 --- contrib/basic_archive/basic_archive.c | 5 ++- src/backend/access/transam/timeline.c | 14 +----- src/backend/access/transam/xlog.c | 8 +--- src/backend/storage/file/fd.c | 63 --------------------------- src/include/pg_config_manual.h | 7 --- src/include/storage/fd.h | 1 - 6 files changed, 7 insertions(+), 91 deletions(-) diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c index e7efbfb9c3..ed33854c57 100644 --- a/contrib/basic_archive/basic_archive.c +++ b/contrib/basic_archive/basic_archive.c @@ -281,9 +281,10 @@ basic_archive_file_internal(const char *file, const char *path) /* * Sync the temporary file to disk and move it to its final destination. - * This will fail if destination already exists. + * Note that this will overwrite any existing file, but this is only + * possible if someone else created the file since the stat() above. */ - (void) durable_rename_excl(temp, destination, ERROR); + (void) durable_rename(temp, destination, ERROR); ereport(DEBUG1, (errmsg("archived \"%s\" via basic_archive", file))); diff --git a/src/backend/access/transam/timeline.c b/src/backend/access/transam/timeline.c index be21968293..128f754e87 100644 --- a/src/backend/access/transam/timeline.c +++ b/src/backend/access/transam/timeline.c @@ -441,12 +441,7 @@ writeTimeLineHistory(TimeLineID newTLI, TimeLineID parentTLI, * Now move the completed history file into place with its final name. */ TLHistoryFilePath(path, newTLI); - - /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). - */ - durable_rename_excl(tmppath, path, ERROR); + durable_rename(tmppath, path, ERROR); /* The history file can be archived immediately. */ if (XLogArchivingActive()) @@ -519,12 +514,7 @@ writeTimeLineHistoryFile(TimeLineID tli, char *content, int size) * Now move the completed history file into place with its final name. */ TLHistoryFilePath(path, tli); - - /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). - */ - durable_rename_excl(tmppath, path, ERROR); + durable_rename(tmppath, path, ERROR); } /* diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index a7814d4019..d19215ab24 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -3323,14 +3323,10 @@ InstallXLogFileSegment(XLogSegNo *segno, char *tmppath, } } - /* - * Perform the rename using link if available, paranoidly trying to avoid - * overwriting an existing file (there shouldn't be one). - */ - if (durable_rename_excl(tmppath, path, LOG) != 0) + if (durable_rename(tmppath, path, LOG) != 0) { LWLockRelease(ControlFileLock); - /* durable_rename_excl already emitted log message */ + /* durable_rename already emitted log message */ return false; } diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 14b77f2861..88645ed83d 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -807,69 +807,6 @@ durable_unlink(const char *fname, int elevel) return 0; } -/* - * durable_rename_excl -- rename a file in a durable manner. - * - * Similar to durable_rename(), except that this routine tries (but does not - * guarantee) not to overwrite the target file. - * - * Note that a crash in an unfortunate moment can leave you with two links to - * the target file. - * - * Log errors with the caller specified severity. - * - * On Windows, using a hard link followed by unlink() causes concurrency - * issues, while a simple rename() does not cause that, so be careful when - * changing the logic of this routine. - * - * Returns 0 if the operation succeeded, -1 otherwise. Note that errno is not - * valid upon return. - */ -int -durable_rename_excl(const char *oldfile, const char *newfile, int elevel) -{ - /* - * Ensure that, if we crash directly after the rename/link, a file with - * valid contents is moved into place. - */ - if (fsync_fname_ext(oldfile, false, false, elevel) != 0) - return -1; - -#ifdef HAVE_WORKING_LINK - if (link(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not link file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } - unlink(oldfile); -#else - if (rename(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } -#endif - - /* - * Make change persistent in case of an OS crash, both the new entry and - * its parent directory need to be flushed. - */ - if (fsync_fname_ext(newfile, false, false, elevel) != 0) - return -1; - - /* Same for parent directory */ - if (fsync_parent_path(newfile, elevel) != 0) - return -1; - - return 0; -} - /* * InitFileAccess --- initialize this module during backend startup * diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 84ce5a4a5d..830804fdfb 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -163,13 +163,6 @@ #define USE_BARRIER_SMGRRELEASE #endif -/* - * Define this if your operating system supports link() - */ -#if !defined(WIN32) && !defined(__CYGWIN__) -#define HAVE_WORKING_LINK 1 -#endif - /* * USE_POSIX_FADVISE controls whether Postgres will attempt to use the * posix_fadvise() kernel call. Usually the automatic configure tests are diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index 69549b000f..2b4a8e0ffe 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -187,7 +187,6 @@ extern void fsync_fname(const char *fname, bool isdir); extern int fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel); extern int durable_rename(const char *oldfile, const char *newfile, int loglevel); extern int durable_unlink(const char *fname, int loglevel); -extern int durable_rename_excl(const char *oldfile, const char *newfile, int loglevel); extern void SyncDataDirectory(void); extern int data_sync_elevel(int elevel); -- 2.25.1 --LQksG6bCIzRHxTLp-- ^ permalink raw reply [nested|flat] 6+ messages in thread
* [PATCH v3 2/2] Remove durable_rename_excl(). @ 2022-04-26 19:38 Nathan Bossart <nathandbossart@gmail.com> 0 siblings, 0 replies; 6+ messages in thread From: Nathan Bossart @ 2022-04-26 19:38 UTC (permalink / raw) A previous commit replaced all calls to this function with durable_rename(), but the function itself was not removed in back- branches since extensions may use it. This change removes the function from v16devel. Do not back-patch. Author: Nathan Bossart Reviewed-by: Robert Haas, Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/20220418182336.GA2298576%40nathanxps13 --- src/backend/storage/file/fd.c | 63 ---------------------------------- src/include/pg_config_manual.h | 7 ---- src/include/storage/fd.h | 1 - 3 files changed, 71 deletions(-) diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 24704b6a02..f904f60c08 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -807,69 +807,6 @@ durable_unlink(const char *fname, int elevel) return 0; } -/* - * durable_rename_excl -- rename a file in a durable manner. - * - * Similar to durable_rename(), except that this routine tries (but does not - * guarantee) not to overwrite the target file. - * - * Note that a crash in an unfortunate moment can leave you with two links to - * the target file. - * - * Log errors with the caller specified severity. - * - * On Windows, using a hard link followed by unlink() causes concurrency - * issues, while a simple rename() does not cause that, so be careful when - * changing the logic of this routine. - * - * Returns 0 if the operation succeeded, -1 otherwise. Note that errno is not - * valid upon return. - */ -int -durable_rename_excl(const char *oldfile, const char *newfile, int elevel) -{ - /* - * Ensure that, if we crash directly after the rename/link, a file with - * valid contents is moved into place. - */ - if (fsync_fname_ext(oldfile, false, false, elevel) != 0) - return -1; - -#ifdef HAVE_WORKING_LINK - if (link(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not link file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } - unlink(oldfile); -#else - if (rename(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } -#endif - - /* - * Make change persistent in case of an OS crash, both the new entry and - * its parent directory need to be flushed. - */ - if (fsync_fname_ext(newfile, false, false, elevel) != 0) - return -1; - - /* Same for parent directory */ - if (fsync_parent_path(newfile, elevel) != 0) - return -1; - - return 0; -} - /* * InitFileAccess --- initialize this module during backend startup * diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 84ce5a4a5d..830804fdfb 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -163,13 +163,6 @@ #define USE_BARRIER_SMGRRELEASE #endif -/* - * Define this if your operating system supports link() - */ -#if !defined(WIN32) && !defined(__CYGWIN__) -#define HAVE_WORKING_LINK 1 -#endif - /* * USE_POSIX_FADVISE controls whether Postgres will attempt to use the * posix_fadvise() kernel call. Usually the automatic configure tests are diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index 69549b000f..2b4a8e0ffe 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -187,7 +187,6 @@ extern void fsync_fname(const char *fname, bool isdir); extern int fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel); extern int durable_rename(const char *oldfile, const char *newfile, int loglevel); extern int durable_unlink(const char *fname, int loglevel); -extern int durable_rename_excl(const char *oldfile, const char *newfile, int loglevel); extern void SyncDataDirectory(void); extern int data_sync_elevel(int elevel); -- 2.25.1 --liOOAslEiF7prFVr-- ^ permalink raw reply [nested|flat] 6+ messages in thread
* [PATCH v4 2/2] Remove durable_rename_excl(). @ 2022-04-26 19:38 Nathan Bossart <nathandbossart@gmail.com> 0 siblings, 0 replies; 6+ messages in thread From: Nathan Bossart @ 2022-04-26 19:38 UTC (permalink / raw) A previous commit replaced all calls to this function with durable_rename(), but the function itself was not removed in back- branches since extensions may use it. This change removes the function from v16devel. Do not back-patch. Author: Nathan Bossart Reviewed-by: Robert Haas, Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/20220418182336.GA2298576%40nathanxps13 --- src/backend/storage/file/fd.c | 63 ---------------------------------- src/include/pg_config_manual.h | 7 ---- src/include/storage/fd.h | 1 - 3 files changed, 71 deletions(-) diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 24704b6a02..f904f60c08 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -807,69 +807,6 @@ durable_unlink(const char *fname, int elevel) return 0; } -/* - * durable_rename_excl -- rename a file in a durable manner. - * - * Similar to durable_rename(), except that this routine tries (but does not - * guarantee) not to overwrite the target file. - * - * Note that a crash in an unfortunate moment can leave you with two links to - * the target file. - * - * Log errors with the caller specified severity. - * - * On Windows, using a hard link followed by unlink() causes concurrency - * issues, while a simple rename() does not cause that, so be careful when - * changing the logic of this routine. - * - * Returns 0 if the operation succeeded, -1 otherwise. Note that errno is not - * valid upon return. - */ -int -durable_rename_excl(const char *oldfile, const char *newfile, int elevel) -{ - /* - * Ensure that, if we crash directly after the rename/link, a file with - * valid contents is moved into place. - */ - if (fsync_fname_ext(oldfile, false, false, elevel) != 0) - return -1; - -#ifdef HAVE_WORKING_LINK - if (link(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not link file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } - unlink(oldfile); -#else - if (rename(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } -#endif - - /* - * Make change persistent in case of an OS crash, both the new entry and - * its parent directory need to be flushed. - */ - if (fsync_fname_ext(newfile, false, false, elevel) != 0) - return -1; - - /* Same for parent directory */ - if (fsync_parent_path(newfile, elevel) != 0) - return -1; - - return 0; -} - /* * InitFileAccess --- initialize this module during backend startup * diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 84ce5a4a5d..830804fdfb 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -163,13 +163,6 @@ #define USE_BARRIER_SMGRRELEASE #endif -/* - * Define this if your operating system supports link() - */ -#if !defined(WIN32) && !defined(__CYGWIN__) -#define HAVE_WORKING_LINK 1 -#endif - /* * USE_POSIX_FADVISE controls whether Postgres will attempt to use the * posix_fadvise() kernel call. Usually the automatic configure tests are diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index 69549b000f..2b4a8e0ffe 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -187,7 +187,6 @@ extern void fsync_fname(const char *fname, bool isdir); extern int fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel); extern int durable_rename(const char *oldfile, const char *newfile, int loglevel); extern int durable_unlink(const char *fname, int loglevel); -extern int durable_rename_excl(const char *oldfile, const char *newfile, int loglevel); extern void SyncDataDirectory(void); extern int data_sync_elevel(int elevel); -- 2.25.1 --k+w/mQv8wyuph6w0-- ^ permalink raw reply [nested|flat] 6+ messages in thread
* [PATCH v5 2/2] Remove durable_rename_excl(). @ 2022-04-26 19:38 Nathan Bossart <nathandbossart@gmail.com> 0 siblings, 0 replies; 6+ messages in thread From: Nathan Bossart @ 2022-04-26 19:38 UTC (permalink / raw) A previous commit replaced all calls to this function with durable_rename(), but the function itself was not removed in back- branches since extensions may use it. This change removes the function from v16devel. Do not back-patch. Author: Nathan Bossart Reviewed-by: Robert Haas, Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/20220418182336.GA2298576%40nathanxps13 --- src/backend/storage/file/fd.c | 63 ---------------------------------- src/include/pg_config_manual.h | 7 ---- src/include/storage/fd.h | 1 - 3 files changed, 71 deletions(-) diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 24704b6a02..f904f60c08 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -807,69 +807,6 @@ durable_unlink(const char *fname, int elevel) return 0; } -/* - * durable_rename_excl -- rename a file in a durable manner. - * - * Similar to durable_rename(), except that this routine tries (but does not - * guarantee) not to overwrite the target file. - * - * Note that a crash in an unfortunate moment can leave you with two links to - * the target file. - * - * Log errors with the caller specified severity. - * - * On Windows, using a hard link followed by unlink() causes concurrency - * issues, while a simple rename() does not cause that, so be careful when - * changing the logic of this routine. - * - * Returns 0 if the operation succeeded, -1 otherwise. Note that errno is not - * valid upon return. - */ -int -durable_rename_excl(const char *oldfile, const char *newfile, int elevel) -{ - /* - * Ensure that, if we crash directly after the rename/link, a file with - * valid contents is moved into place. - */ - if (fsync_fname_ext(oldfile, false, false, elevel) != 0) - return -1; - -#ifdef HAVE_WORKING_LINK - if (link(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not link file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } - unlink(oldfile); -#else - if (rename(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } -#endif - - /* - * Make change persistent in case of an OS crash, both the new entry and - * its parent directory need to be flushed. - */ - if (fsync_fname_ext(newfile, false, false, elevel) != 0) - return -1; - - /* Same for parent directory */ - if (fsync_parent_path(newfile, elevel) != 0) - return -1; - - return 0; -} - /* * InitFileAccess --- initialize this module during backend startup * diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 84ce5a4a5d..830804fdfb 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -163,13 +163,6 @@ #define USE_BARRIER_SMGRRELEASE #endif -/* - * Define this if your operating system supports link() - */ -#if !defined(WIN32) && !defined(__CYGWIN__) -#define HAVE_WORKING_LINK 1 -#endif - /* * USE_POSIX_FADVISE controls whether Postgres will attempt to use the * posix_fadvise() kernel call. Usually the automatic configure tests are diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index 69549b000f..2b4a8e0ffe 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -187,7 +187,6 @@ extern void fsync_fname(const char *fname, bool isdir); extern int fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel); extern int durable_rename(const char *oldfile, const char *newfile, int loglevel); extern int durable_unlink(const char *fname, int loglevel); -extern int durable_rename_excl(const char *oldfile, const char *newfile, int loglevel); extern void SyncDataDirectory(void); extern int data_sync_elevel(int elevel); -- 2.25.1 --ZGiS0Q5IWpPtfppv-- ^ permalink raw reply [nested|flat] 6+ messages in thread
* [PATCH v5 2/2] Remove durable_rename_excl(). @ 2022-04-26 19:38 Nathan Bossart <nathandbossart@gmail.com> 0 siblings, 0 replies; 6+ messages in thread From: Nathan Bossart @ 2022-04-26 19:38 UTC (permalink / raw) A previous commit replaced all calls to this function with durable_rename(), but the function itself was not removed in back- branches since extensions may use it. This change removes the function from v16devel. Do not back-patch. Author: Nathan Bossart Reviewed-by: Robert Haas, Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/20220418182336.GA2298576%40nathanxps13 --- src/backend/storage/file/fd.c | 63 ---------------------------------- src/include/pg_config_manual.h | 7 ---- src/include/storage/fd.h | 1 - 3 files changed, 71 deletions(-) diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 24704b6a02..f904f60c08 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -807,69 +807,6 @@ durable_unlink(const char *fname, int elevel) return 0; } -/* - * durable_rename_excl -- rename a file in a durable manner. - * - * Similar to durable_rename(), except that this routine tries (but does not - * guarantee) not to overwrite the target file. - * - * Note that a crash in an unfortunate moment can leave you with two links to - * the target file. - * - * Log errors with the caller specified severity. - * - * On Windows, using a hard link followed by unlink() causes concurrency - * issues, while a simple rename() does not cause that, so be careful when - * changing the logic of this routine. - * - * Returns 0 if the operation succeeded, -1 otherwise. Note that errno is not - * valid upon return. - */ -int -durable_rename_excl(const char *oldfile, const char *newfile, int elevel) -{ - /* - * Ensure that, if we crash directly after the rename/link, a file with - * valid contents is moved into place. - */ - if (fsync_fname_ext(oldfile, false, false, elevel) != 0) - return -1; - -#ifdef HAVE_WORKING_LINK - if (link(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not link file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } - unlink(oldfile); -#else - if (rename(oldfile, newfile) < 0) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - oldfile, newfile))); - return -1; - } -#endif - - /* - * Make change persistent in case of an OS crash, both the new entry and - * its parent directory need to be flushed. - */ - if (fsync_fname_ext(newfile, false, false, elevel) != 0) - return -1; - - /* Same for parent directory */ - if (fsync_parent_path(newfile, elevel) != 0) - return -1; - - return 0; -} - /* * InitFileAccess --- initialize this module during backend startup * diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 84ce5a4a5d..830804fdfb 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -163,13 +163,6 @@ #define USE_BARRIER_SMGRRELEASE #endif -/* - * Define this if your operating system supports link() - */ -#if !defined(WIN32) && !defined(__CYGWIN__) -#define HAVE_WORKING_LINK 1 -#endif - /* * USE_POSIX_FADVISE controls whether Postgres will attempt to use the * posix_fadvise() kernel call. Usually the automatic configure tests are diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index 69549b000f..2b4a8e0ffe 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -187,7 +187,6 @@ extern void fsync_fname(const char *fname, bool isdir); extern int fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel); extern int durable_rename(const char *oldfile, const char *newfile, int loglevel); extern int durable_unlink(const char *fname, int loglevel); -extern int durable_rename_excl(const char *oldfile, const char *newfile, int loglevel); extern void SyncDataDirectory(void); extern int data_sync_elevel(int elevel); -- 2.25.1 --ZGiS0Q5IWpPtfppv-- ^ permalink raw reply [nested|flat] 6+ messages in thread
* [PATCH v24 6/8] Row pattern recognition patch (docs). @ 2024-12-19 06:06 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2024-12-19 06:06 UTC (permalink / raw) --- doc/src/sgml/advanced.sgml | 82 ++++++++++++++++++++++++++++++++++++ doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++ doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++- 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml index 755c9f1485..b0b1d1c51e 100644 --- a/doc/src/sgml/advanced.sgml +++ b/doc/src/sgml/advanced.sgml @@ -537,6 +537,88 @@ WHERE pos < 3; <literal>rank</literal> less than 3. </para> + <para> + Row pattern common syntax can be used to perform row pattern recognition + in a query. The row pattern common syntax includes two sub + clauses: <literal>DEFINE</literal> + and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines + definition variables along with an expression. The expression must be a + logical expression, which means it must + return <literal>TRUE</literal>, <literal>FALSE</literal> + or <literal>NULL</literal>. The expression may comprise column references + and functions. Window functions, aggregate functions and subqueries are + not allowed. An example of <literal>DEFINE</literal> is as follows. + +<programlisting> +DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +</programlisting> + + Note that <function>PREV</function> returns the price column in the + previous row if it's called in a context of row pattern recognition. Thus in + the second line the definition variable "UP" is <literal>TRUE</literal> + when the price column in the current row is greater than the price column + in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when + the price column in the current row is lower than the price column in the + previous row. + </para> + <para> + Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be + used. <literal>PATTERN</literal> defines a sequence of rows that satisfies + certain conditions. For example following <literal>PATTERN</literal> + defines that a row starts with the condition "LOWPRICE", then one or more + rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that + "+" means one or more matches. Also you can use "*", which means zero or + more matches. If a sequence of rows which satisfies the PATTERN is found, + in the starting row of the sequence of rows all window functions and + aggregates are shown in the target list. Note that aggregations only look + into the matched rows, rather than whole frame. On the second or + subsequent rows all window functions are NULL. Aggregates are NULL or 0 + (count case) depending on its aggregation definition. For rows that do not + match on the PATTERN, all window functions and aggregates are shown AS + NULL too, except count showing 0. This is because the rows do not match, + thus they are in an empty frame. Example of a <literal>SELECT</literal> + using the <literal>DEFINE</literal> and <literal>PATTERN</literal> clause + is as follows. + +<programlisting> +SELECT company, tdate, price, + first_value(price) OVER w, + max(price) OVER w, + count(price) OVER w +FROM stock + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN (LOWPRICE UP+ DOWN+) + DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +); +</programlisting> +<screen> + company | tdate | price | first_value | max | count +----------+------------+-------+-------------+-----+------- + company1 | 2023-07-01 | 100 | 100 | 200 | 4 + company1 | 2023-07-02 | 200 | | | 0 + company1 | 2023-07-03 | 150 | | | 0 + company1 | 2023-07-04 | 140 | | | 0 + company1 | 2023-07-05 | 150 | | | 0 + company1 | 2023-07-06 | 90 | 90 | 130 | 4 + company1 | 2023-07-07 | 110 | | | 0 + company1 | 2023-07-08 | 130 | | | 0 + company1 | 2023-07-09 | 120 | | | 0 + company1 | 2023-07-10 | 130 | | | 0 +(10 rows) +</screen> + </para> + <para> When a query involves multiple window functions, it is possible to write out each one with a separate <literal>OVER</literal> clause, but this is diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 47370e581a..17a38c4046 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -23338,6 +23338,7 @@ SELECT count(*) FROM sometable; returns <literal>NULL</literal> if there is no such row. </para></entry> </row> + </tbody> </tgroup> </table> @@ -23377,6 +23378,59 @@ SELECT count(*) FROM sometable; Other frame specifications can be used to obtain other effects. </para> + <para> + Row pattern recognition navigation functions are listed in + <xref linkend="functions-rpr-navigation-table"/>. These functions + can be used to describe DEFINE clause of Row pattern recognition. + </para> + + <table id="functions-rpr-navigation-table"> + <title>Row Pattern Navigation Functions</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="func_table_entry"><para role="func_signature"> + Function + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>prev</primary> + </indexterm> + <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the previous row; + returns NULL if there is no previous row in the window frame. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>next</primary> + </indexterm> + <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the next row; + returns NULL if there is no next row in the window frame. + </para></entry> + </row> + + </tbody> + </tgroup> + </table> + <note> <para> The SQL standard defines a <literal>RESPECT NULLS</literal> or diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index d7089eac0b..7e1c9989ba 100644 --- a/doc/src/sgml/ref/select.sgml +++ b/doc/src/sgml/ref/select.sgml @@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl The <replaceable class="parameter">frame_clause</replaceable> can be one of <synopsis> -{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] -{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] +{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] +{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] </synopsis> where <replaceable>frame_start</replaceable> @@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS a given peer group will be in the frame or excluded from it. </para> + <para> + The + optional <replaceable class="parameter">row_pattern_common_syntax</replaceable> + defines the <firstterm>row pattern recognition condition</firstterm> for + this + window. <replaceable class="parameter">row_pattern_common_syntax</replaceable> + includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST + ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls + how to proceed to next row position after a match + found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the + default) next row position is next to the last row of previous match. On + the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next + row position is always next to the last row of previous + match. <literal>DEFINE</literal> defines definition variables along with a + boolean expression. <literal>PATTERN</literal> defines a sequence of rows + that satisfies certain conditions using variables defined + in <literal>DEFINE</literal> clause. If the variable is not defined in + the <literal>DEFINE</literal> clause, it is implicitly assumed + following is defined in the <literal>DEFINE</literal> clause. + +<synopsis> +<literal>variable_name</literal> AS TRUE +</synopsis> + + Note that the maximu number of variables defined + in <literal>DEFINE</literal> clause is 26. + +<synopsis> +[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ] +PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...] +DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...] +</synopsis> + </para> + <para> The purpose of a <literal>WINDOW</literal> clause is to specify the behavior of <firstterm>window functions</firstterm> appearing in the query's -- 2.25.1 ----Next_Part(Thu_Dec_19_15_19_50_2024_894)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v24-0007-Row-pattern-recognition-patch-tests.patch" ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2024-12-19 06:06 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-04-08 18:48 [PATCH v2 1/1] Remove durable_rename_excl(). Nathan Bossart <nathandbossart@gmail.com> 2022-04-26 19:38 [PATCH v3 2/2] Remove durable_rename_excl(). Nathan Bossart <nathandbossart@gmail.com> 2022-04-26 19:38 [PATCH v4 2/2] Remove durable_rename_excl(). Nathan Bossart <nathandbossart@gmail.com> 2022-04-26 19:38 [PATCH v5 2/2] Remove durable_rename_excl(). Nathan Bossart <nathandbossart@gmail.com> 2022-04-26 19:38 [PATCH v5 2/2] Remove durable_rename_excl(). Nathan Bossart <nathandbossart@gmail.com> 2024-12-19 06:06 [PATCH v24 6/8] Row pattern recognition patch (docs). Tatsuo Ishii <ishii@postgresql.org>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox