pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix issues with Windows' stat() for files pending on deletion 3+ messages / 2 participants [nested] [flat]
* pgsql: Fix issues with Windows' stat() for files pending on deletion @ 2021-07-12 04:03 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 3+ messages in thread From: Michael Paquier @ 2021-07-12 04:03 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix issues with Windows' stat() for files pending on deletion The code introduced by bed9075 to enhance the stat() implementation on Windows for file sizes larger than 4GB fails to properly detect files pending for deletion with its method based on NtQueryInformationFile() or GetFileInformationByHandleEx(), as proved by Alexander Lakhin in a custom TAP test of his own. The method used in the implementation of open() to sleep and loop when when failing on ERROR_ACCESS_DENIED (EACCES) is showing much more stability, so switch to this method. This could still lead to issues if the permission problem stays around for much longer than the timeout of 1 second used, but that should (hopefully) never happen in performance-critical paths. Still, there could be a point in increasing the timeouts for the sake of machines that handle heavy loads. Note that WIN32's open() now uses microsoft_native_stat() as it should be similar to stat() when working around issues with concurrent file deletions. I have spent some time testing this patch with pgbench in combination of the SQL functions from genfile.c, as well as running the TAP test provided on the thread with MSVC builds, and this looks much more stable than the previous method. Author: Alexander Lakhin Reviewed-by: Tom Lane, Michael Paquier, Justin Pryzby Discussion: https://postgr.es/m/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com Backpatch-through: 14 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/54fb8c7ddf152629021cab3ac3596354217b7d81 Modified Files -------------- src/port/open.c | 4 +- src/port/win32stat.c | 177 ++++++++++++++------------------------------------- 2 files changed, 49 insertions(+), 132 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
* pgsql: Fix issues with Windows' stat() for files pending on deletion @ 2021-07-12 04:03 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 3+ messages in thread From: Michael Paquier @ 2021-07-12 04:03 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix issues with Windows' stat() for files pending on deletion The code introduced by bed9075 to enhance the stat() implementation on Windows for file sizes larger than 4GB fails to properly detect files pending for deletion with its method based on NtQueryInformationFile() or GetFileInformationByHandleEx(), as proved by Alexander Lakhin in a custom TAP test of his own. The method used in the implementation of open() to sleep and loop when when failing on ERROR_ACCESS_DENIED (EACCES) is showing much more stability, so switch to this method. This could still lead to issues if the permission problem stays around for much longer than the timeout of 1 second used, but that should (hopefully) never happen in performance-critical paths. Still, there could be a point in increasing the timeouts for the sake of machines that handle heavy loads. Note that WIN32's open() now uses microsoft_native_stat() as it should be similar to stat() when working around issues with concurrent file deletions. I have spent some time testing this patch with pgbench in combination of the SQL functions from genfile.c, as well as running the TAP test provided on the thread with MSVC builds, and this looks much more stable than the previous method. Author: Alexander Lakhin Reviewed-by: Tom Lane, Michael Paquier, Justin Pryzby Discussion: https://postgr.es/m/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com Backpatch-through: 14 Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/de1510e2f5a1826890b206253016ebfc592c2f0a Modified Files -------------- src/port/open.c | 4 +- src/port/win32stat.c | 177 ++++++++++++++------------------------------------- 2 files changed, 49 insertions(+), 132 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
* pgsql: Fix issues with Windows' stat() for files pending on deletion @ 2024-11-07 23:19 Andrew Dunstan <andrew@dunslane.net> 0 siblings, 0 replies; 3+ messages in thread From: Andrew Dunstan @ 2024-11-07 23:19 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix issues with Windows' stat() for files pending on deletion The code introduced by bed9075 to enhance the stat() implementation on Windows for file sizes larger than 4GB fails to properly detect files pending for deletion with its method based on NtQueryInformationFile() or GetFileInformationByHandleEx(), as proved by Alexander Lakhin in a custom TAP test of his own. The method used in the implementation of open() to sleep and loop when when failing on ERROR_ACCESS_DENIED (EACCES) is showing much more stability, so switch to this method. This could still lead to issues if the permission problem stays around for much longer than the timeout of 1 second used, but that should (hopefully) never happen in performance-critical paths. Still, there could be a point in increasing the timeouts for the sake of machines that handle heavy loads. Note that WIN32's open() now uses microsoft_native_stat() as it should be similar to stat() when working around issues with concurrent file deletions. I have spent some time testing this patch with pgbench in combination of the SQL functions from genfile.c, as well as running the TAP test provided on the thread with MSVC builds, and this looks much more stable than the previous method. Author: Alexander Lakhin Reviewed-by: Tom Lane, Michael Paquier, Justin Pryzby Discussion: https://postgr.es/m/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com Backpatch-through: 14 (cherry picked from commit 54fb8c7ddf152629021cab3ac3596354217b7d81) Author: Alexandra Wang <alexandra.wang.oss@gmail.com> Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/f1cf64167fc9a09f54f69ec1d82865ba6aca5fe6 Author: Michael Paquier <michael@paquier.xyz> Modified Files -------------- src/port/open.c | 4 +- src/port/win32stat.c | 173 +++++++++++++++------------------------------------ 2 files changed, 51 insertions(+), 126 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2024-11-07 23:19 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-07-12 04:03 pgsql: Fix issues with Windows' stat() for files pending on deletion Michael Paquier <michael@paquier.xyz> 2021-07-12 04:03 pgsql: Fix issues with Windows' stat() for files pending on deletion Michael Paquier <michael@paquier.xyz> 2024-11-07 23:19 pgsql: Fix issues with Windows' stat() for files pending on deletion Andrew Dunstan <andrew@dunslane.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox