From: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
To: Andres Freund <andres@anarazel.de>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: pgsql-hackers@lists.postgresql.org <pgsql-hackers@lists.postgresql.org>
Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>
Cc: Thomas Munro <thomas.munro@gmail.com>
Subject: RE: pg_upgrade test failure
Date: Sun, 29 Oct 2023 05:43:46 +0000
Message-ID: <TYAPR01MB58666BF46F9AB5FB9D57436CF5A2A@TYAPR01MB5866.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <20230206234301.33c5vvtd7f3rs6qv@awork3.anarazel.de>
References: <CA+hUKGLfruDhC_8QRGDoCndsdzDVjrf7TYwgsv+OfoAmK2syEg@mail.gmail.com>
<20230131172806.GM22427@telsasoft.com>
<CA+hUKGKCVy2=o=d8c2Va6a_3Rpf_KkhUitkWCZ3hzuO2VwLMXA@mail.gmail.com>
<F0554EC4-4545-4224-B373-9B01EA8D25F1@anarazel.de>
<CA+hUKG+e788-7YMxFEdScS+V_rVYoM6-H+mTS6caoo8oJBzf=A@mail.gmail.com>
<CA+hUKGJMynw7BNpsaF3c7hPBEBpzP8T_8WbO-93s=XzcoEsoxw@mail.gmail.com>
<CA+hUKG+08L=QNq8Zn6xG7f=m41=w=x52YViT6fyRMoCD9RnMzA@mail.gmail.com>
<3FC7ACA2-5A65-4CAB-AC3D-8F9695A97A2C@anarazel.de>
<CA+hUKGJnbNm_WmtzmCwPXZpYtqLnMy1=atCHUtNki8pbDhVXSQ@mail.gmail.com>
<20230206221422.ykidlne7pmywu4ay@awork3.anarazel.de>
<20230206234301.33c5vvtd7f3rs6qv@awork3.anarazel.de>
Dear Andres,
While tracking BF failures related with pg_ugprade, I found the same failure has still happened [1] - [4].
According to the log, the output directory was remained even after the successful upgrade [5].
I analyzed and attached the fix patch, and below is my analysis... how do you think?
=====
lstat() seemed fail while doing the second try of rmtree(). This error message is
output from get_dirent_type().
Apart from pgunlink(), get_dirent_type() does not have an retry mechanism when
lstat()->_pglstat64() detects STATUS_DELETE_PENDING. Therefore, I think rmtree()
may not wait the file until it would be really removed, if the status is deceted
in the get_dirent_type().
One solution is to retry stat() or lstat() even in get_dirent_type(), like attached.
[1]: 2023-07-21 02:21:53 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-07-21%2002%3A21%3A53
[2]: 2023-10-21 13:39:15 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-10-21%2013%3A39%3A15
[3]: 2023-10-23 09:03:07 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-10-23%2009%3A03%3A07
[4]: 2023-10-27 23:06:17 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-10-27%2023%3A06%3A17
[5]
```
...
*Clusters are compatible*
pg_upgrade: warning: could not remove directory "C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867/log": Directory not empty
pg_upgrade: warning: could not remove directory "C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867": Directory not empty
pg_upgrade: warning: could not stat file "C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867/log/pg_upgrade_internal.log": No such file or directory
pg_upgrade: warning: could not remove directory "C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867/log": Directory not empty
pg_upgrade: warning: could not remove directory "C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867": Directory not empty
[23:46:07.585](17.106s) ok 12 - run of pg_upgrade --check for new instance
[23:46:07.587](0.002s) not ok 13 - pg_upgrade_output.d/ removed after pg_upgrade --check success
...
```
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:0001-Retry-stat-and-lstat-even-in-get_dirent_type.patch (application/octet-stream, 2.0K) download
From 085753f24c0fce71464f0b5e67049c1c7f9c8baa Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Sun, 29 Oct 2023 04:32:53 +0000
Subject: [PATCH] Retry stat() and lstat() even in get_dirent_type()
---
src/common/file_utils.c | 13 ++++++++++++-
src/include/port.h | 1 +
src/port/dirmod.c | 2 +-
3 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/src/common/file_utils.c b/src/common/file_utils.cindex abe5129412..704a269406 100644--- a/src/common/file_utils.c+++ b/src/common/file_utils.c@@ -552,8 +552,9 @@ get_dirent_type(const char *path,
{
struct stat fst;
int sret;
+ int count = 0;-+retry:
if (look_through_symlinks)
sret = stat(path, &fst);
else
@@ -561,6 +562,16 @@ get_dirent_type(const char *path,
if (sret < 0)
{
+#if defined(WIN32) || defined(__CYGWIN__)+ /* Retry is STATUS_DELETE_PENDING error. Timeout after 10 sec. */+ if (lstat_error_was_status_delete_pending() &&+ ++count <= 100)+ {+ pg_usleep(100000); /* us */+ goto retry;+ }+#endif+
result = PGFILETYPE_ERROR;
#ifdef FRONTEND
pg_log_generic(elevel, PG_LOG_PRIMARY, "could not stat file \"%s\": %m", path);
diff --git a/src/include/port.h b/src/include/port.hindex 5979139ebc..0ffa9f802e 100644--- a/src/include/port.h+++ b/src/include/port.h@@ -275,6 +275,7 @@ extern int pclose_check(FILE *stream);
*/
extern int pgrename(const char *from, const char *to);
extern int pgunlink(const char *path);
+extern bool lstat_error_was_status_delete_pending(void);
/* Include this first so later includes don't see these defines */
#ifdef _MSC_VER
diff --git a/src/port/dirmod.c b/src/port/dirmod.cindex 07dd190cbc..3db1a0f3d1 100644--- a/src/port/dirmod.c+++ b/src/port/dirmod.c@@ -100,7 +100,7 @@ pgrename(const char *from, const char *to)
* This doesn't apply to Cygwin, which has its own lstat() that would report
* the case as EACCES.
*/
-static bool+bool
lstat_error_was_status_delete_pending(void)
{
if (errno != ENOENT)
--
2.27.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: kuroda.hayato@fujitsu.com, andres@anarazel.de, pryzby@telsasoft.com, michael@paquier.xyz, pgsql-hackers@lists.postgresql.org, byavuz81@gmail.com, thomas.munro@gmail.com
Subject: RE: pg_upgrade test failure
In-Reply-To: <TYAPR01MB58666BF46F9AB5FB9D57436CF5A2A@TYAPR01MB5866.jpnprd01.prod.outlook.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox