Subject: Re: pg_basebackup check vs Windows file path limits

From: Andrew Dunstan <[email-masked]>
Date: Sat, 11 Nov 2023 10:18:51 -0500

Hi, Alexander

On 2023-11-11 Sa 08:00, Alexander Lakhin wrote:
> Hello Andrew,
>
> 08.07.2023 18:52, Andrew Dunstan wrote:
>>> Since this test is passing on HEAD which has slightly shorter paths, I'm wondering if we should change this:
>>>
>>>     rename("$pgdata/pg_replslot", "$tempdir/pg_replslot")
>>>       or BAIL_OUT "could not move $pgdata/pg_replslot";
>>>     dir_symlink("$tempdir/pg_replslot", "$pgdata/pg_replslot")
>>>       or BAIL_OUT "could not symlink to $pgdata/pg_replslot";
>>>
>>> to use the much shorter $sys_tempdir created a few lines below.
>>>
>> Pushed a tested fix along those lines.
>>
>
> Today I've started up my Windows VM to run some tests and discovered a 
> test
> failure caused by that fix (e213de8e7):
> >meson test
> Ok:                 246
> Expected Fail:      0
> Fail:               1
> Unexpected Pass:    0
> Skipped:            14
> Timeout:            0
>
> ...\010_pg_basebackup\log\regress_log_010_pg_basebackup.txt contains:
> [04:42:45.321](0.291s) Bail out!  could not move 
> T:\postgresql\build/testrun/pg_basebackup/010_pg_basebackup\data/t_010_pg_basebackup_main_data/pgdata/pg_replslot
>
> With a diagnostic print added before rename() in 010_pg_basebackup.pl, 
> I see:
> rename("T:\postgresql\build/testrun/pg_basebackup/010_pg_basebackup\data/t_010_pg_basebackup_main_data/pgdata/pg_replslot", 
> "C:\Users\User\AppData\Local\Temp\fGT76tZUWr/pg_replslot")
> That is, I have the postgres source tree and the user tempdir placed on
> different disks.
>
> perldoc on rename() says that it usually doesn't work across filesystem
> boundaries, so I think it's not a Windows-specific issue.
>
>

Hmm, maybe we should be using File::Copy::move() instead of rename(). 
The docco for that says:

         If possible, move() will simply rename the file. Otherwise, it
         copies the file to the new location and deletes the original. If an
         error occurs during this copy-and-delete process, you may be left
         with a (possibly partial) copy of the file under the destination
         name.

Can you try it out?

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

---

From: Alexander Lakhin <[email-masked]>
Date: Sat, 11 Nov 2023 20:00:01 +0300

11.11.2023 18:18, Andrew Dunstan wrote:
>
> Hmm, maybe we should be using File::Copy::move() instead of rename(). The docco for that says:
>

Unfortunately, I've stumbled upon inability of File::Copy::move()
to move directories across filesystems, exactly as described here:
https://stackoverflow.com/questions/17628039/filecopy-move-directories-accross-drives-in-windows-not-working

(I'm sorry for not looking above rename() where this stated explicitly:
# On Windows use the short location to avoid path length issues.
# Elsewhere use $tempdir to avoid file system boundary issues with moving.
So this issue affects Windows only.)

Best regards,
Alexander

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 12 Nov 2023 09:09:27 -0500

> So this issue affects Windows only.)

*sigh*

A probable workaround is to use a temp directory on the same device the 
test is building on. Just set it up and set your environment TEMPDIR to 
point to it, and I think it will be OK (i.e. I havent tested it).

But that doesn't mean I'm not searching for a better solution. Maybe 
Alvaro's suggestion nearby will help.

cheers

andrew