pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_basebackup: errors on macOS on directories with ".DS_Store" files
34+ messages / 7 participants
[nested] [flat]
* pg_basebackup: errors on macOS on directories with ".DS_Store" files
@ 2023-04-19 15:45 Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
0 siblings, 1 reply; 34+ messages in thread
From: Mark Guertin @ 2023-04-19 15:45 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
pg_basebackup: error: backup failed: ERROR: invalid segment number 0 in file ".DS_Store"
This one is pretty self explanatory. When running pg_basebackup on macOS it should probably be ignoring any .DS_Store files found inside data directories instead of transferring/processing them.
Steps to reproduce:
- on a macOS based pg server open a directory containing pg data files and set any kind of view options or move an icon
- this can also happen with inheritance, so even if you do not directly set these kinds of options on data folders, for example if a parent folder has a view option set when a new folder is created it can inherit the same view options, or if spotlight search sets up some metadata this .DS_Store may also be created
My workaround was to manually delete all of the .DS_Store files that the OS created and then immediately run pb_basebackup using the following method:
find <dir> -iname .DS_Store -exec rm {} \;
Some background on .DS_Store files:
https://en.wikipedia.org/wiki/.DS_Store=
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
@ 2023-04-19 17:51 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 34+ messages in thread
From: Tobias Bussmann @ 2023-04-19 17:51 UTC (permalink / raw)
To: Mark Guertin <markguertin@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
> pg_basebackup: error: backup failed: ERROR: invalid segment number 0 in file ".DS_Store"
This seem to be the same problem as already discussed in [1]: Tools like pg_basebackup and pg_checksums can get upset if there are random extra files in $PGDATA/base, /global or /pg_tblspc. Unfortunately, macOS has the habit to create such files quite easily. If I read the outcome correctly, we are back at a blacklist approach, so adding ".DS_Store" may be something to consider.
[1]: <https://www.postgresql.org/message-id/flat/20181021134206.GA14282%40paquier.xyz;
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
@ 2023-04-19 18:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2023-04-19 18:15 UTC (permalink / raw)
To: Tobias Bussmann <t.bussmann@gmx.net>; +Cc: Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
Tobias Bussmann <t.bussmann@gmx.net> writes:
>> pg_basebackup: error: backup failed: ERROR: invalid segment number 0 in file ".DS_Store"
> This seem to be the same problem as already discussed in [1]: Tools like pg_basebackup and pg_checksums can get upset if there are random extra files in $PGDATA/base, /global or /pg_tblspc. Unfortunately, macOS has the habit to create such files quite easily. If I read the outcome correctly, we are back at a blacklist approach, so adding ".DS_Store" may be something to consider.
> [1]: <https://www.postgresql.org/message-id/flat/20181021134206.GA14282%40paquier.xyz;
Yeah. I wonder if we ought to do something more general, and
ignore all files whose names start with ".".
regards, tom lane
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
@ 2023-04-20 00:15 ` Michael Paquier <michael@paquier.xyz>
2023-04-20 07:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 2 replies; 34+ messages in thread
From: Michael Paquier @ 2023-04-20 00:15 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Tobias Bussmann <t.bussmann@gmx.net>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Wed, Apr 19, 2023 at 02:15:51PM -0400, Tom Lane wrote:
> Yeah. I wonder if we ought to do something more general, and
> ignore all files whose names start with ".".
Indeed. I don't see any point is adding hidden files for the
checksum, rewind and base backup lists, so we could just make it a
hardcoded rule. That's currently what we do for configuration files
when using include_dir for postgresql.conf (as well as hba and ident
files in 16~).
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZECELdGIt2xGSWte@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-20 07:34 ` Daniel Gustafsson <daniel@yesql.se>
1 sibling, 0 replies; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-20 07:34 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Tobias Bussmann <t.bussmann@gmx.net>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 20 Apr 2023, at 02:15, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Apr 19, 2023 at 02:15:51PM -0400, Tom Lane wrote:
>> Yeah. I wonder if we ought to do something more general, and
>> ignore all files whose names start with ".".
>
> Indeed. I don't see any point is adding hidden files for the
> checksum, rewind and base backup lists, so we could just make it a
> hardcoded rule. That's currently what we do for configuration files
> when using include_dir for postgresql.conf (as well as hba and ident
> files in 16~).
+1, it will keep editor leftovers away as well.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-20 08:50 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
1 sibling, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-20 08:50 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Tobias Bussmann <t.bussmann@gmx.net>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 20 Apr 2023, at 02:15, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Apr 19, 2023 at 02:15:51PM -0400, Tom Lane wrote:
>> Yeah. I wonder if we ought to do something more general, and
>> ignore all files whose names start with ".".
>
> Indeed. I don't see any point is adding hidden files for the
> checksum, rewind and base backup lists, so we could just make it a
> hardcoded rule. That's currently what we do for configuration files
> when using include_dir for postgresql.conf (as well as hba and ident
> files in 16~).
The attached trivial diff skips hidden files for pg_basebackup and pg_checksums
as part of the checks already performed by these tools for skipping other
files. Skimming other callsites of ReadDir and readdir I didn't see any others
which could benefit from skipping hidden files.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] skip_hidden_files.diff (1.1K, ../../5781DAEE-66E2-41F5-BE03-E82C729F1340@yesql.se/2-skip_hidden_files.diff)
download | inline diff:
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 5baea7535b..75b6af80cb 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1202,6 +1202,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..d3c16acf0d 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -337,6 +337,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-04-20 09:51 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Tobias Bussmann @ 2023-04-20 09:51 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> Am 20.04.2023 um 10:50 schrieb Daniel Gustafsson <daniel@yesql.se>:
>
> Skimming other callsites of ReadDir and readdir I didn't see any others
> which could benefit from skipping hidden files.
Could /src/bin/pg_rewind/filemap.c be another candidate for skipping hidden files? The 'copy all other files'-behaviour seems to be similar to what pg_basebackup does and it uses the same exclude_list_item blacklist mechanism.
--
Tobias Bussmann
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
@ 2023-04-20 11:15 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-20 11:15 UTC (permalink / raw)
To: Tobias Bussmann <t.bussmann@gmx.net>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 20 Apr 2023, at 11:51, Tobias Bussmann <t.bussmann@gmx.net> wrote:
>
>> Am 20.04.2023 um 10:50 schrieb Daniel Gustafsson <daniel@yesql.se>:
>>
>> Skimming other callsites of ReadDir and readdir I didn't see any others
>> which could benefit from skipping hidden files.
>
> Could /src/bin/pg_rewind/filemap.c be another candidate for skipping hidden files? The 'copy all other files'-behaviour seems to be similar to what pg_basebackup does and it uses the same exclude_list_item blacklist mechanism.
Maybe. I'm a bit hesitant to add too many smarts to pg_rewind. It's a tool
for when something has gone wrong with a cluster (albeit probably not at the
filesystem level), and at that point I feel it's better to put the user fully
in charge. Perhaps I'm overly cautious, curious to hear from others.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-04-20 22:33 ` Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 09:08 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 2 replies; 34+ messages in thread
From: Michael Paquier @ 2023-04-20 22:33 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Thu, Apr 20, 2023 at 01:15:40PM +0200, Daniel Gustafsson wrote:
> Maybe. I'm a bit hesitant to add too many smarts to pg_rewind. It's a tool
> for when something has gone wrong with a cluster (albeit probably not at the
> filesystem level), and at that point I feel it's better to put the user fully
> in charge. Perhaps I'm overly cautious, curious to hear from others.
Hmm. pg_rewind is mostly a differential block-level backup tool, so
applying the same rules everywhere across the board would be sensible
here. See that exclude_list_item is able to handle prefixes, and we
may want to extend the same logic for the directory list, as well..
By the way, the patch ought to add some tests? For pg_basebackup,
this would be around "These files should not be copied" in
010_pg_basebackup.pl. pg_checksums has also its own checks in
002_actions.pl.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZEG9pTrrb5fdqRpS@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-21 03:02 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
1 sibling, 1 reply; 34+ messages in thread
From: Kyotaro Horiguchi @ 2023-04-21 03:02 UTC (permalink / raw)
To: michael@paquier.xyz; +Cc: daniel@yesql.se; t.bussmann@gmx.net; tgl@sss.pgh.pa.us; markguertin@gmail.com; pgsql-bugs@lists.postgresql.org
At Fri, 21 Apr 2023 07:33:09 +0900, Michael Paquier <michael@paquier.xyz> wrote in
> On Thu, Apr 20, 2023 at 01:15:40PM +0200, Daniel Gustafsson wrote:
> > Maybe. I'm a bit hesitant to add too many smarts to pg_rewind. It's a tool
> > for when something has gone wrong with a cluster (albeit probably not at the
> > filesystem level), and at that point I feel it's better to put the user fully
> > in charge. Perhaps I'm overly cautious, curious to hear from others.
>
> Hmm. pg_rewind is mostly a differential block-level backup tool, so
> applying the same rules everywhere across the board would be sensible
> here. See that exclude_list_item is able to handle prefixes, and we
> may want to extend the same logic for the directory list, as well..
After applying the change, sendDir excludes some files in the the
following steps.
1. Excludes "." and ".." quietly.
2. Excludes files starting with "pgsql_tmp", quietly.
3. (new) Excludes files begins with '.', quietly.
(checks signal)
4. Excludes everything suggested by excludeFiles then reports the
excluded files using DEBUG1.
5. Excludes files based on more complex conditions.
These steps seem a bit arbitrary. Is there any reason we keep step 1
seprate from step 3, and step 2 seprate from step 4?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
@ 2023-04-27 14:22 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-27 14:22 UTC (permalink / raw)
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; +Cc: michael@paquier.xyz; t.bussmann@gmx.net; tgl@sss.pgh.pa.us; markguertin@gmail.com; pgsql-bugs@lists.postgresql.org
> On 21 Apr 2023, at 05:02, Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote:
> After applying the change, sendDir excludes some files in the the
> following steps.
>
> 1. Excludes "." and ".." quietly.
>
> 2. Excludes files starting with "pgsql_tmp", quietly.
>
> 3. (new) Excludes files begins with '.', quietly.
>
> (checks signal)
>
> 4. Excludes everything suggested by excludeFiles then reports the
> excluded files using DEBUG1.
>
> 5. Excludes files based on more complex conditions.
>
> These steps seem a bit arbitrary. Is there any reason we keep step 1
> seprate from step 3,
I'm not sure I follow. While "." and ".." and hidden files share a leading '.'
character they are vastly different and combining them in the same check would
trade lines of code for readability. I'm not sure that's a net win. If you
mean that the checks should be reordered such that hidden files are checked for
before "pgsql_tmp", then that seems equally arbitrary no? There is no reason
to believe that hidden files would be more frequently found than "pgsql_tmp
files. I don't have strong opinions on the order, I'm just not sure either is
better than the other.
> and step 2 seprate from step 4?
The commit message for 6ad8ac60262 doesn't explain why pgsql_tmp was left out
of the excludeFiles table, but my guess is that it's either an optimization or
a deliberate choice to not DEBUG1 log skipping temporary files. I didn't go
digging in the archives to find the corresponding thread but there might be
clues to be had there.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-04-28 05:32 ` Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-04-28 05:32 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; t.bussmann@gmx.net; tgl@sss.pgh.pa.us; markguertin@gmail.com; pgsql-bugs@lists.postgresql.org
On Thu, Apr 27, 2023 at 04:22:08PM +0200, Daniel Gustafsson wrote:
> I'm not sure I follow. While "." and ".." and hidden files share a leading '.'
> character they are vastly different and combining them in the same check would
> trade lines of code for readability. I'm not sure that's a net win. If you
> mean that the checks should be reordered such that hidden files are checked for
> before "pgsql_tmp", then that seems equally arbitrary no? There is no reason
> to believe that hidden files would be more frequently found than "pgsql_tmp
> files. I don't have strong opinions on the order, I'm just not sure either is
> better than the other.
FWIW, I don't have an issue with the order of the checks as presented
in v2.
> The commit message for 6ad8ac60262 doesn't explain why pgsql_tmp was left out
> of the excludeFiles table, but my guess is that it's either an optimization or
> a deliberate choice to not DEBUG1 log skipping temporary files. I didn't go
> digging in the archives to find the corresponding thread but there might be
> clues to be had there.
FWIW, here is the thread:
https://www.postgresql.org/message-id/CAB7nPqSNFm2Lz6jASj1RGvAdod1W8ZmHfvML3M7gDnBQ3x6QMw@mail.gmail...
I think that you're right, the idea is to avoid the random noise
caused by these temp files and their names. This elog has been useful
for debugging in the past for the fixed entries, at least for me.
While on it, it strikes me that we should have a check on
PG_TEMP_FILES_DIR in basebackup.c's sendDir()? Okay, that's the same
as PG_TEMP_FILE_PREFIX, but pg_checksums and pg_rewind check for
*both* patterns so that feels inconsistent to me. This should not be
in excludeDirContents, because we don't want an empty folder in this
case.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZEtahOGT60kql%2FrS@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-28 12:01 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-28 12:01 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 28 Apr 2023, at 07:32, Michael Paquier <michael@paquier.xyz> wrote:
> On Thu, Apr 27, 2023 at 04:22:08PM +0200, Daniel Gustafsson wrote:
>> The commit message for 6ad8ac60262 doesn't explain why pgsql_tmp was left out
>> of the excludeFiles table, but my guess is that it's either an optimization or
>> a deliberate choice to not DEBUG1 log skipping temporary files. I didn't go
>> digging in the archives to find the corresponding thread but there might be
>> clues to be had there.
>
> FWIW, here is the thread:
> https://www.postgresql.org/message-id/CAB7nPqSNFm2Lz6jASj1RGvAdod1W8ZmHfvML3M7gDnBQ3x6QMw@mail.gmail...
>
> I think that you're right, the idea is to avoid the random noise
> caused by these temp files and their names. This elog has been useful
> for debugging in the past for the fixed entries, at least for me.
Aha, thanks for the digging!
> While on it, it strikes me that we should have a check on
> PG_TEMP_FILES_DIR in basebackup.c's sendDir()? Okay, that's the same
> as PG_TEMP_FILE_PREFIX, but pg_checksums and pg_rewind check for
> *both* patterns so that feels inconsistent to me. This should not be
> in excludeDirContents, because we don't want an empty folder in this
> case.
That makes sense, even though it's a bit duplicative as it stands in core
today. Given that these *can* be different at some point in the future (or in
a fork), we should check both of course. Attached v5 does that as well as
incorporates a version of the doc change proposed in v4 upthread.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] skip_hidden_files_v5.diff (6.8K, ../../D2C15F6D-D133-43D4-BC46-40120E605611@yesql.se/2-skip_hidden_files_v5.diff)
download | inline diff:
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index b11d9a6ba3..08f740f6ca 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3048,7 +3048,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<listitem>
<para>
Files other than regular files and directories, such as symbolic
- links (other than for the directories listed above) and special
+ links (other than for the directories listed above), hidden files and special
device files, are skipped. (Symbolic links
in <filename>pg_tblspc</filename> are maintained.)
</para>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 79d3e657c3..a7c8ed7370 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -909,7 +909,7 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and hidden files. But only regular files and directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 9118f05bf2..005fa83b9e 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -381,8 +381,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename>, and
+ hidden files as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 5baea7535b..e3d041c35d 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1196,10 +1196,13 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0)
continue;
- /* Skip temporary files */
- if (strncmp(de->d_name,
- PG_TEMP_FILE_PREFIX,
- strlen(PG_TEMP_FILE_PREFIX)) == 0)
+ /* Skip temporary files and folders */
+ if (strstr(de->d_name, PG_TEMP_FILE_PREFIX) != NULL ||
+ strstr(de->d_name, PG_TEMP_FILES_DIR "/") != NULL)
+ continue;
+
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
continue;
/*
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 4d130a7f94..95dbcf7338 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -173,7 +173,7 @@ SKIP:
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
- current_logfiles.tmp global/pg_internal.init.123))
+ current_logfiles.tmp global/pg_internal.init.123 pgsql_tmp.donotcopy .donotcopy))
{
open my $file, '>>', "$pgdata/$filename";
print $file "DONOTCOPY";
@@ -244,7 +244,7 @@ foreach my $dirname (
# These files should not be copied.
foreach my $filename (
qw(postgresql.auto.conf.tmp postmaster.opts postmaster.pid tablespace_map current_logfiles.tmp
- global/pg_internal.init global/pg_internal.init.123))
+ global/pg_internal.init global/pg_internal.init.123 pgsql_tmp.donotcopy .donotcopy))
{
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..68c255877f 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -325,16 +325,13 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strcmp(de->d_name, "..") == 0)
continue;
- /* Skip temporary files */
- if (strncmp(de->d_name,
- PG_TEMP_FILE_PREFIX,
- strlen(PG_TEMP_FILE_PREFIX)) == 0)
+ /* Skip temporary files and folders */
+ if (strstr(de->d_name, PG_TEMP_FILE_PREFIX) != NULL ||
+ strstr(de->d_name, PG_TEMP_FILES_DIR "/") != NULL)
continue;
- /* Skip temporary folders */
- if (strncmp(de->d_name,
- PG_TEMP_FILES_DIR,
- strlen(PG_TEMP_FILES_DIR)) == 0)
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
continue;
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index 2316f611b2..ff975e5601 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -114,6 +114,9 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres files, which should be ignored by the scan
+append_to_file "$pgdata/global/.donotscan", "foo";
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/file_ops.c b/src/bin/pg_rewind/file_ops.c
index 25996b4da4..77ff8ac5d2 100644
--- a/src/bin/pg_rewind/file_ops.c
+++ b/src/bin/pg_rewind/file_ops.c
@@ -398,6 +398,10 @@ recurse_dir(const char *datadir, const char *parentpath,
strcmp(xlde->d_name, "..") == 0)
continue;
+ /* Skip hidden files */
+ if (xlde->d_name[0] == '.')
+ continue;
+
snprintf(fullpath, sizeof(fullpath), "%s/%s", fullparentpath, xlde->d_name);
if (lstat(fullpath, &fst) < 0)
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index fd2bee5d20..95ae15f946 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -35,6 +35,9 @@ sub run_test
append_to_file
"$test_primary_datadir/tst_both_dir/both_subdir/both_file3",
"in both3";
+ append_to_file
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
+ "in both4";
RewindTest::create_standby($test_mode);
@@ -82,6 +85,7 @@ sub run_test
\@paths,
[
"$test_primary_datadir/tst_both_dir",
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
"$test_primary_datadir/tst_both_dir/both_file1",
"$test_primary_datadir/tst_both_dir/both_file2",
"$test_primary_datadir/tst_both_dir/both_subdir",
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-04-28 12:06 ` Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-04-28 12:06 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Fri, Apr 28, 2023 at 02:01:47PM +0200, Daniel Gustafsson wrote:
> That makes sense, even though it's a bit duplicative as it stands in core
> today. Given that these *can* be different at some point in the future (or in
> a fork), we should check both of course.
Yes, who knows what the future is made of.
> Attached v5 does that as well as incorporates a version of the doc
> change proposed in v4 upthread.
Looks rather OK on a quick pass.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZEu21Ud7FitLiu7F@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-28 12:11 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-28 12:11 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 28 Apr 2023, at 14:06, Michael Paquier <michael@paquier.xyz> wrote:
> On Fri, Apr 28, 2023 at 02:01:47PM +0200, Daniel Gustafsson wrote:
>> Attached v5 does that as well as incorporates a version of the doc
>> change proposed in v4 upthread.
>
> Looks rather OK on a quick pass.
Thanks! I've parked it in the July commitfest to keep it from being forgotten
about when the tree re-opens.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-04-28 23:42 ` Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-04-28 23:42 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Fri, Apr 28, 2023 at 02:11:00PM +0200, Daniel Gustafsson wrote:
> Thanks! I've parked it in the July commitfest to keep it from being forgotten
> about when the tree re-opens.
I'm OK to hold on this patch until v17 opens for business, but isn't
that something we'd better backpatch at some point? The behavior of
the stable branches is kind of annoying, in my opinion.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZExZ510ZcfpZ5VdE@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-29 20:35 ` Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-29 20:35 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 29 Apr 2023, at 01:42, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Apr 28, 2023 at 02:11:00PM +0200, Daniel Gustafsson wrote:
>> Thanks! I've parked it in the July commitfest to keep it from being forgotten
>> about when the tree re-opens.
>
> I'm OK to hold on this patch until v17 opens for business, but isn't
> that something we'd better backpatch at some point? The behavior of
> the stable branches is kind of annoying, in my opinion.
My thinking is that skipping non-postgres files which really have no business
being there in the first place is a new feature and not a bugfix, but I can see
merit in your argument too. If the RMT and -hackers have other opinions then
I'm not going to object.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-05-01 05:25 ` Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-05-01 05:25 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Sat, Apr 29, 2023 at 10:35:52PM +0200, Daniel Gustafsson wrote:
> My thinking is that skipping non-postgres files which really have no business
> being there in the first place is a new feature and not a bugfix, but I can see
> merit in your argument too. If the RMT and -hackers have other opinions then
> I'm not going to object.
I am fine to hear more opinions, then. .DS_Store is created in a
directory when browsed by Finder, from what I can read, and there's
been few complaints across the years about that.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZE9NXovq7cNLfPnM@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-05-01 21:23 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Tobias Bussmann @ 2023-05-01 21:23 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Daniel Gustafsson <daniel@yesql.se>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
>> [...] is a new feature and not a bugfix [...] other opinions [...]
> I am fine to hear more opinions, then.
IMHO, while generally ignoring hidden files can be seen as a change in behaviour and a new feature, specifically skipping the '.DS_Store' files is clearly a bugfix that should be backported. With checksumming enabled, pg_basebackup is simply broken on macOS, depending on whether or not you have ever browsed parts of the data directory in the Finder.
The safe solution for the back branches would be to add '{".DS_Store", false}' to the exclude_list_item arrays in backend/backup/basebackup.c, bin/pg_checksums/pg_checksums.c and bin/pg_rewind/filemap.c, but I can't judge whether this would justify using a different codepath for the older versions or not.
--
Tobias Bussmann
Attachments:
[application/pgp-signature] signature.asc (832B, ../../817432D1-864B-449F-B895-3A829ADE2A98@gmx.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
@ 2023-05-02 10:34 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-05-02 10:34 UTC (permalink / raw)
To: Tobias Bussmann <t.bussmann@gmx.net>; +Cc: Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 1 May 2023, at 23:23, Tobias Bussmann <t.bussmann@gmx.net> wrote:
> The safe solution for the back branches would be to add '{".DS_Store", false}' to the exclude_list_item arrays in backend/backup/basebackup.c, bin/pg_checksums/pg_checksums.c and bin/pg_rewind/filemap.c, but I can't judge whether this would justify using a different codepath for the older versions or not.
If we deem this to be a bugfix then we should backpatch the solution from HEAD
imho. So far there are mainly voices for considering it a bugfix, but given
where we are in the cycle I the RMT has the final say on that.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-07-06 21:14 ` Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 34+ messages in thread
From: Jacob Champion @ 2023-07-06 21:14 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tobias Bussmann <t.bussmann@gmx.net>; Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Tue, May 2, 2023 at 3:35 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> If we deem this to be a bugfix then we should backpatch the solution from HEAD
> imho. So far there are mainly voices for considering it a bugfix, but given
> where we are in the cycle I the RMT has the final say on that.
(Was there an RMT decision?)
My two cents: backpatching an exception for .DS_Store seems good, but
creating an exclusion for all hidden files feels like a surprising
change for stable branches. If your postgresql.conf has something like
include '.my.conf'
then a basebackup of that setup seems to work okay today.
--Jacob
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
@ 2023-07-06 23:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2023-07-06 23:05 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: Daniel Gustafsson <daniel@yesql.se>; Tobias Bussmann <t.bussmann@gmx.net>; Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
Jacob Champion <jchampion@timescale.com> writes:
> My two cents: backpatching an exception for .DS_Store seems good, but
> creating an exclusion for all hidden files feels like a surprising
> change for stable branches.
On the whole, I think I'd vote for blocking .DS_Store only, even
in HEAD. (IIRC, I thought differently to start with, but today
I'm feeling conservative about it.) If there are other special
file names on other platforms, we could add some more targeted
exceptions; but dropping all hidden files seems more likely to
break things than be helpful.
regards, tom lane
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
@ 2023-07-07 07:23 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-07-07 07:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 7 Jul 2023, at 01:05, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Jacob Champion <jchampion@timescale.com> writes:
>> My two cents: backpatching an exception for .DS_Store seems good, but
>> creating an exclusion for all hidden files feels like a surprising
>> change for stable branches.
>
> On the whole, I think I'd vote for blocking .DS_Store only, even
> in HEAD. (IIRC, I thought differently to start with, but today
> I'm feeling conservative about it.) If there are other special
> file names on other platforms, we could add some more targeted
> exceptions; but dropping all hidden files seems more likely to
> break things than be helpful.
I think the case for skipping all hidden files is that it would offer more
consistency with other serverside filesystem reads are performed. After
.DS_Store I would think that editor swapfiles would be other likely culprit of
hidden-but-not-belonging files.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-07-08 00:05 ` Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-07-08 00:05 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Fri, Jul 07, 2023 at 09:23:38AM +0200, Daniel Gustafsson wrote:
> On 7 Jul 2023, at 01:05, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> On the whole, I think I'd vote for blocking .DS_Store only, even
>> in HEAD. (IIRC, I thought differently to start with, but today
>> I'm feeling conservative about it.) If there are other special
>> file names on other platforms, we could add some more targeted
>> exceptions; but dropping all hidden files seems more likely to
>> break things than be helpful.
>
> I think the case for skipping all hidden files is that it would offer more
> consistency with other serverside filesystem reads are performed. After
> .DS_Store I would think that editor swapfiles would be other likely culprit of
> hidden-but-not-belonging files.
.DS_Store is not the only hidden file pattern that could be used by a
filesystem for its metadata. And I don't quite see what we gain by
only ignoring it, letting the others be. My take would be to just
ignore all of them, and I'm OK even if it means to do so only on
HEAD per the argument of being careful with stable branches.
I cannot get excited about the potential use case where someone
decides that it would be a good idea to include a hidden file, as
well. It's not like it offers any kind of additional protection.
Actually, it can make the debugging of a configuration a worse
experience.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZKioRpLcqW1oqWhO@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-07-14 13:07 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-07-14 13:07 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 8 Jul 2023, at 02:05, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Jul 07, 2023 at 09:23:38AM +0200, Daniel Gustafsson wrote:
>> On 7 Jul 2023, at 01:05, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> On the whole, I think I'd vote for blocking .DS_Store only, even
>>> in HEAD. (IIRC, I thought differently to start with, but today
>>> I'm feeling conservative about it.) If there are other special
>>> file names on other platforms, we could add some more targeted
>>> exceptions; but dropping all hidden files seems more likely to
>>> break things than be helpful.
>>
>> I think the case for skipping all hidden files is that it would offer more
>> consistency with other serverside filesystem reads are performed. After
>> .DS_Store I would think that editor swapfiles would be other likely culprit of
>> hidden-but-not-belonging files.
>
> .DS_Store is not the only hidden file pattern that could be used by a
> filesystem for its metadata. And I don't quite see what we gain by
> only ignoring it, letting the others be. My take would be to just
> ignore all of them, and I'm OK even if it means to do so only on
> HEAD per the argument of being careful with stable branches.
Judging by the thread there seems to be concensus on skipping .DS_Store files
in all branches, with a few +1's for skipping all hidden files in HEAD. I'll
prepare a patch for the former and we can pick up the discussion on the latter
ones that's done.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-07-17 08:29 ` Michael Paquier <michael@paquier.xyz>
2023-07-19 23:16 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-07-17 08:29 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Fri, Jul 14, 2023 at 03:07:08PM +0200, Daniel Gustafsson wrote:
> Judging by the thread there seems to be concensus on skipping .DS_Store files
> in all branches, with a few +1's for skipping all hidden files in HEAD. I'll
> prepare a patch for the former and we can pick up the discussion on the latter
> ones that's done.
Yes, I am also getting that this is the consensus reached here.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZLT73vi21KQBesD1@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-07-19 23:16 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-20 07:47 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-07-19 23:16 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 17 Jul 2023, at 10:29, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Jul 14, 2023 at 03:07:08PM +0200, Daniel Gustafsson wrote:
>> Judging by the thread there seems to be concensus on skipping .DS_Store files
>> in all branches, with a few +1's for skipping all hidden files in HEAD. I'll
>> prepare a patch for the former and we can pick up the discussion on the latter
>> ones that's done.
>
> Yes, I am also getting that this is the consensus reached here.
Attached is a version that just skips .DS_Store files as per the discussion
here. I opted for not testing this on macOS since I don't want to create
invalid .DS_Store files where they potentially can be seen by the Finder
causing unwanted side effects.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] v6-0001-Skip-.DS_Store-files-in-server-side-utils.patch (7.4K, ../../5E11C21A-D3D9-4655-B208-4BFF310AF551@yesql.se/2-v6-0001-Skip-.DS_Store-files-in-server-side-utils.patch)
download | inline diff:
From b9128e948fb85505f5bdbc097ea9c972c837ce8b Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 20 Jul 2023 00:39:39 +0200
Subject: [PATCH v6] Skip .DS_Store files in server side utils
The macOS Finder application creates .DS_Store files in directories
when opened, which creates problems for serverside utilities which
expect all files to be PostgreSQL specific files. Skip these files
when encountered in pg_checksums, pg_rewind and pg_basebackup.
Reported-by: Mark Guertin <markguertin@gmail.com>
Discussion: https://postgr.es/m/E258CE50-AB0E-455D-8AAD-BB4FE8F882FB@gmail.com
---
doc/src/sgml/protocol.sgml | 2 +-
doc/src/sgml/ref/pg_basebackup.sgml | 3 ++-
doc/src/sgml/ref/pg_rewind.sgml | 5 +++--
src/backend/backup/basebackup.c | 4 ++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 17 +++++++++++++++++
src/bin/pg_checksums/pg_checksums.c | 4 ++++
src/bin/pg_checksums/t/002_actions.pl | 7 +++++++
src/bin/pg_rewind/file_ops.c | 4 ++++
8 files changed, 42 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index b11d9a6ba3..d8765100e3 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3049,7 +3049,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
Files other than regular files and directories, such as symbolic
links (other than for the directories listed above) and special
- device files, are skipped. (Symbolic links
+ device and operating system files, are skipped. (Symbolic links
in <filename>pg_tblspc</filename> are maintained.)
</para>
</listitem>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 79d3e657c3..c2fd27ca64 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -909,7 +909,8 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and operating system files. But only regular files and
+ directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 15cddd086b..2de747ec37 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -381,8 +381,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename> and
+ <filename>.DS_Store</filename> as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 45be21131c..52f0cd3be1 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1202,6 +1202,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(de->d_name, ".DS_Store") == 0)
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b9f5e1266b..c8cef688d4 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -3,6 +3,7 @@
use strict;
use warnings;
+use Config;
use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
@@ -179,6 +180,16 @@ foreach my $filename (
close $file;
}
+# Test that macOS system files are skipped. Only test on non-macOS systems
+# however since creating incorrect .DS_Store files on a macOS system may have
+# unintended side effects.
+if ($Config{osname} ne 'darwin')
+{
+ open my $file, '>>', "$pgdata/.DS_Store";
+ print $file "DONOTCOPY";
+ close $file;
+}
+
# Connect to a database to create global/pg_internal.init. If this is removed
# the test to ensure global/pg_internal.init is not copied will return a false
# positive.
@@ -248,6 +259,12 @@ foreach my $filename (
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
+# We only test .DS_Store files being skipped on non-macOS systems
+if ($Config{osname} ne 'darwin')
+{
+ ok(!-f "$tempdir/backup/.DS_Store", ".DS_Store not copied");
+}
+
# Unlogged relation forks other than init should not be copied
ok(-f "$tempdir/backup/${baseUnloggedPath}_init",
'unlogged init fork in backup');
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..64b31d57d8 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -337,6 +337,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(de->d_name, ".DS_Store") == 0)
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index 2d63182d59..1d4d39db58 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -6,6 +6,7 @@
use strict;
use warnings;
+use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
@@ -114,6 +115,12 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres macOS files, which should be ignored by the scan.
+# Only perform this test on non-macOS systems though as creating incorrect
+# system files may have side effects on macOS.
+append_to_file "$pgdata/global/.DS_Store", "foo"
+ unless ($Config{osname} eq 'darwin');
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/file_ops.c b/src/bin/pg_rewind/file_ops.c
index 25996b4da4..f7a2abb48e 100644
--- a/src/bin/pg_rewind/file_ops.c
+++ b/src/bin/pg_rewind/file_ops.c
@@ -398,6 +398,10 @@ recurse_dir(const char *datadir, const char *parentpath,
strcmp(xlde->d_name, "..") == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(xlde->d_name, ".DS_Store") == 0)
+ continue;
+
snprintf(fullpath, sizeof(fullpath), "%s/%s", fullparentpath, xlde->d_name);
if (lstat(fullpath, &fst) < 0)
--
2.32.1 (Apple Git-133)
=
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-19 23:16 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-07-20 07:47 ` Michael Paquier <michael@paquier.xyz>
2023-07-20 10:36 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-07-20 07:47 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Thu, Jul 20, 2023 at 01:16:45AM +0200, Daniel Gustafsson wrote:
> Attached is a version that just skips .DS_Store files as per the discussion
> here. I opted for not testing this on macOS since I don't want to create
> invalid .DS_Store files where they potentially can be seen by the Finder
> causing unwanted side effects.
@@ -398,6 +398,10 @@ recurse_dir(const char *datadir, const char *parentpath,
strcmp(xlde->d_name, "..") == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(xlde->d_name, ".DS_Store") == 0)
+ continue;
Hmm. Are you sure that this is correct for pg_rewind? This should
work when using a local source with --source-pgdata, but under a
remote source with --source-pgdata, I get the impression that we'd
still copy a .DS_store into the target's data folder.
For example, see this little tweak in 003_extrafiles.pl that adds a
.DS_Store in the standby:
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -53,6 +53,9 @@ sub run_test
append_to_file
"$test_standby_datadir/tst_standby_dir/standby_subdir/standby_file4",
"in standby4";
+ append_to_file
+ "$test_standby_datadir/tst_standby_dir/.DS_Store",
+ "in standby4";
Note that I have not updated the list associated to is_deeply(), to
check that this file is not copied over to the primary. This leads me
to a failure of the test for the remote case. Then, looking from
src/bin/pg_rewind/:
$ find . -name ".DS_Store"
./tmp_check/t_003_extrafiles_primary_remote_data/pgdata/tst_standby_dir/.DS_Store
./tmp_check/t_003_extrafiles_standby_local_data/pgdata/tst_standby_dir/.DS_Store
./tmp_check/t_003_extrafiles_standby_remote_data/pgdata/tst_standby_dir/.DS_Store
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZLjmnOdz5jT+2s4g@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-19 23:16 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-20 07:47 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-07-20 10:36 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-20 23:53 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-07-20 10:36 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 20 Jul 2023, at 09:47, Michael Paquier <michael@paquier.xyz> wrote:
> Hmm. Are you sure that this is correct for pg_rewind? This should
> work when using a local source with --source-pgdata, but under a
> remote source with --source-pgdata, I get the impression that we'd
> still copy a .DS_store into the target's data folder.
Ah, good point. The updated patch should work for both by moving the check
into the filemap.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] v7-0001-Skip-.DS_Store-files-in-server-side-utils.patch (8.2K, ../../3D7967A7-CB8A-428C-BBEA-D87AAA875055@yesql.se/2-v7-0001-Skip-.DS_Store-files-in-server-side-utils.patch)
download | inline diff:
From dbeb1b70c900786f5f7bcf60f19fd5928248c39d Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 20 Jul 2023 00:39:39 +0200
Subject: [PATCH v7] Skip .DS_Store files in server side utils
The macOS Finder application creates .DS_Store files in directories
when opened, which creates problems for serverside utilities which
expect all files to be PostgreSQL specific files. Skip these files
when encountered in pg_checksums, pg_rewind and pg_basebackup.
Reported-by: Mark Guertin <markguertin@gmail.com>
Discussion: https://postgr.es/m/E258CE50-AB0E-455D-8AAD-BB4FE8F882FB@gmail.com
---
doc/src/sgml/protocol.sgml | 2 +-
doc/src/sgml/ref/pg_basebackup.sgml | 3 ++-
doc/src/sgml/ref/pg_rewind.sgml | 5 +++--
src/backend/backup/basebackup.c | 4 ++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 17 +++++++++++++++++
src/bin/pg_checksums/pg_checksums.c | 4 ++++
src/bin/pg_checksums/t/002_actions.pl | 7 +++++++
src/bin/pg_rewind/filemap.c | 4 ++++
src/bin/pg_rewind/t/003_extrafiles.pl | 5 +++++
9 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index b11d9a6ba3..d8765100e3 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3049,7 +3049,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
Files other than regular files and directories, such as symbolic
links (other than for the directories listed above) and special
- device files, are skipped. (Symbolic links
+ device and operating system files, are skipped. (Symbolic links
in <filename>pg_tblspc</filename> are maintained.)
</para>
</listitem>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 79d3e657c3..c2fd27ca64 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -909,7 +909,8 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and operating system files. But only regular files and
+ directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 15cddd086b..2de747ec37 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -381,8 +381,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename> and
+ <filename>.DS_Store</filename> as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 45be21131c..52f0cd3be1 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1202,6 +1202,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(de->d_name, ".DS_Store") == 0)
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b9f5e1266b..c8cef688d4 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -3,6 +3,7 @@
use strict;
use warnings;
+use Config;
use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
@@ -179,6 +180,16 @@ foreach my $filename (
close $file;
}
+# Test that macOS system files are skipped. Only test on non-macOS systems
+# however since creating incorrect .DS_Store files on a macOS system may have
+# unintended side effects.
+if ($Config{osname} ne 'darwin')
+{
+ open my $file, '>>', "$pgdata/.DS_Store";
+ print $file "DONOTCOPY";
+ close $file;
+}
+
# Connect to a database to create global/pg_internal.init. If this is removed
# the test to ensure global/pg_internal.init is not copied will return a false
# positive.
@@ -248,6 +259,12 @@ foreach my $filename (
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
+# We only test .DS_Store files being skipped on non-macOS systems
+if ($Config{osname} ne 'darwin')
+{
+ ok(!-f "$tempdir/backup/.DS_Store", ".DS_Store not copied");
+}
+
# Unlogged relation forks other than init should not be copied
ok(-f "$tempdir/backup/${baseUnloggedPath}_init",
'unlogged init fork in backup');
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..64b31d57d8 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -337,6 +337,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(de->d_name, ".DS_Store") == 0)
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index 2d63182d59..1d4d39db58 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -6,6 +6,7 @@
use strict;
use warnings;
+use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
@@ -114,6 +115,12 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres macOS files, which should be ignored by the scan.
+# Only perform this test on non-macOS systems though as creating incorrect
+# system files may have side effects on macOS.
+append_to_file "$pgdata/global/.DS_Store", "foo"
+ unless ($Config{osname} eq 'darwin');
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index bd5c598e20..435742d20d 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -647,6 +647,10 @@ decide_file_action(file_entry_t *entry)
if (strcmp(path, "global/pg_control") == 0)
return FILE_ACTION_NONE;
+ /* Skip macOS system files */
+ if (strstr(path, ".DS_Store") != NULL)
+ return FILE_ACTION_NONE;
+
/*
* Remove all files matching the exclusion filters in the target.
*/
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index fd2bee5d20..77438341d6 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -5,6 +5,7 @@
use strict;
use warnings;
+use Config;
use PostgreSQL::Test::Utils;
use Test::More;
@@ -53,6 +54,10 @@ sub run_test
append_to_file
"$test_standby_datadir/tst_standby_dir/standby_subdir/standby_file4",
"in standby4";
+ # Skip testing .DS_Store files on macOS to avoid risk of side effects
+ append_to_file
+ "$test_standby_datadir/tst_standby_dir/.DS_Store",
+ "in standby4" unless ($Config{osname} eq 'darwin');
mkdir "$test_primary_datadir/tst_primary_dir";
append_to_file "$test_primary_datadir/tst_primary_dir/primary_file1",
--
2.32.1 (Apple Git-133)
=
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-19 23:16 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-20 07:47 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-20 10:36 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-07-20 23:53 ` Michael Paquier <michael@paquier.xyz>
2024-01-17 14:37 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 34+ messages in thread
From: Michael Paquier @ 2023-07-20 23:53 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
On Thu, Jul 20, 2023 at 12:36:52PM +0200, Daniel Gustafsson wrote:
> On 20 Jul 2023, at 09:47, Michael Paquier <michael@paquier.xyz> wrote:
>> Hmm. Are you sure that this is correct for pg_rewind? This should
>> work when using a local source with --source-pgdata, but under a
>> remote source with --source-pgdata, I get the impression that we'd
>> still copy a .DS_store into the target's data folder.
>
> Ah, good point. The updated patch should work for both by moving the check
> into the filemap.
Sounds OK to me to do that in the filemap. The checksum and base
backup parts look OK as well. Thanks for the quick update!
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZLnI3vLFVnpXj0sE@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-19 23:16 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-20 07:47 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-07-20 10:36 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-07-20 23:53 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2024-01-17 14:37 ` Daniel Gustafsson <daniel@yesql.se>
0 siblings, 0 replies; 34+ messages in thread
From: Daniel Gustafsson @ 2024-01-17 14:37 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jacob Champion <jchampion@timescale.com>; Tobias Bussmann <t.bussmann@gmx.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 21 Jul 2023, at 01:53, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Jul 20, 2023 at 12:36:52PM +0200, Daniel Gustafsson wrote:
>> On 20 Jul 2023, at 09:47, Michael Paquier <michael@paquier.xyz> wrote:
>>> Hmm. Are you sure that this is correct for pg_rewind? This should
>>> work when using a local source with --source-pgdata, but under a
>>> remote source with --source-pgdata, I get the impression that we'd
>>> still copy a .DS_store into the target's data folder.
>>
>> Ah, good point. The updated patch should work for both by moving the check
>> into the filemap.
>
> Sounds OK to me to do that in the filemap. The checksum and base
> backup parts look OK as well. Thanks for the quick update!
Reviving an old thread which had fallen off the end of the TODO. The attached
is a rebase over nearby changes causing conflicts but no functional changes
from the previous which was marked ready for committer, so will go ahead with
this once it's been through a CI cycle in the cfbot.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] v8-0001-Skip-.DS_Store-files-in-server-side-utils.patch (8.2K, ../../6408F630-3533-4D98-B0FE-89D6AED5AB52@yesql.se/2-v8-0001-Skip-.DS_Store-files-in-server-side-utils.patch)
download | inline diff:
From 90c60d5e05b21a633fe9c378e41feb66367a91d7 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 20 Jul 2023 00:39:39 +0200
Subject: [PATCH v8] Skip .DS_Store files in server side utils
The macOS Finder application creates .DS_Store files in directories
when opened, which creates problems for serverside utilities which
expect all files to be PostgreSQL specific files. Skip these files
when encountered in pg_checksums, pg_rewind and pg_basebackup.
Reported-by: Mark Guertin <markguertin@gmail.com>
Discussion: https://postgr.es/m/E258CE50-AB0E-455D-8AAD-BB4FE8F882FB@gmail.com
---
doc/src/sgml/protocol.sgml | 2 +-
doc/src/sgml/ref/pg_basebackup.sgml | 3 ++-
doc/src/sgml/ref/pg_rewind.sgml | 5 +++--
src/backend/backup/basebackup.c | 4 ++++
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 17 +++++++++++++++++
src/bin/pg_checksums/pg_checksums.c | 4 ++++
src/bin/pg_checksums/t/002_actions.pl | 7 +++++++
src/bin/pg_rewind/filemap.c | 4 ++++
src/bin/pg_rewind/t/003_extrafiles.pl | 5 +++++
9 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 6c3e8a631d..b33a530b94 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3080,7 +3080,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
Files other than regular files and directories, such as symbolic
links (other than for the directories listed above) and special
- device files, are skipped. (Symbolic links
+ device and operating system files, are skipped. (Symbolic links
in <filename>pg_tblspc</filename> are maintained.)
</para>
</listitem>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index e411ddbf45..88c689e725 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -965,7 +965,8 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and operating system files. But only regular files and
+ directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 8e0000d39f..841bd49fc7 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -403,8 +403,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename> and
+ <filename>.DS_Store</filename> as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index d5b8ca21b7..c7a2cbb13e 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1265,6 +1265,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(de->d_name, ".DS_Store") == 0)
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 42a09d0da3..86cc01a640 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -3,6 +3,7 @@
use strict;
use warnings FATAL => 'all';
+use Config;
use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
@@ -179,6 +180,16 @@ foreach my $filename (
close $file;
}
+# Test that macOS system files are skipped. Only test on non-macOS systems
+# however since creating incorrect .DS_Store files on a macOS system may have
+# unintended side effects.
+if ($Config{osname} ne 'darwin')
+{
+ open my $file, '>>', "$pgdata/.DS_Store";
+ print $file "DONOTCOPY";
+ close $file;
+}
+
# Connect to a database to create global/pg_internal.init. If this is removed
# the test to ensure global/pg_internal.init is not copied will return a false
# positive.
@@ -248,6 +259,12 @@ foreach my $filename (
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
+# We only test .DS_Store files being skipped on non-macOS systems
+if ($Config{osname} ne 'darwin')
+{
+ ok(!-f "$tempdir/backup/.DS_Store", ".DS_Store not copied");
+}
+
# Unlogged relation forks other than init should not be copied
ok(-f "$tempdir/backup/${baseUnloggedPath}_init",
'unlogged init fork in backup');
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 6fc1326418..9e6fd435f6 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -329,6 +329,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip macOS system files */
+ if (strcmp(de->d_name, ".DS_Store") == 0)
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index f83cf67a0f..2faf2f4590 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -6,6 +6,7 @@
use strict;
use warnings FATAL => 'all';
+use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
@@ -114,6 +115,12 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres macOS files, which should be ignored by the scan.
+# Only perform this test on non-macOS systems though as creating incorrect
+# system files may have side effects on macOS.
+append_to_file "$pgdata/global/.DS_Store", "foo"
+ unless ($Config{osname} eq 'darwin');
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 0894c317bf..255ddf2ffa 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -647,6 +647,10 @@ decide_file_action(file_entry_t *entry)
if (strcmp(path, "global/pg_control") == 0)
return FILE_ACTION_NONE;
+ /* Skip macOS system files */
+ if (strstr(path, ".DS_Store") != NULL)
+ return FILE_ACTION_NONE;
+
/*
* Remove all files matching the exclusion filters in the target.
*/
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d54dcddcbb..8405cd2de7 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -5,6 +5,7 @@
use strict;
use warnings FATAL => 'all';
+use Config;
use PostgreSQL::Test::Utils;
use Test::More;
@@ -53,6 +54,10 @@ sub run_test
append_to_file
"$test_standby_datadir/tst_standby_dir/standby_subdir/standby_file4",
"in standby4";
+ # Skip testing .DS_Store files on macOS to avoid risk of side effects
+ append_to_file
+ "$test_standby_datadir/tst_standby_dir/.DS_Store",
+ "in standby4" unless ($Config{osname} eq 'darwin');
mkdir "$test_primary_datadir/tst_primary_dir";
append_to_file "$test_primary_datadir/tst_primary_dir/primary_file1",
--
2.32.1 (Apple Git-133)
=
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
@ 2023-04-27 09:08 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-27 15:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
1 sibling, 1 reply; 34+ messages in thread
From: Daniel Gustafsson @ 2023-04-27 09:08 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tobias Bussmann <t.bussmann@gmx.net>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> On 21 Apr 2023, at 00:33, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Apr 20, 2023 at 01:15:40PM +0200, Daniel Gustafsson wrote:
>> Maybe. I'm a bit hesitant to add too many smarts to pg_rewind. It's a tool
>> for when something has gone wrong with a cluster (albeit probably not at the
>> filesystem level), and at that point I feel it's better to put the user fully
>> in charge. Perhaps I'm overly cautious, curious to hear from others.
>
> Hmm. pg_rewind is mostly a differential block-level backup tool, so
> applying the same rules everywhere across the board would be sensible
> here. See that exclude_list_item is able to handle prefixes, and we
> may want to extend the same logic for the directory list, as well..
>
> By the way, the patch ought to add some tests? For pg_basebackup,
> this would be around "These files should not be copied" in
> 010_pg_basebackup.pl. pg_checksums has also its own checks in
> 002_actions.pl.
Skipping hidden files in pg_rewind added as well as tests for all three
utilities and mentions of this in the docs. I'll park this in the next
commitfest for now.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] skip_hidden_files_v2.diff (5.6K, ../../7E3C671C-F9DA-42D9-8F1D-447A20F48537@yesql.se/2-skip_hidden_files_v2.diff)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 79d3e657c3..a7c8ed7370 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -909,7 +909,7 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and hidden files. But only regular files and directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 9118f05bf2..005fa83b9e 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -381,8 +381,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename>, and
+ hidden files as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 5baea7535b..75b6af80cb 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1202,6 +1202,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 4d130a7f94..ea22d439ae 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -173,7 +173,7 @@ SKIP:
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
- current_logfiles.tmp global/pg_internal.init.123))
+ current_logfiles.tmp global/pg_internal.init.123 .donotcopy))
{
open my $file, '>>', "$pgdata/$filename";
print $file "DONOTCOPY";
@@ -244,7 +244,7 @@ foreach my $dirname (
# These files should not be copied.
foreach my $filename (
qw(postgresql.auto.conf.tmp postmaster.opts postmaster.pid tablespace_map current_logfiles.tmp
- global/pg_internal.init global/pg_internal.init.123))
+ global/pg_internal.init global/pg_internal.init.123 .donotcopy))
{
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..d3c16acf0d 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -337,6 +337,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index 2316f611b2..ff975e5601 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -114,6 +114,9 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres files, which should be ignored by the scan
+append_to_file "$pgdata/global/.donotscan", "foo";
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/file_ops.c b/src/bin/pg_rewind/file_ops.c
index 25996b4da4..77ff8ac5d2 100644
--- a/src/bin/pg_rewind/file_ops.c
+++ b/src/bin/pg_rewind/file_ops.c
@@ -398,6 +398,10 @@ recurse_dir(const char *datadir, const char *parentpath,
strcmp(xlde->d_name, "..") == 0)
continue;
+ /* Skip hidden files */
+ if (xlde->d_name[0] == '.')
+ continue;
+
snprintf(fullpath, sizeof(fullpath), "%s/%s", fullparentpath, xlde->d_name);
if (lstat(fullpath, &fst) < 0)
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index fd2bee5d20..95ae15f946 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -35,6 +35,9 @@ sub run_test
append_to_file
"$test_primary_datadir/tst_both_dir/both_subdir/both_file3",
"in both3";
+ append_to_file
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
+ "in both4";
RewindTest::create_standby($test_mode);
@@ -82,6 +85,7 @@ sub run_test
\@paths,
[
"$test_primary_datadir/tst_both_dir",
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
"$test_primary_datadir/tst_both_dir/both_file1",
"$test_primary_datadir/tst_both_dir/both_file2",
"$test_primary_datadir/tst_both_dir/both_subdir",
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-27 09:08 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
@ 2023-04-27 15:34 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-04-27 15:48 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
0 siblings, 1 reply; 34+ messages in thread
From: Tobias Bussmann @ 2023-04-27 15:34 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> Skipping hidden files in pg_rewind added as well as tests for all three
> utilities and mentions of this in the docs. I'll park this in the next
> commitfest for now.
thanks for the update!
I gave the v2 patch a quick review:
* it applies and builds cleanly
* the new TAP tests pass with the changes and fail without
* in a real world test on macOS, the issue with the .DS_Store files on macOS is no longer showing with the patch and reproducible without
* the documentation changes apply and are understandable. One thing seems to be missing, thus: in protocol-replication.html the change should be mentioned as well:
> Various temporary files and directories created during the operation of the PostgreSQL server, such as any file or directory beginning with pgsql_tmp and temporary relations.
In the attached v3 I propose the following change:
- with <filename>pgsql_tmp</filename> and temporary relations.
+ with <filename>pgsql_tmp</filename>, hidden files and temporary relations.
So beside the documentation detail, +1 from my side.
Best,
Tobias
Attachments:
[application/octet-stream] skip_hidden_files_v3.diff (6.2K, ../../50DEEBD6-FB7A-4200-8B8E-8DCA59BB668D@gmx.net/2-skip_hidden_files_v3.diff)
download | inline diff:
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index b11d9a6ba3..8c566889c0 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3019,7 +3019,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
Various temporary files and directories created during the operation
of the PostgreSQL server, such as any file or directory beginning
- with <filename>pgsql_tmp</filename> and temporary relations.
+ with <filename>pgsql_tmp</filename>, hidden files and temporary relations.
</para>
</listitem>
<listitem>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 79d3e657c3..a7c8ed7370 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -909,7 +909,7 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and hidden files. But only regular files and directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 9118f05bf2..005fa83b9e 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -381,8 +381,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename>, and
+ hidden files as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 5baea7535b..75b6af80cb 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1202,6 +1202,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 4d130a7f94..ea22d439ae 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -173,7 +173,7 @@ SKIP:
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
- current_logfiles.tmp global/pg_internal.init.123))
+ current_logfiles.tmp global/pg_internal.init.123 .donotcopy))
{
open my $file, '>>', "$pgdata/$filename";
print $file "DONOTCOPY";
@@ -244,7 +244,7 @@ foreach my $dirname (
# These files should not be copied.
foreach my $filename (
qw(postgresql.auto.conf.tmp postmaster.opts postmaster.pid tablespace_map current_logfiles.tmp
- global/pg_internal.init global/pg_internal.init.123))
+ global/pg_internal.init global/pg_internal.init.123 .donotcopy))
{
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..d3c16acf0d 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -337,6 +337,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index 2316f611b2..ff975e5601 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -114,6 +114,9 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres files, which should be ignored by the scan
+append_to_file "$pgdata/global/.donotscan", "foo";
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/file_ops.c b/src/bin/pg_rewind/file_ops.c
index 25996b4da4..77ff8ac5d2 100644
--- a/src/bin/pg_rewind/file_ops.c
+++ b/src/bin/pg_rewind/file_ops.c
@@ -398,6 +398,10 @@ recurse_dir(const char *datadir, const char *parentpath,
strcmp(xlde->d_name, "..") == 0)
continue;
+ /* Skip hidden files */
+ if (xlde->d_name[0] == '.')
+ continue;
+
snprintf(fullpath, sizeof(fullpath), "%s/%s", fullparentpath, xlde->d_name);
if (lstat(fullpath, &fst) < 0)
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index fd2bee5d20..95ae15f946 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -35,6 +35,9 @@ sub run_test
append_to_file
"$test_primary_datadir/tst_both_dir/both_subdir/both_file3",
"in both3";
+ append_to_file
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
+ "in both4";
RewindTest::create_standby($test_mode);
@@ -82,6 +85,7 @@ sub run_test
\@paths,
[
"$test_primary_datadir/tst_both_dir",
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
"$test_primary_datadir/tst_both_dir/both_file1",
"$test_primary_datadir/tst_both_dir/both_file2",
"$test_primary_datadir/tst_both_dir/both_subdir",
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-20 08:50 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Michael Paquier <michael@paquier.xyz>
2023-04-27 09:08 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Daniel Gustafsson <daniel@yesql.se>
2023-04-27 15:34 ` Re: pg_basebackup: errors on macOS on directories with ".DS_Store" files Tobias Bussmann <t.bussmann@gmx.net>
@ 2023-04-27 15:48 ` Tobias Bussmann <t.bussmann@gmx.net>
0 siblings, 0 replies; 34+ messages in thread
From: Tobias Bussmann @ 2023-04-27 15:48 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Mark Guertin <markguertin@gmail.com>; pgsql-bugs@lists.postgresql.org
> Various temporary files and directories created during the operation of the PostgreSQL server, such as any file or directory beginning with pgsql_tmp and temporary relations.
On a second read, I realised this is likely not the proper paragraph to stuff it in, as these files are not 'created during the operation of PostgreSQL'.
Attached v4 where I propose to modify the last paragraph instead:
- Files other than regular files and directories, such as symbolic
+ Hidden files and files other than regular files and directories, such as symbolic
Tobias
Attachments:
[application/octet-stream] skip_hidden_files_v4.diff (6.3K, ../../7A3398A7-0990-4CE9-B832-AB429D886703@gmx.net/2-skip_hidden_files_v4.diff)
download | inline diff:
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index b11d9a6ba3..052cc6438e 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3047,7 +3047,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
<listitem>
<para>
- Files other than regular files and directories, such as symbolic
+ Hidden files and files other than regular files and directories, such as symbolic
links (other than for the directories listed above) and special
device files, are skipped. (Symbolic links
in <filename>pg_tblspc</filename> are maintained.)
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 79d3e657c3..a7c8ed7370 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -909,7 +909,7 @@ PostgreSQL documentation
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties, except certain temporary files managed by
- PostgreSQL. But only regular files and directories are copied, except that
+ PostgreSQL and hidden files. But only regular files and directories are copied, except that
symbolic links used for tablespaces are preserved. Symbolic links pointing
to certain directories known to PostgreSQL are copied as empty directories.
Other symbolic links and special device files are skipped.
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 9118f05bf2..005fa83b9e 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -381,8 +381,9 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b
<filename>backup_label</filename>,
<filename>tablespace_map</filename>,
<filename>pg_internal.init</filename>,
- <filename>postmaster.opts</filename>, and
- <filename>postmaster.pid</filename>, as well as any file or directory
+ <filename>postmaster.opts</filename>,
+ <filename>postmaster.pid</filename>, and
+ hidden files as well as any file or directory
beginning with <filename>pgsql_tmp</filename>, are omitted.
</para>
</step>
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 5baea7535b..75b6af80cb 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1202,6 +1202,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
/*
* Check if the postmaster has signaled us to exit, and abort with an
* error in that case. The error handler further up will call
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 4d130a7f94..ea22d439ae 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -173,7 +173,7 @@ SKIP:
# Write some files to test that they are not copied.
foreach my $filename (
qw(backup_label tablespace_map postgresql.auto.conf.tmp
- current_logfiles.tmp global/pg_internal.init.123))
+ current_logfiles.tmp global/pg_internal.init.123 .donotcopy))
{
open my $file, '>>', "$pgdata/$filename";
print $file "DONOTCOPY";
@@ -244,7 +244,7 @@ foreach my $dirname (
# These files should not be copied.
foreach my $filename (
qw(postgresql.auto.conf.tmp postmaster.opts postmaster.pid tablespace_map current_logfiles.tmp
- global/pg_internal.init global/pg_internal.init.123))
+ global/pg_internal.init global/pg_internal.init.123 .donotcopy))
{
ok(!-f "$tempdir/backup/$filename", "$filename not copied");
}
diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c
index 19eb67e485..d3c16acf0d 100644
--- a/src/bin/pg_checksums/pg_checksums.c
+++ b/src/bin/pg_checksums/pg_checksums.c
@@ -337,6 +337,10 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly)
strlen(PG_TEMP_FILES_DIR)) == 0)
continue;
+ /* Skip hidden files */
+ if (de->d_name[0] == '.')
+ continue;
+
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
if (lstat(fn, &st) < 0)
pg_fatal("could not stat file \"%s\": %m", fn);
diff --git a/src/bin/pg_checksums/t/002_actions.pl b/src/bin/pg_checksums/t/002_actions.pl
index 2316f611b2..ff975e5601 100644
--- a/src/bin/pg_checksums/t/002_actions.pl
+++ b/src/bin/pg_checksums/t/002_actions.pl
@@ -114,6 +114,9 @@ append_to_file "$pgdata/global/pgsql_tmp/1.1", "foo";
append_to_file "$pgdata/global/pg_internal.init", "foo";
append_to_file "$pgdata/global/pg_internal.init.123", "foo";
+# These are non-postgres files, which should be ignored by the scan
+append_to_file "$pgdata/global/.donotscan", "foo";
+
# Enable checksums.
command_ok([ 'pg_checksums', '--enable', '--no-sync', '-D', $pgdata ],
"checksums successfully enabled in cluster");
diff --git a/src/bin/pg_rewind/file_ops.c b/src/bin/pg_rewind/file_ops.c
index 25996b4da4..77ff8ac5d2 100644
--- a/src/bin/pg_rewind/file_ops.c
+++ b/src/bin/pg_rewind/file_ops.c
@@ -398,6 +398,10 @@ recurse_dir(const char *datadir, const char *parentpath,
strcmp(xlde->d_name, "..") == 0)
continue;
+ /* Skip hidden files */
+ if (xlde->d_name[0] == '.')
+ continue;
+
snprintf(fullpath, sizeof(fullpath), "%s/%s", fullparentpath, xlde->d_name);
if (lstat(fullpath, &fst) < 0)
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index fd2bee5d20..95ae15f946 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -35,6 +35,9 @@ sub run_test
append_to_file
"$test_primary_datadir/tst_both_dir/both_subdir/both_file3",
"in both3";
+ append_to_file
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
+ "in both4";
RewindTest::create_standby($test_mode);
@@ -82,6 +85,7 @@ sub run_test
\@paths,
[
"$test_primary_datadir/tst_both_dir",
+ "$test_primary_datadir/tst_both_dir/.donotcopy",
"$test_primary_datadir/tst_both_dir/both_file1",
"$test_primary_datadir/tst_both_dir/both_file2",
"$test_primary_datadir/tst_both_dir/both_subdir",
^ permalink raw reply [nested|flat] 34+ messages in thread
end of thread, other threads:[~2024-01-17 14:37 UTC | newest]
Thread overview: 34+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-04-19 15:45 pg_basebackup: errors on macOS on directories with ".DS_Store" files Mark Guertin <markguertin@gmail.com>
2023-04-19 17:51 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-04-19 18:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-04-20 00:15 ` Michael Paquier <michael@paquier.xyz>
2023-04-20 07:34 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-20 08:50 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-20 09:51 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-04-20 11:15 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-20 22:33 ` Michael Paquier <michael@paquier.xyz>
2023-04-21 03:02 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-04-27 14:22 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-28 05:32 ` Michael Paquier <michael@paquier.xyz>
2023-04-28 12:01 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-28 12:06 ` Michael Paquier <michael@paquier.xyz>
2023-04-28 12:11 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-28 23:42 ` Michael Paquier <michael@paquier.xyz>
2023-04-29 20:35 ` Daniel Gustafsson <daniel@yesql.se>
2023-05-01 05:25 ` Michael Paquier <michael@paquier.xyz>
2023-05-01 21:23 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-05-02 10:34 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-06 21:14 ` Jacob Champion <jchampion@timescale.com>
2023-07-06 23:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-07-07 07:23 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-08 00:05 ` Michael Paquier <michael@paquier.xyz>
2023-07-14 13:07 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-17 08:29 ` Michael Paquier <michael@paquier.xyz>
2023-07-19 23:16 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-20 07:47 ` Michael Paquier <michael@paquier.xyz>
2023-07-20 10:36 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-20 23:53 ` Michael Paquier <michael@paquier.xyz>
2024-01-17 14:37 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-27 09:08 ` Daniel Gustafsson <daniel@yesql.se>
2023-04-27 15:34 ` Tobias Bussmann <t.bussmann@gmx.net>
2023-04-27 15:48 ` Tobias Bussmann <t.bussmann@gmx.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox