agora inbox for pgsql-hackers@postgresql.org  

help / color / mirror / Atom feed

Re: pg_ls_tmpdir to show directories and shared filesets
70+ messages / 12 participants
[nested] [flat]

* Re: pg_ls_tmpdir to show directories and shared filesets
@ 2020-03-05 16:18 Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-05 16:18 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Fabien COELHO <coelho@cri.ensmp.fr>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>

On Tue, Mar 03, 2020 at 05:23:13PM -0300, Alvaro Herrera wrote:
> On 2020-Mar-03, Justin Pryzby wrote:
> 
> > But I don't think it makes sense to go through more implementation/review
> > cycles without some agreement from a larger group regarding the
> > desired/intended interface.  Should there be a column for "parent dir" ?  Or a
> > column for "is_dir" ?  Should dirs be shown at all, or only files ?
> 
> IMO: is_dir should be there (and subdirs should be listed), but
> parent_dir should not appear.  Also, the "path" should show the complete
> pathname, including containing dirs, starting from whatever the "root"
> is for the operation.
> 
> So for the example in the initial email, it would look like
> 
> path					isdir
> pgsql_tmp11025.0.sharedfileset/		t
> pgsql_tmp11025.0.sharedfileset/0.0	f
> pgsql_tmp11025.0.sharedfileset/1.0	f
> 
> plus additional columns, same as pg_ls_waldir et al.
> 
> I'd rather not have the code assume that there's a single level of
> subdirs, or assuming that an entry in the subdir cannot itself be a dir;
> that might end up hiding files for no good reason.
> 

Thanks for your input, see attached.

I'm not sure if prefer the 0002 patch alone (which recurses into dirs all at
once during the initial call), or 0002+3+4, which incrementally reads the dirs
on each call (but requires keeping dirs opened).

> I don't understand what purpose is served by having pg_ls_waldir() hide
> directories.

We could talk about whether the other functions should show dirs, if it's worth
breaking their return type.  Or if they should show hidden or special files,
which doesn't require breaking the return.  But until then I am to leave the
behavior alone.

-- 
Justin

Attachments:

  [text/x-diff] v6-0001-BUG-in-errmsg.patch (948B, ../../20200305161838.GJ684@telsasoft.com/2-v6-0001-BUG-in-errmsg.patch)
  download | inline diff:
From a5b9a03445d1c768662cafebd8ab3bd7a62890aa Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 27 Dec 2019 23:34:14 -0600
Subject: [PATCH v6 1/4] BUG: in errmsg

Note there's two changes here.
Should backpatch to v12, where pg_ls_tmpdir was added.
---
 src/backend/utils/adt/genfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 3741b87486..897b11a77d 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -590,7 +590,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 		if (stat(path, &attrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
-					 errmsg("could not stat directory \"%s\": %m", dir)));
+					 errmsg("could not stat file \"%s\": %m", path)));
 
 		/* Ignore anything but regular files */
 		if (!S_ISREG(attrib.st_mode))
-- 
2.17.0

  [text/x-diff] v6-0002-pg_ls_tmpdir-to-show-directories.patch (10.7K, ../../20200305161838.GJ684@telsasoft.com/3-v6-0002-pg_ls_tmpdir-to-show-directories.patch)
  download | inline diff:
From 2437276b9c7525981d4a70b804c81021b2f5fa1f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 14 Dec 2019 16:22:15 -0600
Subject: [PATCH v6 2/4] pg_ls_tmpdir to show directories

See also 9cd92d1a33699f86aa53d44ab04cc3eb50c18d11
---
 doc/src/sgml/func.sgml           |  13 +--
 src/backend/utils/adt/genfile.c  | 133 +++++++++++++++++++++++--------
 src/include/catalog/catversion.h |   2 +-
 src/include/catalog/pg_proc.dat  |   8 +-
 4 files changed, 114 insertions(+), 42 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..35abff16c9 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21382,8 +21382,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
+        For the temporary directory for <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and boolean
+        indicating if it is a directory.  Directories are shown recursively.  If
         <parameter>tablespace</parameter> is not provided, the
         <literal>pg_default</literal> tablespace is used.  Access is granted
         to members of the <literal>pg_monitor</literal> role and may be
@@ -21479,9 +21480,11 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and boolean indicating if it is a
+    directory.  Directories are used for temporary files used by parallel
+    processes, and are shown recursively.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 897b11a77d..c5148f547b 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -40,6 +40,11 @@ typedef struct
 	char	   *location;
 	DIR		   *dirdesc;
 	bool		include_dot_dirs;
+
+	/* Used in ls_dir_files: */
+	int			npaths;
+	char		**path;
+	struct stat	*stat;
 } directory_fctx;
 
 
@@ -522,12 +527,73 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 	return pg_ls_dir(fcinfo);
 }
 
-/* Generic function to return a directory listing of files */
+/*
+ * Populate fctx with list of pathnames and stat structues.  Generate a full
+ * list of the stat structures all at once - the alternative is to re-stat
+ * everything later on, which then requires somehow handling cases like the 2nd
+ * stat fails, or the 2nd stat returns a dir which didn't exist at the time we
+ * originally looked, or existed but wasn't a dir.  stat()ing everything all at
+ * once gets as close as we can to a consistent view of the filesystem, from
+ * which files might be removed or renamed or reordered.
+ */
+static void
+populate_paths(directory_fctx *fctx, bool recurse)
+{
+	struct dirent *de;
+	struct stat	attrib;
+	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	{
+		char		path[MAXPGPATH];
+
+		/* Skip hidden files */
+		if (de->d_name[0] == '.')
+			continue;
+
+		/* Get the file info */
+		snprintf(path, sizeof(path), "%s/%s", fctx->location, de->d_name);
+		if (stat(path, &attrib) < 0)
+			ereport(ERROR,
+					(errcode_for_file_access(),
+					 errmsg("could not stat file \"%s\": %m", path)));
+
+		/* Ignore anything but regular files, or dirs, if requested */
+		if (S_ISDIR(attrib.st_mode))
+		{
+			/* Save current dir while recursing */
+			directory_fctx oldfctx = *fctx;
+
+			fctx->location = path;
+			fctx->dirdesc = AllocateDir(path);
+
+			if (recurse)
+				populate_paths(fctx, recurse);
+
+			/* Restore previous dir */
+			FreeDir(fctx->dirdesc);
+			fctx->dirdesc = oldfctx.dirdesc;
+			fctx->location = oldfctx.location;
+		} else if (!S_ISREG(attrib.st_mode))
+			continue;
+
+		/* Reallocate path and stat if npaths is a power of two */
+		if (!(fctx->npaths & (fctx->npaths-1)))
+		{
+			int nalloc = fctx->npaths ? 2 * fctx->npaths : 1;
+			fctx->path = repalloc(fctx->path, nalloc*sizeof(*fctx->path));
+			fctx->stat = repalloc(fctx->stat, nalloc*sizeof(*fctx->stat));
+		}
+
+		fctx->path[fctx->npaths] = pstrdup(path);
+		fctx->stat[fctx->npaths] = attrib;
+		fctx->npaths++;
+	}
+}
+
+/* Generic function to return a directory listing of files (and optionally dirs) */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok)
 {
 	FuncCallContext *funcctx;
-	struct dirent *de;
 	directory_fctx *fctx;
 
 	if (SRF_IS_FIRSTCALL())
@@ -540,17 +606,24 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc(3);
+		tupdesc = CreateTemplateTupleDesc(dir_ok ? 4:3);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
 						   INT8OID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
 						   TIMESTAMPTZOID, -1, 0);
+		if (dir_ok)
+			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+						   BOOLOID, -1, 0);
+
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
 
 		fctx->location = pstrdup(dir);
 		fctx->dirdesc = AllocateDir(fctx->location);
+		fctx->npaths = 0;
+		fctx->path = palloc(sizeof(*fctx->path));
+		fctx->stat = palloc(sizeof(*fctx->stat));
 
 		if (!fctx->dirdesc)
 		{
@@ -566,6 +639,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 								fctx->location)));
 		}
 
+		populate_paths(fctx, dir_ok);
+		funcctx->max_calls = fctx->npaths;
 		funcctx->user_fctx = fctx;
 		MemoryContextSwitchTo(oldcontext);
 	}
@@ -573,38 +648,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	/*
+	 * Having already stat()ed all files, each call now just returns the
+	 * nth element.
+	 */
+	if (funcctx->call_cntr < funcctx->max_calls)
 	{
-		Datum		values[3];
-		bool		nulls[3];
-		char		path[MAXPGPATH * 2];
-		struct stat attrib;
+		struct stat	*stat = &fctx->stat[funcctx->call_cntr];
+		Datum		values[4];
+		bool		nulls[4] = {0};
 		HeapTuple	tuple;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
-			continue;
-
-		/* Get the file info */
-		snprintf(path, sizeof(path), "%s/%s", fctx->location, de->d_name);
-		if (stat(path, &attrib) < 0)
-			ereport(ERROR,
-					(errcode_for_file_access(),
-					 errmsg("could not stat file \"%s\": %m", path)));
-
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-		memset(nulls, 0, sizeof(nulls));
+		values[0] = CStringGetTextDatum(fctx->path[funcctx->call_cntr]);
+		values[1] = Int64GetDatum((int64) stat->st_size);
+		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(stat->st_mtime));
+		if (dir_ok)
+			values[3] = BoolGetDatum(S_ISDIR(stat->st_mode));
 
 		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
 		SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple));
 	}
 
+	for (int i = 0; i < funcctx->max_calls; ++i)
+		pfree(fctx->path[i]);
+
+	pfree(fctx->path);
+	pfree(fctx->stat);
 	FreeDir(fctx->dirdesc);
 	SRF_RETURN_DONE(funcctx);
 }
@@ -613,14 +682,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, false, false);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, false, false);
 }
 
 /*
@@ -638,7 +707,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path, true, true);
 }
 
 /*
@@ -667,5 +736,5 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false);
 }
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index d4fe84a037..6e1e2b961b 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
  */
 
 /*							yyyymmddN */
-#define CATALOG_VERSION_NO	202003031
+#define CATALOG_VERSION_NO	202003032
 
 #endif
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 07a86c7b7b..0dbce4ba09 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10739,13 +10739,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 
 # hash partitioning constraint function
-- 
2.17.0

  [text/x-diff] v6-0003-Convert-data-structure-to-use-a-list.patch (5.9K, ../../20200305161838.GJ684@telsasoft.com/4-v6-0003-Convert-data-structure-to-use-a-list.patch)
  download | inline diff:
From 864fa87522c3c0a062c11adfc0421cba92ac6efd Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 4 Mar 2020 17:38:03 -0600
Subject: [PATCH v6 3/4] Convert data structure to use a list

---
 src/backend/utils/adt/genfile.c | 78 +++++++++++++++++----------------
 1 file changed, 41 insertions(+), 37 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index c5148f547b..bc473e79ef 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -37,8 +37,9 @@
 
 typedef struct
 {
-	char	   *location;
-	DIR		   *dirdesc;
+	/* List of opened dirs */
+	List		*location;
+	List		*dirdesc;
 	bool		include_dot_dirs;
 
 	/* Used in ls_dir_files: */
@@ -474,10 +475,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
 
 		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
+		fctx->location = lappend(NIL, convert_and_check_filename(PG_GETARG_TEXT_PP(0)));
+		fctx->dirdesc = lappend(NIL, AllocateDir(linitial(fctx->location)));
 		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
 
 		if (!fctx->dirdesc)
 		{
@@ -490,7 +490,7 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 				ereport(ERROR,
 						(errcode_for_file_access(),
 						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
+								(char*)linitial(fctx->location))));
 		}
 		funcctx->user_fctx = fctx;
 		MemoryContextSwitchTo(oldcontext);
@@ -499,7 +499,7 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	while ((de = ReadDir(linitial(fctx->dirdesc), linitial(fctx->location))) != NULL)
 	{
 		if (!fctx->include_dot_dirs &&
 			(strcmp(de->d_name, ".") == 0 ||
@@ -509,7 +509,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
 	}
 
-	FreeDir(fctx->dirdesc);
+	FreeDir(linitial(fctx->dirdesc));
+	list_free(fctx->dirdesc);
+	list_free_deep(fctx->location);
 
 	SRF_RETURN_DONE(funcctx);
 }
@@ -528,20 +530,25 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Populate fctx with list of pathnames and stat structues.  Generate a full
- * list of the stat structures all at once - the alternative is to re-stat
- * everything later on, which then requires somehow handling cases like the 2nd
- * stat fails, or the 2nd stat returns a dir which didn't exist at the time we
- * originally looked, or existed but wasn't a dir.  stat()ing everything all at
- * once gets as close as we can to a consistent view of the filesystem, from
- * which files might be removed or renamed or reordered.
+ * Populate fctx with list of pathnames and stat structues.
+ *
+ * Generate a full list of the stat structures all at once.  That gets as close
+ * as we can to a consistent view of the filesystem, from which files might be
+ * removed or renamed or reordered.
+ *
+ * recurse is the recursion level into a stack of dirs, or -1 to not decend
+ * into directories.
  */
 static void
-populate_paths(directory_fctx *fctx, bool recurse)
+populate_paths(directory_fctx *fctx, int recurse)
 {
 	struct dirent *de;
 	struct stat	attrib;
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+
+	DIR *dirdesc = llast(fctx->dirdesc);
+	char *location = llast(fctx->location);
+
+	while ((de = ReadDir(dirdesc, location)) != NULL)
 	{
 		char		path[MAXPGPATH];
 
@@ -550,28 +557,23 @@ populate_paths(directory_fctx *fctx, bool recurse)
 			continue;
 
 		/* Get the file info */
-		snprintf(path, sizeof(path), "%s/%s", fctx->location, de->d_name);
+		snprintf(path, sizeof(path), "%s/%s", location, de->d_name);
 		if (stat(path, &attrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
 		/* Ignore anything but regular files, or dirs, if requested */
-		if (S_ISDIR(attrib.st_mode))
+		if (recurse == -1)
+			; /* Do nothing, fall through */
+		else if (S_ISDIR(attrib.st_mode))
 		{
-			/* Save current dir while recursing */
-			directory_fctx oldfctx = *fctx;
-
-			fctx->location = path;
-			fctx->dirdesc = AllocateDir(path);
-
-			if (recurse)
-				populate_paths(fctx, recurse);
-
-			/* Restore previous dir */
-			FreeDir(fctx->dirdesc);
-			fctx->dirdesc = oldfctx.dirdesc;
-			fctx->location = oldfctx.location;
+			/* Reallocate location and dirdesc whenever recursing */
+			fctx->location = lappend(fctx->location, path);
+			fctx->dirdesc = lappend(fctx->dirdesc, AllocateDir(path));
+			populate_paths(fctx, 1+recurse);
+			FreeDir(llast(fctx->dirdesc));
+			list_delete_last(fctx->dirdesc);
 		} else if (!S_ISREG(attrib.st_mode))
 			continue;
 
@@ -619,8 +621,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
 
-		fctx->location = pstrdup(dir);
-		fctx->dirdesc = AllocateDir(fctx->location);
+		fctx->location = lappend(NIL, pstrdup(dir));
+		fctx->dirdesc = lappend(NIL, AllocateDir(dir));
 		fctx->npaths = 0;
 		fctx->path = palloc(sizeof(*fctx->path));
 		fctx->stat = palloc(sizeof(*fctx->stat));
@@ -636,10 +638,10 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 				ereport(ERROR,
 						(errcode_for_file_access(),
 						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
+								dir)));
 		}
 
-		populate_paths(fctx, dir_ok);
+		populate_paths(fctx, dir_ok ? 0 : -1);
 		funcctx->max_calls = fctx->npaths;
 		funcctx->user_fctx = fctx;
 		MemoryContextSwitchTo(oldcontext);
@@ -674,7 +676,9 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 
 	pfree(fctx->path);
 	pfree(fctx->stat);
-	FreeDir(fctx->dirdesc);
+	FreeDir(linitial(fctx->dirdesc));
+	list_free(fctx->dirdesc);
+	list_free(fctx->location);
 	SRF_RETURN_DONE(funcctx);
 }
 
-- 
2.17.0

  [text/x-diff] v6-0004-Read-each-dir-incrementally-during-each-SRF-call.patch (6.2K, ../../20200305161838.GJ684@telsasoft.com/5-v6-0004-Read-each-dir-incrementally-during-each-SRF-call.patch)
  download | inline diff:
From 1a0c463d317d0e63885e7442b53875e9a1fd64e3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 5 Mar 2020 06:04:35 -0600
Subject: [PATCH v6 4/4] Read each dir incrementally during each SRF call..

This requires a stack and keeping state and opened dirs in the data structure.
It also requires squirreling around with memory contexts.
The SRF implementation seems to preclude use of recursion.
---
 src/backend/utils/adt/genfile.c | 107 ++++++++++++++------------------
 1 file changed, 47 insertions(+), 60 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index bc473e79ef..32d5b6f8b9 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -37,15 +37,15 @@
 
 typedef struct
 {
-	/* List of opened dirs */
+	/* Stack of opened dirs */
 	List		*location;
 	List		*dirdesc;
 	bool		include_dot_dirs;
 
 	/* Used in ls_dir_files: */
-	int			npaths;
-	char		**path;
-	struct stat	*stat;
+	bool		dir_ok;
+	char		*path;
+	struct stat	stat;
 } directory_fctx;
 
 
@@ -530,27 +530,37 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Populate fctx with list of pathnames and stat structues.
- *
- * Generate a full list of the stat structures all at once.  That gets as close
- * as we can to a consistent view of the filesystem, from which files might be
- * removed or renamed or reordered.
- *
- * recurse is the recursion level into a stack of dirs, or -1 to not decend
- * into directories.
+ * Update fctx->path and stat with next filename.
+ * Directories are descended into.
+ * The current dir to read from is at fctx[fctx->depth].
  */
-static void
-populate_paths(directory_fctx *fctx, int recurse)
+static int
+populate_paths(directory_fctx *fctx, FuncCallContext *funcctx)
 {
 	struct dirent *de;
-	struct stat	attrib;
-
-	DIR *dirdesc = llast(fctx->dirdesc);
-	char *location = llast(fctx->location);
 
-	while ((de = ReadDir(dirdesc, location)) != NULL)
+	for (;;)
 	{
 		char		path[MAXPGPATH];
+		DIR			*dirdesc = llast(fctx->dirdesc);
+		char		*location = llast(fctx->location);
+
+		Assert(list_length(fctx->dirdesc) == list_length(fctx->location));
+
+		if ((de = ReadDir(dirdesc, location)) == NULL)
+		{
+			/*
+			 * Read to the end of the dir on the top of the stack, now move to
+			 * the next dir.
+			 */
+			if (list_length(fctx->dirdesc) == 1)
+				return 0;
+			FreeDir(llast(fctx->dirdesc));
+			fctx->dirdesc = list_delete_last(fctx->dirdesc);
+			pfree(location);
+			fctx->location = list_delete_last(fctx->location);
+			continue;
+		}
 
 		/* Skip hidden files */
 		if (de->d_name[0] == '.')
@@ -558,36 +568,26 @@ populate_paths(directory_fctx *fctx, int recurse)
 
 		/* Get the file info */
 		snprintf(path, sizeof(path), "%s/%s", location, de->d_name);
-		if (stat(path, &attrib) < 0)
+		if (stat(path, &fctx->stat) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
 		/* Ignore anything but regular files, or dirs, if requested */
-		if (recurse == -1)
+		if (!fctx->dir_ok)
 			; /* Do nothing, fall through */
-		else if (S_ISDIR(attrib.st_mode))
+		else if (S_ISDIR(fctx->stat.st_mode))
 		{
-			/* Reallocate location and dirdesc whenever recursing */
-			fctx->location = lappend(fctx->location, path);
+			/* Reallocate location and dirdesc whenever descending */
+			MemoryContext oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+			fctx->location = lappend(fctx->location, pstrdup(path));
 			fctx->dirdesc = lappend(fctx->dirdesc, AllocateDir(path));
-			populate_paths(fctx, 1+recurse);
-			FreeDir(llast(fctx->dirdesc));
-			list_delete_last(fctx->dirdesc);
-		} else if (!S_ISREG(attrib.st_mode))
+			MemoryContextSwitchTo(oldcontext);
+		} else if (!S_ISREG(fctx->stat.st_mode))
 			continue;
 
-		/* Reallocate path and stat if npaths is a power of two */
-		if (!(fctx->npaths & (fctx->npaths-1)))
-		{
-			int nalloc = fctx->npaths ? 2 * fctx->npaths : 1;
-			fctx->path = repalloc(fctx->path, nalloc*sizeof(*fctx->path));
-			fctx->stat = repalloc(fctx->stat, nalloc*sizeof(*fctx->stat));
-		}
-
-		fctx->path[fctx->npaths] = pstrdup(path);
-		fctx->stat[fctx->npaths] = attrib;
-		fctx->npaths++;
+		fctx->path = pstrdup(path);
+		return 1;
 	}
 }
 
@@ -623,9 +623,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 
 		fctx->location = lappend(NIL, pstrdup(dir));
 		fctx->dirdesc = lappend(NIL, AllocateDir(dir));
-		fctx->npaths = 0;
-		fctx->path = palloc(sizeof(*fctx->path));
-		fctx->stat = palloc(sizeof(*fctx->stat));
+		fctx->dir_ok = dir_ok;
 
 		if (!fctx->dirdesc)
 		{
@@ -641,8 +639,6 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 								dir)));
 		}
 
-		populate_paths(fctx, dir_ok ? 0 : -1);
-		funcctx->max_calls = fctx->npaths;
 		funcctx->user_fctx = fctx;
 		MemoryContextSwitchTo(oldcontext);
 	}
@@ -650,35 +646,26 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	/*
-	 * Having already stat()ed all files, each call now just returns the
-	 * nth element.
-	 */
-	if (funcctx->call_cntr < funcctx->max_calls)
+	if (populate_paths(fctx, funcctx))
 	{
-		struct stat	*stat = &fctx->stat[funcctx->call_cntr];
 		Datum		values[4];
 		bool		nulls[4] = {0};
 		HeapTuple	tuple;
 
-		values[0] = CStringGetTextDatum(fctx->path[funcctx->call_cntr]);
-		values[1] = Int64GetDatum((int64) stat->st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(stat->st_mtime));
+		values[0] = CStringGetTextDatum(fctx->path);
+		values[1] = Int64GetDatum((int64) fctx->stat.st_size);
+		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(fctx->stat.st_mtime));
 		if (dir_ok)
-			values[3] = BoolGetDatum(S_ISDIR(stat->st_mode));
+			values[3] = BoolGetDatum(S_ISDIR(fctx->stat.st_mode));
 
 		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
+		pfree(fctx->path);
 		SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple));
 	}
 
-	for (int i = 0; i < funcctx->max_calls; ++i)
-		pfree(fctx->path[i]);
-
-	pfree(fctx->path);
-	pfree(fctx->stat);
 	FreeDir(linitial(fctx->dirdesc));
 	list_free(fctx->dirdesc);
-	list_free(fctx->location);
+	list_free_deep(fctx->location);
 	SRF_RETURN_DONE(funcctx);
 }
 
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-06 23:35 ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  0 siblings, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-06 23:35 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Fabien COELHO <coelho@cri.ensmp.fr>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>

On Thu, Mar 05, 2020 at 10:18:38AM -0600, Justin Pryzby wrote:
> I'm not sure if prefer the 0002 patch alone (which recurses into dirs all at
> once during the initial call), or 0002+3+4, which incrementally reads the dirs
> on each call (but requires keeping dirs opened).

I fixed an issue that leading dirs were being shown which should not have been,
which was easier in the 0004 patch, so squished.  And fixed a bug that
"special" files weren't excluded, and "missing_ok" wasn't effective.

> > I don't understand what purpose is served by having pg_ls_waldir() hide
> > directories.
> 
> We could talk about whether the other functions should show dirs, if it's worth
> breaking their return type.  Or if they should show hidden or special files,
> which doesn't require breaking the return.  But until then I am to leave the
> behavior alone.

I don't see why any of the functions would exclude dirs, but ls_tmpdir deserves
to be fixed since core postgres dynamically creates dirs there.

Also ... I accidentally changed the behavior: master not only doesn't decend
into dirs, it hides them - that was my original complaint.  I propose to *also*
change at least tmpdir and logdir to show dirs, but don't decend.  I left
waldir alone for now.

Since v12 ls_tmpdir and since v10 logdir and waldir exclude dirs, I think we
should backpatch documentation to say so.

ISTM pg_ls_tmpdir and ls_logdir should be called with missing_ok=true, since
they're not created until they're used.

-- 
Justin

Attachments:

  [text/x-diff] v7-0001-BUG-in-errmsg.patch (948B, ../../20200306233507.GN684@telsasoft.com/2-v7-0001-BUG-in-errmsg.patch)
  download | inline diff:
From a5b9a03445d1c768662cafebd8ab3bd7a62890aa Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 27 Dec 2019 23:34:14 -0600
Subject: [PATCH v7 1/6] BUG: in errmsg

Note there's two changes here.
Should backpatch to v12, where pg_ls_tmpdir was added.
---
 src/backend/utils/adt/genfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 3741b87486..897b11a77d 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -590,7 +590,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 		if (stat(path, &attrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
-					 errmsg("could not stat directory \"%s\": %m", dir)));
+					 errmsg("could not stat file \"%s\": %m", path)));
 
 		/* Ignore anything but regular files */
 		if (!S_ISREG(attrib.st_mode))
-- 
2.17.0

  [text/x-diff] v7-0002-Document-historic-behavior-about-hiding-directori.patch (1.7K, ../../20200306233507.GN684@telsasoft.com/3-v7-0002-Document-historic-behavior-about-hiding-directori.patch)
  download | inline diff:
From 6ea85ec0a267930320b8454a33bca368a8544a2d Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 16:50:07 -0600
Subject: [PATCH v7 2/6] Document historic behavior about hiding directories
 and special files

Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
 doc/src/sgml/func.sgml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v7-0003-Document-historic-behavior-about-hiding-directori.patch (903B, ../../20200306233507.GN684@telsasoft.com/4-v7-0003-Document-historic-behavior-about-hiding-directori.patch)
  download | inline diff:
From 5250d637493627f1ff3587bc73dd598bc1ca3ffc Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:12:04 -0600
Subject: [PATCH v7 3/6] Document historic behavior about hiding directories
 and special files

Should backpatch to v12: tmpdir
---
 doc/src/sgml/func.sgml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v7-0004-pg_ls_tmpdir-to-show-directories.patch (12.2K, ../../20200306233507.GN684@telsasoft.com/5-v7-0004-pg_ls_tmpdir-to-show-directories.patch)
  download | inline diff:
From 8d00a1c80679d9a754a3988786e70c0385b46b30 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 14 Dec 2019 16:22:15 -0600
Subject: [PATCH v7 4/6] pg_ls_tmpdir to show directories

See also 9cd92d1a33699f86aa53d44ab04cc3eb50c18d11
---
 doc/src/sgml/func.sgml          |  14 +--
 src/backend/utils/adt/genfile.c | 167 +++++++++++++++++++++++---------
 src/include/catalog/pg_proc.dat |   8 +-
 3 files changed, 134 insertions(+), 55 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..15f4908924 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21382,8 +21382,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
+        For the temporary directory for <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and boolean
+        indicating if it is a directory.  Directories are shown recursively.  If
         <parameter>tablespace</parameter> is not provided, the
         <literal>pg_default</literal> tablespace is used.  Access is granted
         to members of the <literal>pg_monitor</literal> role and may be
@@ -21482,14 +21483,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and boolean indicating if it is a
+    directory.  Directories are used for temporary files used by parallel
+    processes, and are shown recursively.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 897b11a77d..0207ac6be8 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -35,11 +35,23 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+enum dir_action {
+	DIR_HIDE,
+	DIR_ONLY,
+	DIR_DESCEND,
+};
+
 typedef struct
 {
-	char	   *location;
-	DIR		   *dirdesc;
+	/* Stack of opened dirs */
+	List		*location;
+	List		*dirdesc;
 	bool		include_dot_dirs;
+
+	/* Used in ls_dir_files: */
+	enum dir_action dir_action;
+	char		*path;
+	struct stat	stat;
 } directory_fctx;
 
 
@@ -469,10 +481,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
 
 		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
+		fctx->location = lappend(NIL, convert_and_check_filename(PG_GETARG_TEXT_PP(0)));
+		fctx->dirdesc = lappend(NIL, AllocateDir(linitial(fctx->location)));
 		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
 
 		if (!fctx->dirdesc)
 		{
@@ -485,7 +496,7 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 				ereport(ERROR,
 						(errcode_for_file_access(),
 						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
+								(char*)linitial(fctx->location))));
 		}
 		funcctx->user_fctx = fctx;
 		MemoryContextSwitchTo(oldcontext);
@@ -494,7 +505,7 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	while ((de = ReadDir(linitial(fctx->dirdesc), linitial(fctx->location))) != NULL)
 	{
 		if (!fctx->include_dot_dirs &&
 			(strcmp(de->d_name, ".") == 0 ||
@@ -504,7 +515,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
 	}
 
-	FreeDir(fctx->dirdesc);
+	FreeDir(linitial(fctx->dirdesc));
+	list_free(fctx->dirdesc);
+	list_free_deep(fctx->location);
 
 	SRF_RETURN_DONE(funcctx);
 }
@@ -522,12 +535,84 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 	return pg_ls_dir(fcinfo);
 }
 
-/* Generic function to return a directory listing of files */
+/*
+ * Update fctx->path and stat with next filename.
+ * Directories are descended into.
+ * The current dir to read from is at fctx[fctx->depth].
+ */
+static int
+populate_paths(directory_fctx *fctx, FuncCallContext *funcctx)
+{
+	struct dirent *de;
+
+	for (;;)
+	{
+		char		path[MAXPGPATH];
+		DIR			*dirdesc = llast(fctx->dirdesc);
+		char		*location = llast(fctx->location);
+
+		Assert(list_length(fctx->dirdesc) == list_length(fctx->location));
+
+		if ((de = ReadDir(dirdesc, location)) == NULL)
+		{
+			/*
+			 * Read to the end of the dir on the top of the stack, now move to
+			 * the next dir.
+			 */
+			if (list_length(fctx->dirdesc) == 1)
+				return 0;
+			FreeDir(llast(fctx->dirdesc));
+			fctx->dirdesc = list_delete_last(fctx->dirdesc);
+			pfree(location);
+			fctx->location = list_delete_last(fctx->location);
+			continue;
+		}
+
+		/* Skip hidden files */
+		if (de->d_name[0] == '.')
+			continue;
+
+		/* Get the file info */
+		snprintf(path, sizeof(path), "%s/%s", location, de->d_name);
+		if (stat(path, &fctx->stat) < 0)
+			ereport(ERROR,
+					(errcode_for_file_access(),
+					 errmsg("could not stat file \"%s\": %m", path)));
+
+		/* Ignore anything but regular files, or dirs, if requested */
+		if (S_ISDIR(fctx->stat.st_mode))
+		{
+			if (fctx->dir_action == DIR_HIDE)
+				continue;
+			else if (fctx->dir_action == DIR_DESCEND)
+			{
+				/* Reallocate location and dirdesc whenever descending */
+				MemoryContext oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+				fctx->location = lappend(fctx->location, pstrdup(path));
+				fctx->dirdesc = lappend(fctx->dirdesc, AllocateDir(path));
+				MemoryContextSwitchTo(oldcontext);
+			}
+		} else if (!S_ISREG(fctx->stat.st_mode))
+			continue;
+
+		/* Do not show the initial dir or slash */
+		fctx->path = pstrdup(path + 1 + strlen(linitial(fctx->location)));
+		return 1;
+	}
+}
+
+/*
+ * Generic function to return a directory listing of files
+ *
+ * if missing_ok, then fail silently if the dir doesn't exist, else error.
+ *
+ * if dir_action!=DIR_HIDE, then include a 4th column indicating whether path
+ * is a directory, and recurse if DIR_DESCEND.
+ */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, enum dir_action dir_action)
 {
 	FuncCallContext *funcctx;
-	struct dirent *de;
 	directory_fctx *fctx;
 
 	if (SRF_IS_FIRSTCALL())
@@ -540,19 +625,24 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc(3);
+		tupdesc = CreateTemplateTupleDesc(dir_action == DIR_HIDE ? 3 : 4);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
 						   INT8OID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
 						   TIMESTAMPTZOID, -1, 0);
+		if (dir_action != DIR_HIDE)
+			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+						   BOOLOID, -1, 0);
+
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
 
-		fctx->location = pstrdup(dir);
-		fctx->dirdesc = AllocateDir(fctx->location);
+		fctx->location = lappend(NIL, pstrdup(dir));
+		fctx->dirdesc = lappend(NIL, AllocateDir(dir));
+		fctx->dir_action = dir_action;
 
-		if (!fctx->dirdesc)
+		if (!linitial(fctx->dirdesc))
 		{
 			if (missing_ok && errno == ENOENT)
 			{
@@ -563,7 +653,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 				ereport(ERROR,
 						(errcode_for_file_access(),
 						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
+								dir)));
 		}
 
 		funcctx->user_fctx = fctx;
@@ -573,39 +663,26 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	if (populate_paths(fctx, funcctx))
 	{
-		Datum		values[3];
-		bool		nulls[3];
-		char		path[MAXPGPATH * 2];
-		struct stat attrib;
+		Datum		values[4];
+		bool		nulls[4] = {0};
 		HeapTuple	tuple;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
-			continue;
-
-		/* Get the file info */
-		snprintf(path, sizeof(path), "%s/%s", fctx->location, de->d_name);
-		if (stat(path, &attrib) < 0)
-			ereport(ERROR,
-					(errcode_for_file_access(),
-					 errmsg("could not stat file \"%s\": %m", path)));
-
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-		memset(nulls, 0, sizeof(nulls));
+		values[0] = CStringGetTextDatum(fctx->path);
+		values[1] = Int64GetDatum((int64) fctx->stat.st_size);
+		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(fctx->stat.st_mtime));
+		if (dir_action != DIR_HIDE)
+			values[3] = BoolGetDatum(S_ISDIR(fctx->stat.st_mode));
 
 		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
+		pfree(fctx->path);
 		SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple));
 	}
 
-	FreeDir(fctx->dirdesc);
+	FreeDir(linitial(fctx->dirdesc));
+	list_free(fctx->dirdesc);
+	list_free_deep(fctx->location);
 	SRF_RETURN_DONE(funcctx);
 }
 
@@ -613,14 +690,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, false, false);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, false, DIR_HIDE);
 }
 
 /*
@@ -638,7 +715,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path, true, DIR_DESCEND);
 }
 
 /*
@@ -667,5 +744,5 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 07a86c7b7b..0dbce4ba09 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10739,13 +10739,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 
 # hash partitioning constraint function
-- 
2.17.0

  [text/x-diff] v7-0005-Change-logdir-and-archive_statusdir-to-include-di.patch (4.1K, ../../20200306233507.GN684@telsasoft.com/6-v7-0005-Change-logdir-and-archive_statusdir-to-include-di.patch)
  download | inline diff:
From 30a6473462e84ae700fb63269bb5669c3d1925be Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 5 Mar 2020 15:17:28 -0600
Subject: [PATCH v7 5/6] Change logdir and archive_statusdir to include dirs..

*and* add a column indicating *whether* a path is a dir..

The previous commit changes to always *show* directories (and optionally
recurse).

XXX: also need to update docs for new column
---
 doc/src/sgml/func.sgml          | 4 ++--
 src/backend/utils/adt/genfile.c | 4 ++--
 src/include/catalog/pg_proc.dat | 8 ++++----
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 15f4908924..2b920ebd12 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21451,7 +21451,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
@@ -21476,7 +21476,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 0207ac6be8..9d99528aa5 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -690,7 +690,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, enum
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, false, DIR_ONLY);
 }
 
 /* Function to return the list of files in the WAL directory */
@@ -744,5 +744,5 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, DIR_ONLY);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0dbce4ba09..fdf86e1eaf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10723,8 +10723,8 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
@@ -10733,8 +10733,8 @@
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
-- 
2.17.0

  [text/x-diff] v7-0006-Change-pg_ls_logdir-to-ignore-error-if-initial-to.patch (880B, ../../20200306233507.GN684@telsasoft.com/7-v7-0006-Change-pg_ls_logdir-to-ignore-error-if-initial-to.patch)
  download | inline diff:
From 458da7d43d5dcd91a1345a3a25ecc07f3274dade Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v7 6/6] Change pg_ls_logdir to ignore error if initial/top dir
 is missing

---
 src/backend/utils/adt/genfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 9d99528aa5..e15ad1827c 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -690,7 +690,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, enum
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false, DIR_ONLY);
+	return pg_ls_dir_files(fcinfo, Log_directory, true, DIR_ONLY);
 }
 
 /* Function to return the list of files in the WAL directory */
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-07 14:14   ` Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 17:10     ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 2 replies; 70+ messages in thread

From: Fabien COELHO @ 2020-03-07 14:14 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>


Hello Justin,

Some feedback about the v7 patch set.

About v7.1, seems ok.

About v7.2 & v7.3 seems ok, altought the two could be merged.

About v7.4:

The documentation sentences could probably be improved "for for", "used 
... used". Maybe:

   For the temporary directory for <parameter>tablespace</parameter>, ...
->
   For <parameter>tablespace</parameter> temporary directory, ...

   Directories are used for temporary files used by parallel
   processes, and are shown recursively.
->
   Directories holding temporary files used by parallel
   processes are shown recursively.

It seems that lists are used as FIFO structures by appending, fetching & 
deleting last, all of which are O(n). ISTM it would be better to use the 
head of the list by inserting, getting and deleting first, which are O(1).

ISTM that several instances of: "pg_ls_dir_files(..., true, false);" 
should be "pg_ls_dir_files(..., true, DIR_HIDE);".

About v7.5 looks like a doc update which should be merged with v7.4.

Alas, ISTM that there are no tests on any of these functions:-(

-- 
Fabien.





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-03-07 17:10     ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 17:40       ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  1 sibling, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-07 17:10 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>

On Sat, Mar 07, 2020 at 03:14:37PM +0100, Fabien COELHO wrote:
> Some feedback about the v7 patch set.

Thanks for looking again

> About v7.1, seems ok.
> 
> About v7.2 & v7.3 seems ok, altought the two could be merged.

These are separate since I proprose that one should be backpatched to v12 and
the other to v10.

> About v7.4:
...
> It seems that lists are used as FIFO structures by appending, fetching &
> deleting last, all of which are O(n). ISTM it would be better to use the
> head of the list by inserting, getting and deleting first, which are O(1).

I think you're referring to linked lists, but pglists are now arrays, for which
that's backwards.  See 1cff1b95a and d97b714a2.  For example, list_delete_last
says:
 * This is the opposite of list_delete_first(), but is noticeably cheaper
 * with a long list, since no data need be moved.

> ISTM that several instances of: "pg_ls_dir_files(..., true, false);" should
> be "pg_ls_dir_files(..., true, DIR_HIDE);".

Oops, that affects an intermediate commit and maybe due to merge conflict.
Thanks.

> About v7.5 looks like a doc update which should be merged with v7.4.

No, v7.5 updates pg_proc.dat and changes the return type of two functions.
It's a short commit since all the infrastructure is implemented to make the
functions do whatever we want.  But it's deliberately separate since I'm
proposing a breaking change, and one that hasn't been discussed until now.

> Alas, ISTM that there are no tests on any of these functions:-(

Yeah.  Everything that includes any output is going to include timestamps;
those could be filtered out.  waldir is going to have random filenames, and a
differing number of rows.  But we should exercize pg_ls_dir_files at least
once..

My previous version had a bug with ignore_missing with pg_ls_tmpdir, which
would've been caught by a test like:
SELECT FROM pg_ls_tmpdir() WHERE name='Does not exist'; -- Never true, so the function runs to completion but returns zero rows.

The 0006 commit changes that for logdir, too.  Without 0006, that will ERROR if
the dir doesn't exist (which I think would be the default during regression
tests).

It'd be nice to run pg_ls_tmpdir before the tmpdir exists, and again
afterwards.  But I'm having trouble finding a single place to put it.  The
closest I can find is dbsize.sql.  Any ideas ?

-- 
Justin





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 17:10     ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-07 17:40       ` Fabien COELHO <coelho@cri.ensmp.fr>
  0 siblings, 0 replies; 70+ messages in thread

From: Fabien COELHO @ 2020-03-07 17:40 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>


>> It seems that lists are used as FIFO structures by appending, fetching &
>> deleting last, all of which are O(n). ISTM it would be better to use the
>> head of the list by inserting, getting and deleting first, which are O(1).
>
> I think you're referring to linked lists, but pglists are now arrays,

Ok… I forgot about this change, so my point is void, you took the right 
one.

-- 
Fabien.

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-03-07 21:40     ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-08 08:02       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  1 sibling, 2 replies; 70+ messages in thread

From: Justin Pryzby @ 2020-03-07 21:40 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>

On Sat, Mar 07, 2020 at 03:14:37PM +0100, Fabien COELHO wrote:
> The documentation sentences could probably be improved "for for", "used ...
> used". Maybe:

> ISTM that several instances of: "pg_ls_dir_files(..., true, false);" should
> be "pg_ls_dir_files(..., true, DIR_HIDE);".

> Alas, ISTM that there are no tests on any of these functions:-(

Addressed these.

And reordered the last two commits to demonstrate and exercize the behavior
change in regress test.

-- 
Justin

Attachments:

  [text/x-diff] v8-0001-BUG-in-errmsg.patch (948B, ../../20200307214010.GB1357@telsasoft.com/2-v8-0001-BUG-in-errmsg.patch)
  download | inline diff:
From a5b9a03445d1c768662cafebd8ab3bd7a62890aa Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 27 Dec 2019 23:34:14 -0600
Subject: [PATCH v8 1/6] BUG: in errmsg

Note there's two changes here.
Should backpatch to v12, where pg_ls_tmpdir was added.
---
 src/backend/utils/adt/genfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 3741b87486..897b11a77d 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -590,7 +590,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 		if (stat(path, &attrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
-					 errmsg("could not stat directory \"%s\": %m", dir)));
+					 errmsg("could not stat file \"%s\": %m", path)));
 
 		/* Ignore anything but regular files */
 		if (!S_ISREG(attrib.st_mode))
-- 
2.17.0

  [text/x-diff] v8-0002-Document-historic-behavior-about-hiding-directori.patch (1.7K, ../../20200307214010.GB1357@telsasoft.com/3-v8-0002-Document-historic-behavior-about-hiding-directori.patch)
  download | inline diff:
From 6ea85ec0a267930320b8454a33bca368a8544a2d Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 16:50:07 -0600
Subject: [PATCH v8 2/6] Document historic behavior about hiding directories
 and special files

Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
 doc/src/sgml/func.sgml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v8-0003-Document-historic-behavior-about-hiding-directori.patch (903B, ../../20200307214010.GB1357@telsasoft.com/4-v8-0003-Document-historic-behavior-about-hiding-directori.patch)
  download | inline diff:
From 5250d637493627f1ff3587bc73dd598bc1ca3ffc Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:12:04 -0600
Subject: [PATCH v8 3/6] Document historic behavior about hiding directories
 and special files

Should backpatch to v12: tmpdir
---
 doc/src/sgml/func.sgml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v8-0004-pg_ls_tmpdir-to-show-directories-recursively.patch (14.9K, ../../20200307214010.GB1357@telsasoft.com/5-v8-0004-pg_ls_tmpdir-to-show-directories-recursively.patch)
  download | inline diff:
From 70183f1ba1eb33e5c279a6d22a56fcaebdbfbb97 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 14 Dec 2019 16:22:15 -0600
Subject: [PATCH v8 4/6] pg_ls_tmpdir to show directories recursively

See also 9cd92d1a33699f86aa53d44ab04cc3eb50c18d11

Discussion
https://www.postgresql.org/message-id/flat/20191213053931.GV2082@telsasoft.com
https://www.postgresql.org/message-id/flat/20191227170220.GE12890@telsasoft.com

Need catversion bump
---
 doc/src/sgml/func.sgml                    |  14 +-
 src/backend/utils/adt/genfile.c           | 167 ++++++++++++++++------
 src/include/catalog/pg_proc.dat           |   8 +-
 src/test/regress/input/tablespace.source  |   5 +
 src/test/regress/output/tablespace.source |   8 ++
 5 files changed, 147 insertions(+), 55 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..234c9c1699 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21382,8 +21382,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and boolean
+        indicating if it is a directory.  Directories are shown recursively.  If
         <parameter>tablespace</parameter> is not provided, the
         <literal>pg_default</literal> tablespace is used.  Access is granted
         to members of the <literal>pg_monitor</literal> role and may be
@@ -21482,14 +21483,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes, and are shown recursively.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 897b11a77d..a534d75dd8 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -35,11 +35,23 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+enum dir_action {
+	DIR_HIDE,
+	DIR_ONLY,
+	DIR_DESCEND,
+};
+
 typedef struct
 {
-	char	   *location;
-	DIR		   *dirdesc;
+	/* Stack of opened dirs */
+	List		*location;
+	List		*dirdesc;
 	bool		include_dot_dirs;
+
+	/* Used in ls_dir_files: */
+	enum dir_action dir_action;
+	char		*path;
+	struct stat	stat;
 } directory_fctx;
 
 
@@ -469,10 +481,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
 
 		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
+		fctx->location = lappend(NIL, convert_and_check_filename(PG_GETARG_TEXT_PP(0)));
+		fctx->dirdesc = lappend(NIL, AllocateDir(linitial(fctx->location)));
 		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
 
 		if (!fctx->dirdesc)
 		{
@@ -485,7 +496,7 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 				ereport(ERROR,
 						(errcode_for_file_access(),
 						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
+								(char*)linitial(fctx->location))));
 		}
 		funcctx->user_fctx = fctx;
 		MemoryContextSwitchTo(oldcontext);
@@ -494,7 +505,7 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	while ((de = ReadDir(linitial(fctx->dirdesc), linitial(fctx->location))) != NULL)
 	{
 		if (!fctx->include_dot_dirs &&
 			(strcmp(de->d_name, ".") == 0 ||
@@ -504,7 +515,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
 	}
 
-	FreeDir(fctx->dirdesc);
+	FreeDir(linitial(fctx->dirdesc));
+	list_free(fctx->dirdesc);
+	list_free_deep(fctx->location);
 
 	SRF_RETURN_DONE(funcctx);
 }
@@ -522,12 +535,84 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 	return pg_ls_dir(fcinfo);
 }
 
-/* Generic function to return a directory listing of files */
+/*
+ * Update fctx->path and stat with next filename.
+ * Directories are descended into.
+ * The current dir to read from is at fctx[fctx->depth].
+ */
+static int
+populate_paths(directory_fctx *fctx, FuncCallContext *funcctx)
+{
+	struct dirent *de;
+
+	for (;;)
+	{
+		char		path[MAXPGPATH];
+		DIR			*dirdesc = llast(fctx->dirdesc);
+		char		*location = llast(fctx->location);
+
+		Assert(list_length(fctx->dirdesc) == list_length(fctx->location));
+
+		if ((de = ReadDir(dirdesc, location)) == NULL)
+		{
+			/*
+			 * Read to the end of the dir on the top of the stack, now move to
+			 * the next dir.
+			 */
+			if (list_length(fctx->dirdesc) == 1)
+				return 0;
+			FreeDir(llast(fctx->dirdesc));
+			fctx->dirdesc = list_delete_last(fctx->dirdesc);
+			pfree(location);
+			fctx->location = list_delete_last(fctx->location);
+			continue;
+		}
+
+		/* Skip hidden files */
+		if (de->d_name[0] == '.')
+			continue;
+
+		/* Get the file info */
+		snprintf(path, sizeof(path), "%s/%s", location, de->d_name);
+		if (stat(path, &fctx->stat) < 0)
+			ereport(ERROR,
+					(errcode_for_file_access(),
+					 errmsg("could not stat file \"%s\": %m", path)));
+
+		/* Ignore anything but regular files, or dirs, if requested */
+		if (S_ISDIR(fctx->stat.st_mode))
+		{
+			if (fctx->dir_action == DIR_HIDE)
+				continue;
+			else if (fctx->dir_action == DIR_DESCEND)
+			{
+				MemoryContext oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+				fctx->location = lappend(fctx->location, pstrdup(path));
+				fctx->dirdesc = lappend(fctx->dirdesc, AllocateDir(path));
+				MemoryContextSwitchTo(oldcontext);
+			}
+		} else if (!S_ISREG(fctx->stat.st_mode))
+			continue;
+
+		/* Do not show the initial dir or slash */
+		fctx->path = pstrdup(path + 1 + strlen(linitial(fctx->location)));
+		return 1;
+	}
+}
+
+/*
+ * Generic function to return a directory listing of files
+ *
+ * if missing_ok, then fail silently if the dir doesn't exist, else error.
+ *
+ * if dir_action!=DIR_HIDE, then include a 4th column indicating whether path
+ * is a directory, and recurse if DIR_DESCEND.
+ */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok,
+		enum dir_action dir_action)
 {
 	FuncCallContext *funcctx;
-	struct dirent *de;
 	directory_fctx *fctx;
 
 	if (SRF_IS_FIRSTCALL())
@@ -540,19 +625,24 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc(3);
+		tupdesc = CreateTemplateTupleDesc(dir_action == DIR_HIDE ? 3 : 4);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
 						   INT8OID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
 						   TIMESTAMPTZOID, -1, 0);
+		if (dir_action != DIR_HIDE)
+			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+						   BOOLOID, -1, 0);
+
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
 
-		fctx->location = pstrdup(dir);
-		fctx->dirdesc = AllocateDir(fctx->location);
+		fctx->location = lappend(NIL, pstrdup(dir));
+		fctx->dirdesc = lappend(NIL, AllocateDir(dir));
+		fctx->dir_action = dir_action;
 
-		if (!fctx->dirdesc)
+		if (!linitial(fctx->dirdesc))
 		{
 			if (missing_ok && errno == ENOENT)
 			{
@@ -563,7 +653,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 				ereport(ERROR,
 						(errcode_for_file_access(),
 						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
+								dir)));
 		}
 
 		funcctx->user_fctx = fctx;
@@ -573,39 +663,26 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	funcctx = SRF_PERCALL_SETUP();
 	fctx = (directory_fctx *) funcctx->user_fctx;
 
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
+	if (populate_paths(fctx, funcctx))
 	{
-		Datum		values[3];
-		bool		nulls[3];
-		char		path[MAXPGPATH * 2];
-		struct stat attrib;
+		Datum		values[4];
+		bool		nulls[4] = {0};
 		HeapTuple	tuple;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
-			continue;
-
-		/* Get the file info */
-		snprintf(path, sizeof(path), "%s/%s", fctx->location, de->d_name);
-		if (stat(path, &attrib) < 0)
-			ereport(ERROR,
-					(errcode_for_file_access(),
-					 errmsg("could not stat file \"%s\": %m", path)));
-
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-		memset(nulls, 0, sizeof(nulls));
+		values[0] = CStringGetTextDatum(fctx->path);
+		values[1] = Int64GetDatum((int64) fctx->stat.st_size);
+		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(fctx->stat.st_mtime));
+		if (dir_action != DIR_HIDE)
+			values[3] = BoolGetDatum(S_ISDIR(fctx->stat.st_mode));
 
 		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
+		pfree(fctx->path);
 		SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple));
 	}
 
-	FreeDir(fctx->dirdesc);
+	FreeDir(linitial(fctx->dirdesc));
+	list_free(fctx->dirdesc);
+	list_free_deep(fctx->location);
 	SRF_RETURN_DONE(funcctx);
 }
 
@@ -613,14 +690,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, false, DIR_HIDE);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, false, DIR_HIDE);
 }
 
 /*
@@ -638,7 +715,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path, true, DIR_DESCEND);
 }
 
 /*
@@ -667,5 +744,5 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, DIR_HIDE);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 07a86c7b7b..0dbce4ba09 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10739,13 +10739,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..a7f34c5dd5 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1)b , pg_ls_tmpdir(oid)c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..edcdb4f32a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1)b , pg_ls_tmpdir(oid)c WHERE c.name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v8-0005-Change-pg_ls_logdir-to-ignore-error-if-initial-to.patch (3.0K, ../../20200307214010.GB1357@telsasoft.com/6-v8-0005-Change-pg_ls_logdir-to-ignore-error-if-initial-to.patch)
  download | inline diff:
From 3ae462152696fe4ac19c0faad9807a2d06849a75 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v8 5/6] Change pg_ls_logdir to ignore error if initial/top dir
 is missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index a534d75dd8..03656a28f9 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -690,7 +690,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok,
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false, DIR_HIDE);
+	return pg_ls_dir_files(fcinfo, Log_directory, true, DIR_HIDE);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a7f34c5dd5..88121f88db 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1)b , pg_ls_tmpdir(oid)c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index edcdb4f32a..25a0081927 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v8-0006-Change-logdir-and-archive_statusdir-to-include-di.patch (7.2K, ../../20200307214010.GB1357@telsasoft.com/7-v8-0006-Change-logdir-and-archive_statusdir-to-include-di.patch)
  download | inline diff:
From 98d4a2c9768770f1325e37fab553198c1fa19da7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 5 Mar 2020 15:17:28 -0600
Subject: [PATCH v8 6/6] Change logdir and archive_statusdir to include dirs..

and add a column indicating *whether* a path is a dir.

Need catversion bump
---
 doc/src/sgml/func.sgml                    | 31 +++++++++++++----------
 src/backend/utils/adt/genfile.c           |  4 +--
 src/include/catalog/pg_proc.dat           |  8 +++---
 src/test/regress/output/tablespace.source |  4 +--
 4 files changed, 26 insertions(+), 21 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 234c9c1699..dd3faa4e7c 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21348,8 +21348,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21370,10 +21371,10 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
-        <literal>pg_monitor</literal> role and may be granted to other
-        non-superuser roles.
+        For each file in the WAL archive status directory, list the file's name,
+        size, last modification time, and boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21447,11 +21448,13 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with the file's name, size, last modified time (mtime) and boolean
+    indicating if the file is a directory.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
@@ -21470,13 +21473,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory <filename>pg_wal/archive_status</filename> along with
+    the file's name, size, last modified time (mtime) and boolean indicating if
+    the file is a directory.
+    By default only
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 03656a28f9..3da901f8d6 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -690,7 +690,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok,
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, true, DIR_HIDE);
+	return pg_ls_dir_files(fcinfo, Log_directory, true, DIR_ONLY);
 }
 
 /* Function to return the list of files in the WAL directory */
@@ -744,5 +744,5 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, DIR_HIDE);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, DIR_ONLY);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0dbce4ba09..fdf86e1eaf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10723,8 +10723,8 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
@@ -10733,8 +10733,8 @@
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 25a0081927..a5017e5f9d 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -24,8 +24,8 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-08 08:02       ` Fabien COELHO <coelho@cri.ensmp.fr>
  1 sibling, 0 replies; 70+ messages in thread

From: Fabien COELHO @ 2020-03-08 08:02 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>


Hello Justin,

Patch series applies cleanly. The last status compiles and passes "make 
check". A few more comments:

* v8.[123] ok.

* v8.4

Avoid using the type name as a field name? "enum dir_action dir_action;" 
-> "enum dir_action action", or maybe rename "dir_action" enum 
"dir_action_t".

About pg_ls_dir:

"if (!fctx->dirdesc)" I do not think that is true even if AllocateDir 
failed, the list exists anyway. ISTM it should be linitial which is NULL 
in that case.

Given the overlap between pg_ls_dir and pg_ls_dir_files, ISTM that the 
former should call the slightly extended later with appropriate flags.

About populate_paths:

function is a little bit strange to me, ISTM it would deserve more 
comments.

I'm not sure the name reflect what it does. For instance, ISTM that it 
does one thing, but the name is plural. Maybe "move_to_next_path" or 
"update_current_path" or something?

It returns an int which can only be 0 or 1, which smells like an bool. 
What is this int/bool is not told in the function head comment. I guess it 
is whether the path was updated. When it returns false, the list length is 
down to one.

Shouldn't AllocateDir be tested for bad result? Maybe it is a dir but you 
do not have perms to open it? Or give a comment about why it cannot 
happen?

later, good, at least the function is called, even if it is only for an 
error case. Maybe some non empty coverage tests could be added with a 
"count(*) > 0" on not is_dir or maybe "count(*) = 0" on is_dir, for 
instance?

   (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace'
   UNION SELECT 0 ORDER BY 1 DESC LIMIT 1)b

The 'b' glued to the ')' looks pretty strange. I'd suggest ") AS b". 
Reusing the same alias twice could be avoided for clarity, maybe.

* v8.[56]

I'd say that a behavior change which adds a column and a possibly a few 
rows is ok, especially as the tmpdir contains subdirs now.

-- 
Fabien.





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-10 18:30       ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  1 sibling, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-10 18:30 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

I took a step back, and I wondered whether we should add a generic function for
listing a dir with metadata, possibly instead of changing the existing
functions.  Then one could do pg_ls_dir_metadata('pg_wal',false,false);

Since pg8.1, we have pg_ls_dir() to show a list of files.  Since pg10, we've
had pg_ls_logdir and pg_ls_waldir, which show not only file names but also
(some) metadata (size, mtime).  And since pg12, we've had pg_ls_tmpfile and
pg_ls_archive_statusdir, which also show metadata.

...but there's no a function which lists the metadata of an directory other
than tmp, wal, log.

One can do this:
|SELECT b.*, c.* FROM (SELECT 'base' a)a, LATERAL (SELECT a||'/'||pg_ls_dir(a.a)b)b, pg_stat_file(b)c;
..but that's not as helpful as allowing:
|SELECT * FROM pg_ls_dir_metadata('.',true,true);

There's also no function which recurses into an arbitrary directory, so it
seems shortsighted to provide a function to recursively list a tmpdir.

Also, since pg_ls_dir_metadata indicates whether the path is a dir, one can
write a SQL function to show the dir recursively.  It'd be trivial to plug in
wal/log/tmp (it seems like tmpdirs of other tablespace's are not entirely
trivial).
|SELECT * FROM pg_ls_dir_recurse('base/pgsql_tmp');

Also, on a neighboring thread[1], Tom indicated that the pg_ls_* functions
should enumerate all files during the initial call, which sounds like a bad
idea when recursively showing directories.  If we add a function recursing into
a directory, we'd need to discuss all the flags to expose to it, like recurse,
ignore_errors, one_filesystem?, show_dotfiles (and eventually bikeshed all the
rest of the flags in find(1)).

My initial patch [2] changed ls_tmpdir to show metadata columns including
is_dir, but not decend.  It's pretty unfortunate if a function called
pg_ls_tmpdir hides shared filesets, so maybe it really is best to change that
(it's new in v12).

I'm interested to in feedback on the alternative approach, as attached.  The
final patch to include all the rest of columns shown by pg_stat_file() is more
of an idea/proposal and not sure if it'll be desirable.  But pg_ls_tmpdir() is
essentially the same as my v1 patch.

This is intended to be mostly independent of any fix to the WARNING I reported
[1].  Since my patch collapses pg_ls_dir into pg_ls_dir_files, we'd only need
to fix one place.  I'm planning to eventually look into Tom's suggestion of
returning tuplestore to fix that, and maybe rebase this patchset on top of
that.

-- 
Justin

[1] https://www.postgresql.org/message-id/flat/20200308173103.GC1357%40telsasoft.com
[2] https://www.postgresql.org/message-id/20191214224735.GA28433%40telsasoft.com

Attachments:

  [text/x-diff] v9-0001-BUG-in-errmsg.patch (950B, ../../20200310183037.GA29065@telsasoft.com/2-v9-0001-BUG-in-errmsg.patch)
  download | inline diff:
From 2c4b2c408490ecde3cfb4e336a78942f7a6f8197 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 27 Dec 2019 23:34:14 -0600
Subject: [PATCH v9 01/11] BUG: in errmsg

Note there's two changes here.
Should backpatch to v12, where pg_ls_tmpdir was added.
---
 src/backend/utils/adt/genfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 3741b87486..897b11a77d 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -590,7 +590,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 		if (stat(path, &attrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
-					 errmsg("could not stat directory \"%s\": %m", dir)));
+					 errmsg("could not stat file \"%s\": %m", path)));
 
 		/* Ignore anything but regular files */
 		if (!S_ISREG(attrib.st_mode))
-- 
2.17.0

  [text/x-diff] v9-0002-Document-historic-behavior-about-hiding-directori.patch (1.7K, ../../20200310183037.GA29065@telsasoft.com/3-v9-0002-Document-historic-behavior-about-hiding-directori.patch)
  download | inline diff:
From f3ef0c6ff664f2f26e95ce97e8b50a813bd1aab8 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 16:50:07 -0600
Subject: [PATCH v9 02/11] Document historic behavior about hiding directories
 and special files

Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
 doc/src/sgml/func.sgml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v9-0003-Document-historic-behavior-about-hiding-directori.patch (905B, ../../20200310183037.GA29065@telsasoft.com/4-v9-0003-Document-historic-behavior-about-hiding-directori.patch)
  download | inline diff:
From b1fb8ef86178fff474f0205143b6f37e156d0d9f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:12:04 -0600
Subject: [PATCH v9 03/11] Document historic behavior about hiding directories
 and special files

Should backpatch to v12: tmpdir
---
 doc/src/sgml/func.sgml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v9-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-me.patch (5.1K, ../../20200310183037.GA29065@telsasoft.com/5-v9-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-me.patch)
  download | inline diff:
From f1350c7880c8ca94358c114808f90d9fc94aa08f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 17:15:02 -0500
Subject: [PATCH v9 04/11] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Need catversion bumped?
---
 src/backend/utils/adt/genfile.c | 51 ++++++++++++++++++++++++++-------
 src/include/catalog/pg_proc.dat |  6 ++++
 2 files changed, 47 insertions(+), 10 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 897b11a77d..4699aea568 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -524,7 +524,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 
 /* Generic function to return a directory listing of files */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok)
 {
 	FuncCallContext *funcctx;
 	struct dirent *de;
@@ -540,13 +540,17 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc(3);
+		tupdesc = CreateTemplateTupleDesc(dir_ok ? 4:3);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
 						   INT8OID, -1, 0);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
 						   TIMESTAMPTZOID, -1, 0);
+		if (dir_ok)
+			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+					BOOLOID, -1, 0);
+
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
 
 		fctx->location = pstrdup(dir);
@@ -575,8 +579,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 		HeapTuple	tuple;
@@ -592,13 +596,21 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (!dir_ok)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+			/* Ignore anything but regular files */
 			continue;
 
 		values[0] = CStringGetTextDatum(de->d_name);
 		values[1] = Int64GetDatum((int64) attrib.st_size);
 		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (dir_ok)
+			values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
@@ -613,14 +625,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, false, false);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, false, false);
 }
 
 /*
@@ -638,7 +650,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path, true, false);
 }
 
 /*
@@ -667,5 +679,24 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+	bool	missing_ok = false;
+	bool	include_dot_dirs = false;
+
+	if (!PG_ARGISNULL(1))
+		missing_ok = PG_GETARG_BOOL(1);
+	if (!PG_ARGISNULL(2))
+		/* XXX: Not implemented */
+		include_dot_dirs = PG_GETARG_BOOL(2);
+
+	return pg_ls_dir_files(fcinfo, dirname, missing_ok, true);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7fb574f9dc..0a1859f709 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10741,6 +10741,12 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
+  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
-- 
2.17.0

  [text/x-diff] v9-0005-Add-tests-exercizing-pg_ls_tmpdir.patch (2.8K, ../../20200310183037.GA29065@telsasoft.com/6-v9-0005-Add-tests-exercizing-pg_ls_tmpdir.patch)
  download | inline diff:
From a90befd3d648bb591af4aba2f903e90b02640bb0 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v9 05/11] Add tests exercizing pg_ls_tmpdir..

..and its backing function pg_ls_dir_files
---
 src/test/regress/input/tablespace.source  | 5 +++++
 src/test/regress/output/tablespace.source | 8 ++++++++
 2 files changed, 13 insertions(+)

diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v9-0006-pg_ls_tmpdir-to-show-isdir-argument.patch (6.5K, ../../20200310183037.GA29065@telsasoft.com/7-v9-0006-pg_ls_tmpdir-to-show-isdir-argument.patch)
  download | inline diff:
From 018b74872089021ba1319aef01ebfb8154ee4b58 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v9 06/11] pg_ls_tmpdir to show "isdir" argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

Note that "dir_ok" means both: do not skip dirs, *and* show "isdir" column.
The other functions both skip dirs and only show 3 columns, and ls_tmpdir is
changing to both show dirs and show 4 columns.  It wouldn't make sense to skip
dirs and then show a 4th column always false.

Need catversion bump
---
 doc/src/sgml/func.sgml                    | 22 ++++++++++++----------
 src/backend/utils/adt/genfile.c           |  4 ++--
 src/include/catalog/pg_proc.dat           |  8 ++++----
 src/test/regress/output/tablespace.source |  4 ++--
 4 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..7c73d3b82a 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21382,12 +21382,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21482,14 +21482,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4699aea568..7168e249a4 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -522,7 +522,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 	return pg_ls_dir(fcinfo);
 }
 
-/* Generic function to return a directory listing of files */
+/* Generic function to return a directory listing of files (and optionally dirs) */
 static Datum
 pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok)
 {
@@ -650,7 +650,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true, false);
+	return pg_ls_dir_files(fcinfo, path, true, true);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0a1859f709..7789d029ea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10733,13 +10733,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v9-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (2.8K, ../../20200310183037.GA29065@telsasoft.com/8-v9-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From 907f88c171bd22f382b707da1cdbbdfd63fb177a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v9 07/11] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

TODO:
src/backend/catalog/system_views.sql:REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;

Need catversion bumped ?
---
 doc/src/sgml/func.sgml          | 13 +++++++++++++
 src/include/catalog/pg_proc.dat |  8 ++++++++
 2 files changed, 21 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 7c73d3b82a..672cbab7b9 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21442,6 +21442,19 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    <parameter>missing_ok</parameter> ???
+    To recusively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp')dir  FROM pg_tablespace b, pg_control_system() pcs, format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) suffix)dir, pg_ls_dir_recurse(dir)a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7789d029ea..cc2c6f6571 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6143,6 +6143,14 @@
   proname => 'pg_ls_dir', prorows => '1000', proretset => 't',
   provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool',
   prosrc => 'pg_ls_dir' },
+
+{ oid => '8511', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false, true) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false, true)a WHERE x.isdir) SELECT * FROM x" },
+
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
   proargtypes => 'float8', prosrc => 'pg_sleep' },
-- 
2.17.0

  [text/x-diff] v9-0008-generalize-pg_ls_dir_files-and-retire-pg_ls_dir.patch (8.4K, ../../20200310183037.GA29065@telsasoft.com/9-v9-0008-generalize-pg_ls_dir_files-and-retire-pg_ls_dir.patch)
  download | inline diff:
From 78e8824063de0170022595644e1be68e341dcb10 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 23:51:22 -0500
Subject: [PATCH v9 08/11] generalize pg_ls_dir_files and retire pg_ls_dir

---
 src/backend/utils/adt/genfile.c | 172 +++++++++++++++-----------------
 1 file changed, 79 insertions(+), 93 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 7168e249a4..888560f78e 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -42,6 +42,14 @@ typedef struct
 	bool		include_dot_dirs;
 } directory_fctx;
 
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	FLAG_ISDIR				(1<<0) /* Show column: isdir */
+#define	FLAG_METADATA			(1<<1) /* Show columns: mtime, size */
+#define	FLAG_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	FLAG_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	FLAG_SKIP_HIDDEN		(1<<4) /* Do not show anything begining with . */
+#define	FLAG_SKIP_DIRS			(1<<5) /* Do not show directories */
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -446,67 +454,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	FuncCallContext *funcctx;
-	struct dirent *de;
-	directory_fctx *fctx;
-	MemoryContext oldcontext;
-
-	if (SRF_IS_FIRSTCALL())
-	{
-		bool		missing_ok = false;
-		bool		include_dot_dirs = false;
-
-		/* check the optional arguments */
-		if (PG_NARGS() == 3)
-		{
-			if (!PG_ARGISNULL(1))
-				missing_ok = PG_GETARG_BOOL(1);
-			if (!PG_ARGISNULL(2))
-				include_dot_dirs = PG_GETARG_BOOL(2);
-		}
-
-		funcctx = SRF_FIRSTCALL_INIT();
-		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
-
-		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
-
-		if (!fctx->dirdesc)
-		{
-			if (missing_ok && errno == ENOENT)
-			{
-				MemoryContextSwitchTo(oldcontext);
-				SRF_RETURN_DONE(funcctx);
-			}
-			else
-				ereport(ERROR,
-						(errcode_for_file_access(),
-						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
-		}
-		funcctx->user_fctx = fctx;
-		MemoryContextSwitchTo(oldcontext);
-	}
-
-	funcctx = SRF_PERCALL_SETUP();
-	fctx = (directory_fctx *) funcctx->user_fctx;
-
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
-	{
-		if (!fctx->include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
-	}
-
-	FreeDir(fctx->dirdesc);
-
-	SRF_RETURN_DONE(funcctx);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, FLAG_SKIP_DOT_DIRS);
 }
 
 /*
@@ -519,17 +469,44 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, FLAG_SKIP_DOT_DIRS);
 }
 
 /* Generic function to return a directory listing of files (and optionally dirs) */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	FuncCallContext *funcctx;
 	struct dirent *de;
 	directory_fctx *fctx;
 
+	/* isdir depends on metadata */
+	Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= FLAG_MISSING_OK;
+			else
+				flags &= ~FLAG_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~FLAG_SKIP_DOT_DIRS;
+			else
+				flags |= FLAG_SKIP_DOT_DIRS;
+		}
+	}
+
 	if (SRF_IS_FIRSTCALL())
 	{
 		MemoryContext oldcontext;
@@ -540,16 +517,20 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc(dir_ok ? 4:3);
+		tupdesc = CreateTemplateTupleDesc((flags&FLAG_ISDIR) ? 4 :
+				(flags&FLAG_METADATA) ? 3 : 1);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
-		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
-						   INT8OID, -1, 0);
-		TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
-						   TIMESTAMPTZOID, -1, 0);
-		if (dir_ok)
-			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
-					BOOLOID, -1, 0);
+		if (flags&FLAG_METADATA)
+		{
+			TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
+							   INT8OID, -1, 0);
+			TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
+							   TIMESTAMPTZOID, -1, 0);
+			if (flags&FLAG_ISDIR)
+				TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+						BOOLOID, -1, 0);
+		}
 
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
 
@@ -558,7 +539,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 
 		if (!fctx->dirdesc)
 		{
-			if (missing_ok && errno == ENOENT)
+			if (flags&FLAG_MISSING_OK && errno == ENOENT)
 			{
 				MemoryContextSwitchTo(oldcontext);
 				SRF_RETURN_DONE(funcctx);
@@ -585,8 +566,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 		struct stat attrib;
 		HeapTuple	tuple;
 
+		if (flags&FLAG_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
 		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		if (flags&FLAG_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
@@ -598,18 +585,21 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 
 		if (S_ISDIR(attrib.st_mode))
 		{
-			if (!dir_ok)
+			if (flags&FLAG_SKIP_DIRS)
 				continue;
 		}
 		else if (!S_ISREG(attrib.st_mode))
-			/* Ignore anything but regular files */
 			continue;
 
-		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-		if (dir_ok)
-			values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+		if (flags & FLAG_METADATA)
+		{
+			values[0] = CStringGetTextDatum(de->d_name);
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & FLAG_ISDIR)
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+		} else
+			SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
 
 		memset(nulls, 0, sizeof(nulls));
 
@@ -625,14 +615,16 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false, false);
+	return pg_ls_dir_files(fcinfo, Log_directory,
+			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR,
+			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /*
@@ -650,7 +642,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true, true);
+	return pg_ls_dir_files(fcinfo, path,
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA|FLAG_ISDIR);
 }
 
 /*
@@ -679,7 +672,8 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /*
@@ -689,14 +683,6 @@ Datum
 pg_ls_dir_metadata(PG_FUNCTION_ARGS)
 {
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-	bool	missing_ok = false;
-	bool	include_dot_dirs = false;
-
-	if (!PG_ARGISNULL(1))
-		missing_ok = PG_GETARG_BOOL(1);
-	if (!PG_ARGISNULL(2))
-		/* XXX: Not implemented */
-		include_dot_dirs = PG_GETARG_BOOL(2);
 
-	return pg_ls_dir_files(fcinfo, dirname, missing_ok, true);
+	return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_ISDIR);
 }
-- 
2.17.0

  [text/x-diff] v9-0009-pg_ls_logdir-to-ignore-error-if-initial-top-dir-i.patch (3.0K, ../../20200310183037.GA29065@telsasoft.com/10-v9-0009-pg_ls_logdir-to-ignore-error-if-initial-top-dir-i.patch)
  download | inline diff:
From 43c8bc8611b53ff172e5c7caa052d80d9e4ac3e9 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v9 09/11] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 888560f78e..4ce39516d7 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -616,7 +616,7 @@ Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, Log_directory,
-			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..dacaafcae6 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v9-0010-pg_ls_-dir-to-show-directories-and-isdir-column.patch (13.3K, ../../20200310183037.GA29065@telsasoft.com/11-v9-0010-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From adf2977cf2553a10a9c2cb6cb3a52590365783bf Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v9 10/11] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

..FLAG_ISDIR is collapsed into FLAG_METADATA.

Need catversion bump
---
 doc/src/sgml/func.sgml          | 39 +++++++++++++++++--------------
 src/backend/utils/adt/genfile.c | 41 +++++++++++----------------------
 src/include/catalog/pg_proc.dat | 20 ++++++++--------
 3 files changed, 46 insertions(+), 54 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 672cbab7b9..d0b782d803 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21348,8 +21348,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21359,8 +21360,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21370,8 +21372,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21459,36 +21462,38 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4ce39516d7..387114d4ee 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -44,12 +44,10 @@ typedef struct
 
 static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
 
-#define	FLAG_ISDIR				(1<<0) /* Show column: isdir */
-#define	FLAG_METADATA			(1<<1) /* Show columns: mtime, size */
-#define	FLAG_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
-#define	FLAG_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
-#define	FLAG_SKIP_HIDDEN		(1<<4) /* Do not show anything begining with . */
-#define	FLAG_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	FLAG_METADATA			(1<<0) /* Show columns: mtime, size */
+#define	FLAG_MISSING_OK			(1<<1) /* Ignore ENOENT if the toplevel dir is missing */
+#define	FLAG_SKIP_DOT_DIRS		(1<<2) /* Do not show . or .. */
+#define	FLAG_SKIP_HIDDEN		(1<<3) /* Do not show anything begining with . */
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -482,11 +480,6 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 	struct dirent *de;
 	directory_fctx *fctx;
 
-	/* isdir depends on metadata */
-	Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA));
-	/* Unreasonable to show isdir and skip dirs */
-	Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS));
-
 	/* check the optional arguments */
 	if (PG_NARGS() == 3)
 	{
@@ -517,8 +510,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc((flags&FLAG_ISDIR) ? 4 :
-				(flags&FLAG_METADATA) ? 3 : 1);
+		tupdesc = CreateTemplateTupleDesc((flags&FLAG_METADATA) ? 4 : 1);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
 		if (flags&FLAG_METADATA)
@@ -527,9 +519,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 							   INT8OID, -1, 0);
 			TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
 							   TIMESTAMPTZOID, -1, 0);
-			if (flags&FLAG_ISDIR)
-				TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
-						BOOLOID, -1, 0);
+			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+								BOOLOID, -1, 0);
 		}
 
 		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
@@ -584,10 +575,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 					 errmsg("could not stat file \"%s\": %m", path)));
 
 		if (S_ISDIR(attrib.st_mode))
-		{
-			if (flags&FLAG_SKIP_DIRS)
-				continue;
-		}
+			; /* Do nothing, fall through */
 		else if (!S_ISREG(attrib.st_mode))
 			continue;
 
@@ -596,8 +584,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 			values[0] = CStringGetTextDatum(de->d_name);
 			values[1] = Int64GetDatum((int64) attrib.st_size);
 			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & FLAG_ISDIR)
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 		} else
 			SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
 
@@ -616,7 +603,7 @@ Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, Log_directory,
-			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /* Function to return the list of files in the WAL directory */
@@ -624,7 +611,7 @@ Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR,
-			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
+			FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /*
@@ -643,7 +630,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA|FLAG_ISDIR);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /*
@@ -673,7 +660,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA);
 }
 
 /*
@@ -684,5 +671,5 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS)
 {
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
 
-	return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_ISDIR);
+	return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index cc2c6f6571..0e5a570285 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6149,7 +6149,7 @@
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
   proallargtypes => '{text,text,int8,timestamptz,bool}',
   proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false, true) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false, true)a WHERE x.isdir) SELECT * FROM x" },
+  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" },
 
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
@@ -10725,18 +10725,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
@@ -10751,9 +10751,9 @@
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
-  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
-  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
-- 
2.17.0

  [text/x-diff] v9-0011-pg_ls_-dir-to-return-all-the-metadata-from-pg_sta.patch (15.4K, ../../20200310183037.GA29065@telsasoft.com/12-v9-0011-pg_ls_-dir-to-return-all-the-metadata-from-pg_sta.patch)
  download | inline diff:
From e8960ff73967e5ca855ee8f392e6a7711e2ee64c Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v9 11/11] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

Need catversion bump
---
 doc/src/sgml/func.sgml                    | 30 +++++++++------------
 src/backend/utils/adt/genfile.c           | 32 ++++++++++++++++-------
 src/include/catalog/pg_proc.dat           | 30 ++++++++++-----------
 src/test/regress/output/tablespace.source |  8 +++---
 4 files changed, 55 insertions(+), 45 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index d0b782d803..ae6c004a6d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21348,8 +21348,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21360,8 +21359,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21372,9 +21370,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21386,8 +21383,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21463,8 +21459,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21475,8 +21471,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21488,8 +21484,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21502,8 +21498,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 387114d4ee..1fe7e0181d 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -510,16 +510,22 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 		fctx = palloc(sizeof(directory_fctx));
 
-		tupdesc = CreateTemplateTupleDesc((flags&FLAG_METADATA) ? 4 : 1);
+		tupdesc = CreateTemplateTupleDesc((flags&FLAG_METADATA) ? 7 : 1);
 		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name",
 						   TEXTOID, -1, 0);
 		if (flags&FLAG_METADATA)
 		{
 			TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size",
 							   INT8OID, -1, 0);
-			TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification",
+			TupleDescInitEntry(tupdesc, (AttrNumber) 3,
+							   "access", TIMESTAMPTZOID, -1, 0);
+			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "modification",
 							   TIMESTAMPTZOID, -1, 0);
-			TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir",
+			TupleDescInitEntry(tupdesc, (AttrNumber) 5,
+							   "change", TIMESTAMPTZOID, -1, 0);
+			TupleDescInitEntry(tupdesc, (AttrNumber) 6,
+							   "creation", TIMESTAMPTZOID, -1, 0);
+			TupleDescInitEntry(tupdesc, (AttrNumber) 7, "isdir",
 								BOOLOID, -1, 0);
 		}
 
@@ -551,8 +557,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 		HeapTuple	tuple;
@@ -579,17 +585,25 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 		else if (!S_ISREG(attrib.st_mode))
 			continue;
 
+		memset(nulls, 0, sizeof(nulls));
 		if (flags & FLAG_METADATA)
 		{
 			values[0] = CStringGetTextDatum(de->d_name);
 			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
+			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 		} else
 			SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
 
-		memset(nulls, 0, sizeof(nulls));
-
 		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
 		SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple));
 	}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0e5a570285..b571666a79 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6147,9 +6147,9 @@
 { oid => '8511', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size,a.access,a.modification,a.change,a.creation,a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" },
 
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
@@ -10725,35 +10725,35 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index dacaafcae6..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-13 13:12         ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  0 siblings, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-13 13:12 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

@cfbot: rebased onto 085b6b6679e73b9b386f209b4d625c7bc60597c0

The merge conflict presents another opportunity to solicit comments on the new
approach.  Rather than making "recurse into tmpdir" the end goal:

  - add a function to show metadata of an arbitrary dir;
  - add isdir arguments to pg_ls_* functions (including pg_ls_tmpdir but not
    pg_ls_dir).
  - maybe add pg_ls_dir_recurse, which satisfies the original need;
  - retire pg_ls_dir (does this work with tuplestore?)
  - profit

The alternative seems to be to go back to Alvaro's earlier proposal:
 - not only add "isdir", but also recurse;

I think I would insist on adding a general function to recurse into any dir.
And *optionally* change ps_ls_* to recurse (either by accepting an argument, or
by making that a separate patch to debate).  tuplestore is certainly better
than keeping a stack/List of DIRs for this.

On Tue, Mar 10, 2020 at 01:30:37PM -0500, Justin Pryzby wrote:
> I took a step back, and I wondered whether we should add a generic function for
> listing a dir with metadata, possibly instead of changing the existing
> functions.  Then one could do pg_ls_dir_metadata('pg_wal',false,false);
> 
> Since pg8.1, we have pg_ls_dir() to show a list of files.  Since pg10, we've
> had pg_ls_logdir and pg_ls_waldir, which show not only file names but also
> (some) metadata (size, mtime).  And since pg12, we've had pg_ls_tmpfile and
> pg_ls_archive_statusdir, which also show metadata.
> 
> ...but there's no a function which lists the metadata of an directory other
> than tmp, wal, log.
> 
> One can do this:
> |SELECT b.*, c.* FROM (SELECT 'base' a)a, LATERAL (SELECT a||'/'||pg_ls_dir(a.a)b)b, pg_stat_file(b)c;
> ..but that's not as helpful as allowing:
> |SELECT * FROM pg_ls_dir_metadata('.',true,true);
> 
> There's also no function which recurses into an arbitrary directory, so it
> seems shortsighted to provide a function to recursively list a tmpdir.
> 
> Also, since pg_ls_dir_metadata indicates whether the path is a dir, one can
> write a SQL function to show the dir recursively.  It'd be trivial to plug in
> wal/log/tmp (it seems like tmpdirs of other tablespace's are not entirely
> trivial).
> |SELECT * FROM pg_ls_dir_recurse('base/pgsql_tmp');
> 
> Also, on a neighboring thread[1], Tom indicated that the pg_ls_* functions
> should enumerate all files during the initial call, which sounds like a bad
> idea when recursively showing directories.  If we add a function recursing into
> a directory, we'd need to discuss all the flags to expose to it, like recurse,
> ignore_errors, one_filesystem?, show_dotfiles (and eventually bikeshed all the
> rest of the flags in find(1)).
> 
> My initial patch [2] changed ls_tmpdir to show metadata columns including
> is_dir, but not decend.  It's pretty unfortunate if a function called
> pg_ls_tmpdir hides shared filesets, so maybe it really is best to change that
> (it's new in v12).
> 
> I'm interested to in feedback on the alternative approach, as attached.  The
> final patch to include all the rest of columns shown by pg_stat_file() is more
> of an idea/proposal and not sure if it'll be desirable.  But pg_ls_tmpdir() is
> essentially the same as my v1 patch.
> 
> This is intended to be mostly independent of any fix to the WARNING I reported
> [1].  Since my patch collapses pg_ls_dir into pg_ls_dir_files, we'd only need
> to fix one place.  I'm planning to eventually look into Tom's suggestion of
> returning tuplestore to fix that, and maybe rebase this patchset on top of
> that.
> 
> -- 
> Justin
> 
> [1] https://www.postgresql.org/message-id/flat/20200308173103.GC1357%40telsasoft.com
> [2] https://www.postgresql.org/message-id/20191214224735.GA28433%40telsasoft.com

Attachments:

  [text/x-diff] v10-0001-Document-historic-behavior-about-hiding-director.patch (1.7K, ../../20200313131232.GO29065@telsasoft.com/2-v10-0001-Document-historic-behavior-about-hiding-director.patch)
  download | inline diff:
From 950eed8812a167a12da49553f7e3a2d1438d779b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 16:50:07 -0600
Subject: [PATCH v10 1/9] Document historic behavior about hiding directories
 and special files

Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
 doc/src/sgml/func.sgml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v10-0002-Document-historic-behavior-about-hiding-director.patch (904B, ../../20200313131232.GO29065@telsasoft.com/3-v10-0002-Document-historic-behavior-about-hiding-director.patch)
  download | inline diff:
From a04d046211dbddfe1cb96fd794ed9ef269e2415f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:12:04 -0600
Subject: [PATCH v10 2/9] Document historic behavior about hiding directories
 and special files

Should backpatch to v12: tmpdir
---
 doc/src/sgml/func.sgml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v10-0003-Add-tests-exercizing-pg_ls_tmpdir.patch (4.3K, ../../20200313131232.GO29065@telsasoft.com/4-v10-0003-Add-tests-exercizing-pg_ls_tmpdir.patch)
  download | inline diff:
From 11100ba3d74051825972f80f5d5f93237762343f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v10 3/9] Add tests exercizing pg_ls_tmpdir..

..and its backing function pg_ls_dir_files
---
 src/test/regress/expected/misc_functions.out | 7 +++++++
 src/test/regress/input/tablespace.source     | 5 +++++
 src/test/regress/output/tablespace.source    | 8 ++++++++
 src/test/regress/sql/misc_functions.sql      | 4 ++++
 4 files changed, 24 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 5eccc97bdb..3b1e62834f 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -179,6 +179,13 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
  t
 (1 row)
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 02d1fc238d..034bbd8d11 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -50,6 +50,10 @@ select (pg_ls_waldir()).size = :segsize as ok limit 1;
 
 select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v10-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (9.9K, ../../20200313131232.GO29065@telsasoft.com/5-v10-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From 969533c27663713d87f6ff2eb0a3c32b13f5c146 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 17:15:02 -0500
Subject: [PATCH v10 4/9] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

Need catversion bumped?
---
 src/backend/utils/adt/genfile.c | 184 +++++++++++++++++---------------
 src/include/catalog/pg_proc.dat |   6 ++
 2 files changed, 106 insertions(+), 84 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index bcf9bd1b97..4f437573ab 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,13 +36,15 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
-typedef struct
-{
-	char	   *location;
-	DIR		   *dirdesc;
-	bool		include_dot_dirs;
-} directory_fctx;
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
 
+#define	FLAG_ISDIR				(1<<0) /* Show column: isdir */
+#define	FLAG_METADATA			(1<<1) /* Show columns: mtime, size */
+#define	FLAG_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	FLAG_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	FLAG_SKIP_HIDDEN		(1<<4) /* Do not show anything begining with . */
+#define	FLAG_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	FLAG_SKIP_SPECIAL		(1<<6) /* Do not show special file types */
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -447,67 +449,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	FuncCallContext *funcctx;
-	struct dirent *de;
-	directory_fctx *fctx;
-	MemoryContext oldcontext;
-
-	if (SRF_IS_FIRSTCALL())
-	{
-		bool		missing_ok = false;
-		bool		include_dot_dirs = false;
-
-		/* check the optional arguments */
-		if (PG_NARGS() == 3)
-		{
-			if (!PG_ARGISNULL(1))
-				missing_ok = PG_GETARG_BOOL(1);
-			if (!PG_ARGISNULL(2))
-				include_dot_dirs = PG_GETARG_BOOL(2);
-		}
-
-		funcctx = SRF_FIRSTCALL_INIT();
-		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
-
-		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
-
-		if (!fctx->dirdesc)
-		{
-			if (missing_ok && errno == ENOENT)
-			{
-				MemoryContextSwitchTo(oldcontext);
-				SRF_RETURN_DONE(funcctx);
-			}
-			else
-				ereport(ERROR,
-						(errcode_for_file_access(),
-						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
-		}
-		funcctx->user_fctx = fctx;
-		MemoryContextSwitchTo(oldcontext);
-	}
-
-	funcctx = SRF_PERCALL_SETUP();
-	fctx = (directory_fctx *) funcctx->user_fctx;
-
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
-	{
-		if (!fctx->include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
-	}
-
-	FreeDir(fctx->dirdesc);
-
-	SRF_RETURN_DONE(funcctx);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, FLAG_SKIP_DOT_DIRS);
 }
 
 /*
@@ -520,7 +464,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, FLAG_SKIP_DOT_DIRS);
 }
 
 /*
@@ -530,7 +476,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -539,6 +485,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= FLAG_MISSING_OK;
+			else
+				flags &= ~FLAG_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~FLAG_SKIP_DOT_DIRS;
+			else
+				flags |= FLAG_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -554,8 +526,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags&FLAG_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	} else {
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -574,7 +556,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags&FLAG_MISSING_OK && errno == ENOENT)
 		{
 			tuplestore_donestoring(tupstore);
 			return (Datum) 0;
@@ -584,13 +566,19 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
+		if (flags&FLAG_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
 		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		if (flags&FLAG_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
@@ -600,13 +588,26 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (flags&FLAG_SKIP_DIRS)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags&FLAG_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & FLAG_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & FLAG_ISDIR)
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -621,14 +622,16 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory,
+			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR,
+			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /*
@@ -646,7 +649,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /*
@@ -675,5 +679,17 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_SKIP_SPECIAL|FLAG_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7fb574f9dc..0a1859f709 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10741,6 +10741,12 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
+  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
-- 
2.17.0

  [text/x-diff] v10-0005-pg_ls_tmpdir-to-show-isdir-argument.patch (7.6K, ../../20200313131232.GO29065@telsasoft.com/6-v10-0005-pg_ls_tmpdir-to-show-isdir-argument.patch)
  download | inline diff:
From 48add2f8cb7697de9735dda714a470708cf8ff44 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v10 5/9] pg_ls_tmpdir to show "isdir" argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Note that "dir_ok" means both: do not skip dirs, *and* show "isdir" column.
The other functions both skip dirs and only show 3 columns, and ls_tmpdir is
changing to both show dirs and show 4 columns.  It wouldn't make sense to skip
dirs and then show a 4th column always false.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..7c73d3b82a 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21382,12 +21382,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21482,14 +21482,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4f437573ab..5e25e31dff 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -470,7 +470,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -650,7 +650,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_ISDIR|FLAG_METADATA);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0a1859f709..7789d029ea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10733,13 +10733,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 3b1e62834f..94b562dcb6 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -182,8 +182,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v10-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch (12.8K, ../../20200313131232.GO29065@telsasoft.com/7-v10-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From 355eddbb6dcadc994b8ce65744e7aab1b96281e6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v10 6/9] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

..FLAG_ISDIR is collapsed into FLAG_METADATA.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 35 ++++++------------
 src/include/catalog/pg_proc.dat              | 19 +++++-----
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 46 insertions(+), 51 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 7c73d3b82a..3d8f7dff96 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21348,8 +21348,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21359,8 +21360,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21370,8 +21372,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21446,36 +21449,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 5e25e31dff..3ea8c5683e 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -38,13 +38,11 @@
 
 static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
 
-#define	FLAG_ISDIR				(1<<0) /* Show column: isdir */
-#define	FLAG_METADATA			(1<<1) /* Show columns: mtime, size */
-#define	FLAG_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
-#define	FLAG_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
-#define	FLAG_SKIP_HIDDEN		(1<<4) /* Do not show anything begining with . */
-#define	FLAG_SKIP_DIRS			(1<<5) /* Do not show directories */
-#define	FLAG_SKIP_SPECIAL		(1<<6) /* Do not show special file types */
+#define	FLAG_METADATA			(1<<0) /* Show columns: mtime, size */
+#define	FLAG_MISSING_OK			(1<<1) /* Ignore ENOENT if the toplevel dir is missing */
+#define	FLAG_SKIP_DOT_DIRS		(1<<2) /* Do not show . or .. */
+#define	FLAG_SKIP_HIDDEN		(1<<3) /* Do not show anything begining with . */
+#define	FLAG_SKIP_SPECIAL		(1<<4) /* Do not show special file types */
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -487,11 +485,6 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 	MemoryContext oldcontext;
 	TypeFuncClass	tuptype ;
 
-	/* isdir depends on metadata */
-	Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA));
-	/* Unreasonable to show isdir and skip dirs */
-	Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS));
-
 	/* check the optional arguments */
 	if (PG_NARGS() == 3)
 	{
@@ -589,10 +582,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 					 errmsg("could not stat file \"%s\": %m", path)));
 
 		if (S_ISDIR(attrib.st_mode))
-		{
-			if (flags&FLAG_SKIP_DIRS)
-				continue;
-		}
+			; /* Do nothing, fall through */
 		else if (!S_ISREG(attrib.st_mode))
 		{
 			if (flags&FLAG_SKIP_SPECIAL)
@@ -604,8 +594,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 		{
 			values[1] = Int64GetDatum((int64) attrib.st_size);
 			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & FLAG_ISDIR)
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 		}
 
 		memset(nulls, 0, sizeof(nulls));
@@ -623,7 +612,7 @@ Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, Log_directory,
-			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
+			FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /* Function to return the list of files in the WAL directory */
@@ -631,7 +620,7 @@ Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR,
-			FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
+			FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /*
@@ -650,7 +639,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_ISDIR|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /*
@@ -680,7 +669,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /*
@@ -691,5 +680,5 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS)
 {
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
 
-	return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_SKIP_SPECIAL|FLAG_ISDIR);
+	return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_SKIP_SPECIAL);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7789d029ea..b4c3d15495 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6143,6 +6143,7 @@
   proname => 'pg_ls_dir', prorows => '1000', proretset => 't',
   provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool',
   prosrc => 'pg_ls_dir' },
+
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
   proargtypes => 'float8', prosrc => 'pg_sleep' },
@@ -10717,18 +10718,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
@@ -10743,9 +10744,9 @@
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
-  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
-  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 94b562dcb6..d11b8669c3 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v10-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (2.7K, ../../20200313131232.GO29065@telsasoft.com/8-v10-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From fc5adf721c5a89c99a0d5caffcb35cf13ab1cf7f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v10 7/9] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

TODO:
src/backend/catalog/system_views.sql:REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;

Need catversion bumped ?
---
 doc/src/sgml/func.sgml          | 13 +++++++++++++
 src/include/catalog/pg_proc.dat |  7 +++++++
 2 files changed, 20 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 3d8f7dff96..d0b782d803 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21445,6 +21445,19 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    <parameter>missing_ok</parameter> ???
+    To recusively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp')dir  FROM pg_tablespace b, pg_control_system() pcs, format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) suffix)dir, pg_ls_dir_recurse(dir)a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b4c3d15495..75423db22e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6144,6 +6144,13 @@
   provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool',
   prosrc => 'pg_ls_dir' },
 
+{ oid => '8511', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false, true) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false, true)a WHERE x.isdir) SELECT * FROM x" },
+
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
   proargtypes => 'float8', prosrc => 'pg_sleep' },
-- 
2.17.0

  [text/x-diff] v10-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.0K, ../../20200313131232.GO29065@telsasoft.com/9-v10-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From f99592c990ebcad27b0cbe1ef6c75f8c8696fad0 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v10 8/9] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 3ea8c5683e..19e0b8a82c 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -612,7 +612,7 @@ Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, Log_directory,
-			FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
+			FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..dacaafcae6 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v10-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (15.3K, ../../20200313131232.GO29065@telsasoft.com/10-v10-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From 63564d624258925fc71bad9796bc3bcff6ceb5ec Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v10 9/9] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 +++++++++-----------
 src/backend/utils/adt/genfile.c              | 17 ++++++++---
 src/include/catalog/pg_proc.dat              | 30 ++++++++++----------
 src/test/regress/expected/misc_functions.out |  8 +++---
 src/test/regress/output/tablespace.source    |  8 +++---
 5 files changed, 49 insertions(+), 44 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index d0b782d803..ae6c004a6d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21348,8 +21348,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21360,8 +21359,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21372,9 +21370,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21386,8 +21383,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21463,8 +21459,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21475,8 +21471,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21488,8 +21484,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21502,8 +21498,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 19e0b8a82c..fb6cdc07a7 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -559,8 +559,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
@@ -593,8 +593,17 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 		if (flags & FLAG_METADATA)
 		{
 			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
+			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 		}
 
 		memset(nulls, 0, sizeof(nulls));
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 75423db22e..6c11aa940e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6147,9 +6147,9 @@
 { oid => '8511', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false, true) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false, true)a WHERE x.isdir) SELECT * FROM x" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size, a.access, a.modification,a.change,a.creation,a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" },
 
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
@@ -10725,35 +10725,35 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d11b8669c3..ea7056cd5c 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -182,8 +182,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index dacaafcae6..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-15 17:15           ` Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 1 reply; 70+ messages in thread

From: Fabien COELHO @ 2020-03-15 17:15 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>


Hello Justin,

Some feedback on v10:

All patches apply cleanly, one on top of the previous. I really wish there 
would be less than 9 patches…

* v10.1 doc change: ok

* v10.2 doc change: ok, not sure why it is not merged with previous

* v10.3 test add: could be merge with both previous

Tests seem a little contrived. I'm wondering whether something more 
straightforward could be proposed. For instance, once the tablespace is 
just created but not used yet, probably we do know that the tmp file 
exists and is empty?

* v10.4 at least, some code!

Compiles, make check ok.

pg_ls_dir_files: I'm fine with the flag approach given the number of 
switches and the internal nature of the function.

I'm not sure of the "FLAG_" prefix which seems too generic, even if it is 
local. I'd suggest "LS_DIR_*", maybe, as a more specific prefix.

ISTM that Pg style requires spaces around operators. I'd suggest some 
parenthesis would help as well, eg: "flags&FLAG_MISSING_OK" -> "(flags & 
FLAG_MISSING_OK)" and other instances.

About:

  if (S_ISDIR(attrib.st_mode)) {
    if (flags&FLAG_SKIP_DIRS)
      continue;
   }

and similars, why not the simpler:

  if (S_ISDIR(attrib.st_mode) && (flags & FLAG_SKIP_DIRS))
     continue;

Especially that it is done like that in previous cases.

Maybe I'd create defines for long and common flag specs, eg:

  #define ..._LS_SIMPLE (FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA)

No attempt at recursing.

I'm not sure about these asserts:

        /* isdir depends on metadata */
        Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA));

Hmmm. Why?

        /* Unreasonable to show isdir and skip dirs */
        Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS));

Hmmm. Why would I prevent that, even if it has little sense, it should 
work. I do not see having false on the isdir column as an actual issue.

* v10.5 add is_dir column, a few tests & doc.

Ok.

* v10.6 behavior change for existing functions, always show isdir column,
and removal of IS_DIR flag.

I'm unsure why the features are removed, some use case may benefit from 
the more complete function?

Maybe flags defs should not be changed anyway?

I do not like much the "if (...) /* empty */;" code. Maybe it could be 
caught more cleanly later in the conditional structure.

* v10.7 adds "pg_ls_dir_recurse" function

Using sql recurse to possibly to implement the feature is pretty elegant
and limits open directories to one at a time, which is pretty neat.

Doc looks incomplete and the example is very contrived and badly indented.

The function definition does not follow the style around: uppercase 
whereas all others are lowercase, "" instead of '', no "as"…

I do not understand why oid 8511 is given to the new function.

I do not understand why UNION ALL and not UNION.

I would have put the definition after "pg_ls_dir_metadata" definition.

pg_ls_dir_metadata seems defined as (text,bool,bool) but called as 
(text,bool,bool,bool).

Maybe a better alias could be given instead of x?

There are no tests for the new function. I'm not sure it would work.

* v10.8 change flags & add test on pg_ls_logdir().

I'm unsure why it is done at this stage.

* v10.9 change some ls functions and fix patch 10.7 issue

I'm unsure why it is done at this stage. "make check" ok.

-- 
Fabien.

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-03-15 21:27             ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  0 siblings, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-15 21:27 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Sun, Mar 15, 2020 at 06:15:02PM +0100, Fabien COELHO wrote:
> Some feedback on v10:

Thanks for looking.  I'm hoping to hear from Alvaro what he thinks of this
approach (all functions to show isdir, rather than one function which lists
recursively).

> All patches apply cleanly, one on top of the previous. I really wish there
> would be less than 9 patches…

I kept them separate to allow the earlier patches to be applied.
And intended to make easier to review, even if it's more work for me..

If you mean that it's a pain to apply 9 patches, I will suggest to use:
|git am -3 ./mailbox
where ./mailbox is either a copy of the mail you received, or retrieved from
the web interface.

To test that each one works (compiles, passes tests, etc), I use git rebase -i
HEAD~11 and "e"edit the target (set of) patches.

> * v10.1 doc change: ok
> 
> * v10.2 doc change: ok, not sure why it is not merged with previous

As I mentioned, separate since I'm proposing that they're backpatched to
different releases.  Those could be applied now (and Tom already applied a
patch identical to 0001 in a prior patchset).

> * v10.3 test add: could be merge with both previous

> Tests seem a little contrived. I'm wondering whether something more
> straightforward could be proposed. For instance, once the tablespace is just
> created but not used yet, probably we do know that the tmp file exists and
> is empty?

The tmpdir *doesn't* exist until someone creates tmpfiles there.
As it mentions:
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet

> * v10.4 at least, some code!
> I'm not sure of the "FLAG_" prefix which seems too generic, even if it is
> local. I'd suggest "LS_DIR_*", maybe, as a more specific prefix.

Done.

> ISTM that Pg style requires spaces around operators. I'd suggest some
> parenthesis would help as well, eg: "flags&FLAG_MISSING_OK" -> "(flags &
> FLAG_MISSING_OK)" and other instances.

Partially took your suggestion.

> About:
> 
>  if (S_ISDIR(attrib.st_mode)) {
>    if (flags&FLAG_SKIP_DIRS)
>      continue;
>   }
> 
> and similars, why not the simpler:
> 
>  if (S_ISDIR(attrib.st_mode) && (flags & FLAG_SKIP_DIRS))
>     continue;

That's not the same - if SKIP_DIRS isn't set, your way would fail that test for
dirs, and then hit the !ISREG test, and skip them anyway.
|else if (!S_ISREG(attrib.st_mode))
|	continue

> Maybe I'd create defines for long and common flag specs, eg:
> 
>  #define ..._LS_SIMPLE (FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA)

Done

> I'm not sure about these asserts:
> 
>        /* isdir depends on metadata */
>        Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA));
> 
> Hmmm. Why?

It's not supported to show isdir without showing metadata (because that case
isn't needed to support the old and the new behaviors).

+               if (flags & FLAG_METADATA)
+               {
+                       values[1] = Int64GetDatum((int64) attrib.st_size);
+                       values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+                       if (flags & FLAG_ISDIR)
+                               values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+               }

>        /* Unreasonable to show isdir and skip dirs */
>        Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS));
> 
> Hmmm. Why would I prevent that, even if it has little sense, it should work.
> I do not see having false on the isdir column as an actual issue.

It's unimportant, but testing for intended use of flags during development.

> * v10.6 behavior change for existing functions, always show isdir column,
> and removal of IS_DIR flag.
> 
> I'm unsure why the features are removed, some use case may benefit from the
> more complete function?
> Maybe flags defs should not be changed anyway?

Maybe.  I put them back...but it means they're not being exercized by any
*used* case.

> I do not like much the "if (...) /* empty */;" code. Maybe it could be
> caught more cleanly later in the conditional structure.

This went away when I put back the SKIP_DIRS flag.

> * v10.7 adds "pg_ls_dir_recurse" function

> Doc looks incomplete and the example is very contrived and badly indented.

Why you think it's contrived?  Listing a tmpdir recursively is the initial
motivation of this patch.  Maybe you think I should list just the tmpdir for
one tablespace ?  Note that for temp_tablespaces parameter:

|When there is more than one name in the list, PostgreSQL chooses a random member of the list each time a temporary object is to be created; except that within a transaction, successively created temporary objects are placed in successive tablespaces from the list.

> The function definition does not follow the style around: uppercase whereas
> all others are lowercase, "" instead of '', no "as"…

I used "" because of this:
| x.name||'/'||a.name
I don't know if there's a better way to join paths in SQL, or if that suggests
this is a bad way to do it.

> I do not understand why oid 8511 is given to the new function.

I used: ./src/include/catalog/unused_oids (maybe not correctly).

> I do not understand why UNION ALL and not UNION.

In general, union ALL can avoid a "distinct" plan node, but it doesn't seem to
have any effect here.

> I would have put the definition after "pg_ls_dir_metadata" definition.

Done

> pg_ls_dir_metadata seems defined as (text,bool,bool) but called as
> (text,bool,bool,bool).

fixed, thanks.

> Maybe a better alias could be given instead of x?
> 
> There are no tests for the new function. I'm not sure it would work.

I added something which would've caught the issue with number of arguments.

> * v10.8 change flags & add test on pg_ls_logdir().
> 
> I'm unsure why it is done at this stage.

I think it makes sense to allow ls_logdir to succeed even if ./log doesn't
exist, since it isn't created by initdb or during postmaster start, and since
we already using MISSING_OK for tmpdir.

But a separate patch since we didn't previous discuss changing logdir.  

> * v10.9 change some ls functions and fix patch 10.7 issue
> I'm unsure why it is done at this stage. "make check" ok.

This is the last patch in the series, since I think it's least likely to be
agreed on.

-- 
Justin

Attachments:

  [text/x-diff] v11-0001-Document-historic-behavior-about-hiding-director.patch (1.7K, ../../20200315212729.GC26184@telsasoft.com/2-v11-0001-Document-historic-behavior-about-hiding-director.patch)
  download | inline diff:
From 1bb8e0efb4f14fa344cd5ee66c3138184a9fa9e2 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 16:50:07 -0600
Subject: [PATCH v11 1/9] Document historic behavior about hiding directories
 and special files

Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
 doc/src/sgml/func.sgml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v11-0002-Document-historic-behavior-about-hiding-director.patch (904B, ../../20200315212729.GC26184@telsasoft.com/3-v11-0002-Document-historic-behavior-about-hiding-director.patch)
  download | inline diff:
From c8b89e104694645873b77f987d795269fc330bf7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:12:04 -0600
Subject: [PATCH v11 2/9] Document historic behavior about hiding directories
 and special files

Should backpatch to v12: tmpdir
---
 doc/src/sgml/func.sgml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v11-0003-Add-tests-exercizing-pg_ls_tmpdir.patch (4.3K, ../../20200315212729.GC26184@telsasoft.com/4-v11-0003-Add-tests-exercizing-pg_ls_tmpdir.patch)
  download | inline diff:
From 96de8a92294eec8c33ecb2ef0dbf48add281b763 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v11 3/9] Add tests exercizing pg_ls_tmpdir..

..and its backing function pg_ls_dir_files
---
 src/test/regress/expected/misc_functions.out | 7 +++++++
 src/test/regress/input/tablespace.source     | 5 +++++
 src/test/regress/output/tablespace.source    | 8 ++++++++
 src/test/regress/sql/misc_functions.sql      | 4 ++++
 4 files changed, 24 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index e217b678d7..9947d9ef9d 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -180,6 +180,13 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
  t
 (1 row)
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 1e11eb3554..656aad93e6 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -51,6 +51,10 @@ from (select pg_ls_waldir() w) ss where length((w).name) = 24 limit 1;
 
 select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v11-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (11.6K, ../../20200315212729.GC26184@telsasoft.com/5-v11-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From 3156f1c19e0389a0a23546971ca8715d378e0370 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 17:15:02 -0500
Subject: [PATCH v11 4/9] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

Need catversion bumped?
---
 doc/src/sgml/func.sgml          |  17 +++
 src/backend/utils/adt/genfile.c | 190 ++++++++++++++++++--------------
 src/include/catalog/pg_proc.dat |   6 +
 3 files changed, 129 insertions(+), 84 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..dfea9f8061 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21342,6 +21342,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         List the contents of a directory.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21442,6 +21451,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_metadata</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_metadata</function> lists the files in the specified
+    directory along with the file's metadata.
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index bcf9bd1b97..b105363903 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,14 +36,23 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
-typedef struct
-{
-	char	   *location;
-	DIR		   *dirdesc;
-	bool		include_dot_dirs;
-} directory_fctx;
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
+#define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	LS_DIR_SKIP_HIDDEN			(1<<4) /* Do not show anything begining with . */
+#define	LS_DIR_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	LS_DIR_SKIP_SPECIAL			(1<<6) /* Do not show special file types */
 
 
+/*
+ * Shortcut for the historic behavior of the pg_ls_* functions (not including
+ * pg_ls_dir, which skips different files and doesn't show metadata.
+ */
+#define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -447,67 +456,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	FuncCallContext *funcctx;
-	struct dirent *de;
-	directory_fctx *fctx;
-	MemoryContext oldcontext;
-
-	if (SRF_IS_FIRSTCALL())
-	{
-		bool		missing_ok = false;
-		bool		include_dot_dirs = false;
-
-		/* check the optional arguments */
-		if (PG_NARGS() == 3)
-		{
-			if (!PG_ARGISNULL(1))
-				missing_ok = PG_GETARG_BOOL(1);
-			if (!PG_ARGISNULL(2))
-				include_dot_dirs = PG_GETARG_BOOL(2);
-		}
-
-		funcctx = SRF_FIRSTCALL_INIT();
-		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
-
-		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
-
-		if (!fctx->dirdesc)
-		{
-			if (missing_ok && errno == ENOENT)
-			{
-				MemoryContextSwitchTo(oldcontext);
-				SRF_RETURN_DONE(funcctx);
-			}
-			else
-				ereport(ERROR,
-						(errcode_for_file_access(),
-						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
-		}
-		funcctx->user_fctx = fctx;
-		MemoryContextSwitchTo(oldcontext);
-	}
-
-	funcctx = SRF_PERCALL_SETUP();
-	fctx = (directory_fctx *) funcctx->user_fctx;
-
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
-	{
-		if (!fctx->include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
-	}
-
-	FreeDir(fctx->dirdesc);
-
-	SRF_RETURN_DONE(funcctx);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -520,7 +471,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -530,7 +483,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -539,6 +492,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= LS_DIR_MISSING_OK;
+			else
+				flags &= ~LS_DIR_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~LS_DIR_SKIP_DOT_DIRS;
+			else
+				flags |= LS_DIR_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -554,8 +533,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags & LS_DIR_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	} else {
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -574,7 +563,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags&LS_DIR_MISSING_OK && errno == ENOENT)
 		{
 			tuplestore_donestoring(tupstore);
 			return (Datum) 0;
@@ -584,13 +573,19 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
+		if (flags & LS_DIR_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
 		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		if (flags & LS_DIR_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
@@ -600,13 +595,26 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_DIRS)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & LS_DIR_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & LS_DIR_ISDIR)
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -621,14 +629,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
 }
 
 /*
@@ -646,7 +654,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
 }
 
 /*
@@ -675,5 +684,18 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7fb574f9dc..0a1859f709 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10741,6 +10741,12 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
+  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
-- 
2.17.0

  [text/x-diff] v11-0005-pg_ls_tmpdir-to-show-isdir-argument.patch (7.6K, ../../20200315212729.GC26184@telsasoft.com/6-v11-0005-pg_ls_tmpdir-to-show-isdir-argument.patch)
  download | inline diff:
From ede11bc6af39a7234c31508bf981ee193dc0e0cb Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v11 5/9] pg_ls_tmpdir to show "isdir" argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Note that "dir_ok" means both: do not skip dirs, *and* show "isdir" column.
The other functions both skip dirs and only show 3 columns, and ls_tmpdir is
changing to both show dirs and show 4 columns.  It wouldn't make sense to skip
dirs and then show a 4th column always false.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index dfea9f8061..82891e2d64 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21391,12 +21391,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21499,14 +21499,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index b105363903..99a540baf2 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -477,7 +477,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -655,7 +655,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+			LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0a1859f709..7789d029ea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10733,13 +10733,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 9947d9ef9d..b0cf145dc5 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -183,8 +183,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v11-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch (10.6K, ../../20200315212729.GC26184@telsasoft.com/7-v11-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From 3398c7eaeb2c3ee2be8d80a726bea6d9a191e470 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v11 6/9] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

XXX: ..FLAG_ISDIR is collapsed into FLAG_METADATA.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 11 ++++--
 src/include/catalog/pg_proc.dat              | 19 +++++-----
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 41 insertions(+), 32 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 82891e2d64..e4ba3fee40 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21357,8 +21357,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21368,8 +21369,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21379,8 +21381,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21463,36 +21466,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 99a540baf2..9c2feb0986 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -53,6 +53,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
  */
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
+/* Shortcut for showing ... */
+#define LS_DIR_MODERN				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -629,14 +632,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_MODERN);
 }
 
 /*
@@ -655,7 +658,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK);
+			LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /*
@@ -685,7 +688,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+			LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7789d029ea..b4c3d15495 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6143,6 +6143,7 @@
   proname => 'pg_ls_dir', prorows => '1000', proretset => 't',
   provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool',
   prosrc => 'pg_ls_dir' },
+
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
   proargtypes => 'float8', prosrc => 'pg_sleep' },
@@ -10717,18 +10718,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
@@ -10743,9 +10744,9 @@
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
-  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
-  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index b0cf145dc5..448901d841 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v11-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (5.3K, ../../20200315212729.GC26184@telsasoft.com/8-v11-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From ed17a9fb0975d35f36f6925e480029c461f03724 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v11 7/9] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

TODO:
src/backend/catalog/system_views.sql:REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;

Need catversion bumped ?
---
 doc/src/sgml/func.sgml                       | 24 ++++++++++++++++++++
 src/include/catalog/pg_proc.dat              |  6 +++++
 src/test/regress/expected/misc_functions.out | 14 ++++++++++++
 src/test/regress/sql/misc_functions.sql      |  6 +++++
 4 files changed, 50 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e4ba3fee40..f3422808d9 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21351,6 +21351,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+         Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21462,6 +21471,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     directory along with the file's metadata.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    To recursively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir
+FROM pg_tablespace b, pg_control_system() pcs,
+format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) AS suffix) AS dir,
+pg_ls_dir_recurse(dir) AS a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b4c3d15495..667ea5721e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10748,6 +10748,12 @@
   proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
   proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
+{ oid => '8511', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 448901d841..82d967a62d 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -187,6 +187,20 @@ select * from pg_ls_tmpdir() where name='Does not exist';
 ------+------+--------------+-------
 (0 rows)
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+         name          | isdir 
+-----------------------+-------
+ pg_wal                | t
+ pg_wal/archive_status | t
+(2 rows)
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 656aad93e6..2cfdac386d 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -55,6 +55,12 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v11-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.0K, ../../20200315212729.GC26184@telsasoft.com/9-v11-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From e5b3bbf8fcf2a89ad5e273a939ef7257c0323fb2 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v11 8/9] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 9c2feb0986..2d4f561ae0 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -632,7 +632,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..ba9a3fe29a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v11-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (15.6K, ../../20200315212729.GC26184@telsasoft.com/10-v11-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From e03b906cc3e00989a3ed8db93ea655b57b71b3ce Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v11 9/9] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 +++++++++-----------
 src/backend/utils/adt/genfile.c              | 18 ++++++++----
 src/include/catalog/pg_proc.dat              | 30 ++++++++++----------
 src/test/regress/expected/misc_functions.out | 12 ++++----
 src/test/regress/output/tablespace.source    |  8 +++---
 5 files changed, 51 insertions(+), 47 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index f3422808d9..7f7d81da6e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21366,8 +21366,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21378,8 +21377,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21390,9 +21388,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21404,8 +21401,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21491,8 +21487,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21503,8 +21499,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21516,8 +21512,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21530,8 +21526,8 @@ pg_ls_dir_recurse(dir) AS a;
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 2d4f561ae0..7459371e48 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -576,8 +576,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
@@ -613,9 +613,17 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 		if (flags & LS_DIR_METADATA)
 		{
 			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & LS_DIR_ISDIR)
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
+			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 		}
 
 		memset(nulls, 0, sizeof(nulls));
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 667ea5721e..b74ed64b71 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10718,42 +10718,42 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '8511', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification,a.change,a.creation,a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 82d967a62d..3544977a51 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -183,8 +183,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
@@ -197,8 +197,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index ba9a3fe29a..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-16 15:20               ` Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 15:41                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 2 replies; 70+ messages in thread

From: Fabien COELHO @ 2020-03-16 15:20 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>


About v11, ISTM that the recursive function should check for symbolic 
links and possibly avoid them:

  sh> cd data/base
  sh> ln -s .. foo

  psql> SELECT * FROM pg_ls_dir_recurse('.');
  ERROR:  could not stat file "./base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo": Too many levels of symbolic links
  CONTEXT:  SQL function "pg_ls_dir_recurse" statement 1

This probably means using lstat instead of (in supplement to?) stat, and 
probably tell if something is a link, and if so not recurse in them.

-- 
Fabien.





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-03-16 15:41                 ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 18:21                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  1 sibling, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-16 15:41 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Mon, Mar 16, 2020 at 04:20:21PM +0100, Fabien COELHO wrote:
> 
> About v11, ISTM that the recursive function should check for symbolic links
> and possibly avoid them:
> 
>  sh> cd data/base
>  sh> ln -s .. foo
> 
>  psql> SELECT * FROM pg_ls_dir_recurse('.');
>  ERROR:  could not stat file "./base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo": Too many levels of symbolic links
>  CONTEXT:  SQL function "pg_ls_dir_recurse" statement 1
> 
> This probably means using lstat instead of (in supplement to?) stat, and
> probably tell if something is a link, and if so not recurse in them.

Thanks for looking.

I think that opens up a can of worms.  I don't want to go into the business of
re-implementing all of find(1) - I count ~128 flags (most of which take
arguments).  You're referring to find -L vs find -P, and some people would want
one and some would want another.  And don't forget about find -H...

pg_stat_file doesn't expose the file type (I guess because it's not portable?),
and I think it's outside the scope of this patch to change that.  Maybe it
suggests that the pg_ls_dir_recurse patch should be excluded.

ISTM if someone wants to recursively list a directory, they should avoid
putting cycles there, or permission errors, or similar.  Or they should write
their own C extension that borrows from pg_ls_dir_files but handles more
arguments.

-- 
Justin





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 15:41                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-16 18:21                   ` Fabien COELHO <coelho@cri.ensmp.fr>
  0 siblings, 0 replies; 70+ messages in thread

From: Fabien COELHO @ 2020-03-16 18:21 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>


Hello Justin,

>>  psql> SELECT * FROM pg_ls_dir_recurse('.');
>>  ERROR:  could not stat file "./base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo/base/foo": Too many levels of symbolic links
>>  CONTEXT:  SQL function "pg_ls_dir_recurse" statement 1
>>
>> This probably means using lstat instead of (in supplement to?) stat, and
>> probably tell if something is a link, and if so not recurse in them.
>
> Thanks for looking.
>
> I think that opens up a can of worms.  I don't want to go into the business of
> re-implementing all of find(1) - I count ~128 flags (most of which take
> arguments).  You're referring to find -L vs find -P, and some people would want
> one and some would want another.  And don't forget about find -H...

This is not the point. The point is that a link can change a finite tree 
into cyclic graph, and you do not want to delve into that, ever.

The "find" command, by default, does not recurse into a link because of 
said problem, and the user *must* ask for it and assume the infinite loop 
if any.

So if you implement one behavior, it should be not recursing into links. 
Franckly, I would not provide the recurse into link alternative, but it 
could be implemented if someone wants it, and the problem that come with 
it.

> pg_stat_file doesn't expose the file type (I guess because it's not portable?),

You are right that Un*x and Windows are not the same wrt link. It seems 
that there is already something about that in port:

   "./src/port/dirmod.c:pgwin32_is_junction(const char *path)"

So most of the details are already hidden.

> and I think it's outside the scope of this patch to change that.  Maybe it
> suggests that the pg_ls_dir_recurse patch should be excluded.

IMHO, I really think that it should be included. Dealing with links is no 
big deal, but you need an additional column in _metadata to tell it is a 
link, and there is a ifdef because testing is a little different between 
unix and windows. I'd guess around 10-20 lines of code added.

> ISTM if someone wants to recursively list a directory, they should avoid
> putting cycles there, or permission errors, or similar.

Hmmm. I'd say the user should like to be able to call the function and 
never have a bad experience with it such as a failure on an infinite loop.

> Or they should write their own C extension that borrows from 
> pg_ls_dir_files but handles more arguments.

ISTM that the point of your patch is to provide the basic tool needed to 
list directories contents, and handling links somehow is a necessary part 
of that.

-- 
Fabien.





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-03-16 21:48                 ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  1 sibling, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-16 21:48 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Mon, Mar 16, 2020 at 04:20:21PM +0100, Fabien COELHO wrote:
> This probably means using lstat instead of (in supplement to?) stat, and
> probably tell if something is a link, and if so not recurse in them.

On Mon, Mar 16, 2020 at 07:21:06PM +0100, Fabien COELHO wrote:
> IMHO, I really think that it should be included. Dealing with links is no
> big deal, but you need an additional column in _metadata to tell it is a
> link

Instead of showing another column, I changed to show links with isdir=false.
At the cost of two more patches, to allow backpatching docs and maybe separate
commit to make the subtle change obvious in commit history, at least.

I see a few places in the backend and a few more in the fronted using the same
logic that I used for islink(), but I'm not sure if there's a good place to put
that to allow factoring out at least the other backend ones.

-- 
Justin

Attachments:

  [text/x-diff] v12-0001-Document-historic-behavior-about-hiding-director.patch (1.7K, ../../20200316214806.GQ26184@telsasoft.com/2-v12-0001-Document-historic-behavior-about-hiding-director.patch)
  download | inline diff:
From 1bb8e0efb4f14fa344cd5ee66c3138184a9fa9e2 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 16:50:07 -0600
Subject: [PATCH v12 01/11] Document historic behavior about hiding directories
 and special files

Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
 doc/src/sgml/func.sgml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     (mtime) of each file in the log directory. By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v12-0002-Document-historic-behavior-of-links-to-directori.patch (1.1K, ../../20200316214806.GQ26184@telsasoft.com/3-v12-0002-Document-historic-behavior-of-links-to-directori.patch)
  download | inline diff:
From b6eb6cd9299e8ae07770790f05da038fede5278c Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 16 Mar 2020 14:12:55 -0500
Subject: [PATCH v12 02/11] Document historic behavior of links to
 directories..

Backpatch to 9.5: pg_stat_file
---
 doc/src/sgml/func.sgml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..ace95fe661 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21527,7 +21527,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory.  Typical usages include:
+    indicating if it is a directory (or a symbolic link to a directory).
+    Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
 SELECT (pg_stat_file('filename')).modification;
-- 
2.17.0

  [text/x-diff] v12-0003-Document-historic-behavior-about-hiding-director.patch (906B, ../../20200316214806.GQ26184@telsasoft.com/4-v12-0003-Document-historic-behavior-about-hiding-director.patch)
  download | inline diff:
From 52f5b76cc278c43983b044b7fb0f04c2d848d61e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:12:04 -0600
Subject: [PATCH v12 03/11] Document historic behavior about hiding directories
 and special files

Should backpatch to v12: tmpdir
---
 doc/src/sgml/func.sgml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index ace95fe661..2c6142a0e0 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
+    Filenames beginning with a dot, directories, and other special files are not shown.
    </para>
 
    <indexterm>
-- 
2.17.0

  [text/x-diff] v12-0004-Add-tests-exercizing-pg_ls_tmpdir.patch (4.3K, ../../20200316214806.GQ26184@telsasoft.com/5-v12-0004-Add-tests-exercizing-pg_ls_tmpdir.patch)
  download | inline diff:
From dd98e2b4f1fb37065a317719f0dd493e8837c7e7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v12 04/11] Add tests exercizing pg_ls_tmpdir..

..and its backing function pg_ls_dir_files
---
 src/test/regress/expected/misc_functions.out | 7 +++++++
 src/test/regress/input/tablespace.source     | 5 +++++
 src/test/regress/output/tablespace.source    | 8 ++++++++
 src/test/regress/sql/misc_functions.sql      | 4 ++++
 4 files changed, 24 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index e217b678d7..9947d9ef9d 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -180,6 +180,13 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
  t
 (1 row)
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 1e11eb3554..656aad93e6 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -51,6 +51,10 @@ from (select pg_ls_waldir() w) ss where length((w).name) = 24 limit 1;
 
 select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v12-0005-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (11.6K, ../../20200316214806.GQ26184@telsasoft.com/6-v12-0005-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From 8e983600c3f85b9e5f8ed15c8b5cba5dd4f5c0fb Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 17:15:02 -0500
Subject: [PATCH v12 05/11] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

Need catversion bumped?
---
 doc/src/sgml/func.sgml          |  17 +++
 src/backend/utils/adt/genfile.c | 194 ++++++++++++++++++--------------
 src/include/catalog/pg_proc.dat |   6 +
 3 files changed, 132 insertions(+), 85 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 2c6142a0e0..4b966ed847 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21342,6 +21342,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         List the contents of a directory.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21442,6 +21451,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_metadata</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_metadata</function> lists the files in the specified
+    directory along with the file's metadata.
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index bcf9bd1b97..98ab9a2b92 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,14 +36,23 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
-typedef struct
-{
-	char	   *location;
-	DIR		   *dirdesc;
-	bool		include_dot_dirs;
-} directory_fctx;
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
+#define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	LS_DIR_SKIP_HIDDEN			(1<<4) /* Do not show anything begining with . */
+#define	LS_DIR_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	LS_DIR_SKIP_SPECIAL			(1<<6) /* Do not show special file types */
 
 
+/*
+ * Shortcut for the historic behavior of the pg_ls_* functions (not including
+ * pg_ls_dir, which skips different files and doesn't show metadata.
+ */
+#define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -447,67 +456,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	FuncCallContext *funcctx;
-	struct dirent *de;
-	directory_fctx *fctx;
-	MemoryContext oldcontext;
-
-	if (SRF_IS_FIRSTCALL())
-	{
-		bool		missing_ok = false;
-		bool		include_dot_dirs = false;
-
-		/* check the optional arguments */
-		if (PG_NARGS() == 3)
-		{
-			if (!PG_ARGISNULL(1))
-				missing_ok = PG_GETARG_BOOL(1);
-			if (!PG_ARGISNULL(2))
-				include_dot_dirs = PG_GETARG_BOOL(2);
-		}
-
-		funcctx = SRF_FIRSTCALL_INIT();
-		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
-
-		fctx = palloc(sizeof(directory_fctx));
-		fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-		fctx->include_dot_dirs = include_dot_dirs;
-		fctx->dirdesc = AllocateDir(fctx->location);
-
-		if (!fctx->dirdesc)
-		{
-			if (missing_ok && errno == ENOENT)
-			{
-				MemoryContextSwitchTo(oldcontext);
-				SRF_RETURN_DONE(funcctx);
-			}
-			else
-				ereport(ERROR,
-						(errcode_for_file_access(),
-						 errmsg("could not open directory \"%s\": %m",
-								fctx->location)));
-		}
-		funcctx->user_fctx = fctx;
-		MemoryContextSwitchTo(oldcontext);
-	}
-
-	funcctx = SRF_PERCALL_SETUP();
-	fctx = (directory_fctx *) funcctx->user_fctx;
-
-	while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL)
-	{
-		if (!fctx->include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name));
-	}
-
-	FreeDir(fctx->dirdesc);
-
-	SRF_RETURN_DONE(funcctx);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -520,7 +471,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -530,7 +483,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -539,6 +492,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= LS_DIR_MISSING_OK;
+			else
+				flags &= ~LS_DIR_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~LS_DIR_SKIP_DOT_DIRS;
+			else
+				flags |= LS_DIR_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -554,8 +533,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags & LS_DIR_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	} else {
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -574,7 +563,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags&LS_DIR_MISSING_OK && errno == ENOENT)
 		{
 			tuplestore_donestoring(tupstore);
 			return (Datum) 0;
@@ -584,13 +573,20 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		/* Skip dot dirs? */
+		if (flags & LS_DIR_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
+		/* Skip hidden files? */
+		if (flags & LS_DIR_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
@@ -600,13 +596,27 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		/* Skip dirs or special files? */
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_DIRS)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & LS_DIR_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & LS_DIR_ISDIR)
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -621,14 +631,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
 }
 
 /*
@@ -646,7 +656,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
 }
 
 /*
@@ -675,5 +686,18 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7fb574f9dc..0a1859f709 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10741,6 +10741,12 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
+  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
-- 
2.17.0

  [text/x-diff] v12-0006-Show-links-to-dirs-with-isdir-false.patch (2.6K, ../../20200316214806.GQ26184@telsasoft.com/7-v12-0006-Show-links-to-dirs-with-isdir-false.patch)
  download | inline diff:
From 38704c29b593e32404ad13f93d0bce6330ee41d1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 17:15:02 -0500
Subject: [PATCH v12 06/11] Show links to dirs with isdir=false..

..this is needed to avoid infinite recursion in pg_ls_dir_recurse.

Change pg_stat_file for consistency.
---
 doc/src/sgml/func.sgml          |  2 +-
 src/backend/utils/adt/genfile.c | 21 +++++++++++++++++++--
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4b966ed847..68c7327e1d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21545,7 +21545,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory (or a symbolic link to a directory).
+    indicating if it is a directory (and not a symbolic link to a directory).
     Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 98ab9a2b92..70b541388e 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -365,6 +365,22 @@ pg_read_binary_file_all(PG_FUNCTION_ARGS)
 	return pg_read_binary_file(fcinfo);
 }
 
+/* Return true iff path is a symbolic link (unix) or junction (win32) */
+static bool
+islink(const char *path)
+{
+#ifndef WIN32
+	struct stat attrib;
+	if (lstat(path, &attrib) < 0)
+		ereport(ERROR,
+				(errcode_for_file_access(),
+				 errmsg("could not stat file \"%s\": %m", path)));
+	return S_ISLNK(attrib.st_mode);
+#else
+	return pgwin32_is_junction(path);
+#endif
+}
+
 /*
  * stat a file
  */
@@ -428,7 +444,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	isnull[3] = true;
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
 #endif
-	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
+	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode) && !islink(filename));
 
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
@@ -614,7 +630,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 			values[1] = Int64GetDatum((int64) attrib.st_size);
 			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
 			if (flags & LS_DIR_ISDIR)
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode) &&
+					!islink(path));
 		}
 
 		memset(nulls, 0, sizeof(nulls));
-- 
2.17.0

  [text/x-diff] v12-0007-pg_ls_tmpdir-to-show-isdir-argument.patch (7.6K, ../../20200316214806.GQ26184@telsasoft.com/8-v12-0007-pg_ls_tmpdir-to-show-isdir-argument.patch)
  download | inline diff:
From c0d3aa1dd505d3da12d4023b59c4d717c3282a9e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v12 07/11] pg_ls_tmpdir to show "isdir" argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Note that "dir_ok" means both: do not skip dirs, *and* show "isdir" column.
The other functions both skip dirs and only show 3 columns, and ls_tmpdir is
changing to both show dirs and show 4 columns.  It wouldn't make sense to skip
dirs and then show a 4th column always false.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 68c7327e1d..b813b19daf 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21391,12 +21391,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21499,14 +21499,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 70b541388e..0cfa055690 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -493,7 +493,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -674,7 +674,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+			LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0a1859f709..7789d029ea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10733,13 +10733,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 9947d9ef9d..b0cf145dc5 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -183,8 +183,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v12-0008-pg_ls_-dir-to-show-directories-and-isdir-column.patch (10.6K, ../../20200316214806.GQ26184@telsasoft.com/9-v12-0008-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From 37084a28fccb68d5f820a7a52eccebfa054ed6d1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v12 08/11] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 11 ++++--
 src/include/catalog/pg_proc.dat              | 19 +++++-----
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 41 insertions(+), 32 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index b813b19daf..4cdd03610a 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21357,8 +21357,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21368,8 +21369,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21379,8 +21381,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21463,36 +21466,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 0cfa055690..de63e77836 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -53,6 +53,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
  */
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
+/* Shortcut for common behavior */
+#define LS_DIR_MODERN				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -648,14 +651,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_MODERN);
 }
 
 /*
@@ -674,7 +677,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK);
+			LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /*
@@ -704,7 +707,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+			LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7789d029ea..b4c3d15495 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6143,6 +6143,7 @@
   proname => 'pg_ls_dir', prorows => '1000', proretset => 't',
   provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool',
   prosrc => 'pg_ls_dir' },
+
 { oid => '2626', descr => 'sleep for the specified time in seconds',
   proname => 'pg_sleep', provolatile => 'v', prorettype => 'void',
   proargtypes => 'float8', prosrc => 'pg_sleep' },
@@ -10717,18 +10718,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
@@ -10743,9 +10744,9 @@
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
-  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
-  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index b0cf145dc5..448901d841 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v12-0009-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (5.3K, ../../20200316214806.GQ26184@telsasoft.com/10-v12-0009-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From 3ef2b34083114e760bb63843bd92fb83f8ac49e5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v12 09/11] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

TODO:
src/backend/catalog/system_views.sql:REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;

Need catversion bumped ?
---
 doc/src/sgml/func.sgml                       | 24 ++++++++++++++++++++
 src/include/catalog/pg_proc.dat              |  6 +++++
 src/test/regress/expected/misc_functions.out | 14 ++++++++++++
 src/test/regress/sql/misc_functions.sql      |  6 +++++
 4 files changed, 50 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4cdd03610a..373e3b647b 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21351,6 +21351,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+         Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21462,6 +21471,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     directory along with the file's metadata.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    To recursively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir
+FROM pg_tablespace b, pg_control_system() pcs,
+format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) AS suffix) AS dir,
+pg_ls_dir_recurse(dir) AS a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b4c3d15495..667ea5721e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10748,6 +10748,12 @@
   proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
   proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
+{ oid => '8511', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 448901d841..82d967a62d 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -187,6 +187,20 @@ select * from pg_ls_tmpdir() where name='Does not exist';
 ------+------+--------------+-------
 (0 rows)
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+         name          | isdir 
+-----------------------+-------
+ pg_wal                | t
+ pg_wal/archive_status | t
+(2 rows)
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 656aad93e6..2cfdac386d 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -55,6 +55,12 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v12-0010-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.0K, ../../20200316214806.GQ26184@telsasoft.com/11-v12-0010-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From 3b03b4e911db0c9b9386667ad0c805dc97ae6607 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v12 10/11] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index de63e77836..810c6b0f23 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -651,7 +651,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..ba9a3fe29a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v12-0011-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (15.8K, ../../20200316214806.GQ26184@telsasoft.com/12-v12-0011-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From 5059e4fb8528657b0b9e81a24c70fe9e56aeb677 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v12 11/11] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 +++++++++-----------
 src/backend/utils/adt/genfile.c              | 24 ++++++++++------
 src/include/catalog/pg_proc.dat              | 30 ++++++++++----------
 src/test/regress/expected/misc_functions.out | 12 ++++----
 src/test/regress/output/tablespace.source    |  8 +++---
 5 files changed, 54 insertions(+), 50 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 373e3b647b..11309f0f9b 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21366,8 +21366,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21378,8 +21377,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21390,9 +21388,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21404,8 +21401,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21491,8 +21487,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21503,8 +21499,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21516,8 +21512,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21530,8 +21526,8 @@ pg_ls_dir_recurse(dir) AS a;
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 810c6b0f23..3a435384f4 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -592,8 +592,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
@@ -627,17 +627,25 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 				continue;
 		}
 
+		memset(nulls, 0, sizeof(nulls));
 		values[0] = CStringGetTextDatum(de->d_name);
 		if (flags & LS_DIR_METADATA)
 		{
 			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & LS_DIR_ISDIR)
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode) &&
-					!islink(path));
-		}
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
 
-		memset(nulls, 0, sizeof(nulls));
+			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode) &&
+				!islink(path));
+		}
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
 	}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 667ea5721e..b74ed64b71 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10718,42 +10718,42 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '8511', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification,a.change,a.creation,a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 82d967a62d..3544977a51 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -183,8 +183,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir();
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
@@ -197,8 +197,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index ba9a3fe29a..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-16 22:17                   ` Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 1 reply; 70+ messages in thread

From: Alvaro Herrera @ 2020-03-16 22:17 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

I pushed 0001 and 0003 (as a single commit).  archive_statusdir didn't
get here until 12, so your commit message was mistaken.  Also, pg10 is
slightly different so it didn't apply there, so I left it alone.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2020-03-17 03:14                     ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 09:21                       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  0 siblings, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-17 03:14 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Mon, Mar 16, 2020 at 07:17:36PM -0300, Alvaro Herrera wrote:
> I pushed 0001 and 0003 (as a single commit).  archive_statusdir didn't
> get here until 12, so your commit message was mistaken.  Also, pg10 is
> slightly different so it didn't apply there, so I left it alone.

Thanks, I appreciate it (and I'm sure Fabien will appreciate having two fewer
patches...).

@cfbot: rebased onto b4570d33aa045df330bb325ba8a2cbf02266a555

I realized that if I lstat() a file to make sure links to dirs show as
isdir=false, it's odd to then show size and timestamps of the dir.  So changed
to use lstat ... and squished.

-- 
Justin

Attachments:

  [text/x-diff] v13-0001-Document-historic-behavior-of-links-to-directori.patch (1.1K, ../../20200317031427.GU26184@telsasoft.com/2-v13-0001-Document-historic-behavior-of-links-to-directori.patch)
  download | inline diff:
From d8294c4747c5ba1f3bec858c137cc2d31e5a0425 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 16 Mar 2020 14:12:55 -0500
Subject: [PATCH v13 1/8] Document historic behavior of links to directories..

Backpatch to 9.5: pg_stat_file
---
 doc/src/sgml/func.sgml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..2c6142a0e0 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21528,7 +21528,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory.  Typical usages include:
+    indicating if it is a directory (or a symbolic link to a directory).
+    Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
 SELECT (pg_stat_file('filename')).modification;
-- 
2.17.0

  [text/x-diff] v13-0002-Add-tests-exercising-pg_ls_tmpdir.patch (4.3K, ../../20200317031427.GU26184@telsasoft.com/3-v13-0002-Add-tests-exercising-pg_ls_tmpdir.patch)
  download | inline diff:
From 27913ef889696ef2c42ed52bf7097e4a5aeaad9e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v13 2/8] Add tests exercising pg_ls_tmpdir..

..and its backing function pg_ls_dir_files
---
 src/test/regress/expected/misc_functions.out | 7 +++++++
 src/test/regress/input/tablespace.source     | 5 +++++
 src/test/regress/output/tablespace.source    | 8 ++++++++
 src/test/regress/sql/misc_functions.sql      | 4 ++++
 4 files changed, 24 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..903f1fe443 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -201,6 +201,13 @@ select count(*) > 0 from
  t
 (1 row)
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..f09c890b5d 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -60,6 +60,10 @@ select count(*) > 0 from
    where spcname = 'pg_default') pts
   join pg_database db on pts.pts = db.oid;
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v13-0003-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (14.3K, ../../20200317031427.GU26184@telsasoft.com/4-v13-0003-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From afc6064e9a93fb0f00eefefe706c59d6325d2727 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 17:15:02 -0500
Subject: [PATCH v13 3/8] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

Change to use lstat() to allow pg_ls_dir_recurse to avoid infinite recursion.
That means:
 - links to dirs are shown with isdir=false;
 - timestamps shown are those of the link;
 - changed pg_stat_file for consistency;

Need catversion bumped?
---
 doc/src/sgml/func.sgml               |  19 ++-
 src/backend/catalog/system_views.sql |   1 +
 src/backend/utils/adt/genfile.c      | 218 +++++++++++++++------------
 src/include/catalog/pg_proc.dat      |   6 +
 4 files changed, 150 insertions(+), 94 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 2c6142a0e0..68c7327e1d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21342,6 +21342,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         List the contents of a directory.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21442,6 +21451,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_metadata</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_metadata</function> lists the files in the specified
+    directory along with the file's metadata.
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
@@ -21528,7 +21545,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory (or a symbolic link to a directory).
+    indicating if it is a directory (and not a symbolic link to a directory).
     Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b8a3f46912..05a644a7c9 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1435,6 +1435,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 01185f218b..4b70a00a35 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,6 +36,21 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
+#define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	LS_DIR_SKIP_HIDDEN			(1<<4) /* Do not show anything begining with . */
+#define	LS_DIR_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	LS_DIR_SKIP_SPECIAL			(1<<6) /* Do not show special file types */
+
+/*
+ * Shortcut for the historic behavior of the pg_ls_* functions (not including
+ * pg_ls_dir, which skips different files and doesn't show metadata.
+ */
+#define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -370,7 +385,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 
 	filename = convert_and_check_filename(filename_t);
 
-	if (stat(filename, &fst) < 0)
+	if (lstat(filename, &fst) < 0)
 	{
 		if (missing_ok && errno == ENOENT)
 			PG_RETURN_NULL();
@@ -413,6 +428,10 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
 #endif
 	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
+#ifdef WIN32
+	if (pgwin32_is_junction(path)) /* Links are not directories */
+		values[5] = BoolGetDatum(false);
+#endif
 
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
@@ -440,79 +459,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
-	char	   *location;
-	bool		missing_ok = false;
-	bool		include_dot_dirs = false;
-	bool		randomAccess;
-	TupleDesc	tupdesc;
-	Tuplestorestate *tupstore;
-	DIR		   *dirdesc;
-	struct dirent *de;
-	MemoryContext oldcontext;
-
-	location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-	/* check the optional arguments */
-	if (PG_NARGS() == 3)
-	{
-		if (!PG_ARGISNULL(1))
-			missing_ok = PG_GETARG_BOOL(1);
-		if (!PG_ARGISNULL(2))
-			include_dot_dirs = PG_GETARG_BOOL(2);
-	}
-
-	/* check to see if caller supports us returning a tuplestore */
-	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
-		ereport(ERROR,
-				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-				 errmsg("set-valued function called in context that cannot accept a set")));
-	if (!(rsinfo->allowedModes & SFRM_Materialize))
-		ereport(ERROR,
-				(errcode(ERRCODE_SYNTAX_ERROR),
-				 errmsg("materialize mode required, but it is not allowed in this context")));
-
-	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
-	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
-
-	tupdesc = CreateTemplateTupleDesc(1);
-	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "pg_ls_dir", TEXTOID, -1, 0);
-
-	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
-	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
-	rsinfo->returnMode = SFRM_Materialize;
-	rsinfo->setResult = tupstore;
-	rsinfo->setDesc = tupdesc;
-
-	MemoryContextSwitchTo(oldcontext);
-
-	dirdesc = AllocateDir(location);
-	if (!dirdesc)
-	{
-		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
-			return (Datum) 0;
-		/* Otherwise, we can let ReadDir() throw the error */
-	}
-
-	while ((de = ReadDir(dirdesc, location)) != NULL)
-	{
-		Datum		values[1];
-		bool		nulls[1];
-
-		if (!include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		nulls[0] = false;
-
-		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
-	}
-
-	FreeDir(dirdesc);
-	return (Datum) 0;
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -525,7 +474,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -535,7 +486,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -544,6 +495,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= LS_DIR_MISSING_OK;
+			else
+				flags &= ~LS_DIR_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~LS_DIR_SKIP_DOT_DIRS;
+			else
+				flags |= LS_DIR_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -558,8 +535,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags & LS_DIR_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	} else {
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -578,36 +565,67 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags&LS_DIR_MISSING_OK && errno == ENOENT)
+		{
+			tuplestore_donestoring(tupstore);
 			return (Datum) 0;
+		}
 		/* Otherwise, we can let ReadDir() throw the error */
 	}
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		/* Skip dot dirs? */
+		if (flags & LS_DIR_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
+		/* Skip hidden files? */
+		if (flags & LS_DIR_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
 		snprintf(path, sizeof(path), "%s/%s", dir, de->d_name);
-		if (stat(path, &attrib) < 0)
+		if (lstat(path, &attrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		/* Skip dirs or special files? */
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_DIRS)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & LS_DIR_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & LS_DIR_ISDIR)
+			{
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+#ifdef WIN32
+				/* Links are not directories */
+				if (pgwin32_is_junction(path))
+					values[3] = BoolGetDatum(false);
+#endif
+			}
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -621,14 +639,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
 }
 
 /*
@@ -646,7 +664,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
 }
 
 /*
@@ -675,5 +694,18 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7fb574f9dc..0a1859f709 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10741,6 +10741,12 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
+  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
-- 
2.17.0

  [text/x-diff] v13-0004-pg_ls_tmpdir-to-show-isdir-argument.patch (7.3K, ../../20200317031427.GU26184@telsasoft.com/5-v13-0004-pg_ls_tmpdir-to-show-isdir-argument.patch)
  download | inline diff:
From 3cee15e9332b3019120fc27d31d0e147d381ea5d Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v13 4/8] pg_ls_tmpdir to show "isdir" argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 68c7327e1d..b813b19daf 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21391,12 +21391,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21499,14 +21499,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4b70a00a35..b35467e522 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -480,7 +480,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -665,7 +665,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+			LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0a1859f709..7789d029ea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10733,13 +10733,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 903f1fe443..dd78d9b20f 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -204,8 +204,8 @@ select count(*) > 0 from
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v13-0005-pg_ls_-dir-to-show-directories-and-isdir-column.patch (10.2K, ../../20200317031427.GU26184@telsasoft.com/6-v13-0005-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From 82cc15b0ae49d82fb51eb27aff8fb221a6335f5b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v13 5/8] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 11 ++++--
 src/include/catalog/pg_proc.dat              | 18 ++++-----
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 40 insertions(+), 32 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index b813b19daf..4cdd03610a 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21357,8 +21357,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21368,8 +21369,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21379,8 +21381,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21463,36 +21466,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index b35467e522..342d6f1205 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -52,6 +52,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
  */
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
+/* Shortcut for common behavior */
+#define LS_DIR_MODERN				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -639,14 +642,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_MODERN);
 }
 
 /*
@@ -665,7 +668,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK);
+			LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /*
@@ -695,7 +698,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			LS_DIR_HISTORIC|LS_DIR_MISSING_OK);
+			LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7789d029ea..be3e820a03 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10717,18 +10717,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
@@ -10743,9 +10743,9 @@
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
-  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool',
-  proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 
 # hash partitioning constraint function
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index dd78d9b20f..3ca4b85dd9 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v13-0006-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (6.1K, ../../20200317031427.GU26184@telsasoft.com/7-v13-0006-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From 0ed2eb8ad917bf5a1130aaf9050b0c3d73f66fba Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v13 6/8] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

XXX: this test case is unstable, if backends/autovacuum remove FSM...
ERROR:  could not stat file "./base/16384/20345_fsm": No such file or directory
CONTEXT:  SQL function "pg_ls_dir_recurse" statement 1

Need catversion bumped ?
---
 doc/src/sgml/func.sgml                       | 25 ++++++++++++++++++++
 src/backend/catalog/system_views.sql         |  1 +
 src/include/catalog/pg_proc.dat              |  6 +++++
 src/test/regress/expected/misc_functions.out | 14 +++++++++++
 src/test/regress/sql/misc_functions.sql      |  6 +++++
 5 files changed, 52 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4cdd03610a..0763db3f50 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21351,6 +21351,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+         Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata.
+         Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21462,6 +21472,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     directory along with the file's metadata.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    To recursively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir
+FROM pg_tablespace b, pg_control_system() pcs,
+format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) AS suffix) AS dir,
+pg_ls_dir_recurse(dir) AS a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 05a644a7c9..e7295d8aaf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1436,6 +1436,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_recurse(text) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be3e820a03..b3509f0e12 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10747,6 +10747,12 @@
   proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
   proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
+{ oid => '8511', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 3ca4b85dd9..ae8716a83f 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -208,6 +208,20 @@ select * from pg_ls_tmpdir() where name='Does not exist';
 ------+------+--------------+-------
 (0 rows)
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+         name          | isdir 
+-----------------------+-------
+ pg_wal                | t
+ pg_wal/archive_status | t
+(2 rows)
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index f09c890b5d..7038bcdb0f 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -64,6 +64,12 @@ select count(*) > 0 from
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v13-0007-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.0K, ../../20200317031427.GU26184@telsasoft.com/8-v13-0007-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From 43137b3dacfadd4f88f16a14972cf7a02c1917cf Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v13 7/8] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 342d6f1205..21c265aab3 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -642,7 +642,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN|LS_DIR_MISSING_OK);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..ba9a3fe29a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v13-0008-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (16.0K, ../../20200317031427.GU26184@telsasoft.com/9-v13-0008-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From f8cfd3da5a2e8f960bca9a550fdd5acf705c59b7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v13 8/8] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 +++++++++-----------
 src/backend/utils/adt/genfile.c              | 29 +++++++++++--------
 src/include/catalog/pg_proc.dat              | 30 ++++++++++----------
 src/test/regress/expected/misc_functions.out | 12 ++++----
 src/test/regress/output/tablespace.source    |  8 +++---
 5 files changed, 55 insertions(+), 54 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 0763db3f50..a3166aaf54 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21367,8 +21367,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21379,8 +21378,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21391,9 +21389,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21405,8 +21402,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21492,8 +21488,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21504,8 +21500,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21517,8 +21513,8 @@ pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21531,8 +21527,8 @@ pg_ls_dir_recurse(dir) AS a;
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 21c265aab3..f794352476 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -578,8 +578,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
@@ -613,24 +613,29 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 				continue;
 		}
 
+		memset(nulls, 0, sizeof(nulls));
 		values[0] = CStringGetTextDatum(de->d_name);
 		if (flags & LS_DIR_METADATA)
 		{
 			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & LS_DIR_ISDIR)
-			{
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
+			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 #ifdef WIN32
-				/* Links are not directories */
-				if (pgwin32_is_junction(path))
-					values[3] = BoolGetDatum(false);
+			/* Links are not directories */
+			if (pgwin32_is_junction(path))
+				values[6] = BoolGetDatum(false);
 #endif
-			}
 		}
 
-		memset(nulls, 0, sizeof(nulls));
-
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
 	}
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b3509f0e12..a50742e870 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10717,42 +10717,42 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '8511', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification,a.change,a.creation,a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index ae8716a83f..5614e564bd 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -204,8 +204,8 @@ select count(*) > 0 from
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
@@ -218,8 +218,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index ba9a3fe29a..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-17 09:21                       ` Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-17 19:04                         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 1 reply; 70+ messages in thread

From: Fabien COELHO @ 2020-03-17 09:21 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>


About v13, seens as one patch:

Function "pg_ls_dir_metadata" documentation suggests a variable number of 
arguments with brackets, but parameters are really mandatory.

  postgres=# SELECT pg_ls_dir_metadata('.');
  ERROR:  function pg_ls_dir_metadata(unknown) does not exist
  LINE 1: SELECT pg_ls_dir_metadata('.');
                ^
  HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
  postgres=# SELECT pg_ls_dir_metadata('.', true, true);
  …

The example in the documentation could be better indented. Also, ISTM that 
there are two implicit laterals (format & pg_ls_dir_recurse) that I would 
make explicit. I'd use the pcs alias explicitely. I'd use meaningful 
aliases (eg ts instead of b, …).

On reflection, I think that a boolean "isdir" column is a bad idea because 
it is not extensible. I'd propose to switch to the standard "ls" approach 
of providing the type as one character: '-' for regular, 'd' for 
directory, 'l' for link, 's' for socket, 'c' for character special…

ISTM that "lstat" is not available on windows, which suggests to call 
"stat" always, and then "lstat" on un*x and pg ports stuff on win.

I'm wondering about the restriction on directories only. Why should it not 
work on a file? Can it be easily extended to work on a simple file? If so, 
it could be just "pg_ls".

-- 
Fabien.

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 09:21                       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-03-17 19:04                         ` Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 19:11                           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Tom Lane <tgl@sss.pgh.pa.us>
  2020-03-31 20:08                           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 2 replies; 70+ messages in thread

From: Justin Pryzby @ 2020-03-17 19:04 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Tue, Mar 17, 2020 at 10:21:48AM +0100, Fabien COELHO wrote:
> 
> About v13, seens as one patch:
> 
> Function "pg_ls_dir_metadata" documentation suggests a variable number of
> arguments with brackets, but parameters are really mandatory.

Fixed, and added tests on 1 and 3 arg versions of both pg_ls_dir() and
pg_ls_dir_metadata().

It seems like the only way to make variable number of arguments is is with
multiple entries in pg_proc.dat, one for each "number of" arguments.  Is that
right ?

> The example in the documentation could be better indented. Also, ISTM that
> there are two implicit laterals (format & pg_ls_dir_recurse) that I would
> make explicit. I'd use the pcs alias explicitely. I'd use meaningful aliases
> (eg ts instead of b, …).

> On reflection, I think that a boolean "isdir" column is a bad idea because
> it is not extensible. I'd propose to switch to the standard "ls" approach of
> providing the type as one character: '-' for regular, 'd' for directory, 'l'
> for link, 's' for socket, 'c' for character special…

I think that's outside the scope of the patch, since I'd want to change
pg_stat_file; that's where I borrowed "isdir" from, for consistency.

Note that both LS_DIR_HISTORIC and LS_DIR_MODERN include LS_DIR_SKIP_SPECIAL,
so only pg_ls_dir itself show specials, so they way to do it would be to 1)
change pg_stat_file to expose the file's "type", 2) use pg_ls_dir() AS a,
lateral pg_stat_file(a) AS b, 3) then consider also changing LS_DIR_MODERN and
all the existing pg_ls_*.

> ISTM that "lstat" is not available on windows, which suggests to call "stat"
> always, and then "lstat" on un*x and pg ports stuff on win.

I believe that's handled here.
src/include/port/win32_port.h:#define lstat(path, sb) stat(path, sb)

> I'm wondering about the restriction on directories only. Why should it not
> work on a file? Can it be easily extended to work on a simple file? If so,
> it could be just "pg_ls".

I think that's a good idea, except it doesn't fit with what the code does:
AllocDir() and ReadDir().  Instead, use pg_stat_file() for that.

Hm, I realized that the existing pg_ls_dir_metadata was skipping links to dirs,
since !ISREG().  So changed to use both stat() and lstat().

-- 
Justin

Attachments:

  [text/x-diff] v14-0001-Document-historic-behavior-of-links-to-directori.patch (1.1K, ../../20200317190401.GY26184@telsasoft.com/2-v14-0001-Document-historic-behavior-of-links-to-directori.patch)
  download | inline diff:
From d8294c4747c5ba1f3bec858c137cc2d31e5a0425 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 16 Mar 2020 14:12:55 -0500
Subject: [PATCH v14 1/8] Document historic behavior of links to directories..

Backpatch to 9.5: pg_stat_file
---
 doc/src/sgml/func.sgml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fc4d7f0f78..2c6142a0e0 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21528,7 +21528,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory.  Typical usages include:
+    indicating if it is a directory (or a symbolic link to a directory).
+    Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
 SELECT (pg_stat_file('filename')).modification;
-- 
2.17.0

  [text/x-diff] v14-0002-Add-tests-on-pg_ls_dir-before-changing-it.patch (2.0K, ../../20200317190401.GY26184@telsasoft.com/3-v14-0002-Add-tests-on-pg_ls_dir-before-changing-it.patch)
  download | inline diff:
From 031ce2684587d6d8d84c6d4b6c87f5399b6efb78 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 17 Mar 2020 13:16:24 -0500
Subject: [PATCH v14 2/8] Add tests on pg_ls_dir before changing it

---
 src/test/regress/expected/misc_functions.out | 18 ++++++++++++++++++
 src/test/regress/sql/misc_functions.sql      |  5 +++++
 2 files changed, 23 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..2e87c548eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -201,6 +201,24 @@ select count(*) > 0 from
  t
 (1 row)
 
+select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true
+ name 
+------
+ .
+(1 row)
+
+select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false
+ name 
+------
+(0 rows)
+
+select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
+ pg_ls_dir 
+-----------
+(0 rows)
+
+select pg_ls_dir('does not exist'); -- fails with missingok=false
+ERROR:  could not open directory "does not exist": No such file or directory
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..f6857ad177 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -60,6 +60,11 @@ select count(*) > 0 from
    where spcname = 'pg_default') pts
   join pg_database db on pts.pts = db.oid;
 
+select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true
+select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false
+select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
+select pg_ls_dir('does not exist'); -- fails with missingok=false
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v14-0003-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (20.4K, ../../20200317190401.GY26184@telsasoft.com/4-v14-0003-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From 8ad4e2f920659df6656b6074b851e5718d8e948b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v14 3/8] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

Change to use lstat() to allow pg_ls_dir_recurse to avoid infinite recursion.
That means:
 - links to dirs are shown with isdir=false;
 - timestamps shown are those of the link;
 - changed pg_stat_file for consistency;

Need catversion bumped?
---
 doc/src/sgml/func.sgml                       |  19 +-
 src/backend/catalog/system_views.sql         |   1 +
 src/backend/utils/adt/genfile.c              | 244 ++++++++++++-------
 src/include/catalog/pg_proc.dat              |  12 +
 src/test/regress/expected/misc_functions.out |  24 ++
 src/test/regress/input/tablespace.source     |   5 +
 src/test/regress/output/tablespace.source    |   8 +
 src/test/regress/sql/misc_functions.sql      |  11 +
 8 files changed, 230 insertions(+), 94 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 2c6142a0e0..68c7327e1d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21342,6 +21342,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         List the contents of a directory.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21442,6 +21451,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_metadata</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_metadata</function> lists the files in the specified
+    directory along with the file's metadata.
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
@@ -21528,7 +21545,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory (or a symbolic link to a directory).
+    indicating if it is a directory (and not a symbolic link to a directory).
     Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b8a3f46912..05a644a7c9 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1435,6 +1435,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 01185f218b..daae06aa9a 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,6 +36,21 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
+#define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	LS_DIR_SKIP_HIDDEN			(1<<4) /* Do not show anything begining with . */
+#define	LS_DIR_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	LS_DIR_SKIP_SPECIAL			(1<<6) /* Do not show special file types */
+
+/*
+ * Shortcut for the historic behavior of the pg_ls_* functions (not including
+ * pg_ls_dir, which skips different files and doesn't show metadata.
+ */
+#define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -370,7 +385,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 
 	filename = convert_and_check_filename(filename_t);
 
-	if (stat(filename, &fst) < 0)
+	if (lstat(filename, &fst) < 0)
 	{
 		if (missing_ok && errno == ENOENT)
 			PG_RETURN_NULL();
@@ -413,6 +428,11 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
 #endif
 	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
+#ifdef WIN32
+	/* Links should have isdir=false */
+	if (pgwin32_is_junction(filename))
+		values[5] = BoolGetDatum(false);
+#endif
 
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
@@ -440,79 +460,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
-	char	   *location;
-	bool		missing_ok = false;
-	bool		include_dot_dirs = false;
-	bool		randomAccess;
-	TupleDesc	tupdesc;
-	Tuplestorestate *tupstore;
-	DIR		   *dirdesc;
-	struct dirent *de;
-	MemoryContext oldcontext;
-
-	location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-	/* check the optional arguments */
-	if (PG_NARGS() == 3)
-	{
-		if (!PG_ARGISNULL(1))
-			missing_ok = PG_GETARG_BOOL(1);
-		if (!PG_ARGISNULL(2))
-			include_dot_dirs = PG_GETARG_BOOL(2);
-	}
-
-	/* check to see if caller supports us returning a tuplestore */
-	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
-		ereport(ERROR,
-				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-				 errmsg("set-valued function called in context that cannot accept a set")));
-	if (!(rsinfo->allowedModes & SFRM_Materialize))
-		ereport(ERROR,
-				(errcode(ERRCODE_SYNTAX_ERROR),
-				 errmsg("materialize mode required, but it is not allowed in this context")));
-
-	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
-	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
-
-	tupdesc = CreateTemplateTupleDesc(1);
-	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "pg_ls_dir", TEXTOID, -1, 0);
-
-	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
-	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
-	rsinfo->returnMode = SFRM_Materialize;
-	rsinfo->setResult = tupstore;
-	rsinfo->setDesc = tupdesc;
-
-	MemoryContextSwitchTo(oldcontext);
-
-	dirdesc = AllocateDir(location);
-	if (!dirdesc)
-	{
-		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
-			return (Datum) 0;
-		/* Otherwise, we can let ReadDir() throw the error */
-	}
-
-	while ((de = ReadDir(dirdesc, location)) != NULL)
-	{
-		Datum		values[1];
-		bool		nulls[1];
-
-		if (!include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		nulls[0] = false;
-
-		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
-	}
-
-	FreeDir(dirdesc);
-	return (Datum) 0;
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -525,7 +475,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -535,7 +487,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -544,6 +496,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= LS_DIR_MISSING_OK;
+			else
+				flags &= ~LS_DIR_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~LS_DIR_SKIP_DOT_DIRS;
+			else
+				flags |= LS_DIR_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -558,8 +536,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags & LS_DIR_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	} else {
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -578,36 +566,77 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags&LS_DIR_MISSING_OK && errno == ENOENT)
+		{
+			tuplestore_donestoring(tupstore);
 			return (Datum) 0;
+		}
 		/* Otherwise, we can let ReadDir() throw the error */
 	}
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
+		struct stat lattrib;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		/* Skip dot dirs? */
+		if (flags & LS_DIR_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
+		/* Skip hidden files? */
+		if (flags & LS_DIR_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
 		snprintf(path, sizeof(path), "%s/%s", dir, de->d_name);
-		if (stat(path, &attrib) < 0)
+		if (stat(path, &attrib) < 0 ||
+			lstat(path, &lattrib) < 0)
 			ereport(ERROR,
 					(errcode_for_file_access(),
 					 errmsg("could not stat file \"%s\": %m", path)));
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		/* Skip dirs or special files? */
+		if (S_ISDIR(lattrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_DIRS)
+				continue;
+		}
+		else if (S_ISDIR(attrib.st_mode))
+		{
+			/*
+			 * Do nothing: links to dirs are not skipped (but are shown as
+			 * isdir=false)
+			 */
+			;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & LS_DIR_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) lattrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(lattrib.st_mtime));
+			if (flags & LS_DIR_ISDIR)
+			{
+				values[3] = BoolGetDatum(S_ISDIR(lattrib.st_mode));
+#ifdef WIN32
+				/* Links should have isdir=false */
+				if (pgwin32_is_junction(path))
+					values[3] = BoolGetDatum(false);
+#endif
+			}
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -621,14 +650,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
 }
 
 /*
@@ -646,7 +675,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
 }
 
 /*
@@ -675,5 +705,33 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ * note: this wrapper is necessary to pass the sanity check in opr_sanity,
+ * which checks that all built-in functions that share the implementing C
+ * function take the same number of arguments.
+ */
+Datum
+pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7fb574f9dc..b1a957fe3e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10741,6 +10741,18 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
+{ oid => '5033', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{dirname,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata_1arg' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2e87c548eb..7930909f02 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -219,6 +219,30 @@ select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
 
 select pg_ls_dir('does not exist'); -- fails with missingok=false
 ERROR:  could not open directory "does not exist": No such file or directory
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
+select name, isdir from pg_ls_dir_metadata('.') where name='.';
+ name | isdir 
+------+-------
+ .    | t
+(1 row)
+
+select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+ name | isdir 
+------+-------
+(0 rows)
+
+-- Check that expected columns are present
+select * from pg_ls_dir_metadata('.') limit 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index f6857ad177..372345720d 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -65,6 +65,17 @@ select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name=
 select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
 select pg_ls_dir('does not exist'); -- fails with missingok=false
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
+select name, isdir from pg_ls_dir_metadata('.') where name='.';
+
+select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+
+-- Check that expected columns are present
+select * from pg_ls_dir_metadata('.') limit 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v14-0004-pg_ls_tmpdir-to-show-isdir-argument.patch (7.4K, ../../20200317190401.GY26184@telsasoft.com/5-v14-0004-pg_ls_tmpdir-to-show-isdir-argument.patch)
  download | inline diff:
From 12c87d95527ec530c52da5272c41038026555897 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v14 4/8] pg_ls_tmpdir to show "isdir" argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 68c7327e1d..b813b19daf 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21391,12 +21391,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21499,14 +21499,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index daae06aa9a..5b10c1df9b 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -481,7 +481,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -676,7 +676,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+			LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b1a957fe3e..fae5b5194a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10733,13 +10733,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 7930909f02..38493de732 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -222,8 +222,8 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select name, isdir from pg_ls_dir_metadata('.') where name='.';
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v14-0005-pg_ls_-dir-to-show-directories-and-isdir-column.patch (9.4K, ../../20200317190401.GY26184@telsasoft.com/6-v14-0005-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From b90d77abe7080233b5cd35680b5d25c7d4418b07 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v14 5/8] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 11 ++++--
 src/include/catalog/pg_proc.dat              | 12 +++---
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 37 insertions(+), 29 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index b813b19daf..4cdd03610a 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21357,8 +21357,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21368,8 +21369,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21379,8 +21381,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21463,36 +21466,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 5b10c1df9b..21611dfbc0 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -52,6 +52,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
  */
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
+/* Shortcut for common behavior */
+#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -650,14 +653,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_COMMON);
 }
 
 /*
@@ -676,7 +679,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
+			LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /*
@@ -706,7 +709,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+			LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fae5b5194a..2c5829bca4 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10717,18 +10717,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 38493de732..64b1417fb8 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v14-0006-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (6.1K, ../../20200317190401.GY26184@telsasoft.com/7-v14-0006-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From 0db676c0c08cedf927b80b330463f21b48a6983b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v14 6/8] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

XXX: this test case is unstable, if backends/autovacuum remove FSM...
ERROR:  could not stat file "./base/16384/20345_fsm": No such file or directory
CONTEXT:  SQL function "pg_ls_dir_recurse" statement 1

Need catversion bumped ?
---
 doc/src/sgml/func.sgml                       | 25 ++++++++++++++++++++
 src/backend/catalog/system_views.sql         |  1 +
 src/include/catalog/pg_proc.dat              |  6 +++++
 src/test/regress/expected/misc_functions.out | 14 +++++++++++
 src/test/regress/sql/misc_functions.sql      |  6 +++++
 5 files changed, 52 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4cdd03610a..b30ac9f10d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21351,6 +21351,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+         Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata.
+         Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21462,6 +21472,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     directory along with the file's metadata.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    To recursively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir
+FROM pg_tablespace b, pg_control_system() pcs,
+lateral format('/PG_%s_%s', left(current_setting('server_version_num'), 2), pcs.catalog_version_no) AS suffix) AS dir,
+lateral pg_ls_dir_recurse(dir) AS a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 05a644a7c9..e7295d8aaf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1436,6 +1436,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_recurse(text) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2c5829bca4..af2c2dd621 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10753,6 +10753,12 @@
   proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
   proargnames => '{dirname,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
+{ oid => '5034', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 64b1417fb8..4188d684f0 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -243,6 +243,20 @@ select * from pg_ls_dir_metadata('.') limit 0;
 ------+------+--------------+-------
 (0 rows)
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+         name          | isdir 
+-----------------------+-------
+ pg_wal                | t
+ pg_wal/archive_status | t
+(2 rows)
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 372345720d..6041c4f3dc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -76,6 +76,12 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; --
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v14-0007-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.0K, ../../20200317190401.GY26184@telsasoft.com/8-v14-0007-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From 5ce104119543729151f1a80b40e9565ad3afa521 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v14 7/8] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 21611dfbc0..00de0c091d 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -653,7 +653,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..ba9a3fe29a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v14-0008-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (17.2K, ../../20200317190401.GY26184@telsasoft.com/9-v14-0008-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From 362e434f64a478a4f74339f4291e0d48eccd1ec6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v14 8/8] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 ++++++++---------
 src/backend/utils/adt/genfile.c              | 29 ++++++++++-------
 src/include/catalog/pg_proc.dat              | 34 ++++++++++----------
 src/test/regress/expected/misc_functions.out | 16 ++++-----
 src/test/regress/output/tablespace.source    |  8 ++---
 5 files changed, 59 insertions(+), 58 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index b30ac9f10d..a815fe7b7d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21367,8 +21367,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21379,8 +21378,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21391,9 +21389,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21405,8 +21402,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21492,8 +21488,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21504,8 +21500,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21517,8 +21513,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21531,8 +21527,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 00de0c091d..5750c5d949 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -579,8 +579,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 		struct stat lattrib;
@@ -624,24 +624,29 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 				continue;
 		}
 
+		memset(nulls, 0, sizeof(nulls));
 		values[0] = CStringGetTextDatum(de->d_name);
 		if (flags & LS_DIR_METADATA)
 		{
 			values[1] = Int64GetDatum((int64) lattrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(lattrib.st_mtime));
-			if (flags & LS_DIR_ISDIR)
-			{
-				values[3] = BoolGetDatum(S_ISDIR(lattrib.st_mode));
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
+			values[6] = BoolGetDatum(S_ISDIR(lattrib.st_mode));
 #ifdef WIN32
-				/* Links should have isdir=false */
-				if (pgwin32_is_junction(path))
-					values[3] = BoolGetDatum(false);
+			/* Links should have isdir=false */
+			if (pgwin32_is_junction(path))
+				values[6] = BoolGetDatum(false);
 #endif
-			}
 		}
 
-		memset(nulls, 0, sizeof(nulls));
-
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
 	}
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index af2c2dd621..954e2cf7dd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10717,48 +10717,48 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '5033', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{dirname,name,size,modification,isdir}',
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
 { oid => '5034', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification,a.change,a.creation,a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 4188d684f0..cce84a60a9 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -222,8 +222,8 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select name, isdir from pg_ls_dir_metadata('.') where name='.';
@@ -239,8 +239,8 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; --
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
@@ -253,8 +253,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index ba9a3fe29a..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 09:21                       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-17 19:04                         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-17 19:11                           ` Tom Lane <tgl@sss.pgh.pa.us>
  1 sibling, 0 replies; 70+ messages in thread

From: Tom Lane @ 2020-03-17 19:11 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>

Justin Pryzby <pryzby@telsasoft.com> writes:
> It seems like the only way to make variable number of arguments is is with
> multiple entries in pg_proc.dat, one for each "number of" arguments.  Is that
> right ?

Another way to do it is to have one entry, put the full set of arguments
into the initial pg_proc.dat data, and then use CREATE OR REPLACE FUNCTION
later during initdb to install some defaults.  See existing cases in
system_views.sql, starting about line 1180.  Neither way is especially
pretty, so take your choice.

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 09:21                       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-17 19:04                         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-03-31 20:08                           ` Justin Pryzby <pryzby@telsasoft.com>
  2020-04-12 11:53                             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  1 sibling, 1 reply; 70+ messages in thread

From: Justin Pryzby @ 2020-03-31 20:08 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Tue, Mar 17, 2020 at 02:04:01PM -0500, Justin Pryzby wrote:
> > The example in the documentation could be better indented. Also, ISTM that
> > there are two implicit laterals (format & pg_ls_dir_recurse) that I would
> > make explicit. I'd use the pcs alias explicitely. I'd use meaningful aliases
> > (eg ts instead of b, …).
> 
> > On reflection, I think that a boolean "isdir" column is a bad idea because
> > it is not extensible. I'd propose to switch to the standard "ls" approach of
> > providing the type as one character: '-' for regular, 'd' for directory, 'l'
> > for link, 's' for socket, 'c' for character special…
> 
> I think that's outside the scope of the patch, since I'd want to change
> pg_stat_file; that's where I borrowed "isdir" from, for consistency.
> 
> Note that both LS_DIR_HISTORIC and LS_DIR_MODERN include LS_DIR_SKIP_SPECIAL,
> so only pg_ls_dir itself show specials, so they way to do it would be to 1)
> change pg_stat_file to expose the file's "type", 2) use pg_ls_dir() AS a,
> lateral pg_stat_file(a) AS b, 3) then consider also changing LS_DIR_MODERN and
> all the existing pg_ls_*.

The patch intends to fix the issue of "failing to show failed filesets"
(because dirs are skipped) while also generalizing existing functions (to show
directories and "isdir" column) and providing some more flexible ones (to list
file and metadata of a dir, which is currently possible [only] for "special"
directories, or by recursively calling pg_stat_file).

I'm still of the opinion that supporting arbitrary file types is out of scope,
but I changed the "isdir" to show "type".  I'm only supporting '[-dl]'.  I
don't want to have to check #ifdef S_ISDOOR or whatever other vendors have.  I
insist that it is a separate patch, since it depends on everything else, and I
have no feedback from anybody else as to whether any of that is desired.

template1=# SELECT * FROM pg_ls_waldir();
           name           |   size   |         access         |      modification      |         change         | creation | type 
--------------------------+----------+------------------------+------------------------+------------------------+----------+------
 barr                     |        0 | 2020-03-31 14:43:11-05 | 2020-03-31 14:43:11-05 | 2020-03-31 14:43:11-05 |          | ?
 baz                      |     4096 | 2020-03-31 14:39:18-05 | 2020-03-31 14:39:18-05 | 2020-03-31 14:39:18-05 |          | d
 foo                      |        0 | 2020-03-31 14:39:37-05 | 2020-03-31 14:39:37-05 | 2020-03-31 14:39:37-05 |          | -
 archive_status           |     4096 | 2020-03-31 14:38:20-05 | 2020-03-31 14:38:18-05 | 2020-03-31 14:38:18-05 |          | d
 000000010000000000000001 | 16777216 | 2020-03-31 14:42:53-05 | 2020-03-31 14:43:08-05 | 2020-03-31 14:43:08-05 |          | -
 bar                      |        3 | 2020-03-31 14:39:16-05 | 2020-03-31 14:39:01-05 | 2020-03-31 14:39:01-05 |          | l

Attachments:

  [text/x-diff] v15-0001-Document-historic-behavior-of-links-to-directori.patch (1.1K, ../../20200331200812.GI14618@telsasoft.com/2-v15-0001-Document-historic-behavior-of-links-to-directori.patch)
  download | inline diff:
From f6b54482cc1d1b9595f49211b14807a20f994a3f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 16 Mar 2020 14:12:55 -0500
Subject: [PATCH v15 01/10] Document historic behavior of links to
 directories..

Backpatch to 9.5: pg_stat_file
---
 doc/src/sgml/func.sgml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 7a0bb0c70a..25b1278459 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21558,7 +21558,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory.  Typical usages include:
+    indicating if it is a directory (or a symbolic link to a directory).
+    Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
 SELECT (pg_stat_file('filename')).modification;
-- 
2.17.0

  [text/x-diff] v15-0002-pg_stat_file-and-pg_ls_dir_-to-use-lstat.patch (2.2K, ../../20200331200812.GI14618@telsasoft.com/3-v15-0002-pg_stat_file-and-pg_ls_dir_-to-use-lstat.patch)
  download | inline diff:
From e1c8750fb777df4bf41f36df7ae6cde39b907ec2 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 30 Mar 2020 18:59:16 -0500
Subject: [PATCH v15 02/10] pg_stat_file and pg_ls_dir_* to use lstat()..

pg_ls_dir_* will now skip (no longer show) symbolic links, same as other
non-regular file types, as we advertize we do since 8b6d94cf6.  That seems to
be the intented behavior, since irregular file types are 1) less portable; and,
2) we don't currently show a file's type except for "bool is_dir".

pg_stat_file will now 1) show metadata of links themselves, rather than their
target; and, 2) specifically, show links to directories with "is_dir=false";
and, 3) not error on broken symlinks.
---
 doc/src/sgml/func.sgml          | 2 +-
 src/backend/utils/adt/genfile.c | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 25b1278459..630f03d7dd 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21558,7 +21558,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory (or a symbolic link to a directory).
+    indicating if it is a directory.
     Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index ceaa6180da..219ac160f8 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -370,7 +370,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 
 	filename = convert_and_check_filename(filename_t);
 
-	if (stat(filename, &fst) < 0)
+	if (lstat(filename, &fst) < 0)
 	{
 		if (missing_ok && errno == ENOENT)
 			PG_RETURN_NULL();
@@ -596,7 +596,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 		/* Get the file info */
 		snprintf(path, sizeof(path), "%s/%s", dir, de->d_name);
-		if (stat(path, &attrib) < 0)
+		if (lstat(path, &attrib) < 0)
 		{
 			/* Ignore concurrently-deleted files, else complain */
 			if (errno == ENOENT)
-- 
2.17.0

  [text/x-diff] v15-0003-Add-tests-on-pg_ls_dir-before-changing-it.patch (2.0K, ../../20200331200812.GI14618@telsasoft.com/4-v15-0003-Add-tests-on-pg_ls_dir-before-changing-it.patch)
  download | inline diff:
From 4c6e234cfee829f0e14f1d4cfb6c0a7c77c86c9b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 17 Mar 2020 13:16:24 -0500
Subject: [PATCH v15 03/10] Add tests on pg_ls_dir before changing it

---
 src/test/regress/expected/misc_functions.out | 18 ++++++++++++++++++
 src/test/regress/sql/misc_functions.sql      |  5 +++++
 2 files changed, 23 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..2e87c548eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -201,6 +201,24 @@ select count(*) > 0 from
  t
 (1 row)
 
+select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true
+ name 
+------
+ .
+(1 row)
+
+select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false
+ name 
+------
+(0 rows)
+
+select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
+ pg_ls_dir 
+-----------
+(0 rows)
+
+select pg_ls_dir('does not exist'); -- fails with missingok=false
+ERROR:  could not open directory "does not exist": No such file or directory
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..f6857ad177 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -60,6 +60,11 @@ select count(*) > 0 from
    where spcname = 'pg_default') pts
   join pg_database db on pts.pts = db.oid;
 
+select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true
+select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false
+select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
+select pg_ls_dir('does not exist'); -- fails with missingok=false
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v15-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (19.4K, ../../20200331200812.GI14618@telsasoft.com/5-v15-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From e255804b9e50e869bb2ba7538349ff2b9cb26133 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v15 04/10] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

XXX: Change to use lstat() to allow pg_ls_dir_recurse to avoid infinite recursion.
That means:
 - links to dirs are NOT SHOWN (was: shown with isdir=false;)
 - NO: timestamps shown are those of the link;
 - changed pg_stat_file for consistency;

Need catversion bumped?
---
 doc/src/sgml/func.sgml                       |  17 ++
 src/backend/catalog/system_views.sql         |   1 +
 src/backend/utils/adt/genfile.c              | 227 +++++++++++--------
 src/include/catalog/pg_proc.dat              |  12 +
 src/test/regress/expected/misc_functions.out |  24 ++
 src/test/regress/input/tablespace.source     |   5 +
 src/test/regress/output/tablespace.source    |   8 +
 src/test/regress/sql/misc_functions.sql      |  11 +
 8 files changed, 214 insertions(+), 91 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 630f03d7dd..ed8e1977ba 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21372,6 +21372,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         List the contents of a directory.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21472,6 +21481,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_metadata</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_metadata</function> lists the files in the specified
+    directory along with the file's metadata.
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 83d00c6cde..4511f9d658 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1436,6 +1436,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 219ac160f8..eaa7c548e0 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,6 +36,21 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
+#define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	LS_DIR_SKIP_HIDDEN			(1<<4) /* Do not show anything begining with . */
+#define	LS_DIR_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	LS_DIR_SKIP_SPECIAL			(1<<6) /* Do not show special file types */
+
+/*
+ * Shortcut for the historic behavior of the pg_ls_* functions (not including
+ * pg_ls_dir, which skips different files and doesn't show metadata.
+ */
+#define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -413,6 +428,11 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
 #endif
 	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
+#ifdef WIN32
+	/* Links should have isdir=false */
+	if (pgwin32_is_junction(filename))
+		values[5] = BoolGetDatum(false);
+#endif
 
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
@@ -440,79 +460,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
-	char	   *location;
-	bool		missing_ok = false;
-	bool		include_dot_dirs = false;
-	bool		randomAccess;
-	TupleDesc	tupdesc;
-	Tuplestorestate *tupstore;
-	DIR		   *dirdesc;
-	struct dirent *de;
-	MemoryContext oldcontext;
-
-	location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-	/* check the optional arguments */
-	if (PG_NARGS() == 3)
-	{
-		if (!PG_ARGISNULL(1))
-			missing_ok = PG_GETARG_BOOL(1);
-		if (!PG_ARGISNULL(2))
-			include_dot_dirs = PG_GETARG_BOOL(2);
-	}
-
-	/* check to see if caller supports us returning a tuplestore */
-	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
-		ereport(ERROR,
-				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-				 errmsg("set-valued function called in context that cannot accept a set")));
-	if (!(rsinfo->allowedModes & SFRM_Materialize))
-		ereport(ERROR,
-				(errcode(ERRCODE_SYNTAX_ERROR),
-				 errmsg("materialize mode required, but it is not allowed in this context")));
-
-	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
-	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
-
-	tupdesc = CreateTemplateTupleDesc(1);
-	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "pg_ls_dir", TEXTOID, -1, 0);
-
-	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
-	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
-	rsinfo->returnMode = SFRM_Materialize;
-	rsinfo->setResult = tupstore;
-	rsinfo->setDesc = tupdesc;
-
-	MemoryContextSwitchTo(oldcontext);
-
-	dirdesc = AllocateDir(location);
-	if (!dirdesc)
-	{
-		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
-			return (Datum) 0;
-		/* Otherwise, we can let ReadDir() throw the error */
-	}
-
-	while ((de = ReadDir(dirdesc, location)) != NULL)
-	{
-		Datum		values[1];
-		bool		nulls[1];
-
-		if (!include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		nulls[0] = false;
-
-		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
-	}
-
-	FreeDir(dirdesc);
-	return (Datum) 0;
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -525,7 +475,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -535,7 +487,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -544,6 +496,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= LS_DIR_MISSING_OK;
+			else
+				flags &= ~LS_DIR_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~LS_DIR_SKIP_DOT_DIRS;
+			else
+				flags |= LS_DIR_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -558,8 +536,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags & LS_DIR_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	} else {
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -578,20 +566,27 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags&LS_DIR_MISSING_OK && errno == ENOENT)
 			return (Datum) 0;
 		/* Otherwise, we can let ReadDir() throw the error */
 	}
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		/* Skip dot dirs? */
+		if (flags & LS_DIR_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
+		/* Skip hidden files? */
+		if (flags & LS_DIR_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
@@ -606,13 +601,34 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 					 errmsg("could not stat file \"%s\": %m", path)));
 		}
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		/* Skip dirs or special files? */
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_DIRS)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & LS_DIR_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & LS_DIR_ISDIR)
+			{
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+#ifdef WIN32
+				/* Links should have isdir=false */
+				if (pgwin32_is_junction(path))
+					values[3] = BoolGetDatum(false);
+#endif
+			}
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -626,14 +642,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
 }
 
 /*
@@ -651,7 +667,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
 }
 
 /*
@@ -680,5 +697,33 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ * note: this wrapper is necessary to pass the sanity check in opr_sanity,
+ * which checks that all built-in functions that share the implementing C
+ * function take the same number of arguments.
+ */
+Datum
+pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a6a708cca9..e8846e1522 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10801,6 +10801,18 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
+{ oid => '5033', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{dirname,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata_1arg' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2e87c548eb..7930909f02 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -219,6 +219,30 @@ select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
 
 select pg_ls_dir('does not exist'); -- fails with missingok=false
 ERROR:  could not open directory "does not exist": No such file or directory
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
+select name, isdir from pg_ls_dir_metadata('.') where name='.';
+ name | isdir 
+------+-------
+ .    | t
+(1 row)
+
+select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+ name | isdir 
+------+-------
+(0 rows)
+
+-- Check that expected columns are present
+select * from pg_ls_dir_metadata('.') limit 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index f6857ad177..372345720d 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -65,6 +65,17 @@ select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name=
 select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
 select pg_ls_dir('does not exist'); -- fails with missingok=false
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
+select name, isdir from pg_ls_dir_metadata('.') where name='.';
+
+select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+
+-- Check that expected columns are present
+select * from pg_ls_dir_metadata('.') limit 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v15-0005-pg_ls_tmpdir-to-show-directories-and-isdir-argum.patch (7.4K, ../../20200331200812.GI14618@telsasoft.com/6-v15-0005-pg_ls_tmpdir-to-show-directories-and-isdir-argum.patch)
  download | inline diff:
From 3294321ae8a5a751b4e81e544e070a9106020b82 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v15 05/10] pg_ls_tmpdir to show directories and "isdir"
 argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index ed8e1977ba..343af5e954 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21421,12 +21421,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -21529,14 +21529,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index eaa7c548e0..78edc85719 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -481,7 +481,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -668,7 +668,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+			LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index e8846e1522..40d951f029 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10793,13 +10793,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 7930909f02..38493de732 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -222,8 +222,8 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select name, isdir from pg_ls_dir_metadata('.') where name='.';
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v15-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch (9.4K, ../../20200331200812.GI14618@telsasoft.com/7-v15-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From 374b0334ab7438c0155242fec26b81ae1ab3fb6b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v15 06/10] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 11 ++++--
 src/include/catalog/pg_proc.dat              | 12 +++---
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 37 insertions(+), 29 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 343af5e954..6a4623d59b 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21387,8 +21387,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21398,8 +21399,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -21409,8 +21411,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21493,36 +21496,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 78edc85719..afef2bf4ab 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -52,6 +52,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
  */
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
+/* Shortcut for common behavior */
+#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -642,14 +645,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_COMMON);
 }
 
 /*
@@ -668,7 +671,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
+			LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /*
@@ -698,7 +701,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+			LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 40d951f029..408bd40256 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,18 +10777,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 38493de732..64b1417fb8 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v15-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (6.1K, ../../20200331200812.GI14618@telsasoft.com/8-v15-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From 9a9f031b65a85b3b4085db3c293321c7f1a8b56f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v15 07/10] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

XXX: this test case is unstable, if backends/autovacuum remove FSM...
ERROR:  could not stat file "./base/16384/20345_fsm": No such file or directory
CONTEXT:  SQL function "pg_ls_dir_recurse" statement 1

Need catversion bumped ?
---
 doc/src/sgml/func.sgml                       | 25 ++++++++++++++++++++
 src/backend/catalog/system_views.sql         |  1 +
 src/include/catalog/pg_proc.dat              |  6 +++++
 src/test/regress/expected/misc_functions.out | 14 +++++++++++
 src/test/regress/sql/misc_functions.sql      |  6 +++++
 5 files changed, 52 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6a4623d59b..7c8b102610 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21381,6 +21381,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         For each file in a directory, list the file and its metadata.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+         Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata.
+         Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -21492,6 +21502,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     directory along with the file's metadata.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    To recursively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir
+FROM pg_tablespace b, pg_control_system() pcs,
+lateral format('/PG_%s_%s', left(current_setting('server_version_num'), 2), pcs.catalog_version_no) AS suffix) AS dir,
+lateral pg_ls_dir_recurse(dir) AS a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4511f9d658..e37ad20013 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1437,6 +1437,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_recurse(text) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 408bd40256..6c3b6f1f5b 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10813,6 +10813,12 @@
   proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
   proargnames => '{dirname,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
+{ oid => '5034', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 64b1417fb8..4188d684f0 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -243,6 +243,20 @@ select * from pg_ls_dir_metadata('.') limit 0;
 ------+------+--------------+-------
 (0 rows)
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+         name          | isdir 
+-----------------------+-------
+ pg_wal                | t
+ pg_wal/archive_status | t
+(2 rows)
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 372345720d..6041c4f3dc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -76,6 +76,12 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; --
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v15-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.1K, ../../20200331200812.GI14618@telsasoft.com/9-v15-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From bf2e00ec7c3e1fda45d0d5d8cb34afd617cdc424 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v15 08/10] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index afef2bf4ab..e8dc1d1be5 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -645,7 +645,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..ba9a3fe29a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v15-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (17.2K, ../../20200331200812.GI14618@telsasoft.com/10-v15-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From 0192275d6918d7a13789fd77a56b55927839881e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v15 09/10] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 ++++++++---------
 src/backend/utils/adt/genfile.c              | 29 ++++++++++-------
 src/include/catalog/pg_proc.dat              | 34 ++++++++++----------
 src/test/regress/expected/misc_functions.out | 16 ++++-----
 src/test/regress/output/tablespace.source    |  8 ++---
 5 files changed, 59 insertions(+), 58 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 7c8b102610..88397ccddb 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21397,8 +21397,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21409,8 +21408,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -21421,9 +21419,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -21435,8 +21432,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -21522,8 +21518,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -21534,8 +21530,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21547,8 +21543,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -21561,8 +21557,8 @@ lateral pg_ls_dir_recurse(dir) AS a;
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index e8dc1d1be5..4d90f37a21 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -576,8 +576,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
@@ -616,24 +616,29 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 				continue;
 		}
 
+		memset(nulls, 0, sizeof(nulls));
 		values[0] = CStringGetTextDatum(de->d_name);
 		if (flags & LS_DIR_METADATA)
 		{
 			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & LS_DIR_ISDIR)
-			{
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime));
+			values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+			values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+			nulls[5] = true;
+#else
+			nulls[4] = true;
+			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
+#endif
+			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
 #ifdef WIN32
-				/* Links should have isdir=false */
-				if (pgwin32_is_junction(path))
-					values[3] = BoolGetDatum(false);
+			/* Links should have isdir=false */
+			if (pgwin32_is_junction(path))
+				values[6] = BoolGetDatum(false);
 #endif
-			}
 		}
 
-		memset(nulls, 0, sizeof(nulls));
-
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
 	}
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6c3b6f1f5b..d97ff8097f 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,48 +10777,48 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '5033', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{dirname,name,size,modification,isdir}',
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
 { oid => '5034', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 4188d684f0..cce84a60a9 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -222,8 +222,8 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select name, isdir from pg_ls_dir_metadata('.') where name='.';
@@ -239,8 +239,8 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; --
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
@@ -253,8 +253,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index ba9a3fe29a..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v15-0010-pg_ls_-to-show-file-type-and-show-special-files.patch (17.6K, ../../20200331200812.GI14618@telsasoft.com/11-v15-0010-pg_ls_-to-show-file-type-and-show-special-files.patch)
  download | inline diff:
From 9091a9138a8c2622e84ea2a06e922b8cb703be62 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 31 Mar 2020 14:40:34 -0500
Subject: [PATCH v15 10/10] pg_ls_* to show file type and show special files

---
 src/backend/utils/adt/genfile.c              | 52 ++++++++++++--------
 src/include/catalog/pg_proc.dat              | 38 +++++++-------
 src/test/regress/expected/misc_functions.out | 40 +++++++--------
 src/test/regress/output/tablespace.source    |  8 +--
 src/test/regress/sql/misc_functions.sql      |  6 +--
 5 files changed, 77 insertions(+), 67 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4d90f37a21..1fdb2bf280 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -38,7 +38,7 @@
 
 static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
 
-#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_TYPE				(1<<0) /* Show column: type */
 #define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
 #define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
 #define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
@@ -53,7 +53,7 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
 /* Shortcut for common behavior */
-#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_METADATA)
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -367,6 +367,26 @@ pg_read_binary_file_all(PG_FUNCTION_ARGS)
 	return pg_read_binary_file(fcinfo);
 }
 
+/* Return a character indicating the type of file, or '?' if unknown type */
+static char
+get_file_type(mode_t mode, const char *path)
+{
+	if (S_ISREG(mode))
+		return '-';
+
+	if (S_ISDIR(mode))
+		return 'd';
+#ifndef WIN32
+	if (S_ISLNK(mode))
+		return 'l';
+#else
+	if (pgwin32_is_junction(path))
+		return 'l';
+#endif
+
+	return '?';
+}
+
 /*
  * stat a file
  */
@@ -414,7 +434,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	TupleDescInitEntry(tupdesc, (AttrNumber) 5,
 					   "creation", TIMESTAMPTZOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 6,
-					   "isdir", BOOLOID, -1, 0);
+					   "type", CHAROID, -1, 0);
 	BlessTupleDesc(tupdesc);
 
 	memset(isnull, false, sizeof(isnull));
@@ -430,12 +450,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	isnull[3] = true;
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
 #endif
-	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
-#ifdef WIN32
-	/* Links should have isdir=false */
-	if (pgwin32_is_junction(filename))
-		values[5] = BoolGetDatum(false);
-#endif
+	values[5] = CharGetDatum(get_file_type(fst.st_mode, filename));
 
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
@@ -501,10 +516,10 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 	MemoryContext oldcontext;
 	TypeFuncClass	tuptype ;
 
-	/* isdir depends on metadata */
-	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
-	/* Unreasonable to show isdir and skip dirs */
-	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+	/* type depends on metadata */
+	Assert(!(flags&LS_DIR_TYPE) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show type and skip dirs XXX */
+	Assert(!(flags&LS_DIR_TYPE) || !(flags&LS_DIR_SKIP_DIRS));
 
 	/* check the optional arguments */
 	if (PG_NARGS() == 3)
@@ -631,12 +646,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 			nulls[4] = true;
 			values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
 #endif
-			values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
-#ifdef WIN32
-			/* Links should have isdir=false */
-			if (pgwin32_is_junction(path))
-				values[6] = BoolGetDatum(false);
-#endif
+			values[6] = CharGetDatum(get_file_type(attrib.st_mode, path));
 		}
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -718,7 +728,7 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS)
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
 
 	return pg_ls_dir_files(fcinfo, dirname,
-			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE);
 }
 
 /*
@@ -733,5 +743,5 @@ pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS)
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
 
 	return pg_ls_dir_files(fcinfo, dirname,
-			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d97ff8097f..8f19220456 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6107,16 +6107,16 @@
 { oid => '2623', descr => 'get information about file',
   proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
   proargmodes => '{i,o,o,o,o,o,o}',
-  proargnames => '{filename,size,access,modification,change,creation,isdir}',
+  proargnames => '{filename,size,access,modification,change,creation,type}',
   prosrc => 'pg_stat_file_1arg' },
 { oid => '3307', descr => 'get information about file',
   proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record',
   proargtypes => 'text bool',
-  proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
   proargmodes => '{i,i,o,o,o,o,o,o}',
-  proargnames => '{filename,missing_ok,size,access,modification,change,creation,isdir}',
+  proargnames => '{filename,missing_ok,size,access,modification,change,creation,type}',
   prosrc => 'pg_stat_file' },
 { oid => '2624', descr => 'read text from a file',
   proname => 'pg_read_file', provolatile => 'v', prorettype => 'text',
@@ -10777,13 +10777,13 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
-  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
-  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
@@ -10793,32 +10793,32 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
-  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
-  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,type}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,type}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '5033', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
-  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}',
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,type}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
 { oid => '5034', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
-  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,type}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.type from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.type='d') select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index cce84a60a9..1bdcca16fc 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -222,39 +222,39 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
-select name, isdir from pg_ls_dir_metadata('.') where name='.';
- name | isdir 
-------+-------
- .    | t
+select name, type from pg_ls_dir_metadata('.') where name='.';
+ name | type 
+------+------
+ .    | d
 (1 row)
 
-select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
- name | isdir 
-------+-------
+select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+ name | type 
+------+------
 (0 rows)
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
-SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
-         name          | isdir 
------------------------+-------
- pg_wal                | t
- pg_wal/archive_status | t
+SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal';
+         name          | type 
+-----------------------+------
+ pg_wal                | d
+ pg_wal/archive_status | d
 (2 rows)
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1e1e02b589..025c9709a1 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 6041c4f3dc..3874625ca1 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -69,15 +69,15 @@ select pg_ls_dir('does not exist'); -- fails with missingok=false
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
 
-select name, isdir from pg_ls_dir_metadata('.') where name='.';
+select name, type from pg_ls_dir_metadata('.') where name='.';
 
-select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
-SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 09:21                       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-17 19:04                         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-31 20:08                           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
@ 2020-04-12 11:53                             ` Fabien COELHO <coelho@cri.ensmp.fr>
  2020-05-03 02:42                               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 1 reply; 70+ messages in thread

From: Fabien COELHO @ 2020-04-12 11:53 UTC (permalink / raw)
  To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>


Hello Justin,

About v15, seen as one patch.

Patches serie applies cleanly, compiles, "make check" ok.

Documentation:
  - indent documentation text around 80 cols, as done around?
  - indent SQL example for readability and capitalize keywords
    (pg_ls_dir_metadata)
  - "For each file in a directory, list the file and its metadata."
    maybe: "List files and their metadata in a directory"?

Code:
  - Most pg_ls_*dir* functions call pg_ls_dir_files(), which looks like
    reasonable refactoring, ISTM that the code is actually smaller.
  - please follow pg style, eg not "} else {"
  - there is a "XXX" (meaning fixme?) tag remaining in a comment.
  - file types: why not do block & character devices, fifo and socket
    as well, before the unkown case?
  - I'm wondering whether could pg_stat_file call pg_ls_dir_files without
    too much effort? ISTM that the output structure nearly the same. I do
    not like much having one function specialized for files and one for
    directories.

Tests:
  - good, there are some!
  - indent SQL code, eg by starting a new line on new clauses?
  - put comments on separate lines (I'm not against it on principle, I do
    that, but I do not think that it is done much in test files).

-- 
Fabien.





^ permalink  raw  reply  [nested|flat] 70+ messages in thread

* Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)
  2020-03-05 16:18 Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-06 23:35 ` Re: pg_ls_tmpdir to show directories and shared filesets Justin Pryzby <pryzby@telsasoft.com>
  2020-03-07 14:14   ` Re: pg_ls_tmpdir to show directories and shared filesets Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-07 21:40     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-10 18:30       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-13 13:12         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-15 17:15           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-15 21:27             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 15:20               ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-16 21:48                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-16 22:17                   ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Alvaro Herrera <alvherre@2ndquadrant.com>
  2020-03-17 03:14                     ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-17 09:21                       ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
  2020-03-17 19:04                         ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-03-31 20:08                           ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-04-12 11:53                             ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Fabien COELHO <coelho@cri.ensmp.fr>
@ 2020-05-03 02:42                               ` Justin Pryzby <pryzby@telsasoft.com>
  2020-05-07 15:08                                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  2020-05-26 02:10                                 ` Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*) Justin Pryzby <pryzby@telsasoft.com>
  0 siblings, 2 replies; 70+ messages in thread

From: Justin Pryzby @ 2020-05-03 02:42 UTC (permalink / raw)
  To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; David Steele <david@pgmasters.net>; pgsql-hackers@postgresql.org, "Bossart, Nathan" <bossartn@amazon.com>; Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>

On Sun, Apr 12, 2020 at 01:53:40PM +0200, Fabien COELHO wrote:
> About v15, seen as one patch.

Thanks for looking.

> - I'm wondering whether could pg_stat_file call pg_ls_dir_files without
>   too much effort? ISTM that the output structure nearly the same. I do
>   not like much having one function specialized for files and one for
>   directories.

I refactored but not like that.  As I mentioned in the commit message, I don't
see a good way to make a function operate on a file when the function's primary
data structure is a DIR*.  Do you ?  I don't think it should call stat() and
then conditionally branch off to pg_stat_file().

There are two functions because they wrap two separate syscalls, which see as
good, transparent goal.  If we want a function that does what "ls -al" does,
that would also be a good example to follow, except that we already didn't
follow it.

/bin/ls first stat()s the path, and then either outputs its metadata (if it's a
file or if -d was specified) or lists a dir.  It's essentially a wrapper around
*two* system calls (stat and readdir/getdents).

Maybe we could invent a new pg_ls() which does that, and then refactor existing
code.  Or, maybe it would be a SQL function which calls stat() and then
conditionally calls pg_ls_dir if isdir=True (or type='d').  That would be easy
if we merge the commit which outputs all stat fields.

I'm still hoping for confirmation from a committer if this approach is worth
pursuing:

https://www.postgresql.org/message-id/20200310183037.GA29065%40telsasoft.com
https://www.postgresql.org/message-id/20200313131232.GO29065%40telsasoft.com
|Rather than making "recurse into tmpdir" the end goal:
|
|  - add a function to show metadata of an arbitrary dir;
|  - add isdir arguments to pg_ls_* functions (including pg_ls_tmpdir but not
|    pg_ls_dir).
|  - maybe add pg_ls_dir_recurse, which satisfies the original need;
|  - retire pg_ls_dir (does this work with tuplestore?)
|  - profit

-- 
Justin

Attachments:

  [text/x-diff] v16-0001-Document-historic-behavior-of-links-to-directori.patch (1.1K, ../../20200503024215.GJ28974@telsasoft.com/2-v16-0001-Document-historic-behavior-of-links-to-directori.patch)
  download | inline diff:
From 6175cecd312296bbb9099834d91ffaa50e059f6c Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 16 Mar 2020 14:12:55 -0500
Subject: [PATCH v16 01/10] Document historic behavior of links to
 directories..

Backpatch to 9.5: pg_stat_file
---
 doc/src/sgml/func.sgml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 96ea57eedd..9b885102da 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25486,7 +25486,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory.  Typical usages include:
+    indicating if it is a directory (or a symbolic link to a directory).
+    Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
 SELECT (pg_stat_file('filename')).modification;
-- 
2.17.0

  [text/x-diff] v16-0002-pg_stat_file-and-pg_ls_dir_-to-use-lstat.patch (2.2K, ../../20200503024215.GJ28974@telsasoft.com/3-v16-0002-pg_stat_file-and-pg_ls_dir_-to-use-lstat.patch)
  download | inline diff:
From 52fd41bfacaba66f26f1e75b34e258b3a0f12372 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 30 Mar 2020 18:59:16 -0500
Subject: [PATCH v16 02/10] pg_stat_file and pg_ls_dir_* to use lstat()..

pg_ls_dir_* will now skip (no longer show) symbolic links, same as other
non-regular file types, as we advertize we do since 8b6d94cf6.  That seems to
be the intented behavior, since irregular file types are 1) less portable; and,
2) we don't currently show a file's type except for "bool is_dir".

pg_stat_file will now 1) show metadata of links themselves, rather than their
target; and, 2) specifically, show links to directories with "is_dir=false";
and, 3) not error on broken symlinks.
---
 doc/src/sgml/func.sgml          | 2 +-
 src/backend/utils/adt/genfile.c | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9b885102da..96b08d0500 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25486,7 +25486,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
     size, last accessed time stamp, last modified time stamp,
     last file status change time stamp (Unix platforms only),
     file creation time stamp (Windows only), and a <type>boolean</type>
-    indicating if it is a directory (or a symbolic link to a directory).
+    indicating if it is a directory.
     Typical usages include:
 <programlisting>
 SELECT * FROM pg_stat_file('filename');
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index ceaa6180da..219ac160f8 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -370,7 +370,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 
 	filename = convert_and_check_filename(filename_t);
 
-	if (stat(filename, &fst) < 0)
+	if (lstat(filename, &fst) < 0)
 	{
 		if (missing_ok && errno == ENOENT)
 			PG_RETURN_NULL();
@@ -596,7 +596,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 
 		/* Get the file info */
 		snprintf(path, sizeof(path), "%s/%s", dir, de->d_name);
-		if (stat(path, &attrib) < 0)
+		if (lstat(path, &attrib) < 0)
 		{
 			/* Ignore concurrently-deleted files, else complain */
 			if (errno == ENOENT)
-- 
2.17.0

  [text/x-diff] v16-0003-Add-tests-on-pg_ls_dir-before-changing-it.patch (2.0K, ../../20200503024215.GJ28974@telsasoft.com/4-v16-0003-Add-tests-on-pg_ls_dir-before-changing-it.patch)
  download | inline diff:
From 66d83278898baf7451c4984ffb0090a08d898880 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 17 Mar 2020 13:16:24 -0500
Subject: [PATCH v16 03/10] Add tests on pg_ls_dir before changing it

---
 src/test/regress/expected/misc_functions.out | 18 ++++++++++++++++++
 src/test/regress/sql/misc_functions.sql      |  5 +++++
 2 files changed, 23 insertions(+)

diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..2e87c548eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -201,6 +201,24 @@ select count(*) > 0 from
  t
 (1 row)
 
+select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true
+ name 
+------
+ .
+(1 row)
+
+select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false
+ name 
+------
+(0 rows)
+
+select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
+ pg_ls_dir 
+-----------
+(0 rows)
+
+select pg_ls_dir('does not exist'); -- fails with missingok=false
+ERROR:  could not open directory "does not exist": No such file or directory
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..f6857ad177 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -60,6 +60,11 @@ select count(*) > 0 from
    where spcname = 'pg_default') pts
   join pg_database db on pts.pts = db.oid;
 
+select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true
+select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false
+select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
+select pg_ls_dir('does not exist'); -- fails with missingok=false
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v16-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch (19.1K, ../../20200503024215.GJ28974@telsasoft.com/5-v16-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-m.patch)
  download | inline diff:
From 71f1e41b20e232ade555dc63f3a6b15c31203d90 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 22:40:24 -0500
Subject: [PATCH v16 04/10] Add pg_ls_dir_metadata to list a dir with file
 metadata..

Generalize pg_ls_dir_files and retire pg_ls_dir

Need catversion bumped?
---
 doc/src/sgml/func.sgml                       |  19 ++
 src/backend/catalog/system_views.sql         |   1 +
 src/backend/utils/adt/genfile.c              | 229 +++++++++++--------
 src/include/catalog/pg_proc.dat              |  12 +
 src/test/regress/expected/misc_functions.out |  24 ++
 src/test/regress/input/tablespace.source     |   5 +
 src/test/regress/output/tablespace.source    |   8 +
 src/test/regress/sql/misc_functions.sql      |  11 +
 8 files changed, 218 insertions(+), 91 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 96b08d0500..9d617f95b9 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25300,6 +25300,17 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         List the contents of a directory.  Restricted to superusers by default, but other users can be granted EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        For each file in a directory, list the file and its metadata.
+        Restricted to superusers by default, but other users can be granted
+        EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -25400,6 +25411,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     empty directory from an non-existent directory.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_metadata</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_metadata</function> lists the files in the specified
+    directory along with the file's metadata.
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2bd5f5ea14..1c77430f0c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1467,6 +1467,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 219ac160f8..4824a55480 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,6 +36,21 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
+
+#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
+#define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
+#define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
+#define	LS_DIR_SKIP_HIDDEN			(1<<4) /* Do not show anything begining with . */
+#define	LS_DIR_SKIP_DIRS			(1<<5) /* Do not show directories */
+#define	LS_DIR_SKIP_SPECIAL			(1<<6) /* Do not show special file types */
+
+/*
+ * Shortcut for the historic behavior of the pg_ls_* functions (not including
+ * pg_ls_dir, which skips different files and doesn't show metadata.
+ */
+#define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -413,6 +428,11 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
 #endif
 	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
+#ifdef WIN32
+	/* Links should have isdir=false */
+	if (pgwin32_is_junction(filename))
+		values[5] = BoolGetDatum(false);
+#endif
 
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
@@ -440,79 +460,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir(PG_FUNCTION_ARGS)
 {
-	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
-	char	   *location;
-	bool		missing_ok = false;
-	bool		include_dot_dirs = false;
-	bool		randomAccess;
-	TupleDesc	tupdesc;
-	Tuplestorestate *tupstore;
-	DIR		   *dirdesc;
-	struct dirent *de;
-	MemoryContext oldcontext;
-
-	location = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
-
-	/* check the optional arguments */
-	if (PG_NARGS() == 3)
-	{
-		if (!PG_ARGISNULL(1))
-			missing_ok = PG_GETARG_BOOL(1);
-		if (!PG_ARGISNULL(2))
-			include_dot_dirs = PG_GETARG_BOOL(2);
-	}
-
-	/* check to see if caller supports us returning a tuplestore */
-	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
-		ereport(ERROR,
-				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-				 errmsg("set-valued function called in context that cannot accept a set")));
-	if (!(rsinfo->allowedModes & SFRM_Materialize))
-		ereport(ERROR,
-				(errcode(ERRCODE_SYNTAX_ERROR),
-				 errmsg("materialize mode required, but it is not allowed in this context")));
-
-	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
-	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
-
-	tupdesc = CreateTemplateTupleDesc(1);
-	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "pg_ls_dir", TEXTOID, -1, 0);
-
-	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
-	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
-	rsinfo->returnMode = SFRM_Materialize;
-	rsinfo->setResult = tupstore;
-	rsinfo->setDesc = tupdesc;
-
-	MemoryContextSwitchTo(oldcontext);
-
-	dirdesc = AllocateDir(location);
-	if (!dirdesc)
-	{
-		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
-			return (Datum) 0;
-		/* Otherwise, we can let ReadDir() throw the error */
-	}
-
-	while ((de = ReadDir(dirdesc, location)) != NULL)
-	{
-		Datum		values[1];
-		bool		nulls[1];
-
-		if (!include_dot_dirs &&
-			(strcmp(de->d_name, ".") == 0 ||
-			 strcmp(de->d_name, "..") == 0))
-			continue;
-
-		values[0] = CStringGetTextDatum(de->d_name);
-		nulls[0] = false;
-
-		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
-	}
-
-	FreeDir(dirdesc);
-	return (Datum) 0;
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -525,7 +475,9 @@ pg_ls_dir(PG_FUNCTION_ARGS)
 Datum
 pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir(fcinfo);
+	text	*filename_t = PG_GETARG_TEXT_PP(0);
+	char	*filename = convert_and_check_filename(filename_t);
+	return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS);
 }
 
 /*
@@ -535,7 +487,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
  * Other unreadable-directory cases throw an error.
  */
 static Datum
-pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
+pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 {
 	ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
 	bool		randomAccess;
@@ -544,6 +496,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	DIR		   *dirdesc;
 	struct dirent *de;
 	MemoryContext oldcontext;
+	TypeFuncClass	tuptype ;
+
+	/* isdir depends on metadata */
+	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show isdir and skip dirs */
+	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+
+	/* check the optional arguments */
+	if (PG_NARGS() == 3)
+	{
+		if (!PG_ARGISNULL(1))
+		{
+			if (PG_GETARG_BOOL(1))
+				flags |= LS_DIR_MISSING_OK;
+			else
+				flags &= ~LS_DIR_MISSING_OK;
+		}
+
+		if (!PG_ARGISNULL(2))
+		{
+			if (PG_GETARG_BOOL(2))
+				flags &= ~LS_DIR_SKIP_DOT_DIRS;
+			else
+				flags |= LS_DIR_SKIP_DOT_DIRS;
+		}
+	}
 
 	/* check to see if caller supports us returning a tuplestore */
 	if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -558,8 +536,20 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	/* The tupdesc and tuplestore must be created in ecxt_per_query_memory */
 	oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
 
-	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
-		elog(ERROR, "return type must be a row type");
+	tuptype = get_call_result_type(fcinfo, NULL, &tupdesc);
+	if (flags & LS_DIR_METADATA)
+	{
+		if (tuptype != TYPEFUNC_COMPOSITE)
+			elog(ERROR, "return type must be a row type");
+	}
+	else
+	{
+		/* pg_ls_dir returns a simple scalar */
+		if (tuptype != TYPEFUNC_SCALAR)
+			elog(ERROR, "return type must be a scalar type");
+		tupdesc = CreateTemplateTupleDesc(1);
+		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0);
+	}
 
 	randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;
 	tupstore = tuplestore_begin_heap(randomAccess, false, work_mem);
@@ -578,20 +568,27 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 	if (!dirdesc)
 	{
 		/* Return empty tuplestore if appropriate */
-		if (missing_ok && errno == ENOENT)
+		if (flags & LS_DIR_MISSING_OK && errno == ENOENT)
 			return (Datum) 0;
 		/* Otherwise, we can let ReadDir() throw the error */
 	}
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[3];
-		bool		nulls[3];
+		Datum		values[4];
+		bool		nulls[4];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
-		/* Skip hidden files */
-		if (de->d_name[0] == '.')
+		/* Skip dot dirs? */
+		if (flags & LS_DIR_SKIP_DOT_DIRS &&
+			(strcmp(de->d_name, ".") == 0 ||
+			 strcmp(de->d_name, "..") == 0))
+			continue;
+
+		/* Skip hidden files? */
+		if (flags & LS_DIR_SKIP_HIDDEN &&
+			de->d_name[0] == '.')
 			continue;
 
 		/* Get the file info */
@@ -606,13 +603,34 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 					 errmsg("could not stat file \"%s\": %m", path)));
 		}
 
-		/* Ignore anything but regular files */
-		if (!S_ISREG(attrib.st_mode))
-			continue;
+		/* Skip dirs or special files? */
+		if (S_ISDIR(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_DIRS)
+				continue;
+		}
+		else if (!S_ISREG(attrib.st_mode))
+		{
+			if (flags & LS_DIR_SKIP_SPECIAL)
+				continue;
+		}
 
 		values[0] = CStringGetTextDatum(de->d_name);
-		values[1] = Int64GetDatum((int64) attrib.st_size);
-		values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+		if (flags & LS_DIR_METADATA)
+		{
+			values[1] = Int64GetDatum((int64) attrib.st_size);
+			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
+			if (flags & LS_DIR_ISDIR)
+			{
+				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
+#ifdef WIN32
+				/* Links should have isdir=false */
+				if (pgwin32_is_junction(path))
+					values[3] = BoolGetDatum(false);
+#endif
+			}
+		}
+
 		memset(nulls, 0, sizeof(nulls));
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -626,14 +644,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, false);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, false);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
 }
 
 /*
@@ -651,7 +669,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 						tblspc)));
 
 	TempTablespacePath(path, tblspc);
-	return pg_ls_dir_files(fcinfo, path, true);
+	return pg_ls_dir_files(fcinfo, path,
+			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
 }
 
 /*
@@ -680,5 +699,33 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS)
 Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
+	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
+			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ */
+Datum
+pg_ls_dir_metadata(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+}
+
+/*
+ * Function to return the list of files and metadata in an arbitrary directory.
+ * note: this wrapper is necessary to pass the sanity check in opr_sanity,
+ * which checks that all built-in functions that share the implementing C
+ * function take the same number of arguments.
+ */
+Datum
+pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS)
+{
+	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
+
+	return pg_ls_dir_files(fcinfo, dirname,
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4bce3ad8de..9f46cba5ed 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10904,6 +10904,18 @@
   proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
   proargnames => '{tablespace,name,size,modification}',
   prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '5032', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata' },
+{ oid => '5033', descr => 'list directory with metadata',
+  proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{dirname,name,size,modification,isdir}',
+  prosrc => 'pg_ls_dir_metadata_1arg' },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2e87c548eb..7930909f02 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -219,6 +219,30 @@ select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
 
 select pg_ls_dir('does not exist'); -- fails with missingok=false
 ERROR:  could not open directory "does not exist": No such file or directory
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
+select name, isdir from pg_ls_dir_metadata('.') where name='.';
+ name | isdir 
+------+-------
+ .    | t
+(1 row)
+
+select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+ name | isdir 
+------+-------
+(0 rows)
+
+-- Check that expected columns are present
+select * from pg_ls_dir_metadata('.') limit 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index a5f61a35dc..0b9cfe615e 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 162b591b31..a42714bf40 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
 DROP TABLESPACE regress_tblspacewith;
 -- create a tablespace we can use
 CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
+SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
+ name | size | modification 
+------+------+--------------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index f6857ad177..372345720d 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -65,6 +65,17 @@ select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name=
 select pg_ls_dir('does not exist', true, false); -- ok with missingok=true
 select pg_ls_dir('does not exist'); -- fails with missingok=false
 
+-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+select * from pg_ls_tmpdir() where name='Does not exist';
+
+select name, isdir from pg_ls_dir_metadata('.') where name='.';
+
+select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+
+-- Check that expected columns are present
+select * from pg_ls_dir_metadata('.') limit 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v16-0005-pg_ls_tmpdir-to-show-directories-and-isdir-argum.patch (7.4K, ../../20200503024215.GJ28974@telsasoft.com/6-v16-0005-pg_ls_tmpdir-to-show-directories-and-isdir-argum.patch)
  download | inline diff:
From 04ed9715a4920828d554770de2c3c48cff1cc548 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:57:54 -0500
Subject: [PATCH v16 05/10] pg_ls_tmpdir to show directories and "isdir"
 argument..

similar to pg_stat_file().

It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonbly complicated.

See following commit which also adds these columns to the other pg_ls_*
functions.  Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 22 +++++++++++---------
 src/backend/utils/adt/genfile.c              |  4 ++--
 src/include/catalog/pg_proc.dat              |  8 +++----
 src/test/regress/expected/misc_functions.out |  4 ++--
 src/test/regress/output/tablespace.source    |  4 ++--
 5 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9d617f95b9..891b990859 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25351,12 +25351,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the
-        temporary directory for <parameter>tablespace</parameter>.  If
-        <parameter>tablespace</parameter> is not provided, the
-        <literal>pg_default</literal> tablespace is used.  Access is granted
-        to members of the <literal>pg_monitor</literal> role and may be
-        granted to other non-superuser roles.
+        For the temporary directory within <parameter>tablespace</parameter>,
+        list each file's name, size, last modification time, and a boolean
+        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        is not provided, the <literal>pg_default</literal> tablespace is used.
+        Access is granted to members of the <literal>pg_monitor</literal> role
+        and may be granted to other non-superuser roles.
        </entry>
       </row>
       <row>
@@ -25459,14 +25459,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_tmpdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_tmpdir</function> returns the name, size, and last modified
-    time (mtime) of each file in the temporary file directory for the specified
-    <parameter>tablespace</parameter>.  If <parameter>tablespace</parameter> is
+    <function>pg_ls_tmpdir</function> lists each file in the temporary file
+    directory for the specified <parameter>tablespace</parameter>, along with
+    its size, last modified time (mtime) and a boolean indicating if the file is a
+    directory.  Directories are used for temporary files shared by parallel
+    processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
     role can use this function.  Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4824a55480..023f1ad605 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -481,7 +481,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
 }
 
 /*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
  *
  * If the directory isn't there, silently return an empty set if missing_ok.
  * Other unreadable-directory cases throw an error.
@@ -670,7 +670,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+			LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9f46cba5ed..dc3b497587 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10896,13 +10896,13 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
-  proargnames => '{tablespace,name,size,modification}',
+  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+  proargnames => '{tablespace,name,size,modification,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 7930909f02..38493de732 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -222,8 +222,8 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select name, isdir from pg_ls_dir_metadata('.') where name='.';
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v16-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch (9.4K, ../../20200503024215.GJ28974@telsasoft.com/7-v16-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch)
  download | inline diff:
From 5ad5a708b1fe7183aff91eca86c29a9c216dac93 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 01:00:42 -0500
Subject: [PATCH v16 06/10] pg_ls_*dir to show directories and "isdir" column..

pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 39 +++++++++++---------
 src/backend/utils/adt/genfile.c              | 11 ++++--
 src/include/catalog/pg_proc.dat              | 12 +++---
 src/test/regress/expected/misc_functions.out |  4 +-
 4 files changed, 37 insertions(+), 29 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 891b990859..c3a6b3277f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25317,8 +25317,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the log
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the log directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -25328,8 +25329,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        directory. Access is granted to members of the <literal>pg_monitor</literal>
+        For each file in the WAL directory, list the file's name, size, last
+        modification time, and a boolean indicating if it is a directory.
+        Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
       </row>
@@ -25339,8 +25341,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        List the name, size, and last modification time of files in the WAL
-        archive status directory. Access is granted to members of the
+        For each file in the WAL archive status directory, list the file's
+        name, size, last modification time, and a boolean indicating if it is a
+        directory.  Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -25423,36 +25426,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     <primary>pg_ls_logdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_logdir</function> returns the name, size, and last modified time
-    (mtime) of each file in the log directory. By default, only superusers
+    <function>pg_ls_logdir</function> lists each file in the log directory,
+    along with file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_waldir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_waldir</function> returns the name, size, and last modified time
-    (mtime) of each file in the write ahead log (WAL) directory. By
-    default only superusers and members of the <literal>pg_monitor</literal> role
+    <function>pg_ls_waldir</function> lists each file in the WAL directory,
+    along with the file's size, last modification time, and a boolean
+    indicating if the file is a directory.  By default, only superusers
+    and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
     <primary>pg_ls_archive_statusdir</primary>
    </indexterm>
    <para>
-    <function>pg_ls_archive_statusdir</function> returns the name, size, and
-    last modified time (mtime) of each file in the WAL archive status
-    directory <filename>pg_wal/archive_status</filename>. By default only
+    <function>pg_ls_archive_statusdir</function> lists each file in the WAL
+    archive status directory, along with the file's size, last modification
+    time, and a boolean indicating if the file is a directory.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
-    Filenames beginning with a dot, directories, and other special files are not shown.
+    Filenames beginning with a dot and special file types are not shown.
    </para>
 
    <indexterm>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 023f1ad605..fba63568b6 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -52,6 +52,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
  */
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
+/* Shortcut for common behavior */
+#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
  *
@@ -644,14 +647,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON);
 }
 
 /* Function to return the list of files in the WAL directory */
 Datum
 pg_ls_waldir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC);
+	return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_COMMON);
 }
 
 /*
@@ -670,7 +673,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
 
 	TempTablespacePath(path, tblspc);
 	return pg_ls_dir_files(fcinfo, path,
-			LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
+			LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /*
@@ -700,7 +703,7 @@ Datum
 pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
 {
 	return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status",
-			LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+			LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index dc3b497587..adfce45d1a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10880,18 +10880,18 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
-  proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz}',
-  proargmodes => '{o,o,o}', proargnames => '{name,size,modification}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
+  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 38493de732..64b1417fb8 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification 
-------+------+--------------
+ name | size | modification | isdir 
+------+------+--------------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
-- 
2.17.0

  [text/x-diff] v16-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch (5.8K, ../../20200503024215.GJ28974@telsasoft.com/8-v16-0007-Add-pg_ls_dir_recurse-to-show-dir-recursively.patch)
  download | inline diff:
From 44951d8c1d07df57806dab12dd77c155de9527ec Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 8 Mar 2020 22:52:14 -0500
Subject: [PATCH v16 07/10] Add pg_ls_dir_recurse to show dir recursively..

..possibly there's a better place to put this, like maybe a doc-only example ?

Need catversion bumped ?
---
 doc/src/sgml/func.sgml                       | 27 ++++++++++++++++++++
 src/backend/catalog/system_views.sql         |  1 +
 src/include/catalog/pg_proc.dat              |  6 +++++
 src/test/regress/expected/misc_functions.out | 14 ++++++++++
 src/test/regress/sql/misc_functions.sql      |  6 +++++
 5 files changed, 54 insertions(+)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index c3a6b3277f..3a2ad3eb50 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25311,6 +25311,18 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
         EXECUTE to run the function.
        </entry>
       </row>
+      <row>
+       <entry>
+        <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal>
+       </entry>
+       <entry><type>setof text</type></entry>
+       <entry>
+        Call pg_ls_dir_metadata to recursively list the files in the specified
+        directory, along with each file's metadata.
+        Restricted to superusers by default, but other users can be granted
+        EXECUTE to run the function.
+       </entry>
+      </row>
       <row>
        <entry>
         <literal><function>pg_ls_logdir()</function></literal>
@@ -25422,6 +25434,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
     directory along with the file's metadata.
    </para>
 
+   <indexterm>
+    <primary>pg_ls_dir_recurse</primary>
+   </indexterm>
+   <para>
+    <function>pg_ls_dir_recurse</function> recursively lists the files 
+    in the specified directory.
+    To recursively list temporary directories in all tablespaces:
+<programlisting>
+SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir
+FROM pg_tablespace b, pg_control_system() pcs,
+LATERAL format('/PG_%s_%s', left(current_setting('server_version_num'), 2), pcs.catalog_version_no) AS suffix) AS dir,
+LATERAL pg_ls_dir_recurse(dir) AS a;
+</programlisting>
+   </para>
+
    <indexterm>
     <primary>pg_ls_logdir</primary>
    </indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 1c77430f0c..215fba3b7d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1468,6 +1468,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
 REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_ls_dir_recurse(text) FROM public;
 
 --
 -- We also set up some things as accessible to standard roles.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index adfce45d1a..f94f403475 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10916,6 +10916,12 @@
   proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
   proargnames => '{dirname,name,size,modification,isdir}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
+{ oid => '5034', descr => 'list all files in a directory recursively',
+  proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
+  provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+  proallargtypes => '{text,text,int8,timestamptz,bool}',
+  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 64b1417fb8..4188d684f0 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -243,6 +243,20 @@ select * from pg_ls_dir_metadata('.') limit 0;
 ------+------+--------------+-------
 (0 rows)
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+         name          | isdir 
+-----------------------+-------
+ pg_wal                | t
+ pg_wal/archive_status | t
+(2 rows)
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 --
 -- Test adding a support function to a subject function
 --
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 372345720d..6041c4f3dc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -76,6 +76,12 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; --
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
 
+-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
+SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+
+-- Check that expected columns are present
+SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
+
 --
 -- Test adding a support function to a subject function
 --
-- 
2.17.0

  [text/x-diff] v16-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch (3.1K, ../../20200503024215.GJ28974@telsasoft.com/9-v16-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch)
  download | inline diff:
From 49fb4f0f6be210667fdd5d27aba26c5a9227fd09 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 6 Mar 2020 17:23:51 -0600
Subject: [PATCH v16 08/10] pg_ls_logdir to ignore error if initial/top dir is
 missing

---
 src/backend/utils/adt/genfile.c           | 2 +-
 src/test/regress/input/tablespace.source  | 4 ++++
 src/test/regress/output/tablespace.source | 7 +++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index fba63568b6..91bf8c69e9 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -647,7 +647,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 Datum
 pg_ls_logdir(PG_FUNCTION_ARGS)
 {
-	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON);
+	return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON | LS_DIR_MISSING_OK);
 }
 
 /* Function to return the list of files in the WAL directory */
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source
index 0b9cfe615e..2a1268e17c 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/input/tablespace.source
@@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1c88e914e3..ba9a3fe29a 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa
 ------+------+--------------+-------
 (0 rows)
 
+-- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
+-- The name='' condition is never true, so the function runs to completion but returns zero rows.
+SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
+ name | size | modification | isdir 
+------+------+--------------+-------
+(0 rows)
+
 -- try setting and resetting some properties for the new tablespace
 ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
 ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true);  -- fail
-- 
2.17.0

  [text/x-diff] v16-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch (18.7K, ../../20200503024215.GJ28974@telsasoft.com/10-v16-0009-pg_ls_-dir-to-return-all-the-metadata-from-pg_st.patch)
  download | inline diff:
From 392dc661c9cc627acab82fd9a35b33061c925349 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 9 Mar 2020 21:56:21 -0500
Subject: [PATCH v16 09/10] pg_ls_*dir to return all the metadata from
 pg_stat_file..

..but it doesn't seem worth factoring out the common bits, since stat_file
doesn't return a name, so all the field numbers are off by one.

NOTE, the atime is now shown where the mtime used to be.

Need catversion bump
---
 doc/src/sgml/func.sgml                       | 30 ++++-----
 src/backend/utils/adt/genfile.c              | 65 +++++++++-----------
 src/include/catalog/pg_proc.dat              | 34 +++++-----
 src/test/regress/expected/misc_functions.out | 16 ++---
 src/test/regress/output/tablespace.source    |  8 +--
 5 files changed, 71 insertions(+), 82 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 3a2ad3eb50..1177799916 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25329,8 +25329,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the log directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the log directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -25341,8 +25340,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL directory, list the file's name, size, last
-        modification time, and a boolean indicating if it is a directory.
+        For each file in the WAL directory, list the file and its metadata.
         Access is granted to members of the <literal>pg_monitor</literal>
         role and may be granted to other non-superuser roles.
        </entry>
@@ -25353,9 +25351,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        </entry>
        <entry><type>setof record</type></entry>
        <entry>
-        For each file in the WAL archive status directory, list the file's
-        name, size, last modification time, and a boolean indicating if it is a
-        directory.  Access is granted to members of the
+        For each file in the WAL archive status directory, list the file and its metadata.
+        Access is granted to members of the
         <literal>pg_monitor</literal> role and may be granted to other
         non-superuser roles.
        </entry>
@@ -25367,8 +25364,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        <entry><type>setof record</type></entry>
        <entry>
         For the temporary directory within <parameter>tablespace</parameter>,
-        list each file's name, size, last modification time, and a boolean
-        indicating if it is a directory.  If <parameter>tablespace</parameter>
+        list each file and its metadata.  If <parameter>tablespace</parameter>
         is not provided, the <literal>pg_default</literal> tablespace is used.
         Access is granted to members of the <literal>pg_monitor</literal> role
         and may be granted to other non-superuser roles.
@@ -25454,8 +25450,8 @@ LATERAL pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_logdir</function> lists each file in the log directory,
-    along with file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role can use this function.
     Access may be granted to others using <command>GRANT</command>.
     Filenames beginning with a dot and special file types are not shown.
@@ -25466,8 +25462,8 @@ LATERAL pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_waldir</function> lists each file in the WAL directory,
-    along with the file's size, last modification time, and a boolean
-    indicating if the file is a directory.  By default, only superusers
+    along with the metadata columns returned by <function>pg_stat_file</function>.
+    By default, only superusers
     and members of the <literal>pg_monitor</literal> role
     can use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -25479,8 +25475,8 @@ LATERAL pg_ls_dir_recurse(dir) AS a;
    </indexterm>
    <para>
     <function>pg_ls_archive_statusdir</function> lists each file in the WAL
-    archive status directory, along with the file's size, last modification
-    time, and a boolean indicating if the file is a directory.  By default, only 
+    archive status directory, along with the metadata columns returned by
+    <function>pg_stat_file</function>.  By default, only 
     superusers and members of the <literal>pg_monitor</literal> role can
     use this function. Access may be granted to others using
     <command>GRANT</command>.
@@ -25493,8 +25489,8 @@ LATERAL pg_ls_dir_recurse(dir) AS a;
    <para>
     <function>pg_ls_tmpdir</function> lists each file in the temporary file
     directory for the specified <parameter>tablespace</parameter>, along with
-    its size, last modified time (mtime) and a boolean indicating if the file is a
-    directory.  Directories are used for temporary files shared by parallel
+    the metadata columns returned by <function>pg_stat_file</function>.
+    Directories are used for temporary files shared by parallel
     processes.  If <parameter>tablespace</parameter> is
     not provided, the <literal>pg_default</literal> tablespace is used.  By
     default only superusers and members of the <literal>pg_monitor</literal>
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 91bf8c69e9..10780d3fb1 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,6 +36,8 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+static void tuple_from_stat(struct stat *fst, const char *path, Datum *values,
+		bool *isnull);
 static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
 
 #define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
@@ -367,6 +369,28 @@ pg_read_binary_file_all(PG_FUNCTION_ARGS)
 	return pg_read_binary_file(fcinfo);
 }
 
+/*
+ * Populate values and isnull from fst and path.
+ * Used for pg_stat_file() and pg_stat_dir_files()
+ * isnull is assumed to have been zeroed.
+ */
+static void
+tuple_from_stat(struct stat *fst, const char *path, Datum *values, bool *isnull)
+{
+	values[0] = Int64GetDatum((int64) fst->st_size);
+	values[1] = TimestampTzGetDatum(time_t_to_timestamptz(fst->st_atime));
+	values[2] = TimestampTzGetDatum(time_t_to_timestamptz(fst->st_mtime));
+	/* Unix has file status change time, while Win32 has creation time */
+#if !defined(WIN32) && !defined(__CYGWIN__)
+	values[3] = TimestampTzGetDatum(time_t_to_timestamptz(fst->st_ctime));
+	isnull[4] = true;
+#else
+	isnull[3] = true;
+	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst->st_ctime));
+#endif
+	values[5] = BoolGetDatum(S_ISDIR(fst->st_mode));
+}
+
 /*
  * stat a file
  */
@@ -418,25 +442,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	BlessTupleDesc(tupdesc);
 
 	memset(isnull, false, sizeof(isnull));
-
-	values[0] = Int64GetDatum((int64) fst.st_size);
-	values[1] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_atime));
-	values[2] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_mtime));
-	/* Unix has file status change time, while Win32 has creation time */
-#if !defined(WIN32) && !defined(__CYGWIN__)
-	values[3] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
-	isnull[4] = true;
-#else
-	isnull[3] = true;
-	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
-#endif
-	values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
-#ifdef WIN32
-	/* Links should have isdir=false */
-	if (pgwin32_is_junction(filename))
-		values[5] = BoolGetDatum(false);
-#endif
-
+	tuple_from_stat(&fst, filename, values, isnull);
 	tuple = heap_form_tuple(tupdesc, values, isnull);
 
 	pfree(filename);
@@ -578,8 +584,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 
 	while ((de = ReadDir(dirdesc, dir)) != NULL)
 	{
-		Datum		values[4];
-		bool		nulls[4];
+		Datum		values[7];
+		bool		nulls[7];
 		char		path[MAXPGPATH * 2];
 		struct stat attrib;
 
@@ -618,23 +624,10 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 				continue;
 		}
 
+		memset(nulls, 0, sizeof(nulls));
 		values[0] = CStringGetTextDatum(de->d_name);
 		if (flags & LS_DIR_METADATA)
-		{
-			values[1] = Int64GetDatum((int64) attrib.st_size);
-			values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime));
-			if (flags & LS_DIR_ISDIR)
-			{
-				values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode));
-#ifdef WIN32
-				/* Links should have isdir=false */
-				if (pgwin32_is_junction(path))
-					values[3] = BoolGetDatum(false);
-#endif
-			}
-		}
-
-		memset(nulls, 0, sizeof(nulls));
+			tuple_from_stat(&attrib, path, 1+values, 1+nulls);
 
 		tuplestore_putvalues(tupstore, tupdesc, values, nulls);
 	}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f94f403475..34ee5a8f97 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10880,48 +10880,48 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
-  proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}',
-  proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}',
+  proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_archive_statusdir' },
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
-  proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{tablespace,name,size,modification,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '5033', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
-  proargnames => '{dirname,name,size,modification,isdir}',
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
 { oid => '5034', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,bool}',
-  proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 4188d684f0..cce84a60a9 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -222,8 +222,8 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 select name, isdir from pg_ls_dir_metadata('.') where name='.';
@@ -239,8 +239,8 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; --
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
@@ -253,8 +253,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index ba9a3fe29a..1e1e02b589 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | modification | isdir 
-------+------+--------------+-------
+ name | size | access | modification | change | creation | isdir 
+------+------+--------+--------------+--------+----------+-------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
-- 
2.17.0

  [text/x-diff] v16-0010-pg_ls_-to-show-file-type-and-show-special-files.patch (17.5K, ../../20200503024215.GJ28974@telsasoft.com/11-v16-0010-pg_ls_-to-show-file-type-and-show-special-files.patch)
  download | inline diff:
From 90b2cdee7516a3ec358ad1f248f45352f782917e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 31 Mar 2020 14:40:34 -0500
Subject: [PATCH v16 10/10] pg_ls_* to show file type and show special files

---
 src/backend/utils/adt/genfile.c              | 58 ++++++++++++++++----
 src/include/catalog/pg_proc.dat              | 38 ++++++-------
 src/test/regress/expected/misc_functions.out | 40 +++++++-------
 src/test/regress/output/tablespace.source    |  8 +--
 src/test/regress/sql/misc_functions.sql      |  6 +-
 5 files changed, 94 insertions(+), 56 deletions(-)

diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 10780d3fb1..57b52625a0 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -36,11 +36,12 @@
 #include "utils/syscache.h"
 #include "utils/timestamp.h"
 
+static char get_file_type(mode_t mode, const char *path);
 static void tuple_from_stat(struct stat *fst, const char *path, Datum *values,
 		bool *isnull);
 static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
 
-#define	LS_DIR_ISDIR				(1<<0) /* Show column: isdir */
+#define	LS_DIR_TYPE				(1<<0) /* Show column: type */
 #define	LS_DIR_METADATA				(1<<1) /* Show columns: mtime, size */
 #define	LS_DIR_MISSING_OK			(1<<2) /* Ignore ENOENT if the toplevel dir is missing */
 #define	LS_DIR_SKIP_DOT_DIRS		(1<<3) /* Do not show . or .. */
@@ -55,7 +56,7 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
 #define LS_DIR_HISTORIC				(LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
 
 /* Shortcut for common behavior */
-#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+#define LS_DIR_COMMON				(LS_DIR_SKIP_HIDDEN|LS_DIR_METADATA)
 
 /*
  * Convert a "text" filename argument to C string, and check it's allowable.
@@ -369,6 +370,43 @@ pg_read_binary_file_all(PG_FUNCTION_ARGS)
 	return pg_read_binary_file(fcinfo);
 }
 
+/* Return a character indicating the type of file, or '?' if unknown type */
+static char
+get_file_type(mode_t mode, const char *path)
+{
+	if (S_ISREG(mode))
+		return '-';
+
+	if (S_ISDIR(mode))
+		return 'd';
+#ifndef WIN32
+	if (S_ISLNK(mode))
+		return 'l';
+#else
+	if (pgwin32_is_junction(path))
+		return 'l';
+#endif
+
+#ifdef S_ISCHR
+	if (S_ISCHR(mode))
+		return 'c';
+#endif
+#ifdef S_ISBLK
+	if (S_ISBLK(mode))
+		return 'b';
+#endif
+#ifdef S_ISFIFO
+	if (S_ISFIFO(mode))
+		return 'p';
+#endif
+#ifdef S_ISSOCK
+	if (S_ISSOCK(mode))
+		return 's';
+#endif
+
+	return '?';
+}
+
 /*
  * Populate values and isnull from fst and path.
  * Used for pg_stat_file() and pg_stat_dir_files()
@@ -388,7 +426,7 @@ tuple_from_stat(struct stat *fst, const char *path, Datum *values, bool *isnull)
 	isnull[3] = true;
 	values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst->st_ctime));
 #endif
-	values[5] = BoolGetDatum(S_ISDIR(fst->st_mode));
+	values[5] = CharGetDatum(get_file_type(fst->st_mode, path));
 }
 
 /*
@@ -438,7 +476,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
 	TupleDescInitEntry(tupdesc, (AttrNumber) 5,
 					   "creation", TIMESTAMPTZOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 6,
-					   "isdir", BOOLOID, -1, 0);
+					   "type", CHAROID, -1, 0);
 	BlessTupleDesc(tupdesc);
 
 	memset(isnull, false, sizeof(isnull));
@@ -507,10 +545,10 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
 	MemoryContext oldcontext;
 	TypeFuncClass	tuptype ;
 
-	/* isdir depends on metadata */
-	Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
-	/* Unreasonable to show isdir and skip dirs */
-	Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+	/* type depends on metadata */
+	Assert(!(flags&LS_DIR_TYPE) || (flags&LS_DIR_METADATA));
+	/* Unreasonable to show type and skip dirs XXX */
+	Assert(!(flags&LS_DIR_TYPE) || !(flags&LS_DIR_SKIP_DIRS));
 
 	/* check the optional arguments */
 	if (PG_NARGS() == 3)
@@ -708,7 +746,7 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS)
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
 
 	return pg_ls_dir_files(fcinfo, dirname,
-			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE);
 }
 
 /*
@@ -723,5 +761,5 @@ pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS)
 	char	*dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
 
 	return pg_ls_dir_files(fcinfo, dirname,
-			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+			LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE);
 }
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 34ee5a8f97..6e1682dafb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6158,16 +6158,16 @@
 { oid => '2623', descr => 'get information about file',
   proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
   proargmodes => '{i,o,o,o,o,o,o}',
-  proargnames => '{filename,size,access,modification,change,creation,isdir}',
+  proargnames => '{filename,size,access,modification,change,creation,type}',
   prosrc => 'pg_stat_file_1arg' },
 { oid => '3307', descr => 'get information about file',
   proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record',
   proargtypes => 'text bool',
-  proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+  proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
   proargmodes => '{i,i,o,o,o,o,o,o}',
-  proargnames => '{filename,missing_ok,size,access,modification,change,creation,isdir}',
+  proargnames => '{filename,missing_ok,size,access,modification,change,creation,type}',
   prosrc => 'pg_stat_file' },
 { oid => '2624', descr => 'read text from a file',
   proname => 'pg_read_file', provolatile => 'v', prorettype => 'text',
@@ -10880,13 +10880,13 @@
 { oid => '3353', descr => 'list files in the log directory',
   proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
-  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_logdir' },
 { oid => '3354', descr => 'list of files in the WAL directory',
   proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
-  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_waldir' },
 { oid => '5031', descr => 'list of files in the archive_status directory',
   proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
   proretset => 't', provolatile => 'v', prorettype => 'record',
@@ -10896,32 +10896,32 @@
 { oid => '5029', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => '',
-  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
-  proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+  proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+  proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_tmpdir_noargs' },
 { oid => '5030', descr => 'list files in the pgsql_tmp directory',
   proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
-  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
-  proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
+  proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{tablespace,name,size,access,modification,change,creation,type}',
   prosrc => 'pg_ls_tmpdir_1arg' },
 { oid => '5032', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
-  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
-  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
+  proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,type}',
   prosrc => 'pg_ls_dir_metadata' },
 { oid => '5033', descr => 'list directory with metadata',
   proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
-  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}',
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,type}',
   prosrc => 'pg_ls_dir_metadata_1arg' },
 { oid => '5034', descr => 'list all files in a directory recursively',
   proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
   provolatile => 'v', prorettype => 'record', proargtypes => 'text',
-  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
-  proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
-  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+  proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
+  proargnames => '{dirname,name,size,access,modification,change,creation,type}', proargmodes => '{i,o,o,o,o,o,o,o}',
+  prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.type from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.type='d') select * from ls" },
 
 # hash partitioning constraint function
 { oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index cce84a60a9..1bdcca16fc 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
 
 -- Test not-run-to-completion cases.
 select * from pg_ls_waldir() limit 0;
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -222,39 +222,39 @@ ERROR:  could not open directory "does not exist": No such file or directory
 -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
-select name, isdir from pg_ls_dir_metadata('.') where name='.';
- name | isdir 
-------+-------
- .    | t
+select name, type from pg_ls_dir_metadata('.') where name='.';
+ name | type 
+------+------
+ .    | d
 (1 row)
 
-select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
- name | isdir 
-------+-------
+select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+ name | type 
+------+------
 (0 rows)
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
-SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
-         name          | isdir 
------------------------+-------
- pg_wal                | t
- pg_wal/archive_status | t
+SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal';
+         name          | type 
+-----------------------+------
+ pg_wal                | d
+ pg_wal/archive_status | d
 (2 rows)
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 --
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1e1e02b589..025c9709a1 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
 SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 -- This tests the missing_ok parameter.  If that's not functioning, this would ERROR if the logdir doesn't exist yet.
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | access | modification | change | creation | isdir 
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type 
+------+------+--------+--------------+--------+----------+------
 (0 rows)
 
 -- try setting and resetting some properties for the new tablespace
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 6041c4f3dc..3874625ca1 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -69,15 +69,15 @@ select pg_ls_dir('does not exist'); -- fails with missingok=false
 -- The name='' condition is never true, so the function runs to completion but returns zero rows.
 select * from pg_ls_tmpdir() where name='Does not exist';
 
-select name, isdir from pg_ls_dir_metadata('.') where name='.';
+select name, type from pg_ls_dir_metadata('.') where name='.';
 
-select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
 
 -- Check that expected columns are present
 select * from pg_ls_dir_metadata('.') limit 0;
 
 -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
-SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal';
 
 -- Check that expected columns are present
 SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
-- 
2.17.0

^ permalink  raw  reply  [nested|flat] 70+ messages in thread