public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v25 05/11] pg_ls_tmpdir to show directories and "isdir" argument..
6+ messages / 4 participants
[nested] [flat]
* [PATCH v25 05/11] pg_ls_tmpdir to show directories and "isdir" argument..
@ 2020-03-09 03:57 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Justin Pryzby @ 2020-03-09 03:57 UTC (permalink / raw)
similar to pg_stat_file().
It's worth breaking the function's return type, since core postgres creates
"shared filesets" underneath the temp dirs, and it's unreasonable to not show
them here, and the alternative query to show them is unreasaonably complicated.
See following commit which also adds these columns to the other pg_ls_*
functions. Although I don't think it matters that they're easily UNIONed, it'd
still make great sense if they returned the same columns.
Need catversion bump
---
doc/src/sgml/func.sgml | 17 +++++++++--------
src/backend/utils/adt/genfile.c | 4 ++--
src/include/catalog/pg_proc.dat | 8 ++++----
src/test/regress/expected/misc_functions.out | 4 ++--
src/test/regress/output/tablespace.source | 4 ++--
5 files changed, 19 insertions(+), 18 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 226923a134..4fb4ec9f33 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25829,16 +25829,17 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<returnvalue>setof record</returnvalue>
( <parameter>name</parameter> <type>text</type>,
<parameter>size</parameter> <type>bigint</type>,
- <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ <parameter>modification</parameter> <type>timestamp with time zone</type>,
+ <parameter>isdir</parameter> <type>boolean</type> )
</para>
<para>
- Returns the name, size, and last modification time (mtime) of each
- ordinary file in the temporary file directory for the
- specified <parameter>tablespace</parameter>.
- If <parameter>tablespace</parameter> is not provided,
- the <literal>pg_default</literal> tablespace is examined. Filenames
- beginning with a dot, directories, and other special files are
- excluded.
+ For each file in the temporary directory within the given
+ <parameter>tablespace</parameter>, return the file's name, size, last
+ modification time (mtime) and a boolean indicating if the file is a directory.
+ Directories are used for temporary files shared by parallel processes.
+ If <parameter>tablespace</parameter> is not provided, the
+ <literal>pg_default</literal> tablespace is examined.
+ Filenames beginning with a dot and special file types are excluded.
</para>
<para>
This function is restricted to superusers and members of
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 7351615f8a..da7d4a7c11 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -517,7 +517,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS)
}
/*
- * Generic function to return a directory listing of files.
+ * Generic function to return a directory listing of files (and optionally dirs).
*
* If the directory isn't there, silently return an empty set if MISSING_OK.
* Other unreadable-directory cases throw an error.
@@ -706,7 +706,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc)
TempTablespacePath(path, tblspc);
return pg_ls_dir_files(fcinfo, path,
- LS_DIR_HISTORIC | LS_DIR_MISSING_OK);
+ LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK);
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 8e88c549d9..a0148e6130 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10968,13 +10968,13 @@
{ oid => '5029', descr => 'list files in the pgsql_tmp directory',
proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
- proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' },
+ proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}',
+ proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
{ oid => '5030', descr => 'list files in the pgsql_tmp directory',
proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
- proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
- proargnames => '{tablespace,name,size,modification}',
+ proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}',
+ proargnames => '{tablespace,name,size,modification,isdir}',
prosrc => 'pg_ls_tmpdir_1arg' },
{ oid => '9979', descr => 'list directory with metadata',
proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index fd7c3c791f..7279c83d89 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -228,8 +228,8 @@ select * from pg_stat_file('.') limit 0;
-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
-- The name='' condition is never true, so the function runs to completion but returns zero rows.
select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | modification
-------+------+--------------
+ name | size | modification | isdir
+------+------+--------------+-------
(0 rows)
select filename, isdir from pg_ls_dir_metadata('.') where filename='.';
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index a42714bf40..1c88e914e3 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
-- The name='' condition is never true, so the function runs to completion but returns zero rows.
-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | modification
-------+------+--------------
+ name | size | modification | isdir
+------+------+--------------+-------
(0 rows)
-- try setting and resetting some properties for the new tablespace
--
2.17.0
--SBikYMzjhZGK9d4p
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v25-0006-pg_ls_-dir-to-show-directories-and-isdir-column.patch"
^ permalink raw reply [nested|flat] 6+ messages in thread
* pgbench error: (setshell) of script 0; execution of meta-command failed
@ 2025-01-10 02:45 Andy Fan <[email protected]>
2025-01-10 07:09 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Andy Fan @ 2025-01-10 02:45 UTC (permalink / raw)
To: pgsql-hackers
Hi:
I run into the {subject} issue with the below setup.
cat foo.sql
\setshell txn_mode echo ${TXN_MODE}
\setshell speed echo ${SPEED}
\setshell sleep_ms echo ${SLEEP_MS}
\setshell subtxn_mode echo ${SUBTXN_MODE}
select 1;
$ TXN_MODE=-1 SPEED=1 SLEEP_MS=0 SUBTXN_MODE=-1 pgbench -n -ffoo.sql postgres -T5 -c4 --exit-on-abort
I *randomly*(7/8) get errors like:
pgbench (18devel)
pgbench: error: client 2 aborted in command 0 (setshell) of script 0; execution of meta-command failed
pgbench: error: Run was aborted due to an error in thread 0
I debug this for 1+ hour and didn't find anything useful, so I'd like
have a ask if there is any known issue or the way I use \setshell is
wrong?
Thanks
--
Best Regards
Andy Fan
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbench error: (setshell) of script 0; execution of meta-command failed
2025-01-10 02:45 pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
@ 2025-01-10 07:09 ` Andy Fan <[email protected]>
2025-01-10 12:41 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Andy Fan @ 2025-01-10 07:09 UTC (permalink / raw)
To: pgsql-hackers
Andy Fan <[email protected]> writes:
> Hi:
>
> I run into the {subject} issue with the below setup.
>
> cat foo.sql
>
> \setshell txn_mode echo ${TXN_MODE}
> \setshell speed echo ${SPEED}
> \setshell sleep_ms echo ${SLEEP_MS}
> \setshell subtxn_mode echo ${SUBTXN_MODE}
>
> select 1;
>
> $ TXN_MODE=-1 SPEED=1 SLEEP_MS=0 SUBTXN_MODE=-1 pgbench -n -ffoo.sql postgres -T5 -c4 --exit-on-abort
>
> I *randomly*(7/8) get errors like:
>
> pgbench (18devel)
> pgbench: error: client 2 aborted in command 0 (setshell) of script 0; execution of meta-command failed
> pgbench: error: Run was aborted due to an error in thread 0
I think I have figured out the issue, if you want reproduce it quicker,
you can change the '-T5' to '-T1' in the pgbench command and run many times.
Here is the patch to fix it, would someone take a look at?
pgbench: Avoid misleading error for \[set]shell when timer_exceeded.
fgets in executeMetaCommand may return NULL if it receives a signal
during the shell command is executing. Before this commit, pgbench
client raises ERROR like below.
pgbench: error: client 3 aborted in command 3 (setshell) of script 0;
execution of meta-command failed
This behavior is misleading since people may think something is
wrong. In this commit, we just ignore fgets return NULL when
timer_exceeded.
--
Best Regards
Andy Fan
Attachments:
[text/x-diff] 0001-pgbench-Avoid-misleading-error-for-set-shell-when-ti.patch (1.9K, ../../[email protected]/2-0001-pgbench-Avoid-misleading-error-for-set-shell-when-ti.patch)
download | inline diff:
From 9b03ff66e45d019631ed9fd3322c0911ce14a8a7 Mon Sep 17 00:00:00 2001
From: Andy Fan <[email protected]>
Date: Fri, 10 Jan 2025 06:56:11 +0000
Subject: [PATCH 1/1] pgbench: Avoid misleading error for \[set]shell when
timer_exceeded.
fgets in executeMetaCommand may return NULL if it receives a signal
during the shell command is executing. Before this commit, pgbench
client raises ERROR like below.
pgbench: error: client 3 aborted in command 3 (setshell) of script 0;
execution of meta-command failed
This behavior is misleading since people may think something is
wrong. In this commit, we just ignore fgets return NULL when
timer_exceeded.
---
src/bin/pgbench/pgbench.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index c415e0f32c..7f61cd3aea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -3851,7 +3851,9 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
Assert(st->state == CSTATE_WAIT_RESULT ||
st->state == CSTATE_END_COMMAND ||
st->state == CSTATE_SLEEP ||
- st->state == CSTATE_ABORTED);
+ st->state == CSTATE_ABORTED ||
+ st->state == CSTATE_FINISHED);
+
break;
/*
@@ -4414,6 +4416,8 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
{
if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
+ if (timer_exceeded)
+ return CSTATE_FINISHED;
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
}
@@ -4422,6 +4426,8 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
{
if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
+ if (timer_exceeded)
+ return CSTATE_FINISHED;
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
}
--
2.45.1
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbench error: (setshell) of script 0; execution of meta-command failed
2025-01-10 02:45 pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 07:09 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
@ 2025-01-10 12:41 ` Fujii Masao <[email protected]>
2025-01-10 14:39 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Fujii Masao @ 2025-01-10 12:41 UTC (permalink / raw)
To: Andy Fan <[email protected]>; pgsql-hackers
On 2025/01/10 16:09, Andy Fan wrote:
> Andy Fan <[email protected]> writes:
>
>> Hi:
>>
>> I run into the {subject} issue with the below setup.
>>
>> cat foo.sql
>>
>> \setshell txn_mode echo ${TXN_MODE}
>> \setshell speed echo ${SPEED}
>> \setshell sleep_ms echo ${SLEEP_MS}
>> \setshell subtxn_mode echo ${SUBTXN_MODE}
>>
>> select 1;
>>
>> $ TXN_MODE=-1 SPEED=1 SLEEP_MS=0 SUBTXN_MODE=-1 pgbench -n -ffoo.sql postgres -T5 -c4 --exit-on-abort
>>
>> I *randomly*(7/8) get errors like:
>>
>> pgbench (18devel)
>> pgbench: error: client 2 aborted in command 0 (setshell) of script 0; execution of meta-command failed
>> pgbench: error: Run was aborted due to an error in thread 0
Interestingly, my git bisect pointed to the following commit
as the cause of this issue, even though it seems unrelated to
the pgbench problem at all. It’s possible my git bisect result
is incorrect, but when I reverted this commit on HEAD,
the pgbench issue didn’t occur during my tests.
----------------------
06843df4abc5a0c24e4bd154a8a1327e074fa3ae is the first bad commit
commit 06843df4abc5a0c24e4bd154a8a1327e074fa3ae
Author: Tom Lane <[email protected]>
Date: Fri Sep 29 14:07:30 2023 -0400
Suppress macOS warnings about duplicate libraries in link commands.
----------------------
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbench error: (setshell) of script 0; execution of meta-command failed
2025-01-10 02:45 pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 07:09 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 12:41 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
@ 2025-01-10 14:39 ` Fujii Masao <[email protected]>
2025-01-10 14:57 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Tom Lane <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Fujii Masao @ 2025-01-10 14:39 UTC (permalink / raw)
To: Andy Fan <[email protected]>; pgsql-hackers
On 2025/01/10 21:41, Fujii Masao wrote:
>
>
> On 2025/01/10 16:09, Andy Fan wrote:
>> Andy Fan <[email protected]> writes:
>>
>>> Hi:
>>>
>>> I run into the {subject} issue with the below setup.
>>>
>>> cat foo.sql
>>>
>>> \setshell txn_mode echo ${TXN_MODE}
>>> \setshell speed echo ${SPEED}
>>> \setshell sleep_ms echo ${SLEEP_MS}
>>> \setshell subtxn_mode echo ${SUBTXN_MODE}
>>>
>>> select 1;
>>>
>>> $ TXN_MODE=-1 SPEED=1 SLEEP_MS=0 SUBTXN_MODE=-1 pgbench -n -ffoo.sql postgres -T5 -c4 --exit-on-abort
>>>
>>> I *randomly*(7/8) get errors like:
>>>
>>> pgbench (18devel)
>>> pgbench: error: client 2 aborted in command 0 (setshell) of script 0; execution of meta-command failed
>>> pgbench: error: Run was aborted due to an error in thread 0
>
> Interestingly, my git bisect pointed to the following commit
> as the cause of this issue, even though it seems unrelated to
> the pgbench problem at all. It’s possible my git bisect result
> is incorrect, but when I reverted this commit on HEAD,
> the pgbench issue didn’t occur during my tests.
>
> ----------------------
> 06843df4abc5a0c24e4bd154a8a1327e074fa3ae is the first bad commit
> commit 06843df4abc5a0c24e4bd154a8a1327e074fa3ae
> Author: Tom Lane <[email protected]>
> Date: Fri Sep 29 14:07:30 2023 -0400
>
> Suppress macOS warnings about duplicate libraries in link commands.
> ----------------------
Before this commit, pgbench used pqsignal() from port/pqsignal.c
to set the signal handler for SIGALRM. This version of pqsignal()
sets SA_RESTART for frontend code, so fgets() in runShellCommand()
wouldn't return NULL even if SIGALRM arrived during fgets(),
preventing the reported error.
On the other hand, currently, pgbench seems to use pqsignal()
from legacy-pqsignal.c, which doesn't set SA_RESTART for SIGALRM.
As a result, SIGALRM can interrupt fgets() in runShellCommand()
and make it return NULL, leading to the reported error.
I'm not sure if this change was an intentional result of that commit...
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgbench error: (setshell) of script 0; execution of meta-command failed
2025-01-10 02:45 pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 07:09 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 12:41 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
2025-01-10 14:39 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
@ 2025-01-10 14:57 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Tom Lane @ 2025-01-10 14:57 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Andy Fan <[email protected]>; pgsql-hackers
Fujii Masao <[email protected]> writes:
> Before this commit, pgbench used pqsignal() from port/pqsignal.c
> to set the signal handler for SIGALRM. This version of pqsignal()
> sets SA_RESTART for frontend code, so fgets() in runShellCommand()
> wouldn't return NULL even if SIGALRM arrived during fgets(),
> preventing the reported error.
> On the other hand, currently, pgbench seems to use pqsignal()
> from legacy-pqsignal.c, which doesn't set SA_RESTART for SIGALRM.
> As a result, SIGALRM can interrupt fgets() in runShellCommand()
> and make it return NULL, leading to the reported error.
Oh, interesting.
> I'm not sure if this change was an intentional result of that commit...
Definitely not. I think we'd better look into how to undo that
effect.
Since legacy-pqsignal is really not supposed to be used by clients
anymore, maybe we could just adjust it to set SA_RESTART for SIGALRM?
The other alternatives I can think of amount to re-introducing
link order dependencies, which would be horrid.
regards, tom lane
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2025-01-10 14:57 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-03-09 03:57 [PATCH v25 05/11] pg_ls_tmpdir to show directories and "isdir" argument.. Justin Pryzby <[email protected]>
2025-01-10 02:45 pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 07:09 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Andy Fan <[email protected]>
2025-01-10 12:41 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
2025-01-10 14:39 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Fujii Masao <[email protected]>
2025-01-10 14:57 ` Re: pgbench error: (setshell) of script 0; execution of meta-command failed Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox