public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v16 07/10] Add pg_ls_dir_recurse to show dir recursively.. 5+ messages / 3 participants [nested] [flat]
* [PATCH v16 07/10] Add pg_ls_dir_recurse to show dir recursively.. @ 2020-03-09 03:52 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Justin Pryzby @ 2020-03-09 03:52 UTC (permalink / raw) ..possibly there's a better place to put this, like maybe a doc-only example ? Need catversion bumped ? --- doc/src/sgml/func.sgml | 27 ++++++++++++++++++++ src/backend/catalog/system_views.sql | 1 + src/include/catalog/pg_proc.dat | 6 +++++ src/test/regress/expected/misc_functions.out | 14 ++++++++++ src/test/regress/sql/misc_functions.sql | 6 +++++ 5 files changed, 54 insertions(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index c3a6b3277f..3a2ad3eb50 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -25311,6 +25311,18 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); EXECUTE to run the function. </entry> </row> + <row> + <entry> + <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal> + </entry> + <entry><type>setof text</type></entry> + <entry> + Call pg_ls_dir_metadata to recursively list the files in the specified + directory, along with each file's metadata. + Restricted to superusers by default, but other users can be granted + EXECUTE to run the function. + </entry> + </row> <row> <entry> <literal><function>pg_ls_logdir()</function></literal> @@ -25422,6 +25434,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); directory along with the file's metadata. </para> + <indexterm> + <primary>pg_ls_dir_recurse</primary> + </indexterm> + <para> + <function>pg_ls_dir_recurse</function> recursively lists the files + in the specified directory. + To recursively list temporary directories in all tablespaces: +<programlisting> +SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir +FROM pg_tablespace b, pg_control_system() pcs, +LATERAL format('/PG_%s_%s', left(current_setting('server_version_num'), 2), pcs.catalog_version_no) AS suffix) AS dir, +LATERAL pg_ls_dir_recurse(dir) AS a; +</programlisting> + </para> + <indexterm> <primary>pg_ls_logdir</primary> </indexterm> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 1c77430f0c..215fba3b7d 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1468,6 +1468,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public; +REVOKE EXECUTE ON FUNCTION pg_ls_dir_recurse(text) FROM public; -- -- We also set up some things as accessible to standard roles. diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index adfce45d1a..f94f403475 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10916,6 +10916,12 @@ proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', proargnames => '{dirname,name,size,modification,isdir}', prosrc => 'pg_ls_dir_metadata_1arg' }, +{ oid => '5034', descr => 'list all files in a directory recursively', + proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text', + proallargtypes => '{text,text,int8,timestamptz,bool}', + proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', + prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 64b1417fb8..4188d684f0 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -243,6 +243,20 @@ select * from pg_ls_dir_metadata('.') limit 0; ------+------+--------------+------- (0 rows) +-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix +SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; + name | isdir +-----------------------+------- + pg_wal | t + pg_wal/archive_status | t +(2 rows) + +-- Check that expected columns are present +SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; + name | size | modification | isdir +------+------+--------------+------- +(0 rows) + -- -- Test adding a support function to a subject function -- diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 372345720d..6041c4f3dc 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -76,6 +76,12 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- -- Check that expected columns are present select * from pg_ls_dir_metadata('.') limit 0; +-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix +SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; + +-- Check that expected columns are present +SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; + -- -- Test adding a support function to a subject function -- -- 2.17.0 --2FkSFaIQeDFoAt0B Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v16-0008-pg_ls_logdir-to-ignore-error-if-initial-top-dir-.patch" ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Missing update of all_hasnulls in BRIN opclasses @ 2023-04-24 21:20 Tomas Vondra <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Tomas Vondra @ 2023-04-24 21:20 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] On 4/24/23 17:36, Alvaro Herrera wrote: > On 2023-Apr-23, Tomas Vondra wrote: > >> here's an updated version of the patch, including a backport version. I >> ended up making the code yet a bit closer to master by introducing >> add_values_to_range(). The current pre-14 code has the loop adding data >> to the BRIN tuple in two places, but with the "fixed" logic handling >> NULLs and the empty_range flag the amount of duplicated code got too >> high, so this seem reasonable. > > In backbranches, the new field to BrinMemTuple needs to be at the end of > the struct, to avoid ABI breakage. > Good point. > There's a comment in add_values_to_range with a typo "If the range was had". > Also, "So we should not see empty range that was not modified" should > perhaps be "should not see an empty range". > OK, I'll check the wording of the comments. > (As for your FIXME comment in brin_memtuple_initialize, I think you're > correct: we definitely need to reset bt_placeholder. Otherwise, we risk > places that call it in a loop using a BrinMemTuple with one range with > the flag set, in a range where that doesn't hold. It might be > impossible for this to happen, given how narrow the conditions are on > which bt_placeholder is used; but it seems safer to reset it anyway.) > Yeah. But isn't that a separate preexisting issue, strictly speaking? > Some pgindent noise would be induced by this patch. I think it's worth > cleaning up ahead of time. > True. Will do. > I did a quick experiment of turning the patches over -- applying test > first, code fix after (requires some conflict fixing). With that I > verified that the behavior of 'hasnulls' indeed changes with the code > fix. > Thanks. Could you do some testing of the union_tuples stuff too? It's a bit tricky part - the behavior is timing sensitive, so testing it requires gdb breakpoints breakpoints or something like that. I think it's correct, but it'd be nice to check that. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Missing update of all_hasnulls in BRIN opclasses @ 2023-05-06 22:13 Tomas Vondra <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Tomas Vondra @ 2023-05-06 22:13 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] On 4/24/23 23:20, Tomas Vondra wrote: > On 4/24/23 17:36, Alvaro Herrera wrote: >> On 2023-Apr-23, Tomas Vondra wrote: >> >>> here's an updated version of the patch, including a backport version. I >>> ended up making the code yet a bit closer to master by introducing >>> add_values_to_range(). The current pre-14 code has the loop adding data >>> to the BRIN tuple in two places, but with the "fixed" logic handling >>> NULLs and the empty_range flag the amount of duplicated code got too >>> high, so this seem reasonable. >> >> In backbranches, the new field to BrinMemTuple needs to be at the end of >> the struct, to avoid ABI breakage. >> Unfortunately, this is not actually possible :-( The BrinMemTuple has a FLEXIBLE_ARRAY_MEMBER at the end, so we can't place anything after it. I think we have three options: a) some other approach? - I really can't see any, except maybe for going back to the previous approach (i.e. encoding the info using the existing BrinValues allnulls/hasnulls flags) b) encoding the info in existing BrinMemTuple flags - e.g. we could use bt_placeholder to store two bits, not just one. Seems a bit ugly. c) ignore the issue - AFAICS this would be an issue only for (external) code accessing BrinMemTuple structs, but I don't think we're aware of any out-of-core BRIN opclasses or anything like that ... regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Missing update of all_hasnulls in BRIN opclasses @ 2023-05-07 05:08 Julien Rouhaud <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Julien Rouhaud @ 2023-05-07 05:08 UTC (permalink / raw) To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Justin Pryzby <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] Hi, On Sun, May 07, 2023 at 12:13:07AM +0200, Tomas Vondra wrote: > > c) ignore the issue - AFAICS this would be an issue only for (external) > code accessing BrinMemTuple structs, but I don't think we're aware of > any out-of-core BRIN opclasses or anything like that ... FTR there's at least postgis that implments BRIN opclasses (for geometries and geographies), but there's nothing fancy in the implementation and it doesn't access BrinMemTuple struct. ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Missing update of all_hasnulls in BRIN opclasses @ 2023-05-07 12:50 Tomas Vondra <[email protected]> parent: Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Tomas Vondra @ 2023-05-07 12:50 UTC (permalink / raw) To: Julien Rouhaud <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Justin Pryzby <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] On 5/7/23 07:08, Julien Rouhaud wrote: > Hi, > > On Sun, May 07, 2023 at 12:13:07AM +0200, Tomas Vondra wrote: >> >> c) ignore the issue - AFAICS this would be an issue only for (external) >> code accessing BrinMemTuple structs, but I don't think we're aware of >> any out-of-core BRIN opclasses or anything like that ... > > FTR there's at least postgis that implments BRIN opclasses (for geometries and > geographies), but there's nothing fancy in the implementation and it doesn't > access BrinMemTuple struct. Right. I believe that should be fine, because opclasses don't access the tuple directly - instead we pass pointers to individual pieces. But maybe it'd be a good idea to test this. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2023-05-07 12:50 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-03-09 03:52 [PATCH v16 07/10] Add pg_ls_dir_recurse to show dir recursively.. Justin Pryzby <[email protected]> 2023-04-24 21:20 Re: Missing update of all_hasnulls in BRIN opclasses Tomas Vondra <[email protected]> 2023-05-06 22:13 ` Re: Missing update of all_hasnulls in BRIN opclasses Tomas Vondra <[email protected]> 2023-05-07 05:08 ` Re: Missing update of all_hasnulls in BRIN opclasses Julien Rouhaud <[email protected]> 2023-05-07 12:50 ` Re: Missing update of all_hasnulls in BRIN opclasses Tomas Vondra <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox