public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v16 02/10] pg_stat_file and pg_ls_dir_* to use lstat()..
6+ messages / 4 participants
[nested] [flat]
* [PATCH v16 02/10] pg_stat_file and pg_ls_dir_* to use lstat()..
@ 2020-03-30 23:59 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Justin Pryzby @ 2020-03-30 23:59 UTC (permalink / raw)
pg_ls_dir_* will now skip (no longer show) symbolic links, same as other
non-regular file types, as we advertize we do since 8b6d94cf6. That seems to
be the intented behavior, since irregular file types are 1) less portable; and,
2) we don't currently show a file's type except for "bool is_dir".
pg_stat_file will now 1) show metadata of links themselves, rather than their
target; and, 2) specifically, show links to directories with "is_dir=false";
and, 3) not error on broken symlinks.
---
doc/src/sgml/func.sgml | 2 +-
src/backend/utils/adt/genfile.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9b885102da..96b08d0500 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25486,7 +25486,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
size, last accessed time stamp, last modified time stamp,
last file status change time stamp (Unix platforms only),
file creation time stamp (Windows only), and a <type>boolean</type>
- indicating if it is a directory (or a symbolic link to a directory).
+ indicating if it is a directory.
Typical usages include:
<programlisting>
SELECT * FROM pg_stat_file('filename');
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index ceaa6180da..219ac160f8 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -370,7 +370,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
filename = convert_and_check_filename(filename_t);
- if (stat(filename, &fst) < 0)
+ if (lstat(filename, &fst) < 0)
{
if (missing_ok && errno == ENOENT)
PG_RETURN_NULL();
@@ -596,7 +596,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok)
/* Get the file info */
snprintf(path, sizeof(path), "%s/%s", dir, de->d_name);
- if (stat(path, &attrib) < 0)
+ if (lstat(path, &attrib) < 0)
{
/* Ignore concurrently-deleted files, else complain */
if (errno == ENOENT)
--
2.17.0
--2FkSFaIQeDFoAt0B
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v16-0003-Add-tests-on-pg_ls_dir-before-changing-it.patch"
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Merging statistics from children instead of re-sampling everything
@ 2022-02-10 11:50 Andrey Lepikhov <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Andrey Lepikhov @ 2022-02-10 11:50 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
On 21/1/2022 01:25, Tomas Vondra wrote:
> But I don't have a very good idea what to do about statistics that we
> can't really merge. For some types of statistics it's rather tricky to
> reasonably merge the results - ndistinct is a simple example, although
> we could work around that by building and merging hyperloglog counters.
I think, as a first step on this way we can reduce a number of pulled
tuples. We don't really needed to pull all tuples from a remote server.
To construct a reservoir, we can pull only a tuple sample. Reservoir
method needs only a few arguments to return a sample like you read
tuples locally. Also, to get such parts of samples asynchronously, we
can get size of each partition on a preliminary step of analysis.
In my opinion, even this solution can reduce heaviness of a problem
drastically.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Merging statistics from children instead of re-sampling everything
@ 2022-02-10 22:37 Tomas Vondra <[email protected]>
parent: Andrey Lepikhov <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Tomas Vondra @ 2022-02-10 22:37 UTC (permalink / raw)
To: Andrey Lepikhov <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2/10/22 12:50, Andrey Lepikhov wrote:
> On 21/1/2022 01:25, Tomas Vondra wrote:
>> But I don't have a very good idea what to do about statistics that we
>> can't really merge. For some types of statistics it's rather tricky to
>> reasonably merge the results - ndistinct is a simple example, although
>> we could work around that by building and merging hyperloglog counters.
>
> I think, as a first step on this way we can reduce a number of pulled
> tuples. We don't really needed to pull all tuples from a remote server.
> To construct a reservoir, we can pull only a tuple sample. Reservoir
> method needs only a few arguments to return a sample like you read
> tuples locally. Also, to get such parts of samples asynchronously, we
> can get size of each partition on a preliminary step of analysis.
> In my opinion, even this solution can reduce heaviness of a problem
> drastically.
>
Oh, wow! I haven't realized we're fetching all the rows from foreign
(postgres_fdw) partitions. For local partitions we already do that,
because that uses the usual acquire function, with a reservoir
proportional to partition size. I have assumed we use tablesample to
fetch just a small fraction of rows from FDW partitions, and I agree
doing that would be a pretty huge benefit.
I actually tried hacking that together - there's a couple problems with
that (e.g. determining what fraction to sample using bernoulli/system),
but in principle it seems quite doable. Some minor changes to the FDW
API may be necessary, not sure.
Not sure about the async execution - that seems way more complicated,
and the sampling reduces the total cost, async just parallelizes it.
That being said, this thread was not really about foreign partitions,
but about re-analyzing inheritance trees in general. And sampling
foreign partitions doesn't really solve that - we'll still do the
sampling over and over.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Merging statistics from children instead of re-sampling everything
@ 2022-02-11 04:29 Andrey V. Lepikhov <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Andrey V. Lepikhov @ 2022-02-11 04:29 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: [email protected]
On 2/11/22 03:37, Tomas Vondra wrote:
> That being said, this thread was not really about foreign partitions,
> but about re-analyzing inheritance trees in general. And sampling
> foreign partitions doesn't really solve that - we'll still do the
> sampling over and over.
IMO, to solve the problem we should do two things:
1. Avoid repeatable partition scans in the case inheritance tree.
2. Avoid to re-analyze everything in the case of active changes in small
subset of partitions.
For (1) i can imagine a solution like multiplexing: on the stage of
defining which relations to scan, group them and prepare parameters of
scanning to make multiple samples in one shot.
It looks like we need a separate logic for analysis of partitioned
tables - we should form and cache samples on each partition before an
analysis.
It requires a prototype to understand complexity of such solution and
can be done separately from (2).
Task (2) is more difficult to solve. Here we can store samples from each
partition in values[] field of pg_statistic or in specific table which
stores a 'most probable values' snapshot of each table.
Most difficult problem here, as you mentioned, is ndistinct value. Is it
possible to store not exactly calculated value of ndistinct, but an
'expected value', based on analysis of samples and histograms on
partitions? Such value can solve also a problem of estimation of a SETOP
result grouping (joining of them, etc), where we have statistics only on
sources of the union.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Merging statistics from children instead of re-sampling everything
@ 2022-02-11 15:12 Tomas Vondra <[email protected]>
parent: Andrey V. Lepikhov <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Tomas Vondra @ 2022-02-11 15:12 UTC (permalink / raw)
To: Andrey V. Lepikhov <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: [email protected]
On 2/11/22 05:29, Andrey V. Lepikhov wrote:
> On 2/11/22 03:37, Tomas Vondra wrote:
>> That being said, this thread was not really about foreign partitions,
>> but about re-analyzing inheritance trees in general. And sampling
>> foreign partitions doesn't really solve that - we'll still do the
>> sampling over and over.
> IMO, to solve the problem we should do two things:
> 1. Avoid repeatable partition scans in the case inheritance tree.
> 2. Avoid to re-analyze everything in the case of active changes in small
> subset of partitions.
>
> For (1) i can imagine a solution like multiplexing: on the stage of
> defining which relations to scan, group them and prepare parameters of
> scanning to make multiple samples in one shot.
>> It looks like we need a separate logic for analysis of partitioned
> tables - we should form and cache samples on each partition before an
> analysis.
> It requires a prototype to understand complexity of such solution and
> can be done separately from (2).
>
I'm not sure I understand what you mean by multiplexing. The term
usually means "sending multiple signals at once" but I'm not sure how
that applies to this issue. Can you elaborate?
I assume you mean something like collecting a sample for a partition
once, and then keeping and reusing the sample for future ANALYZE runs,
until invalidated in some sense.
Yeah, I agree that'd be useful - and not just for partitions, actually.
I've been pondering something like that for regular tables, because the
sample might be used for estimation of clauses directly.
But it requires storing the sample somewhere, and I haven't found a good
and simple way to do that. We could serialize that into bytea, or we
could create a new fork, or something, but what should that do with
oversized attributes (how would TOAST work for a fork) and/or large
samples (which might not fit into 1GB bytea)?
> Task (2) is more difficult to solve. Here we can store samples from each
> partition in values[] field of pg_statistic or in specific table which
> stores a 'most probable values' snapshot of each table.
I think storing samples in pg_statistic is problematic, because values[]
is subject to 1GB limit etc. Not an issue for small MCV list of a single
attribute, certainly an issue for larger samples. Even if the data fit,
the size of pg_statistic would explode.
> Most difficult problem here, as you mentioned, is ndistinct value. Is it
> possible to store not exactly calculated value of ndistinct, but an
> 'expected value', based on analysis of samples and histograms on
> partitions? Such value can solve also a problem of estimation of a SETOP
> result grouping (joining of them, etc), where we have statistics only on
> sources of the union.
>
I think ndistinct is problem only when merging final estimates. But if
we're able to calculate and store some intermediate results, we can
easily use HLL and merge that.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Merging statistics from children instead of re-sampling everything
@ 2022-02-14 10:22 Andrey V. Lepikhov <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Andrey V. Lepikhov @ 2022-02-14 10:22 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: [email protected]
On 2/11/22 20:12, Tomas Vondra wrote:
>
>
> On 2/11/22 05:29, Andrey V. Lepikhov wrote:
>> On 2/11/22 03:37, Tomas Vondra wrote:
>>> That being said, this thread was not really about foreign partitions,
>>> but about re-analyzing inheritance trees in general. And sampling
>>> foreign partitions doesn't really solve that - we'll still do the
>>> sampling over and over.
>> IMO, to solve the problem we should do two things:
>> 1. Avoid repeatable partition scans in the case inheritance tree.
>> 2. Avoid to re-analyze everything in the case of active changes in
>> small subset of partitions.
>>
>> For (1) i can imagine a solution like multiplexing: on the stage of
>> defining which relations to scan, group them and prepare parameters of
>> scanning to make multiple samples in one shot.
> I'm not sure I understand what you mean by multiplexing. The term
> usually means "sending multiple signals at once" but I'm not sure how
> that applies to this issue. Can you elaborate?
I suppose to make a set of samples in one scan: one sample for plane
table, another - for a parent and so on, according to the inheritance
tree. And cache these samples in memory. We can calculate all parameters
of reservoir method to do it.
> sample might be used for estimation of clauses directly.
You mean, to use them in difficult cases, such of estimation of grouping
over APPEND ?
>
> But it requires storing the sample somewhere, and I haven't found a good
> and simple way to do that. We could serialize that into bytea, or we
> could create a new fork, or something, but what should that do with
> oversized attributes (how would TOAST work for a fork) and/or large
> samples (which might not fit into 1GB bytea)?
This feature looks like meta-info over a database. It can be stored in
separate relation. It is not obvious that we need to use it for each
relation, for example, with large samples. I think, it can be controlled
by a table parameter.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2022-02-14 10:22 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-03-30 23:59 [PATCH v16 02/10] pg_stat_file and pg_ls_dir_* to use lstat().. Justin Pryzby <[email protected]>
2022-02-10 11:50 Re: Merging statistics from children instead of re-sampling everything Andrey Lepikhov <[email protected]>
2022-02-10 22:37 ` Re: Merging statistics from children instead of re-sampling everything Tomas Vondra <[email protected]>
2022-02-11 04:29 ` Re: Merging statistics from children instead of re-sampling everything Andrey V. Lepikhov <[email protected]>
2022-02-11 15:12 ` Re: Merging statistics from children instead of re-sampling everything Tomas Vondra <[email protected]>
2022-02-14 10:22 ` Re: Merging statistics from children instead of re-sampling everything Andrey V. Lepikhov <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox