public inbox for [email protected]help / color / mirror / Atom feed
Re: ANALYZE on partitioned tables vs on individual partitions 3+ messages / 2 participants [nested] [flat]
* Re: ANALYZE on partitioned tables vs on individual partitions @ 2024-08-07 07:20 Michael Harris <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Michael Harris @ 2024-08-07 07:20 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: [email protected]; Christophe Pettus <[email protected]> Many thanks David for the comprehensive response. > I think the complaint was about no autovacuum on the partitioned > table, not the partitions. Yes, exactly. One other piece of information: these tables contain a lot of columns, of which only 4 are normally used for WHERE clauses or joins. The table I was experimenting with has 150 columns, 156026832 rows and occupies 166GB. I found that running an ANALYZE specifying only those 4 columns only took 5 minutes, compared to the 30 minutes for the whole table. That was a bit of a surprise as I imagined actually reading the table would take most of the time and would be the same regardless of the number of columns being analyzed, but I guess that is wrong. Regards, Mike On Wed, 7 Aug 2024 at 15:23, David Rowley <[email protected]> wrote: > > On Wed, 7 Aug 2024 at 16:44, Christophe Pettus <[email protected]> wrote: > > Child partitions should be autovacuumed and autoanalyzed just like any other table; they are not prohibited from autovacuum in any way by default. It's probably a good idea to investigate why they are not being picked up by autovacuum. If they are created by a bulk load process, it's not a bad idea to do a VACUUM ANALYZE on them once the bulk load is complete. > > I think the complaint was about no autovacuum on the partitioned > table, not the partitions. This is expected as we don't track the > counters (in particular n_mod_since_analyze) shown in > pg_stat_all_tables at the partitioned table level, so the trigger > points that normally cause autovacuum to analyze or vacuum a table > just won't be triggered for a partitioned table. For VACUUM, that's > fine as, as you mentioned, no rows are stored. But for analyze, that > does present a problem. > > To name the aspects of planning that rely on statistics of the > partitioned table, basically anything above the Append or MergeAppend > which joins the partitioned results together. So that doesn't include > the scans of each partition and any quals that are pushed down to the > scan level as those are able to use the partition level statistics. > However, it does include things like joins, group by, distinct as > those require n_distinct estimates for the partitioned table. It's not > all bad though as the row estimates for each individual partition will > be totalled up through the Append / MergeAppend simply by adding up > the row estimates for each Append / MergeAppend child plan. So, it's > really only an estimation problem for any node that comes after a join > node or a group by node as the output rows for those nodes will depend > on a good n_distinct estimate for the partitioned table. > > Partition-wise joins and aggregates do change things a bit as those > features do permit moving those operations below the Append / Merge > Append, in which case the statistics for the individual partition can > be used. > > You could consider manually setting the n_distinct_inherited estimates > for the columns that you join on or group by in the partitioned table. > You might find that you're able to choose a suitable value for that if > you review the documentation for that setting. In particular, please > review what is mentioned about using negative numbers for that > setting. You may be able to choose a value that scales correctly with > the row estimate that doesn't get outdated as you add more rows to the > partitions. You'll need to determine that based on the data you're > storing. > > David ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: ANALYZE on partitioned tables vs on individual partitions @ 2024-08-07 08:09 David Rowley <[email protected]> parent: Michael Harris <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: David Rowley @ 2024-08-07 08:09 UTC (permalink / raw) To: Michael Harris <[email protected]>; +Cc: [email protected]; Christophe Pettus <[email protected]> On Wed, 7 Aug 2024 at 19:20, Michael Harris <[email protected]> wrote: > I found that running an ANALYZE specifying only those 4 columns only took > 5 minutes, compared to the 30 minutes for the whole table. > > That was a bit of a surprise as I imagined actually reading the table would take > most of the time and would be the same regardless of the number of columns > being analyzed, but I guess that is wrong. ANALYZE does do sampling of the data in the table. It would only read all of the rows for fairly small tables. The docs in [1] mention this: "For large tables, ANALYZE takes a random sample of the table contents, rather than examining every row. This allows even very large tables to be analyzed in a small amount of time." I think the reason it's taking so long is not because of it performing ANALYZE on the partitioned table which results in gathering statistics for the partitioned table which means proportionately (based on the size of the partition) sampling rows from each partition, it's more likely due to the fact that each partition is also analysed and the statistics for each of those is updated. There is no "ANALYZE ONLY" command similar to "FROM ONLY" in SELECT queries. You could probably do some sampling of the pg_stat_progress_analyze view to figure out what's taking the most time. If you find that the majority of the time is spent analysing the partitions and not the partitioned table then maybe we should expand ANALYZE to add the ONLY option... David [1] https://www.postgresql.org/docs/current/sql-analyze.html [2] https://www.postgresql.org/docs/current/progress-reporting.html ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: ANALYZE on partitioned tables vs on individual partitions @ 2024-08-08 01:02 Michael Harris <[email protected]> parent: David Rowley <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Michael Harris @ 2024-08-08 01:02 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: [email protected] > You could probably do some sampling of the pg_stat_progress_analyze > view to figure out what's taking the most time. I did another run, sampling the pg_stat_progress_analyze every 30s. For the first 4 minutes it was working on the partitioned table. After that it began analyzing all the partitions, which took approx 29 minutes. I think you are correct - an ONLY option for ANALYZE would be a huge benefit. In my use case, the autovacuum processes are keeping the partitions analyzed so there would seem to be little benefit to including them in the manual table level ANALYZE. Regards Mike On Wed, 7 Aug 2024 at 18:09, David Rowley <[email protected]> wrote: > > On Wed, 7 Aug 2024 at 19:20, Michael Harris <[email protected]> wrote: > > I found that running an ANALYZE specifying only those 4 columns only took > > 5 minutes, compared to the 30 minutes for the whole table. > > > > That was a bit of a surprise as I imagined actually reading the table would take > > most of the time and would be the same regardless of the number of columns > > being analyzed, but I guess that is wrong. > > ANALYZE does do sampling of the data in the table. It would only read > all of the rows for fairly small tables. The docs in [1] mention > this: > > "For large tables, ANALYZE takes a random sample of the table > contents, rather than examining every row. This allows even very large > tables to be analyzed in a small amount of time." > > I think the reason it's taking so long is not because of it performing > ANALYZE on the partitioned table which results in gathering statistics > for the partitioned table which means proportionately (based on the > size of the partition) sampling rows from each partition, it's more > likely due to the fact that each partition is also analysed and the > statistics for each of those is updated. There is no "ANALYZE ONLY" > command similar to "FROM ONLY" in SELECT queries. > > You could probably do some sampling of the pg_stat_progress_analyze > view to figure out what's taking the most time. If you find that the > majority of the time is spent analysing the partitions and not the > partitioned table then maybe we should expand ANALYZE to add the ONLY > option... > > David > > [1] https://www.postgresql.org/docs/current/sql-analyze.html > [2] https://www.postgresql.org/docs/current/progress-reporting.html ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2024-08-08 01:02 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-08-07 07:20 Re: ANALYZE on partitioned tables vs on individual partitions Michael Harris <[email protected]> 2024-08-07 08:09 ` David Rowley <[email protected]> 2024-08-08 01:02 ` Michael Harris <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox