pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: pg_dump: Retrieve attribute statistics in batches.
2+ messages / 2 participants
[nested] [flat]
* pgsql: pg_dump: Retrieve attribute statistics in batches.
@ 2025-04-04 19:51 Nathan Bossart <nathan@postgresql.org>
0 siblings, 1 reply; 2+ messages in thread
From: Nathan Bossart @ 2025-04-04 19:51 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
pg_dump: Retrieve attribute statistics in batches.
Currently, pg_dump gathers attribute statistics with a query per
relation, which can cause pg_dump to take significantly longer,
especially when there are many relations. This commit addresses
this by teaching pg_dump to gather attribute statistics for 64
relations at a time. Some simple tests showed this was the optimal
batch size, but performance may vary depending on the workload.
Our lookahead code determines the next batch of relations by
searching the TOC sequentially for relevant entries. This approach
assumes that we will dump all such entries in TOC order, which
unfortunately isn't true for dump formats that use
RestoreArchive(). RestoreArchive() does multiple passes through
the TOC and selectively dumps certain groups of entries each time.
This is particularly problematic for index stats and a subset of
matview stats; both are in SECTION_POST_DATA, but matview stats
that depend on matview data are dumped in RESTORE_PASS_POST_ACL,
while all other stats are dumped in RESTORE_PASS_MAIN. To handle
this, this commit moves all statistics data entries in
SECTION_POST_DATA to RESTORE_PASS_POST_ACL, which ensures that we
always dump them in TOC order. A convenient side effect of this
change is that we can revert a decent chunk of commit a0a4601765,
but that is left for a follow-up commit.
Author: Corey Huinker <corey.huinker@gmail.com>
Co-authored-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Jeff Davis <pgsql@j-davis.com>
Discussion: https://postgr.es/m/CADkLM%3Dc%2Br05srPy9w%2B-%2BnbmLEo15dKXYQ03Q_xyK%2BriJerigLQ%40mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/9c02e3a986daa865ecdc2e3d8183e2d83b8f4824
Modified Files
--------------
src/bin/pg_dump/pg_backup.h | 5 +-
src/bin/pg_dump/pg_backup_archiver.c | 29 +++----
src/bin/pg_dump/pg_dump.c | 147 ++++++++++++++++++++++++++++++-----
3 files changed, 142 insertions(+), 39 deletions(-)
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: pgsql: pg_dump: Retrieve attribute statistics in batches.
@ 2025-04-04 20:07 Nathan Bossart <nathandbossart@gmail.com>
parent: Nathan Bossart <nathan@postgresql.org>
0 siblings, 0 replies; 2+ messages in thread
From: Nathan Bossart @ 2025-04-04 20:07 UTC (permalink / raw)
To: Nathan Bossart <nathan@postgresql.org>; +Cc: pgsql-committers@lists.postgresql.org
On Fri, Apr 04, 2025 at 07:51:56PM +0000, Nathan Bossart wrote:
> pg_dump: Retrieve attribute statistics in batches.
>
> [...]
>
> Discussion: https://postgr.es/m/CADkLM%3Dc%2Br05srPy9w%2B-%2BnbmLEo15dKXYQ03Q_xyK%2BriJerigLQ%40mail.gmail.com
I see the buildfarm failure and am working on a fix. I will post further
updates in the original thread.
--
nathan
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2025-04-04 20:07 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-04-04 19:51 pgsql: pg_dump: Retrieve attribute statistics in batches. Nathan Bossart <nathan@postgresql.org>
2025-04-04 20:07 ` Nathan Bossart <nathandbossart@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox