From: Stephen Frost <sfrost@snowman.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: pg_dump test instability
Date: Mon, 27 Aug 2018 11:59:44 -0400
Message-ID: <20180827155943.GP3326@tamriel.snowman.net> (raw)
In-Reply-To: <1665.1535381158@sss.pgh.pa.us>
References: <5137fe12-d0a2-4971-61b6-eb4e7e8875f8@2ndquadrant.com>
<32399.1535379095@sss.pgh.pa.us>
<20180827144138.GO3326@tamriel.snowman.net>
<1665.1535381158@sss.pgh.pa.us>
Greetings,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> >> However, at least for the directory-format case (which I think is the
> >> only one supported for parallel restore), we could make it compare the
> >> file sizes of the TABLE DATA items. That'd work pretty well as a proxy
> >> for both the amount of effort needed for table restore, and the amount
> >> of effort needed to build indexes on the tables afterwards.
>
> > Parallel restore also works w/ custom-format dumps.
>
> Really. Well then the existing code is even more broken, because it
> only does this sorting for directory output:
>
> /* If we do a parallel dump, we want the largest tables to go first */
> if (archiveFormat == archDirectory && numWorkers > 1)
> sortDataAndIndexObjectsBySize(dobjs, numObjs);
>
> so that parallel restore is completely left in the lurch with a
> custom-format dump.
Sorry for not being clear- it's only possible to parallel *dump* to a
directory-format dump, and the above code is for performing that
sort-by-size before executing a parallel dump. One might wonder why
there's the check for archiveFormat at all though- numWorkers shouldn't
be able to be >1 except in the case where the archiveFormat supports
parallel dump, and if it supports parallel dump, then we should try to
dump out the tables largest-first.
Parallel *restore* can be done from either a custom-format dump or from
a directory-format dump. I agree that we should seperate the concerns
and perform independent sorting on the restore side of things based on
the relative sizes of tables in the dump (be it custom format or
directory format). While compression might make us not exactly correct
on the restore side, I expect that we'll generally be close enough to
avoid most cases where a single worker gets stuck working on a large
table at the end after all the other work is done.
> But I imagine we can get some measure of table data size out of a custom
> dump too.
I would think so.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: sfrost@snowman.net, tgl@sss.pgh.pa.us, peter.eisentraut@2ndquadrant.com
Subject: Re: pg_dump test instability
In-Reply-To: <20180827155943.GP3326@tamriel.snowman.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox