Date: Mon, 27 Aug 2018 11:59:44 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: pg_dump test instability
Message-ID: <20180827155943.GP3326@tamriel.snowman.net>
References: <5137fe12-d0a2-4971-61b6-eb4e7e8875f8@2ndquadrant.com>
 <32399.1535379095@sss.pgh.pa.us>
 <20180827144138.GO3326@tamriel.snowman.net>
 <1665.1535381158@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="WpdwlRvXBOxKVUXx"
Content-Disposition: inline
In-Reply-To: <1665.1535381158@sss.pgh.pa.us>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--WpdwlRvXBOxKVUXx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> >> However, at least for the directory-format case (which I think is the
> >> only one supported for parallel restore), we could make it compare the
> >> file sizes of the TABLE DATA items.  That'd work pretty well as a proxy
> >> for both the amount of effort needed for table restore, and the amount
> >> of effort needed to build indexes on the tables afterwards.
>=20
> > Parallel restore also works w/ custom-format dumps.
>=20
> Really.  Well then the existing code is even more broken, because it
> only does this sorting for directory output:
>=20
> 	/* If we do a parallel dump, we want the largest tables to go first */
> 	if (archiveFormat =3D=3D archDirectory && numWorkers > 1)
> 		sortDataAndIndexObjectsBySize(dobjs, numObjs);
>=20
> so that parallel restore is completely left in the lurch with a
> custom-format dump.

Sorry for not being clear- it's only possible to parallel *dump* to a
directory-format dump, and the above code is for performing that
sort-by-size before executing a parallel dump.  One might wonder why
there's the check for archiveFormat at all though- numWorkers shouldn't
be able to be >1 except in the case where the archiveFormat supports
parallel dump, and if it supports parallel dump, then we should try to
dump out the tables largest-first.

Parallel *restore* can be done from either a custom-format dump or from
a directory-format dump.  I agree that we should seperate the concerns
and perform independent sorting on the restore side of things based on
the relative sizes of tables in the dump (be it custom format or
directory format).  While compression might make us not exactly correct
on the restore side, I expect that we'll generally be close enough to
avoid most cases where a single worker gets stuck working on a large
table at the end after all the other work is done.

> But I imagine we can get some measure of table data size out of a custom
> dump too.

I would think so.

Thanks!

Stephen

--WpdwlRvXBOxKVUXx
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJbhB/vAAoJEO1sijiDR2RVjLwP/3wvb21KdVqoTgP29IqqdzGK
pj00Ycl9XHwHMK7pYyCjusRLBh+SYDEvzJ2NugyS6rJ5EKRj59TLZsgICOXwWd24
6ab6Y34x7TFsT6Ody0/Oj/RYvbgrsRMrL5oItns/0Ze65I+xSfZI9wRICMBOWBkt
yxUDN/3ofE6RSMSNswVzKBHYQIJzjOPo/uqlG2BWxnnacv4kEg0HpOZX4syzDSgJ
cGNX57wSKquNpmzkHEJedKWwmTgrzPlYoSE4sAt0FpMuSPzlyD25F7w2xWSv0xa5
oLCWZSPVv/WPkRU1XORVwxHQ37G4hG0B5VCEirMTCg20x5nJA7u+xj28hqoj6zRe
6TrU2ewqtN5k480ruoTYKz9VyxGGJEu3C3trlvUao5npAwqbM0eKxRUNO60k9HRH
aO55NboXMoaxaXT3/GDGm1toF43Q6rUa5wTruuxfD2jmDMNTXpMBTvQa99shA4t7
8d5uYHRPhnIeJJHf+P7/ar/emeIQOet2rKzqFhhL0moQSY+lYz0b7kWU1lsC/7bj
LpWC2hPIQvjkUdlwtlcNhg1qGrNaTJtwfUTvmlDM8pmEqhd4I6E0Rf8Mjp5OJP85
wVdcxhzD4AxWFrckI8DUMO5mcbrtPuqAqznCDy6hfmVzvzNJXksVE6EVwo6kArP1
w7WNrVVvN2bhRQcmxbSX
=RDn9
-----END PGP SIGNATURE-----

--WpdwlRvXBOxKVUXx--