Date: Mon, 27 Aug 2018 10:41:38 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: pg_dump test instability
Message-ID: <20180827144138.GO3326@tamriel.snowman.net>
References: <5137fe12-d0a2-4971-61b6-eb4e7e8875f8@2ndquadrant.com>
 <32399.1535379095@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="HUY/6dRnBRT+g9H1"
Content-Disposition: inline
In-Reply-To: <32399.1535379095@sss.pgh.pa.us>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--HUY/6dRnBRT+g9H1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> > In a non-data_only dump, the order of the tables doesn't matter, because
> > the foreign keys are added at the very end.  In parallel dumps, the
> > tables are in addition sorted by size, so the resultant order is
> > different from a single-threaded dump.  This can be seen by comparing
> > the dumped TOCs of the defaults_dir_format and defaults_parallel cases.
> > But it all happens to pass the tests right now.
>=20
> I noticed that business about sorting the TOC by size yesterday.
> I think that's a completely bletcherous hack, and we ought to get
> rid of it in favor of keeping the TOC order the same between parallel
> and non-parallel cases, and instead doing size comparisons during
> parallel worker dispatch.

So instead of dumping things by the order of the TOC, we'll perform the
sorting later on before handing out jobs to workers?  That seems alright
to me for the most part.  One thing I do wonder about is if we should
also be sorting by tablespace and not just size, to try and maximize
throughput (that is, assign out parallel workers to each tablespace,
each going after the largest table in that tablespace, before coming
back around to assigning the next-largest file to the second worker on a
given tablespace, presuming we have more workers than tablespaces),
that's what we've seen works rather well in pgbackrest.

> However, at least for the directory-format case (which I think is the
> only one supported for parallel restore), we could make it compare the
> file sizes of the TABLE DATA items.  That'd work pretty well as a proxy
> for both the amount of effort needed for table restore, and the amount
> of effort needed to build indexes on the tables afterwards.

Parallel restore also works w/ custom-format dumps.

Thanks!

Stephen

--HUY/6dRnBRT+g9H1
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJbhA2iAAoJEO1sijiDR2RVfskQAIkzq+CgXDlvGcDkvlq5QGB1
4jhb5ir/6xiWCfHLb+2A/W5hzvSHQii2SXZXPOzDyLFA/z658bC3ej4dND2tcX/0
ANw3Lc55GyONdbYakWSTeoVxZr2T32PGUf+mLcQH3bf/HJsL+cC3qdxNtXHhQTqx
SsqnPq9kHlaZTevCpmEzqvwo3SXdoNButHYE9q7kjv1+4DJ24oGEwirB74kk8iai
r3ZE11YxFYC0nUnHwdH8Ov8A4kWOOIQVshe/AFkbxtAO0uA8QF94zp76d2i0PI3s
bNYEaxl3lB8JXOmOdYGbJzrA6LdEKZ7AgwLSP1Fe7D9uOh+iDbujWMaay1J9cgvv
cxPzF0tch3yYE2yGAs+GKH3GBNgftubbyGL9UAqlVZgV+5fqZDCW1VceoMcvaTXl
tYiLSHkIFBYaml6boUbMW8GLpL7Hx4aZfZ9rFnO8aEdCLp5f6uNdUJa5a8+Ef1M6
Ug1so4muBGTlebtaDVNu6N/vBGmOegTXth9tm6UKCkrWu9eYxaqS3ZNKSbkS2FB2
CFkQZyMMtJP/PM75XXidwf9X2JdV0UUyt1BSdsMqy1wPLtS7YAZ79qzwjlDRb6em
9pyq8umN//6dYP1qSBNKPitEoGivhipyRp3T9Qunnla9e10vi5n6ectw8yJpnxHV
FzzNST0+u1WaMMcJR2ur
=SeH4
-----END PGP SIGNATURE-----

--HUY/6dRnBRT+g9H1--