Date: Thu, 6 Mar 2025 13:33:13 -0600
From: Nathan Bossart <nathandbossart@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>,
	Corey Huinker <corey.huinker@gmail.com>,
	Robert Haas <robertmhaas@gmail.com>, Jeff Davis <pgsql@j-davis.com>,
	Michael Paquier <michael@paquier.xyz>,
	jian he <jian.universality@gmail.com>,
	Bruce Momjian <bruce@momjian.us>,
	Matthias van de Meent <boekewurm+postgres@gmail.com>,
	Magnus Hagander <magnus@hagander.net>,
	Stephen Frost <sfrost@snowman.net>,
	Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
	Peter Smith <smithpb2250@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
	alvherre@alvh.no-ip.org
Subject: Re: Statistics Import and Export
Message-ID: <Z8n4eUTS_iEq3mxn@nathan>
References: <CADkLM=e-H5H+pY8iA58ie28PVSRJ1HBpzyy1=FWmy0Hm+s4boQ@mail.gmail.com>
 <6rpmhyrtci4epuzay7y5xvd2cwdwb6zmtt6ofxrvakyawxhm7s@6grajbpr4kij>
 <CADkLM=ct+i2G=Z+c_8np1+C75uvzjUK_SBsouUvgWMHRs2cnyg@mail.gmail.com>
 <xe3yjkcthy7f377zjayuckxyd62z5cekgjau3j4vp6elbicson@cx2uiilwmxyy>
 <CADkLM=f1n2_Vomq0gKab7xdxDHmJGgn=DE48P8fzQOp3Mrs1Qg@mail.gmail.com>
 <pbjbvxnq36xo2ooufcnlfqsqnxuvggbqxbx3myjtswokshfjbl@hfihx57z4j4t>
 <CA+TgmoYGuEC=E4TbomRyqO8+Uav=90G0dYXw=rA1d5UmRB76Yg@mail.gmail.com>
 <CADkLM=dTTf9tonKsQnDCDp5oyODE2mFV7K6nFuCp84QE3GBWuQ@mail.gmail.com>
 <urk5wcvgvnqzmvdvbe6b2gthhf3pphfqmbwculciy3lm4kvnay@avvgvhxo544p>
 <714295.1741286854@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <714295.1741286854@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/Z8n4eUTS_iEq3mxn%40nathan>
Precedence: bulk

On Thu, Mar 06, 2025 at 01:47:34PM -0500, Tom Lane wrote:
> 1. pg_upgrade has made a policy judgement to apply parallelism across
> databases not within a database, ie it will launch concurrent dump/
> restore tasks in different DBs but not authorize any one of them to
> eat multiple CPUs.  That needs to be re-thought probably, as I think
> that decision dates to before we had useful parallelism in pg_dump and
> pg_restore.  I wonder if we could just rip out pg_upgrade's support
> for DB-level parallelism, which is not terribly pretty anyway, and
> simply pass the -j switch straight to pg_dump and pg_restore.

That would certainly help for clusters with one big database with many LOs
or something, but I worry it would hurt the many database case quite a bit.
Maybe we could add a --jobs-per-db option that indicates how to parallelize
dump/restore.  If you set --jobs=8 --jobs-per-db=8, the databases would be
dumped serially, but pg_dump would get -j8.  If you set --jobs=8 and
--jobs-per-db=2, we'd process 4 databases at a time, each with -j2.

-- 
nathan