Date: Wed, 24 Jan 2018 10:20:18 -0800
From: Andres Freund <andres@anarazel.de>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>,
	Thomas Munro <thomas.munro@enterprisedb.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: pgsql: Add parallel-aware hash joins.
Message-ID: <20180124182018.jx2h4wp6twqo7xix@alap3.anarazel.de>
References: <20171222082203.fl6rpzlqnm4h7h6h@alap3.anarazel.de>
 <CAEepm=04WhyY65BpE2g_8E==ayhBXjV9JbyR-+9N4McmY3SQkQ@mail.gmail.com>
 <30219.1514428346@sss.pgh.pa.us>
 <CAEepm=1OyZcCAxAEWYhmeg0qmWyYF2O2yW7s7A4uVTt-dgRRyQ@mail.gmail.com>
 <4503.1514435195@sss.pgh.pa.us>
 <CAEepm=2rZ6NzdR7Rk5jsXsJR8M_kRmi+0JSY1x6DLSdmBb-=VA@mail.gmail.com>
 <CAEepm=1KTKEKW3KBbZwN6Ohh+rhuXjxC_f-F4VHR8Y6UWC=xVQ@mail.gmail.com>
 <21339.1516633310@sss.pgh.pa.us>
 <24384.1516665190@sss.pgh.pa.us>
 <CA+TgmoZeUY9W30Zzmi6q9JbnF6cGOpaea381M2XmDro+umYVhA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmoZeUY9W30Zzmi6q9JbnF6cGOpaea381M2XmDro+umYVhA@mail.gmail.com>
Precedence: bulk

On 2018-01-23 14:24:56 -0500, Robert Haas wrote:
> Right, but this doesn't seem to show any big spike in the runtime at
> the time when parallel hash was committed, or when the preparatory
> patch to add test coverage for hash joins got committed.  Rather,
> there's a gradual increase over time.  Either we're making the server
> slower (which would be bad) or we're adding proper test coverage for
> all the new features that we're adding (which would be good).  We
> can't expect every feature patch to preserve the runtime of the tests
> absolutely unchanged; figuring out what can be optimized is a separate
> exercise from adding test coverage either for new things or for things
> that weren't previously covered.

Agreed.

One the improvement front, my observation is that we rarely are actually
cpu bound across processes. One thing I've been wondering is whether we
can get a pretty large win from just rescheduling
parallel_schedule. There definitely are individual testfiles that take a
lot longer than others, but their positining in groups doesn't
necessarily reflect that.

Besides manually reordering the schedule, I think it might be time that
we improve pg_regress's scheduling. One big first step would e.g. be to
not manually limit the number of parallel tests in a group to 20, but
instead allow larger groups and only run a limited number of them in
parallel. If done right we could start the next test in a group as soon
as *one* task in a group has finished, rather than waiting for all of
them to finish as we currently do for (sub-)groups.

Besides larger groups, starting the next test(s) earlier, another way to
gain pretty large improvements would be a test schedule feature that
allowed to stat dependencies between tests. So instead of manually
grouping the schedule, have 'numerology' state that it depends on int2,
int4, int8, float4, float8, which means it can actually be started
earlier than it currently can in many cases.

Greetings,

Andres Freund