From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Tatsuro Yamada <yamada.tatsuro@lab.ntt.co.jp>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] CLUSTER command progress monitor
Date: Tue, 18 Dec 2018 18:01:59 -0300
Message-ID: <20181218210159.xtkltzm7flrwsm55@alvherre.pgsql> (raw)
In-Reply-To: <CAH2-WzknWhguFqC17Ax512C2UwoHqAE14bsrE+S-Za9u8pDAyQ@mail.gmail.com>
On 2017-Nov-21, Peter Geoghegan wrote:
> On Mon, Oct 2, 2017 at 6:04 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > Progress reporting on sorts seems like a tricky problem to me, as I
> > said before. In most cases, a sort is going to involve an initial
> > stage where it reads all the input tuples and writes out quicksorted
> > runs, and then a merge phase where it merges all the output tapes into
> > a sorted result. There are some complexities; for example, if the
> > number of tapes is really large, then we might need multiple merge
> > phases, only the last of which will produce tuples.
>
> This would ordinarily be the point at which I'd say "but you're very
> unlikely to require multiple passes for an external sort these days".
> But I won't say that on this thread, because CLUSTER generally has
> unusually wide tuples, and so is much more likely to be I/O bound, to
> require multiple passes, etc. (I bet the v10 enhancements
> disproportionately improved CLUSTER performance.)
When the seqscan-and-sort strategy is used, we feed tuplesort with every
tuple from the scan. Once that's completed, we call `performsort`, then
retrieve tuples.
If we see this in terms of tapes and merges, we can report the total
number of each of those that we have completed. As far as I understand,
we write one tape to completion, and only then start another one, right?
Since there's no way to know how many tapes/merges are needed in total,
it's not possible to compute a percentage of completion. That's seems
okay -- we're just telling the user that progress is being made, and we
only report facts not theory. Perhaps we can (also?) indicate disk I/O
utilization, in terms of the number of blocks written by tuplesort.
I suppose that in order to have tuplesort.c report progress, we would
have to have some kind of API that tuplesort would invoke internally to
indicate events such as "tape started/completed", "merge started/completed".
One idea is to use a callback system; each tuplesort caller could
optionally pass a callback to the "begin" function, for progress
reporting purposes. Initially only cluster.c would use it, but I
suppose eventually every tuplesort caller would want that.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: alvherre@2ndquadrant.com, pg@bowt.ie, robertmhaas@gmail.com, yamada.tatsuro@lab.ntt.co.jp
Subject: Re: [HACKERS] CLUSTER command progress monitor
In-Reply-To: <20181218210159.xtkltzm7flrwsm55@alvherre.pgsql>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox