Date: Mon, 24 Jun 2019 22:20:10 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Simon Riggs <simon@2ndquadrant.com>
Cc: James Coleman <jtc331@gmail.com>,
	Rafia Sabih <rafia.pghackers@gmail.com>,
	Shaun Thomas <shaun.thomas@2ndquadrant.com>,
	Dmitry Dolgov <9erthalion6@gmail.com>,
	Alexander Korotkov <a.korotkov@postgrespro.ru>,
	Tom Lane <tgl@sss.pgh.pa.us>,
	PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>
Subject: Re: [PATCH] Incremental sort (was: PoC: Partial sort)
Message-ID: <20190624202010.7n7lolmrcs67udcg@development>
References: 
 <CAAaqYe_3GY8EpN=n2sBEHdXBTG==tfKqDTGbdKakCunWz490MQ@mail.gmail.com>
 <20190602211753.rxor4apm7zfiltt3@development>
 <CAAaqYe_a7Umxn6J+GGXGu00f+Jwmcq83gh3dj6tC4RR75+yjAw@mail.gmail.com>
 <CA+FpmFffDyU_FTf8c3a9+GWqMmsQ9xy6Dibc0Ja9v4BeYzjXow@mail.gmail.com>
 <20190614181135.tdj7aq4iffy5ghl2@development>
 <CAAaqYe9mifZ9wv-Jm=+J3FWEOrXM-6SgY7z_YJhObfiG7dcxnA@mail.gmail.com>
 <CAAaqYe_PXXHnOavS_X+C+yJnWesQzWeCv80jsp33SW4osGRUWg@mail.gmail.com>
 <CANP8+jJaGHCQb8aBuywkz8Ev9WJwSZrMkmtr2WciCFazhtsgcw@mail.gmail.com>
 <CAAaqYe9N=BZDJ2U9FsRX1-foJu-a8u1t1KGYtvf=zJdUAO9JdA@mail.gmail.com>
 <CANP8+jLw+WQEUN5zZwQ_egr2WSbsR2bWqY79fS5BzAMJmLJCDA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: 
 <CANP8+jLw+WQEUN5zZwQ_egr2WSbsR2bWqY79fS5BzAMJmLJCDA@mail.gmail.com>
User-Agent: NeoMutt/20180716-1444-295967
Precedence: bulk

On Mon, Jun 24, 2019 at 07:05:24PM +0100, Simon Riggs wrote:
>On Mon, 24 Jun 2019 at 18:01, James Coleman <jtc331@gmail.com> wrote:
>
>> On Mon, Jun 24, 2019 at 12:56 PM Simon Riggs <simon@2ndquadrant.com>
>> wrote:
>>
>> > What is the specific use case for this? This sounds quite general case.
>>
>> They are both general cases in some sense, but the concerns lie mostly
>> with what happens when they're unexpectedly encountered. For example,
>> if the expected row count or group size is off by a good bit and we
>> effectively have to perform a sort of all (or most) possible rows.
>>
>> If we can get the performance to a point where that misestimated row
>> count or group size doesn't much matter, then ISTM including the patch
>> becomes a much more obvious total win.
>>
>
>I was trying to think of ways of using external information/circumstance to
>knowingly avoid negative use cases. i.e. don't treat sort as a black box,
>use its context.
>

Like what, for example? I'm not saying there's no such additional
information, but I can't think of anything at the moment.

One of the issues is that while the decision whether to use incremental
sort is done during planning, most of the additional (and reliable)
infomation about the data is only available at execution time. And it's
likely not available explicitly - we need to deduce it. Which I think is
exactly what the min group size heuristics is about, no?


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services