MIME-Version: 1.0
References: <7886a68f-b466-2131-1747-f69f0fb71a37@gmx.net> <69077f15-4125-2d63-733f-21ce6eac4f01@gmx.net>
 <559b0e40-63e6-fa9a-6b03-d1eba10f30f8@gmx.net> <1629463.1715372568@sss.pgh.pa.us>
 <6a2f3906-3d7a-6924-7403-8f77d57a18e4@gmx.net> <CAApHDvrtTKfh7HgAyXBd3KN0s-jxiHzW7sWdm-sFEjP6fGPCkg@mail.gmail.com>
 <3efce60b-48c0-160d-0444-474b02f76739@gmx.net> <CAApHDvo8yYvqa1+bkW_f5xHX-gmKGYfaGwH+Y_KP-=9TOuF+-g@mail.gmail.com>
 <98c715d1-22f4-0fc1-1997-6236873c13de@gmx.net>
In-Reply-To: <98c715d1-22f4-0fc1-1997-6236873c13de@gmx.net>
From: David Rowley <dgrowleyml@gmail.com>
Date: Tue, 14 May 2024 02:15:45 +1200
Message-ID: <CAApHDvpn9ukU8ZfUe_jPU6roFLS2tECx1fH6sbQYnAyfW_z6DA@mail.gmail.com>
Subject: Re: SELECT DISTINCT chooses parallel seqscan instead of indexscan on
 huge table with 1000 partitions
To: Dimitrios Apostolou <jimis@gmx.net>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-general@lists.postgresql.org
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://www.postgresql.org/message-id/CAApHDvpn9ukU8ZfUe_jPU6roFLS2tECx1fH6sbQYnAyfW_z6DA%40mail.gmail.com>
Precedence: bulk

On Tue, 14 May 2024 at 01:52, Dimitrios Apostolou <jimis@gmx.net> wrote:
>
> On Tue, 14 May 2024, David Rowley wrote:
> > The query does contain an ORDER BY, so if the index is not chosen to
> > provide pre-sorted input, then something has to put the results in the
> > correct order before the LIMIT is applied.
>
> The last query I tried was:
>
> SELECT DISTINCT workitem_n FROM test_runs_raw LIMIT 10;

I was looking at the original query.   In that case, we have 2 ways to
remove duplicate rows with DISTINCT, "Hash Aggregate" and "Sort" ->
"Unique". Both of these will consume all of their input rows before
outputting any rows.

DISTINCT with LIMIT is a special case that we don't have a good
operator for.  In theory, we could have some "Hash Distinct" node type
that was less eager to consume all of its input rows.  When invoked
"Hash Distinct" could consume input rows until it found one that
didn't exist in the hash table.  I've no idea how that would work when
we exceed work_mem.  However, most queries with a LIMIT will have an
ORDER BY, so such a node likely wouldn't get much use.

David