Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ai1pZ-0004so-96 for pgsql-performance@arkaria.postgresql.org; Mon, 21 Mar 2016 15:33:21 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1ai1pY-0002nc-SC for pgsql-performance@arkaria.postgresql.org; Mon, 21 Mar 2016 15:33:20 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ai1pY-0002nW-DF for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 15:33:20 +0000 Received: from post.visena.com ([46.226.10.50]) by magus.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1ai1pU-0004H1-LC for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 15:33:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=visena.com; s=20141101.wh; h=Content-Type:MIME-Version:Subject:In-Reply-To:Message-ID:To:From:Date; bh=kPoFyi5iYXoKSx8Bo1kX/Pm4uSj/hc2VG/hnJGkF0qY=; b=U/RsGhJOpVtC6EVNS/NBN28mz0U/crIVSXc2snZxYaM7b7V/P0b71bDpYnrQlT7iJvfmd/b60bZDki0qh3uRD54TeR33J7ndQikJBttmKMPxWdlk6kYxEM/5YeclKeHBHgN27gRGwX7KEM+B/WsIppOC1aLfLMEYJRILhZeFKwk=; Received: from [10.0.1.10] (helo=tc7-visena.wh.internal.visena.com) by post.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1ai1pQ-0006yw-JW for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 16:33:15 +0100 Received: from localhost ([127.0.0.1] helo=tc7-visena.wh.internal.visena.com) by tc7-visena.wh.internal.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1ai1pQ-0003Yj-IY for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 16:33:12 +0100 Date: Mon, 21 Mar 2016 16:33:12 +0100 (CET) From: Andreas Joseph Krogh To: pgsql-performance@postgresql.org Message-ID: In-Reply-To: Subject: Re: Searching GIN-index (FTS) and sort by timestamp-column MIME-Version: 1.0 X-Mailer: Visena Mail 2.0.1-SNAPSHOT X-Spam-Score: -1.0 X-Spam-Report: SpamAssasin (score=-1.0, required 5.0 ALL_TRUSTED=-1,HTML_MESSAGE=0.001) X-Pg-Spam-Score: -2.0 (--) Content-Type: multipart/related; boundary="----=_Part_26_1684957786.1458574392506" List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org ------=_Part_25_1230076740.1458574392506 Content-Type: multipart/related; boundary="----=_Part_26_1684957786.1458574392506" ------=_Part_26_1684957786.1458574392506 Content-Type: multipart/alternative; boundary="----=_Part_27_588950635.1458574392520" ------=_Part_27_588950635.1458574392520 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable P=C3=A5 mandag 21. mars 2016 kl. 16:13:07, skrev Oleg Bartunov >: =C2=A0 =C2=A0 On Mon, Mar 21, 2016 at 5:41 PM, Andreas Joseph Krogh > wrote: P=C3=A5 l=C3=B8rdag 19. mars 2016 kl. 0= 3:44:55, skrev=20 Jeff Janes >: On Wed, Mar 16, 2016 at 6:53 AM, Andreas Joseph Krogh > wrote: P=C3=A5 onsdag 16. mars 2016 kl. 14:37:= 27, skrev=20 Tom Lane >: Andreas Joseph Krogh > write= s: > 1. Why isnt' folder_id part of the index-cond? Because a GIN index is useless for sorting. > 2. Is there a way to make it use the (same) index to sort by > received_timestamp? No. > 3. Using a GIN-index, is there a way to use the index at all for sorting= ? No. > 4. It doesn't seem like ts_rank uses the index for sorting either. Same reason. regards, tom lane =C2=A0 So it's basically impossible to use FTS/GIN with sorting on large datasets? Are there any plans to improve this situation? =C2=A0 I don't see why it would not be possible to create a new execution node typ= e=20 that does an index scan to obtain order (or just to satisfy an equality or= =20 range expression), and takes a bitmap (as produced by the FTS/GIN) to apply= as=20 a filter.=C2=A0 But, I don't know of anyone planning on doing that. =C2=A0 Isn't this what Postgres Pro are planning?=20 http://postgrespro.com/roadmap/mssearch=20 =C2=A0 "Unlike external special-purpose search engines, a full-text search engine= =20 built in a DBMS is capable of combining full-text and attributive search=20 criteria in SQL query syntax. It is planned to improve the existing Postgre= SQL=20 full-text search engine byextending the functionality of Generalized Invert= ed=20 Index (GIN) to make it capable of storing extra information required for=20 ranging query results. This search acceleration will allow to go back from= =20 external full-text search engines, thus facilitating system administration = and=20 use, reducing technology risks, and improving information security." =C2=A0 This is different feature ! Actually, we already have prototype of what Jef= f=20 suggested, we called it bitmap filtering, but failed to find use case where= it=20 provides benefits. Teodor will comment this idea more detail. =C2=A0 The feature I'm missing is the ability to do FTS (or use GIN in general) an= d=20 then sort on some other column (also indexed by the same GIN-index, using t= he=20 btree-gin extention), often of type BIGINT or TIMESTAMP. Are you planning to work on such a feature for GIN? =C2=A0 Thanks. =C2=A0 -- Andreas Joseph Krogh CTO / Partner - Visena AS Mobile: +47 909 56 963 andreas@visena.com www.visena.com =C2=A0 ------=_Part_27_588950635.1458574392520 Content-Type: text/html;charset=UTF-8 Content-Transfer-Encoding: quoted-printable
P=C3=A5 mandag 21. mars 2016 kl. 16:13:07, skrev Oleg Bartunov <obartunov@gmail.com>:
=C2=A0
=C2=A0
On Mon, Mar 21, 2016 at 5:41 PM, Andreas Joseph = Krogh <andreas@visena.com> wrote:
P=C3=A5 l=C3=B8rdag 19. mars 2016 kl. 03:44:55, skrev Jeff Janes= <jeff.janes@g= mail.com>:
On Wed, Mar 16, 2016 at 6:53 AM, Andreas J= oseph Krogh <andreas@visena.com> wrote:
P=C3=A5 onsdag 16. mars 2016 kl. 14:37:27, skrev Tom Lane = <tgl@sss.pgh.pa.u= s>:
Andreas Joseph Krogh <andreas@visena.com> writes:
> 1. Why isnt' folder_id part of the index-cond?

Because a GIN index is useless for sorting.

> 2. Is there a way to make it use the (same) index to sort by
> received_timestamp?

No.

> 3. Using a GIN-index, is there a way to use the index at all for sorti= ng?

No.

> 4. It doesn't seem like ts_rank uses the index for sorting either.

Same reason.

regards, tom lane
=C2=A0
So it's basically impossible to use FTS/GIN with sorting on larg= e datasets?
Are there any plans to improve this situation?
=C2=A0
I don't see why it would not be possible to create a new executi= on node type that does an index scan to obtain order (or just to satisfy an= equality or range expression), and takes a bitmap (as produced by the FTS/= GIN) to apply as a filter.=C2=A0 But, I don't know of anyone planning on do= ing that.
=C2=A0
Isn't this what Postgres Pro are planning? http://postgrespro.com/roadmap/m= ssearch
=C2=A0
"Unlike external special-purpose search e= ngines, a full-text search engine built in a DBMS is capable of combining f= ull-text and attributive search criteria in SQL query syntax. It is planned= to improve the existing PostgreSQL full-text search engine by extending= the functionality of Generalized Inverted Index (GIN) to make it capable o= f storing extra information required for ranging query results. This se= arch acceleration will allow to go back from external full-text search engi= nes, thus facilitating system administration and use, reducing technology r= isks, and improving information security."
=C2=A0
This is different feature ! Actually, we already have prototype of wha= t Jeff suggested, we called it bitmap filtering, but failed to find use cas= e where it provides benefits. Teodor will comment this idea more detail.
=C2=A0
The feature I'm missing is the ability to do FTS (or use GIN in genera= l) and then sort on some other column (also indexed by the same GIN-index, = using the btree-gin extention), often of type BIGINT or TIMESTAMP.
Are you planning to work on such a feature for GIN?
=C2=A0
Thanks.
=C2=A0
--
Andrea= s Joseph Krogh
CTO / Partner<= /span> - Visena AS
Mobile: +47 90= 9 56 963
=3D""
=C2=A0
------=_Part_27_588950635.1458574392520-- ------=_Part_26_1684957786.1458574392506 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-ID: iVBORw0KGgoAAAANSUhEUgAAAIUAAAAYCAYAAADUIj6hAAAABHNCSVQICAgIfAhkiAAABzBJREFU aEPtmNFxHDcMhmVP3i1VECpvnjzkVIHWFfhcgVcVRKrAUgWRK/C6Al8H3lTgy0PGbzFdQc4VJP/H ADs43q6kROeJNbOYgQACIAgCWJKng4MZ5gxUGXh0U0b+SIuF9G+EWXj2Q15vbrE/NPtk9uub7Gfd t5mBx1NhqSEo8HshjbEUtlO2yIM9tsx5dZP9rPt2MzDZFAqZE4LGcFjdsg3saQaHt7fY70X949On aS+OZidDBkavD33157L4xaw2os+Mp/DAC10l2XhOCeStj0W5ajrJL8WfCi80Xgf9vVlrhg9yRONe /f7xI2vNsIcM7JwU9o6oGyJrLT8JOA1aX9sKP4wl94bAniukEbo/n7YPmuTETzIab4Y9ZWCrKexd 8M58lxPCvvD6aihfvexbEQoPYO8NQRO0JnddGO6FJYaVsBde7cXj7KRk4LsqDxQzCYeGsJNgaXZe +JXkyGgWINq3Gp+bHNILz+wEYs5qH1eJrouNrpDXtk4O6x1IvtD40GWyJYYdkF0jIbYZlN16xygI gl9smbMDsmFdfAKb23xGB8H/mv3tOJfArs1kukk79DEPdQ5u0g1vCvvqKXIW8mZYW+F3Tg4r8HvZ kQCCLydK8EFMQCe5N8RgL9mRG0SqQFuNvdE6beSs0qPDBuCdg0+gvClso8SbTO4ki7mQzQqBrcMH QPwReg2wW8umEe/+L8T/LEzBGF9nXjzZ46s+ITHvhegW8LL39xm6App7KYL/GE+vcYnFbJiP/0aY hdiCnRC7jaj7eiK2ETInC5PRF6LYkSN0+HabF75WuT6syCyI0YkVGGMvUC33AmfZTDUEj8u6IVhu EhRUJ2U2g6UlugyNb03Hl9obHwlxJbcRzcYjO4QPjVfGFTQav4vrmp7cpMp2qbHnBxWJbisbho2Q XI6C1sIHDUFhH4Hij4UbIev66eANeiwbkA+LBsO36zAHzoXU7AhbqLAXEiOYTXcSdO8VS9L4wN8U BIYhBd6oSUgYMijOkecRuTdQa/YiBXhbXFcnCnI2SrfeBG9NydrLYMhGHa5qB9pQIxlzAE6Okjzx JO5afGe6kmhBicWKQNKuTZ5EW+MjQY8v/9rQ0bhJSJyNGUe/JH1t8h1iMbdSPAvxHYin6YmN9YBX QmTYZZNh14vHhhhal5vtcIrJjmvszPRJdExH3MWHvykoBEc9CoCGWJisOLOGoCORs1FvIMbYA8z3 kwM59oemy6LlWrLxFLmWgiQAfEGd8S+NssbK+EhyGLxUkhhiy717waBqHOJYSEacwBejkFNhjJNj v/gANOdQxPfciP/JVBCK2cOIcg3RRJ+CPrLPNVhhN6F38VKMF3XLlIJrjU5C8gMFxvKDvOcPc4rV NjCn7KM0BV+161V8viSCuJZ8SITGJGEh7IUUlxOFMYUH1kJOCN4Wh+I5pqCuK01k40kSNtnKiKIl UfxAgW5sU5JlS05rtt5YFJF1SWoqHv6BRgQcA4/bdb9WRjmMk3jyUMAbIoyJq9e4cVmgzKt9j5gN J/aYDhk+hhjExwaPcz5PObA5Zd9bvz5UzCTZuZDidu5AchpiKSwPR+TV1bCWqBQ9nCjJ5neivC82 Nr4LeS2j1gw5LUqwBuhGQQU5UwHeSslXkwIynz2U2A2yKDgG7OffwLA3TpGRpk0Tzpj3ZEJXi/GR a6GN0e0NtppChePdcAz1FTS+FN8Kpxoiykk+J8fC5tenjbu9kSqpHLu9jBrhUuhNwVGbpyZrzrl0 HPVD8SXj5EOOD/eDC64VjvYBZLuUbIVAfBN1t/C/SU+cAGtdGo+fVnzycUX5wmn6eCKPmWYJG2E/ ppTsuRCbvUD9fwquksG5GqLVKhzDQ3GrEyLKSXhsiK3T5j9EyxffCFOYi2wUlPyFFDQAhehESDjQ GoX0ho0oj8RPou6TxHJdcT0NTcWkO0AnG7+uXsnHqcas/72wvWF+mSf7N/Watp9kTXoluzeS0fB9 9CcZ/hvhSZTfh99pispZOXL9KrGrARkNUMu9ITbScZWs7xOYNt9pwxSZtYDsX/GE3xTkrXgwAsXm fud08FiTeC+m2/o7ppo+PTS/NBK5ARpDG5YHr++DpmVfreYdiX8mnp+DzPEG9WbqJON0JBenZofs sxBA1gj5NXGvfJu/Qh7HwQh/VDUEyUxCit5hH94QfKkEdu+GwK8BX0hvCF+D67xhjmXQCXMwJCZ+ olK0A1EKRCE4snsh42w8Mv/Zhxw9iD7Cjo7CyQC/KyF6oBfShL4PYgG+CHsYKyZxvxZSZBAgjhIz YDz+AbfD37GtbaoSKzgGt+lKfI/GZtay6vG4VXTpPsh+IcQhOk9I7WYeP5AM3HZ9+DaSGIp9HIuu huC4pCGGx+YD2fcc5tfIAA0h/Et4/jX8zz4fWAasIf4UbR9Y6HO4d8jAnd4U0Y8ageuCB+c+H5R3 CHU2mTMwZ+B/y8DfSMBLLOYXVuEAAAAASUVORK5CYII= ------=_Part_26_1684957786.1458574392506-- ------=_Part_25_1230076740.1458574392506--