Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ai13P-0002IN-G2 for pgsql-performance@arkaria.postgresql.org; Mon, 21 Mar 2016 14:43:36 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1ai13P-0003Xv-2n for pgsql-performance@arkaria.postgresql.org; Mon, 21 Mar 2016 14:43:35 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ai11j-0000yn-3C for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 14:41:51 +0000 Received: from post.visena.com ([46.226.10.50]) by makus.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1ai11f-0006tg-MD for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 14:41:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=visena.com; s=20141101.wh; h=Content-Type:MIME-Version:Subject:In-Reply-To:Message-ID:To:From:Date; bh=2symFk/ecQaWjnEHFB35XfVdsWUeBzV4WN2wZdCrgeg=; b=KZGO3ZHu8sqU5IROkA6+bmluU//Ut3Yu7j2bgya8snSOEcfI9agUyAFgTDzc5R5oC9bD0npghuLl6S1yoERxW7iQ5VJVEf/PUd9da2ZKhE6YH2jxd2rP3AlFv2J8ZP/4ljNdz5goXtm5M9zVnRpw/eKIPO5rsH0I13u9MP2mn6U=; Received: from [10.0.1.10] (helo=tc7-visena.wh.internal.visena.com) by post.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1ai11a-0003Hw-6F for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 15:41:44 +0100 Received: from localhost ([127.0.0.1] helo=tc7-visena.wh.internal.visena.com) by tc7-visena.wh.internal.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1ai11a-0001LO-5K for pgsql-performance@postgresql.org; Mon, 21 Mar 2016 15:41:42 +0100 Date: Mon, 21 Mar 2016 15:41:42 +0100 (CET) From: Andreas Joseph Krogh To: pgsql-performance@postgresql.org Message-ID: In-Reply-To: Subject: Re: Searching GIN-index (FTS) and sort by timestamp-column MIME-Version: 1.0 X-Mailer: Visena Mail 2.0.1-SNAPSHOT X-Spam-Score: -1.0 X-Spam-Report: SpamAssasin (score=-1.0, required 5.0 ALL_TRUSTED=-1,HTML_MESSAGE=0.001) X-Pg-Spam-Score: -2.0 (--) Content-Type: multipart/related; boundary="----=_Part_16_2074544689.1458571302062" List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org ------=_Part_15_1551055497.1458571302062 Content-Type: multipart/related; boundary="----=_Part_16_2074544689.1458571302062" ------=_Part_16_2074544689.1458571302062 Content-Type: multipart/alternative; boundary="----=_Part_17_1161280096.1458571302100" ------=_Part_17_1161280096.1458571302100 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable P=C3=A5 l=C3=B8rdag 19. mars 2016 kl. 03:44:55, skrev Jeff Janes >: On Wed, Mar 16, 2016 at 6:53 AM, Andreas Joseph Krogh > wrote: P=C3=A5 onsdag 16. mars 2016 kl. 14:37:= 27, skrev=20 Tom Lane >: Andreas Joseph Krogh > write= s: > 1. Why isnt' folder_id part of the index-cond? Because a GIN index is useless for sorting. > 2. Is there a way to make it use the (same) index to sort by > received_timestamp? No. > 3. Using a GIN-index, is there a way to use the index at all for sorting= ? No. > 4. It doesn't seem like ts_rank uses the index for sorting either. Same reason. regards, tom lane =C2=A0 So it's basically impossible to use FTS/GIN with sorting on large datasets? Are there any plans to improve this situation? =C2=A0 I don't see why it would not be possible to create a new execution node typ= e=20 that does an index scan to obtain order (or just to satisfy an equality or= =20 range expression), and takes a bitmap (as produced by the FTS/GIN) to apply= as=20 a filter.=C2=A0 But, I don't know of anyone planning on doing that. =C2=A0 Isn't this what Postgres Pro are planning?=20 http://postgrespro.com/roadmap/mssearch =C2=A0 "Unlike external special-purpose search engines, a full-text search engine= =20 built in a DBMS is capable of combining full-text and attributive search=20 criteria in SQL query syntax. It is planned to improve the existing Postgre= SQL=20 full-text search engine byextending the functionality of Generalized Invert= ed=20 Index (GIN) to make it capable of storing extra information required for=20 ranging query results. This search acceleration will allow to go back from= =20 external full-text search engines, thus facilitating system administration = and=20 use, reducing technology risks, and improving information security." =C2=A0 -- Andreas Joseph Krogh CTO / Partner - Visena AS Mobile: +47 909 56 963 andreas@visena.com www.visena.com =C2=A0 ------=_Part_17_1161280096.1458571302100 Content-Type: text/html;charset=UTF-8 Content-Transfer-Encoding: quoted-printable
P=C3=A5 l=C3=B8rdag 19. mars 2016 kl. 03:44:55, skrev Jeff Janes <<= a href=3D"mailto:jeff.janes@gmail.com">jeff.janes@gmail.com>:
On Wed, Mar 16, 2016 at 6:53 AM, Andreas Joseph = Krogh <andreas@visena.com> wrote:
P=C3=A5 onsdag 16. mars 2016 kl. 14:37:27, skrev Tom Lane <tgl@sss.pgh.pa.us&= gt;:
An= dreas Joseph Krogh <andreas@visena.com> writes:
> 1. Why isnt' folder_id part of the index-cond?

Because a GIN index is useless for sorting.

> 2. Is there a way to make it use the (same) index to sort by
> received_timestamp?

No.

> 3. Using a GIN-index, is there a way to use the index at all for sorti= ng?

No.

> 4. It doesn't seem like ts_rank uses the index for sorting either.

Same reason.

regards, tom lane
=C2=A0
So it's basically impossible to use FTS/GIN with sorting on large data= sets?
Are there any plans to improve this situation?
=C2=A0
I don't see why it would not be possible to create a new execution nod= e type that does an index scan to obtain order (or just to satisfy an equal= ity or range expression), and takes a bitmap (as produced by the FTS/GIN) t= o apply as a filter.=C2=A0 But, I don't know of anyone planning on doing th= at.
=C2=A0
Isn't this what Postgres Pro are planning? http://postgrespro.com/road= map/mssearch
=C2=A0
"Unlike external special-purpose search = engines, a full-text search engine built in a DBMS is capable of combining = full-text and attributive search criteria in SQL query syntax. It is planne= d to improve the existing PostgreSQL full-text search engine by extendin= g the functionality of Generalized Inverted Index (GIN) to make it capable = of storing extra information required for ranging query results. This s= earch acceleration will allow to go back from external full-text search eng= ines, thus facilitating system administration and use, reducing technology = risks, and improving information security."
=C2=A0
--
Andrea= s Joseph Krogh
CTO / Partner<= /span> - Visena AS
Mobile: +47 90= 9 56 963
=3D""
=C2=A0
------=_Part_17_1161280096.1458571302100-- ------=_Part_16_2074544689.1458571302062 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-ID: iVBORw0KGgoAAAANSUhEUgAAAIUAAAAYCAYAAADUIj6hAAAABHNCSVQICAgIfAhkiAAABzBJREFU aEPtmNFxHDcMhmVP3i1VECpvnjzkVIHWFfhcgVcVRKrAUgWRK/C6Al8H3lTgy0PGbzFdQc4VJP/H ADs43q6kROeJNbOYgQACIAgCWJKng4MZ5gxUGXh0U0b+SIuF9G+EWXj2Q15vbrE/NPtk9uub7Gfd t5mBx1NhqSEo8HshjbEUtlO2yIM9tsx5dZP9rPt2MzDZFAqZE4LGcFjdsg3saQaHt7fY70X949On aS+OZidDBkavD33157L4xaw2os+Mp/DAC10l2XhOCeStj0W5ajrJL8WfCi80Xgf9vVlrhg9yRONe /f7xI2vNsIcM7JwU9o6oGyJrLT8JOA1aX9sKP4wl94bAniukEbo/n7YPmuTETzIab4Y9ZWCrKexd 8M58lxPCvvD6aihfvexbEQoPYO8NQRO0JnddGO6FJYaVsBde7cXj7KRk4LsqDxQzCYeGsJNgaXZe +JXkyGgWINq3Gp+bHNILz+wEYs5qH1eJrouNrpDXtk4O6x1IvtD40GWyJYYdkF0jIbYZlN16xygI gl9smbMDsmFdfAKb23xGB8H/mv3tOJfArs1kukk79DEPdQ5u0g1vCvvqKXIW8mZYW+F3Tg4r8HvZ kQCCLydK8EFMQCe5N8RgL9mRG0SqQFuNvdE6beSs0qPDBuCdg0+gvClso8SbTO4ki7mQzQqBrcMH QPwReg2wW8umEe/+L8T/LEzBGF9nXjzZ46s+ITHvhegW8LL39xm6App7KYL/GE+vcYnFbJiP/0aY hdiCnRC7jaj7eiK2ETInC5PRF6LYkSN0+HabF75WuT6syCyI0YkVGGMvUC33AmfZTDUEj8u6IVhu EhRUJ2U2g6UlugyNb03Hl9obHwlxJbcRzcYjO4QPjVfGFTQav4vrmp7cpMp2qbHnBxWJbisbho2Q XI6C1sIHDUFhH4Hij4UbIev66eANeiwbkA+LBsO36zAHzoXU7AhbqLAXEiOYTXcSdO8VS9L4wN8U BIYhBd6oSUgYMijOkecRuTdQa/YiBXhbXFcnCnI2SrfeBG9NydrLYMhGHa5qB9pQIxlzAE6Okjzx JO5afGe6kmhBicWKQNKuTZ5EW+MjQY8v/9rQ0bhJSJyNGUe/JH1t8h1iMbdSPAvxHYin6YmN9YBX QmTYZZNh14vHhhhal5vtcIrJjmvszPRJdExH3MWHvykoBEc9CoCGWJisOLOGoCORs1FvIMbYA8z3 kwM59oemy6LlWrLxFLmWgiQAfEGd8S+NssbK+EhyGLxUkhhiy717waBqHOJYSEacwBejkFNhjJNj v/gANOdQxPfciP/JVBCK2cOIcg3RRJ+CPrLPNVhhN6F38VKMF3XLlIJrjU5C8gMFxvKDvOcPc4rV NjCn7KM0BV+161V8viSCuJZ8SITGJGEh7IUUlxOFMYUH1kJOCN4Wh+I5pqCuK01k40kSNtnKiKIl UfxAgW5sU5JlS05rtt5YFJF1SWoqHv6BRgQcA4/bdb9WRjmMk3jyUMAbIoyJq9e4cVmgzKt9j5gN J/aYDhk+hhjExwaPcz5PObA5Zd9bvz5UzCTZuZDidu5AchpiKSwPR+TV1bCWqBQ9nCjJ5neivC82 Nr4LeS2j1gw5LUqwBuhGQQU5UwHeSslXkwIynz2U2A2yKDgG7OffwLA3TpGRpk0Tzpj3ZEJXi/GR a6GN0e0NtppChePdcAz1FTS+FN8Kpxoiykk+J8fC5tenjbu9kSqpHLu9jBrhUuhNwVGbpyZrzrl0 HPVD8SXj5EOOD/eDC64VjvYBZLuUbIVAfBN1t/C/SU+cAGtdGo+fVnzycUX5wmn6eCKPmWYJG2E/ ppTsuRCbvUD9fwquksG5GqLVKhzDQ3GrEyLKSXhsiK3T5j9EyxffCFOYi2wUlPyFFDQAhehESDjQ GoX0ho0oj8RPou6TxHJdcT0NTcWkO0AnG7+uXsnHqcas/72wvWF+mSf7N/Watp9kTXoluzeS0fB9 9CcZ/hvhSZTfh99pispZOXL9KrGrARkNUMu9ITbScZWs7xOYNt9pwxSZtYDsX/GE3xTkrXgwAsXm fud08FiTeC+m2/o7ppo+PTS/NBK5ARpDG5YHr++DpmVfreYdiX8mnp+DzPEG9WbqJON0JBenZofs sxBA1gj5NXGvfJu/Qh7HwQh/VDUEyUxCit5hH94QfKkEdu+GwK8BX0hvCF+D67xhjmXQCXMwJCZ+ olK0A1EKRCE4snsh42w8Mv/Zhxw9iD7Cjo7CyQC/KyF6oBfShL4PYgG+CHsYKyZxvxZSZBAgjhIz YDz+AbfD37GtbaoSKzgGt+lKfI/GZtay6vG4VXTpPsh+IcQhOk9I7WYeP5AM3HZ9+DaSGIp9HIuu huC4pCGGx+YD2fcc5tfIAA0h/Et4/jX8zz4fWAasIf4UbR9Y6HO4d8jAnd4U0Y8ageuCB+c+H5R3 CHU2mTMwZ+B/y8DfSMBLLOYXVuEAAAAASUVORK5CYII= ------=_Part_16_2074544689.1458571302062-- ------=_Part_15_1551055497.1458571302062--