Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WfyOQ-0006nt-8b for pgsql-performance@arkaria.postgresql.org; Thu, 01 May 2014 21:19:46 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1WfyOP-0007ra-Ol for pgsql-performance@arkaria.postgresql.org; Thu, 01 May 2014 21:19:45 +0000 Received: from makus.postgresql.org ([2001:4800:7903:4::125]) by malur.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WfyON-0007rR-B7 for pgsql-performance@postgresql.org; Thu, 01 May 2014 21:19:43 +0000 Received: from prod2.officenet.no ([195.159.87.246]) by makus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WfyOE-00052w-QL for pgsql-performance@postgresql.org; Thu, 01 May 2014 21:19:42 +0000 Received: from localhost ([127.0.0.1] helo=prod2) by prod2.officenet.no with esmtp (Exim 4.76) (envelope-from ) id 1WfyOD-0005zz-Ck for pgsql-performance@postgresql.org; Thu, 01 May 2014 23:19:33 +0200 Date: Thu, 1 May 2014 23:19:33 +0200 (CEST) From: Andreas Joseph Krogh To: pgsql-performance@postgresql.org Message-ID: In-Reply-To: Subject: Re: Optimize query for listing un-read messages MIME-Version: 1.0 X-Mailer: OfficeNet Mail 1.9.0-SNAPSHOT X-Pg-Spam-Score: -1.9 (-) Content-Type: multipart/related; boundary="----=_Part_41_158633025.1398979173212" List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org ------=_Part_40_924192768.1398979173212 Content-Type: multipart/related; boundary="----=_Part_41_158633025.1398979173212" ------=_Part_41_158633025.1398979173212 Content-Type: multipart/alternative; boundary="----=_Part_42_1316188218.1398979173229" ------=_Part_42_1316188218.1398979173229 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable P=C3=A5 torsdag 01. mai 2014 kl. 23:02:13, skrev Pavel Stehule < pavel.stehule@gmail.com >: =C2=A0 =C2=A0 20= 14-05-01 22:30=20 GMT+02:00 Andreas Joseph Krogh>:=20 P=C3=A5 torsdag 01. mai 2014 kl. 21:53:32, skrev Pavel Stehule < pavel.stehule@gmail.com >: =C2=A0 =C2=A0 20= 14-05-01 21:39=20 GMT+02:00 Andreas Joseph Krogh>:=20 P=C3=A5 torsdag 01. mai 2014 kl. 21:30:39, skrev Pavel Stehule < pavel.stehule@gmail.com >: Hello [snip] =C2= =A0 I had=20 a perfect success on similar use case with descent ordered partial index http://www.postgresql.org/docs/9.3/interactive/sql-createindex.html=20 =C2= =A0 I'm not=20 getting good performance. Are you able to craft an example using my schema = and=20 partial index? =C2=A0 maybe some like =C2=A0 CREATE INDEX ON message_property (person_id, message_id) WHERE pr.i= s_read =C2=A0 When I am thinking about your schema, it is designed well, but it i= s not=20 index friendly, so for some fast access you should to hold a cache (table) = of=20 unread messages =C2=A0 Ah, that's what I was hoping to not having to do. In= my=20 system, messages arrive all the time and having to update a cache for all n= ew=20 messages for all users seems messy... Seems I could just as well create a= =20 message_property for all users when a new message arrives, so I can INNER J= OIN=20 it and get good performance. But that table will quickly grow *very* large.= .. =C2=A0=20 What you need is a JOIN index, that is not possible in Postgres. =C2=A0 I a= fraid so=20 some "ugly" solutions is necessary (when you require extra fast access). Yo= u=20 need a index (small index) and it require some existing set - you cannot do= =20 index on the difference two sets. =C2=A0 I expect so some flag on the relation "message" - like "it should n= ot be=20 not read" can helps little bit - and can be used in partial index as=20 conditions. Other possibility is some variant of partitioning - you can div= ide=20 a messages and users to distinct sets and then you decrease a number of=20 possible combinations. =C2=A0 Just curious: Is such a JOIN index possible i= n other=20 DBs, if so - which? Can other DBs do index on difference between two sets? = Will=20 PG ever have this, is it even possible? =C2=A0 In my real system the=20 message_property holds other properties for a message also, so I have to LE= FT=20 OUTER JOIN with it to get the properties where they exist when listing=20 messages. The system works by assuming that when an entry in message_proper= ty=20 does not exist for a given message for a given user then the property is eq= ual=20 to "false". =C2=A0 I don't quite see how maintaining a message_properrty en= try for=20 all users, for all messages, is a good idea. That's quite some work to be d= one=20 when adding/removing users etc. =C2=A0 Thanks for having this discussion. = =C2=A0 --=20 Andreas Jospeh Krogh CTO / Partner - Visena AS Mobile: +47 909 56 963=20 andreas@visena.com www.visena.com=20 =C2=A0 ------=_Part_42_1316188218.1398979173229 Content-Type: text/html;charset=UTF-8 Content-Transfer-Encoding: quoted-printable
P=C3=A5 torsdag 01. mai 2014 kl. 23:02:13, skrev Pavel Stehule <pavel.stehule@gma= il.com>:
=C2=A0
=C2=A0
2014-05-01 22:30 GMT+02:00 Andreas Joseph Krogh = <andreas@visena.com>:
P=C3=A5 torsdag 01. mai 2014 kl. 21:53:32, skrev Pavel Stehule <pavel.stehule@gma= il.com>:
=C2=A0
=C2=A0
2014-05-01 21:39 GMT+02:00 Andreas Joseph Krogh = <andreas@visena.com>:
P=C3=A5 torsdag 01. mai 2014 kl. 21:30:39, skrev Pavel Stehule <pavel.stehule@gma= il.com>:
Hello
[snip]
=C2=A0
I had a perfect success on similar use case with descent ordered parti= al index

http://www.postgresql.org/docs/9.3/interactive/sql-c= reateindex.html
=C2=A0
I'm not getting good performance. Are you able to craft an example usi= ng my schema and partial index?
=C2=A0
maybe some like
=C2=A0
CREATE INDEX ON message_property (person_id, message_id) WHERE pr.is_r= ead
=C2=A0
When I am thinking about your schema, it is designed well, but it is n= ot index friendly, so for some fast access you should to hold a cache (tabl= e) of unread messages
=C2=A0
Ah, that's what I was hoping to not having to do. In my system, messag= es arrive all the time and having to update a cache for all new messages fo= r all users seems messy... Seems I could just as well create a message_prop= erty for all users when a new message arrives, so I can INNER JOIN it and g= et good performance. But that table will quickly grow *very* large...
=C2=A0
What you need is a JOIN index, that is not possible in Postgres.
=C2=A0
I afraid so some "ugly" solutions is necessary (when you req= uire extra fast access). You need a index (small index) and it require some= existing set - you cannot do index on the difference two sets.
=C2=A0
I expect so some flag on the relation "message" - like "= ;it should not be not read" can helps little bit - and can be used in = partial index as conditions. Other possibility is some variant of partition= ing - you can divide a messages and users to distinct sets and then you dec= rease a number of possible combinations.
=C2=A0
Just curious:
Is such a JOIN index possible in other DBs, if so - which?
Can other DBs do index on difference between two sets?
Will PG ever have this, is it even possible?
=C2=A0
In my real system the message_property holds other properties for a me= ssage also, so I have to LEFT OUTER JOIN with it to get the properties wher= e they exist when listing messages. The system works by assuming that when = an entry in message_property does not exist for a given message for a given= user then the property is equal to "false".
=C2=A0
I don't quite see how maintaining a message_properrty entry for all us= ers, for all messages, is a good idea. That's quite some work to be done wh= en adding/removing users etc.
=C2=A0
Thanks for having this discussion.
=C2=A0
--
Andrea= s Jospeh Krogh
CTO / Partner<= /span> - Visena AS
Mobile: +47 90= 9 56 963
3D""
=C2=A0
------=_Part_42_1316188218.1398979173229-- ------=_Part_41_158633025.1398979173212 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-ID: iVBORw0KGgoAAAANSUhEUgAAAIUAAAAYCAYAAADUIj6hAAAABHNCSVQICAgIfAhkiAAABzBJREFU aEPtmNFxHDcMhmVP3i1VECpvnjzkVIHWFfhcgVcVRKrAUgWRK/C6Al8H3lTgy0PGbzFdQc4VJP/H ADs43q6kROeJNbOYgQACIAgCWJKng4MZ5gxUGXh0U0b+SIuF9G+EWXj2Q15vbrE/NPtk9uub7Gfd t5mBx1NhqSEo8HshjbEUtlO2yIM9tsx5dZP9rPt2MzDZFAqZE4LGcFjdsg3saQaHt7fY70X949On aS+OZidDBkavD33157L4xaw2os+Mp/DAC10l2XhOCeStj0W5ajrJL8WfCi80Xgf9vVlrhg9yRONe /f7xI2vNsIcM7JwU9o6oGyJrLT8JOA1aX9sKP4wl94bAniukEbo/n7YPmuTETzIab4Y9ZWCrKexd 8M58lxPCvvD6aihfvexbEQoPYO8NQRO0JnddGO6FJYaVsBde7cXj7KRk4LsqDxQzCYeGsJNgaXZe +JXkyGgWINq3Gp+bHNILz+wEYs5qH1eJrouNrpDXtk4O6x1IvtD40GWyJYYdkF0jIbYZlN16xygI gl9smbMDsmFdfAKb23xGB8H/mv3tOJfArs1kukk79DEPdQ5u0g1vCvvqKXIW8mZYW+F3Tg4r8HvZ kQCCLydK8EFMQCe5N8RgL9mRG0SqQFuNvdE6beSs0qPDBuCdg0+gvClso8SbTO4ki7mQzQqBrcMH QPwReg2wW8umEe/+L8T/LEzBGF9nXjzZ46s+ITHvhegW8LL39xm6App7KYL/GE+vcYnFbJiP/0aY hdiCnRC7jaj7eiK2ETInC5PRF6LYkSN0+HabF75WuT6syCyI0YkVGGMvUC33AmfZTDUEj8u6IVhu EhRUJ2U2g6UlugyNb03Hl9obHwlxJbcRzcYjO4QPjVfGFTQav4vrmp7cpMp2qbHnBxWJbisbho2Q XI6C1sIHDUFhH4Hij4UbIev66eANeiwbkA+LBsO36zAHzoXU7AhbqLAXEiOYTXcSdO8VS9L4wN8U BIYhBd6oSUgYMijOkecRuTdQa/YiBXhbXFcnCnI2SrfeBG9NydrLYMhGHa5qB9pQIxlzAE6Okjzx JO5afGe6kmhBicWKQNKuTZ5EW+MjQY8v/9rQ0bhJSJyNGUe/JH1t8h1iMbdSPAvxHYin6YmN9YBX QmTYZZNh14vHhhhal5vtcIrJjmvszPRJdExH3MWHvykoBEc9CoCGWJisOLOGoCORs1FvIMbYA8z3 kwM59oemy6LlWrLxFLmWgiQAfEGd8S+NssbK+EhyGLxUkhhiy717waBqHOJYSEacwBejkFNhjJNj v/gANOdQxPfciP/JVBCK2cOIcg3RRJ+CPrLPNVhhN6F38VKMF3XLlIJrjU5C8gMFxvKDvOcPc4rV NjCn7KM0BV+161V8viSCuJZ8SITGJGEh7IUUlxOFMYUH1kJOCN4Wh+I5pqCuK01k40kSNtnKiKIl UfxAgW5sU5JlS05rtt5YFJF1SWoqHv6BRgQcA4/bdb9WRjmMk3jyUMAbIoyJq9e4cVmgzKt9j5gN J/aYDhk+hhjExwaPcz5PObA5Zd9bvz5UzCTZuZDidu5AchpiKSwPR+TV1bCWqBQ9nCjJ5neivC82 Nr4LeS2j1gw5LUqwBuhGQQU5UwHeSslXkwIynz2U2A2yKDgG7OffwLA3TpGRpk0Tzpj3ZEJXi/GR a6GN0e0NtppChePdcAz1FTS+FN8Kpxoiykk+J8fC5tenjbu9kSqpHLu9jBrhUuhNwVGbpyZrzrl0 HPVD8SXj5EOOD/eDC64VjvYBZLuUbIVAfBN1t/C/SU+cAGtdGo+fVnzycUX5wmn6eCKPmWYJG2E/ ppTsuRCbvUD9fwquksG5GqLVKhzDQ3GrEyLKSXhsiK3T5j9EyxffCFOYi2wUlPyFFDQAhehESDjQ GoX0ho0oj8RPou6TxHJdcT0NTcWkO0AnG7+uXsnHqcas/72wvWF+mSf7N/Watp9kTXoluzeS0fB9 9CcZ/hvhSZTfh99pispZOXL9KrGrARkNUMu9ITbScZWs7xOYNt9pwxSZtYDsX/GE3xTkrXgwAsXm fud08FiTeC+m2/o7ppo+PTS/NBK5ARpDG5YHr++DpmVfreYdiX8mnp+DzPEG9WbqJON0JBenZofs sxBA1gj5NXGvfJu/Qh7HwQh/VDUEyUxCit5hH94QfKkEdu+GwK8BX0hvCF+D67xhjmXQCXMwJCZ+ olK0A1EKRCE4snsh42w8Mv/Zhxw9iD7Cjo7CyQC/KyF6oBfShL4PYgG+CHsYKyZxvxZSZBAgjhIz YDz+AbfD37GtbaoSKzgGt+lKfI/GZtay6vG4VXTpPsh+IcQhOk9I7WYeP5AM3HZ9+DaSGIp9HIuu huC4pCGGx+YD2fcc5tfIAA0h/Et4/jX8zz4fWAasIf4UbR9Y6HO4d8jAnd4U0Y8ageuCB+c+H5R3 CHU2mTMwZ+B/y8DfSMBLLOYXVuEAAAAASUVORK5CYII= ------=_Part_41_158633025.1398979173212-- ------=_Part_40_924192768.1398979173212--