Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1YTc1A-0004H0-Lh for pgsql-sql@arkaria.postgresql.org; Thu, 05 Mar 2015 20:05:12 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1YTc19-0005ye-PS for pgsql-sql@arkaria.postgresql.org; Thu, 05 Mar 2015 20:05:11 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YTc18-0005xk-Gc for pgsql-sql@postgresql.org; Thu, 05 Mar 2015 20:05:10 +0000 Received: from post.visena.com ([46.226.10.50]) by magus.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1YTc13-0006Qa-Gz for pgsql-sql@postgresql.org; Thu, 05 Mar 2015 20:05:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=visena.com; s=20141101.wh; h=Content-Type:MIME-Version:Subject:In-Reply-To:Message-ID:To:From:Date; bh=Kht2epwugk2xIGtwnkkTwh/NVgSNjmgXyGn7R+7azAA=; b=c9evEcUsiCwQqaDuQdubKY/mvMFti6UguggtR0kz5d5K9DKzUa4VdDcRUvVFEcutq3k2sK0N0RgYhSTk1ht0+CjWWkMTdmNIUovYvJGYdaL9Q9Mn76sBs7im5LYmp+zuF2gCGbn3qfkQZcVQipvlFt5ynVDHau7rL0DRfY1a0Zo=; Received: from [10.0.1.10] (helo=tc7-visena.wh.internal.visena.com) by post.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1YTc0w-0003wh-Ua for pgsql-sql@postgresql.org; Thu, 05 Mar 2015 21:05:04 +0100 Received: from localhost ([127.0.0.1] helo=tc7-visena.wh.internal.visena.com) by tc7-visena.wh.internal.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1YTc0x-0002Xq-GZ for pgsql-sql@postgresql.org; Thu, 05 Mar 2015 21:04:59 +0100 Date: Thu, 5 Mar 2015 21:04:59 +0100 (CET) From: Andreas Joseph Krogh To: pgsql-sql@postgresql.org Message-ID: In-Reply-To: <54F8B5A0.4020607@aklaver.com> Subject: Re: Schema for caching message-count in folders using triggers MIME-Version: 1.0 X-Mailer: Visena Mail 1.9.0-SNAPSHOT X-Spam-Score: -1.0 X-Spam-Report: SpamAssasin (score=-1.0, required 5.0 ALL_TRUSTED=-1, HTML_MESSAGE=0.001) X-Pg-Spam-Score: -2.0 (--) Content-Type: multipart/related; boundary="----=_Part_21_2052206080.1425585899377" List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-sql Precedence: bulk Sender: pgsql-sql-owner@postgresql.org ------=_Part_20_960072607.1425585899377 Content-Type: multipart/related; boundary="----=_Part_21_2052206080.1425585899377" ------=_Part_21_2052206080.1425585899377 Content-Type: multipart/alternative; boundary="----=_Part_22_877179209.1425585899393" ------=_Part_22_877179209.1425585899393 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable P=C3=A5 torsdag 05. mars 2015 kl. 20:59:28, skrev Adrian Klaver < adrian.klaver@aklaver.com >: On 03/05/201= 5=20 11:45 AM, Andreas Joseph Krogh wrote: > Hi all. > I'm facing a problem with my current schema for email where folders > start containing several 100K of messages and count(*) in them taks > noticeable time. This schema is accessible from IMAP and a web-app so > lots of queries of the type "list folders with message count" are perfor= med. > So, I'm toying with this idea of caching the message-count in the > folder-table itself. > I currently have this: > > CREATE or replace FUNCTIONcount_increment_tf()RETURNS TRIGGER AS$_$ > BEGIN > UPDATE folder SETmessage_count=C2=A0 =3Dmessage_count=C2=A0 + 1 WHEREid= =C2=A0 =3DNEW.folder_id; > RETURNNEW; > END $_$LANGUAGE'plpgsql'; > > CREATE or replace FUNCTIONcount_decrement_tf()RETURNS TRIGGER AS$_$ > BEGIN >=C2=A0 =C2=A0 =C2=A0 UPDATE folder SETmessage_count=C2=A0 =3Dmessage_coun= t=C2=A0 - 1 WHEREid=C2=A0=20 =3DOLD.folder_id; > RETURNOLD; > END $_$LANGUAGE'plpgsql'; > > CREATE or replace FUNCTIONcount_update_tf()RETURNS TRIGGER AS$_$ > BEGIN >=C2=A0 =C2=A0 =C2=A0 UPDATE folder SETmessage_count=C2=A0 =3Dmessage_coun= t=C2=A0 - 1 WHEREid=C2=A0=20 =3DOLD.folder_id; >=C2=A0 =C2=A0 =C2=A0 UPDATE folder SETmessage_count=C2=A0 =3Dmessage_coun= t=C2=A0 + 1 WHEREid=C2=A0=20 =3DNEW.folder_id; > RETURNNEW; > END $_$LANGUAGE'plpgsql'; > > CREATE TRIGGERincrement_folder_msg_tAFTER INSERT ON message FOR EACH ROW= =20 EXECUTE PROCEDUREcount_increment_tf(); > CREATE TRIGGERdecrement_folder_msg_tAFTER DELETE ON message FOR EACH ROW= =20 EXECUTE PROCEDUREcount_decrement_tf(); > CREATE TRIGGERupdate_folder_msg_tAFTER UPDATE ON message FOR EACH ROW=20 EXECUTE PROCEDUREcount_update_tf(); > > The problem with this is locking (waiting for another TX to commit when > updating the same folder) and deadlock issues when trying to > simultaneously insert/delete/update messages=C2=A0 in a folder. > Does anyone have any better ideas for safely caching the message-count > in each folder without locking and deadlock issues? How accurate does this have to be? Not exactly following what is folder? Is it a table that contains the messages? A top of the head idea would be to use sequences. Create a sequence for each folder starting at current count and then use nextval, setval to change the value: http://www.postgresql.org/docs/9.4/interactive/functions-sequence.html It is not transactional, so it would probably not be spot on, which is why I asked about accuracy earlier. =C2=A0 Yes, 'folder' is a table which = contains=20 'message': =C2=A0 create table folder( id serial PRIMARY KEY, name varchar = not null=20 unique, message_count integer not null default 0 ); create table message( i= d=20 serial PRIMARY KEY, folder_id INTEGER NOT NULL REFERENCES folder(id), messa= ge=20 varchar not null); =C2=A0 The count has to be exact, no estimate from EXPLA= IN or=20 such... =C2=A0 -- Andreas Joseph Krogh CTO / Partner - Visena AS Mobile: +4= 7 909 56=20 963 andreas@visena.com www.visena.com=20 =C2=A0 ------=_Part_22_877179209.1425585899393 Content-Type: text/html;charset=UTF-8 Content-Transfer-Encoding: quoted-printable
P=C3=A5 torsdag 05. mars 2015 kl. 20:59:28, skrev Adrian Klaver <adrian.klaver@aklaver.com>= ;:
On = 03/05/2015 11:45 AM, Andreas Joseph Krogh wrote:
> Hi all.
> I'm facing a problem with my current schema for email where folders > start containing several 100K of messages and count(*) in them taks > noticeable time. This schema is accessible from IMAP and a web-app so<= br> > lots of queries of the type "list folders with message count"= ; are performed.
> So, I'm toying with this idea of caching the message-count in the
> folder-table itself.
> I currently have this:
>
> CREATE or replace FUNCTIONcount_increment_tf()RETURNS TRIGGER AS$_$ > BEGIN
> UPDATE folder SETmessage_count=C2=A0 =3Dmessage_count=C2=A0 + 1 WHEREi= d=C2=A0 =3DNEW.folder_id;
> RETURNNEW;
> END $_$LANGUAGE'plpgsql';
>
> CREATE or replace FUNCTIONcount_decrement_tf()RETURNS TRIGGER AS$_$ > BEGIN
>=C2=A0 =C2=A0 =C2=A0 UPDATE folder SETmessage_count=C2=A0 =3Dmessage_co= unt=C2=A0 - 1 WHEREid=C2=A0 =3DOLD.folder_id;
> RETURNOLD;
> END $_$LANGUAGE'plpgsql';
>
> CREATE or replace FUNCTIONcount_update_tf()RETURNS TRIGGER AS$_$
> BEGIN
>=C2=A0 =C2=A0 =C2=A0 UPDATE folder SETmessage_count=C2=A0 =3Dmessage_co= unt=C2=A0 - 1 WHEREid=C2=A0 =3DOLD.folder_id;
>=C2=A0 =C2=A0 =C2=A0 UPDATE folder SETmessage_count=C2=A0 =3Dmessage_co= unt=C2=A0 + 1 WHEREid=C2=A0 =3DNEW.folder_id;
> RETURNNEW;
> END $_$LANGUAGE'plpgsql';
>
> CREATE TRIGGERincrement_folder_msg_tAFTER INSERT ON message FOR EACH R= OW EXECUTE PROCEDUREcount_increment_tf();
> CREATE TRIGGERdecrement_folder_msg_tAFTER DELETE ON message FOR EACH R= OW EXECUTE PROCEDUREcount_decrement_tf();
> CREATE TRIGGERupdate_folder_msg_tAFTER UPDATE ON message FOR EACH ROW = EXECUTE PROCEDUREcount_update_tf();
>
> The problem with this is locking (waiting for another TX to commit whe= n
> updating the same folder) and deadlock issues when trying to
> simultaneously insert/delete/update messages=C2=A0 in a folder.
> Does anyone have any better ideas for safely caching the message-count=
> in each folder without locking and deadlock issues?

How accurate does this have to be?

Not exactly following what is folder?
Is it a table that contains the messages?

A top of the head idea would be to use sequences. Create a sequence for
each folder starting at current count and then use nextval, setval to
change the value:

http://www.postgresql.org/docs/9.4/interactive/functions-sequence.html

It is not transactional, so it would probably not be spot on, which is
why I asked about accuracy earlier.
=C2=A0
Yes, 'folder' is a table which contains 'message':
=C2=A0
create table folder(
    id =
serial PRIMARY KE=
Y,
    name varchar not nul=
l unique,
    message_co=
unt intege=
r not null default 0
);

create table mess=
age(
    id =
serial PRIMARY KE=
Y,
    folder_id =
INTEGER NO=
T NULL REFERENCES folder(id),
    message varchar not =
null
);
=C2=A0
The count has to be exact, no estimate from EXPLAIN or such...
=C2=A0
--
Andrea= s Joseph Krogh
CTO / Partner<= /span> - Visena AS
Mobile: +47 90= 9 56 963
=3D""
=C2=A0
------=_Part_22_877179209.1425585899393-- ------=_Part_21_2052206080.1425585899377 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-ID: iVBORw0KGgoAAAANSUhEUgAAAIUAAAAYCAYAAADUIj6hAAAABHNCSVQICAgIfAhkiAAABzBJREFU aEPtmNFxHDcMhmVP3i1VECpvnjzkVIHWFfhcgVcVRKrAUgWRK/C6Al8H3lTgy0PGbzFdQc4VJP/H ADs43q6kROeJNbOYgQACIAgCWJKng4MZ5gxUGXh0U0b+SIuF9G+EWXj2Q15vbrE/NPtk9uub7Gfd t5mBx1NhqSEo8HshjbEUtlO2yIM9tsx5dZP9rPt2MzDZFAqZE4LGcFjdsg3saQaHt7fY70X949On aS+OZidDBkavD33157L4xaw2os+Mp/DAC10l2XhOCeStj0W5ajrJL8WfCi80Xgf9vVlrhg9yRONe /f7xI2vNsIcM7JwU9o6oGyJrLT8JOA1aX9sKP4wl94bAniukEbo/n7YPmuTETzIab4Y9ZWCrKexd 8M58lxPCvvD6aihfvexbEQoPYO8NQRO0JnddGO6FJYaVsBde7cXj7KRk4LsqDxQzCYeGsJNgaXZe +JXkyGgWINq3Gp+bHNILz+wEYs5qH1eJrouNrpDXtk4O6x1IvtD40GWyJYYdkF0jIbYZlN16xygI gl9smbMDsmFdfAKb23xGB8H/mv3tOJfArs1kukk79DEPdQ5u0g1vCvvqKXIW8mZYW+F3Tg4r8HvZ kQCCLydK8EFMQCe5N8RgL9mRG0SqQFuNvdE6beSs0qPDBuCdg0+gvClso8SbTO4ki7mQzQqBrcMH QPwReg2wW8umEe/+L8T/LEzBGF9nXjzZ46s+ITHvhegW8LL39xm6App7KYL/GE+vcYnFbJiP/0aY hdiCnRC7jaj7eiK2ETInC5PRF6LYkSN0+HabF75WuT6syCyI0YkVGGMvUC33AmfZTDUEj8u6IVhu EhRUJ2U2g6UlugyNb03Hl9obHwlxJbcRzcYjO4QPjVfGFTQav4vrmp7cpMp2qbHnBxWJbisbho2Q XI6C1sIHDUFhH4Hij4UbIev66eANeiwbkA+LBsO36zAHzoXU7AhbqLAXEiOYTXcSdO8VS9L4wN8U BIYhBd6oSUgYMijOkecRuTdQa/YiBXhbXFcnCnI2SrfeBG9NydrLYMhGHa5qB9pQIxlzAE6Okjzx JO5afGe6kmhBicWKQNKuTZ5EW+MjQY8v/9rQ0bhJSJyNGUe/JH1t8h1iMbdSPAvxHYin6YmN9YBX QmTYZZNh14vHhhhal5vtcIrJjmvszPRJdExH3MWHvykoBEc9CoCGWJisOLOGoCORs1FvIMbYA8z3 kwM59oemy6LlWrLxFLmWgiQAfEGd8S+NssbK+EhyGLxUkhhiy717waBqHOJYSEacwBejkFNhjJNj v/gANOdQxPfciP/JVBCK2cOIcg3RRJ+CPrLPNVhhN6F38VKMF3XLlIJrjU5C8gMFxvKDvOcPc4rV NjCn7KM0BV+161V8viSCuJZ8SITGJGEh7IUUlxOFMYUH1kJOCN4Wh+I5pqCuK01k40kSNtnKiKIl UfxAgW5sU5JlS05rtt5YFJF1SWoqHv6BRgQcA4/bdb9WRjmMk3jyUMAbIoyJq9e4cVmgzKt9j5gN J/aYDhk+hhjExwaPcz5PObA5Zd9bvz5UzCTZuZDidu5AchpiKSwPR+TV1bCWqBQ9nCjJ5neivC82 Nr4LeS2j1gw5LUqwBuhGQQU5UwHeSslXkwIynz2U2A2yKDgG7OffwLA3TpGRpk0Tzpj3ZEJXi/GR a6GN0e0NtppChePdcAz1FTS+FN8Kpxoiykk+J8fC5tenjbu9kSqpHLu9jBrhUuhNwVGbpyZrzrl0 HPVD8SXj5EOOD/eDC64VjvYBZLuUbIVAfBN1t/C/SU+cAGtdGo+fVnzycUX5wmn6eCKPmWYJG2E/ ppTsuRCbvUD9fwquksG5GqLVKhzDQ3GrEyLKSXhsiK3T5j9EyxffCFOYi2wUlPyFFDQAhehESDjQ GoX0ho0oj8RPou6TxHJdcT0NTcWkO0AnG7+uXsnHqcas/72wvWF+mSf7N/Watp9kTXoluzeS0fB9 9CcZ/hvhSZTfh99pispZOXL9KrGrARkNUMu9ITbScZWs7xOYNt9pwxSZtYDsX/GE3xTkrXgwAsXm fud08FiTeC+m2/o7ppo+PTS/NBK5ARpDG5YHr++DpmVfreYdiX8mnp+DzPEG9WbqJON0JBenZofs sxBA1gj5NXGvfJu/Qh7HwQh/VDUEyUxCit5hH94QfKkEdu+GwK8BX0hvCF+D67xhjmXQCXMwJCZ+ olK0A1EKRCE4snsh42w8Mv/Zhxw9iD7Cjo7CyQC/KyF6oBfShL4PYgG+CHsYKyZxvxZSZBAgjhIz YDz+AbfD37GtbaoSKzgGt+lKfI/GZtay6vG4VXTpPsh+IcQhOk9I7WYeP5AM3HZ9+DaSGIp9HIuu huC4pCGGx+YD2fcc5tfIAA0h/Et4/jX8zz4fWAasIf4UbR9Y6HO4d8jAnd4U0Y8ageuCB+c+H5R3 CHU2mTMwZ+B/y8DfSMBLLOYXVuEAAAAASUVORK5CYII= ------=_Part_21_2052206080.1425585899377-- ------=_Part_20_960072607.1425585899377--