Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1YTbiX-0003Qa-0Q for pgsql-sql@arkaria.postgresql.org; Thu, 05 Mar 2015 19:45:57 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1YTbiW-0002SX-9h for pgsql-sql@arkaria.postgresql.org; Thu, 05 Mar 2015 19:45:56 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YTbiV-0002SR-Eu for pgsql-sql@postgresql.org; Thu, 05 Mar 2015 19:45:55 +0000 Received: from post.visena.com ([46.226.10.50]) by magus.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1YTbiD-000650-Kn for pgsql-sql@postgresql.org; Thu, 05 Mar 2015 19:45:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=visena.com; s=20141101.wh; h=Content-Type:MIME-Version:Subject:Message-ID:To:From:Date; bh=ohUo75fqucyma/zexEa1SL5Rfkbhu2yki6r7YDWHEko=; b=W0cfOnLsGO5vfApjkM2q2JQ0jJRyo0iiCkrw8imkKyk8zPJc1UdPHMl6Y5KXeJB7K3g7ToDi+b/ADka9q8rwYrr+1sm4LrHnm4FP9OqvV1FHY3mkhCcf5QvjKAIoQwnNMnJjPjLHRnPGdOW4MEGCm1h0ngZhiv5ERkvX/LZ1OmA=; Received: from [10.0.1.10] (helo=tc7-visena.wh.internal.visena.com) by post.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1YTbi8-0001yv-My; Thu, 05 Mar 2015 20:45:34 +0100 Received: from localhost ([127.0.0.1] helo=tc7-visena.wh.internal.visena.com) by tc7-visena.wh.internal.visena.com with esmtp (Exim 4.82) (envelope-from ) id 1YTbi9-0001do-0j; Thu, 05 Mar 2015 20:45:33 +0100 Date: Thu, 5 Mar 2015 20:45:32 +0100 (CET) From: Andreas Joseph Krogh To: "pgsql-sql@postgresql.org" Message-ID: Subject: Schema for caching message-count in folders using triggers MIME-Version: 1.0 X-Mailer: Visena Mail 1.9.0-SNAPSHOT X-Spam-Score: -1.0 X-Spam-Report: SpamAssasin (score=-1.0, required 5.0 ALL_TRUSTED=-1, HTML_MESSAGE=0.001) X-Pg-Spam-Score: -2.0 (--) Content-Type: multipart/related; boundary="----=_Part_16_725842074.1425584732868" List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-sql Precedence: bulk Sender: pgsql-sql-owner@postgresql.org ------=_Part_15_1490037815.1425584732868 Content-Type: multipart/related; boundary="----=_Part_16_725842074.1425584732868" ------=_Part_16_725842074.1425584732868 Content-Type: multipart/alternative; boundary="----=_Part_17_95487984.1425584732915" ------=_Part_17_95487984.1425584732915 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi all. =C2=A0 I'm facing a problem with my current schema for email where = folders=20 start containing several 100K of messages and count(*) in them taks noticea= ble=20 time. This schema is accessible from IMAP and a web-app so lots of queries = of=20 the type "list folders with message count" are performed. =C2=A0 So, I'm to= ying with=20 this idea of caching the message-count in the folder-table itself. =C2=A0 I= =20 currently have this: =C2=A0 CREATE or replace FUNCTION count_increment_tf()= RETURNS=20 TRIGGER AS$_$ BEGIN UPDATE folder SET message_count =3D message_count + 1 W= HERE id =3DNEW.folder_id; RETURN NEW; END $_$ LANGUAGE 'plpgsql'; CREATE or replac= e=20 FUNCTIONcount_decrement_tf() RETURNS TRIGGER AS $_$ BEGIN UPDATE folder SE= T=20 message_count =3D message_count - 1 WHERE id =3D OLD.folder_id; RETURN OLD;= END $_$=20 LANGUAGE'plpgsql'; CREATE or replace FUNCTION count_update_tf() RETURNS TRI= GGER=20 AS$_$ BEGIN UPDATE folder SET message_count =3D message_count - 1 WHERE id= =3D OLD. folder_id; UPDATE folder SET message_count =3D message_count + 1 WHERE id = =3D NEW. folder_id; RETURN NEW; END $_$ LANGUAGE 'plpgsql'; CREATE TRIGGER=20 increment_folder_msg_tAFTER INSERT ON message FOR EACH ROW EXECUTE PROCEDUR= E=20 count_increment_tf(); CREATE TRIGGER decrement_folder_msg_t AFTER DELETE ON= =20 message FOR EACH ROW EXECUTE PROCEDUREcount_decrement_tf(); CREATE TRIGGER= =20 update_folder_msg_tAFTER UPDATE ON message FOR EACH ROW EXECUTE PROCEDURE= =20 count_update_tf(); =C2=A0 The problem with this is locking (waiting for ano= ther TX=20 to commit when updating the same folder) and deadlock issues when trying to= =20 simultaneously insert/delete/update messages=C2=A0 in a folder. =C2=A0 Does= anyone have=20 any better ideas for safely caching the message-count in each folder withou= t=20 locking and deadlock issues? =C2=A0 Thanks. =C2=A0 -- Andreas Joseph Krogh = CTO / Partner=20 - Visena AS Mobile: +47 909 56 963 andreas@visena.com=20 www.visena.com =20 ------=_Part_17_95487984.1425584732915 Content-Type: text/html;charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi all.
=C2=A0
I'm facing a problem with my current schema for email where folders st= art containing several 100K of messages and count(*) in them taks noticeabl= e time. This schema is accessible from IMAP and a web-app so lots of querie= s of the type "list folders with message count" are performed.
=C2=A0
So, I'm toying with this idea of caching the message-count in the fold= er-table itself.
=C2=A0
I currently have this:
=C2=A0
CREATE or replace FUNC=
TION count_inc=
rement_tf() RETURNS TRIGGER AS $_$
BEGIN
UPDATE fol=
der SET message_count =3D  + 1 WHERE id =3D NEW=
.folder_id=
;
RETURN NEW;
END $_$ LANGUAGE 'plpgsql';

CREATE or replace=
 FUNCTION count_decrement_tf() RETURNS TRI=
GGER AS $_=
$
BEGIN
    UPDATE=
 folder SET message_count =3D message_count - 1 WHERE id =3D OLD.folder_id;
RETURN OLD;
END $_$ LANGUAGE 'plpgsql';

CREATE or replace=
 FUNCTION count_update_tf=
() RETURNS TRIGGE=
R AS $_$
BEGIN
    UPDATE=
 folder SET message_count =3D message_count - 1 WHERE id =3D OLD.folder_id;
    UPDATE=
 folder SET message_count =3D message_count + 1 WHERE id =3D NEW.folder_id;
RETURN NEW;
END $_$ LANGUAGE 'plpgsql';

CREATE TRIGGER increment_folder_msg_t AFTER INSERT ON message FOR EACH ROW EXECUTE PROCEDURE count_increment_tf();
CREATE TRIGGER decrement_folder_msg_t AFTER DELETE ON message FOR EACH ROW EXECUTE PROCEDURE count_decrement_tf();
CREATE TRIGGER update_folder_msg_t AFTER=
 UPDATE ON message FOR EACH ROW EXECUTE PROCEDURE count_update_tf();
=C2=A0
The problem with this is locking (waiting for another TX to commit whe= n updating the same folder) and deadlock issues when trying to simultaneous= ly insert/delete/update messages=C2=A0 in a folder.
=C2=A0
Does anyone have any better ideas for safely caching the message-count= in each folder without locking and deadlock issues?
=C2=A0
Thanks.
=C2=A0
--
Andrea= s Joseph Krogh
CTO / Partner<= /span> - Visena AS
Mobile: +47 90= 9 56 963
=3D""
------=_Part_17_95487984.1425584732915-- ------=_Part_16_725842074.1425584732868 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-ID: iVBORw0KGgoAAAANSUhEUgAAAIUAAAAYCAYAAADUIj6hAAAABHNCSVQICAgIfAhkiAAABzBJREFU aEPtmNFxHDcMhmVP3i1VECpvnjzkVIHWFfhcgVcVRKrAUgWRK/C6Al8H3lTgy0PGbzFdQc4VJP/H ADs43q6kROeJNbOYgQACIAgCWJKng4MZ5gxUGXh0U0b+SIuF9G+EWXj2Q15vbrE/NPtk9uub7Gfd t5mBx1NhqSEo8HshjbEUtlO2yIM9tsx5dZP9rPt2MzDZFAqZE4LGcFjdsg3saQaHt7fY70X949On aS+OZidDBkavD33157L4xaw2os+Mp/DAC10l2XhOCeStj0W5ajrJL8WfCi80Xgf9vVlrhg9yRONe /f7xI2vNsIcM7JwU9o6oGyJrLT8JOA1aX9sKP4wl94bAniukEbo/n7YPmuTETzIab4Y9ZWCrKexd 8M58lxPCvvD6aihfvexbEQoPYO8NQRO0JnddGO6FJYaVsBde7cXj7KRk4LsqDxQzCYeGsJNgaXZe +JXkyGgWINq3Gp+bHNILz+wEYs5qH1eJrouNrpDXtk4O6x1IvtD40GWyJYYdkF0jIbYZlN16xygI gl9smbMDsmFdfAKb23xGB8H/mv3tOJfArs1kukk79DEPdQ5u0g1vCvvqKXIW8mZYW+F3Tg4r8HvZ kQCCLydK8EFMQCe5N8RgL9mRG0SqQFuNvdE6beSs0qPDBuCdg0+gvClso8SbTO4ki7mQzQqBrcMH QPwReg2wW8umEe/+L8T/LEzBGF9nXjzZ46s+ITHvhegW8LL39xm6App7KYL/GE+vcYnFbJiP/0aY hdiCnRC7jaj7eiK2ETInC5PRF6LYkSN0+HabF75WuT6syCyI0YkVGGMvUC33AmfZTDUEj8u6IVhu EhRUJ2U2g6UlugyNb03Hl9obHwlxJbcRzcYjO4QPjVfGFTQav4vrmp7cpMp2qbHnBxWJbisbho2Q XI6C1sIHDUFhH4Hij4UbIev66eANeiwbkA+LBsO36zAHzoXU7AhbqLAXEiOYTXcSdO8VS9L4wN8U BIYhBd6oSUgYMijOkecRuTdQa/YiBXhbXFcnCnI2SrfeBG9NydrLYMhGHa5qB9pQIxlzAE6Okjzx JO5afGe6kmhBicWKQNKuTZ5EW+MjQY8v/9rQ0bhJSJyNGUe/JH1t8h1iMbdSPAvxHYin6YmN9YBX QmTYZZNh14vHhhhal5vtcIrJjmvszPRJdExH3MWHvykoBEc9CoCGWJisOLOGoCORs1FvIMbYA8z3 kwM59oemy6LlWrLxFLmWgiQAfEGd8S+NssbK+EhyGLxUkhhiy717waBqHOJYSEacwBejkFNhjJNj v/gANOdQxPfciP/JVBCK2cOIcg3RRJ+CPrLPNVhhN6F38VKMF3XLlIJrjU5C8gMFxvKDvOcPc4rV NjCn7KM0BV+161V8viSCuJZ8SITGJGEh7IUUlxOFMYUH1kJOCN4Wh+I5pqCuK01k40kSNtnKiKIl UfxAgW5sU5JlS05rtt5YFJF1SWoqHv6BRgQcA4/bdb9WRjmMk3jyUMAbIoyJq9e4cVmgzKt9j5gN J/aYDhk+hhjExwaPcz5PObA5Zd9bvz5UzCTZuZDidu5AchpiKSwPR+TV1bCWqBQ9nCjJ5neivC82 Nr4LeS2j1gw5LUqwBuhGQQU5UwHeSslXkwIynz2U2A2yKDgG7OffwLA3TpGRpk0Tzpj3ZEJXi/GR a6GN0e0NtppChePdcAz1FTS+FN8Kpxoiykk+J8fC5tenjbu9kSqpHLu9jBrhUuhNwVGbpyZrzrl0 HPVD8SXj5EOOD/eDC64VjvYBZLuUbIVAfBN1t/C/SU+cAGtdGo+fVnzycUX5wmn6eCKPmWYJG2E/ ppTsuRCbvUD9fwquksG5GqLVKhzDQ3GrEyLKSXhsiK3T5j9EyxffCFOYi2wUlPyFFDQAhehESDjQ GoX0ho0oj8RPou6TxHJdcT0NTcWkO0AnG7+uXsnHqcas/72wvWF+mSf7N/Watp9kTXoluzeS0fB9 9CcZ/hvhSZTfh99pispZOXL9KrGrARkNUMu9ITbScZWs7xOYNt9pwxSZtYDsX/GE3xTkrXgwAsXm fud08FiTeC+m2/o7ppo+PTS/NBK5ARpDG5YHr++DpmVfreYdiX8mnp+DzPEG9WbqJON0JBenZofs sxBA1gj5NXGvfJu/Qh7HwQh/VDUEyUxCit5hH94QfKkEdu+GwK8BX0hvCF+D67xhjmXQCXMwJCZ+ olK0A1EKRCE4snsh42w8Mv/Zhxw9iD7Cjo7CyQC/KyF6oBfShL4PYgG+CHsYKyZxvxZSZBAgjhIz YDz+AbfD37GtbaoSKzgGt+lKfI/GZtay6vG4VXTpPsh+IcQhOk9I7WYeP5AM3HZ9+DaSGIp9HIuu huC4pCGGx+YD2fcc5tfIAA0h/Et4/jX8zz4fWAasIf4UbR9Y6HO4d8jAnd4U0Y8ageuCB+c+H5R3 CHU2mTMwZ+B/y8DfSMBLLOYXVuEAAAAASUVORK5CYII= ------=_Part_16_725842074.1425584732868-- ------=_Part_15_1490037815.1425584732868--