Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mARTK-00017n-Ou for pgsql-hackers@arkaria.postgresql.org; Mon, 02 Aug 2021 06:30:47 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mARTJ-0000ya-OJ for pgsql-hackers@arkaria.postgresql.org; Mon, 02 Aug 2021 06:30:45 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mARTJ-0000xB-6X for pgsql-hackers@lists.postgresql.org; Mon, 02 Aug 2021 06:30:45 +0000 Received: from mx0b-0050f201.pphosted.com ([148.163.141.54]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mARTA-0004dV-Eb for pgsql-hackers@lists.postgresql.org; Mon, 02 Aug 2021 06:30:44 +0000 Received: from pps.filterd (m0203370.ppops.net [127.0.0.1]) by mx0b-0050f201.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 171M0ao7025164 for ; Mon, 2 Aug 2021 15:30:34 +0900 Received: from sraihe.sra.co.jp (sraihe.sra.co.jp [221.255.117.38]) by mx0b-0050f201.pphosted.com with ESMTP id 3a5rpygvn7-1 for ; Mon, 02 Aug 2021 15:30:29 +0900 Received: from srascb.sra.co.jp (srascb [133.137.8.65]) by sraihe.sra.co.jp (Postfix) with ESMTP id 68AD12A6863 for ; Mon, 2 Aug 2021 15:30:28 +0900 (JST) Received: from sranhm.sra.co.jp (osspc25 [133.137.174.97]) by srascb.sra.co.jp (Postfix) with ESMTP id 4BC87580295 for ; Mon, 2 Aug 2021 15:30:28 +0900 (JST) Received: from yugon-CFSV7-1 (sraihb-hub.sra.co.jp [133.137.8.6]) by sranhm.sra.co.jp (Postfix) with SMTP id 304E9A16DD; Mon, 2 Aug 2021 15:30:27 +0900 (JST) Date: Mon, 2 Aug 2021 15:28:34 +0900 From: Yugo NAGATA To: Yugo NAGATA Cc: vignesh C , Tom Lane , Andrew Dunstan , Andy Fan , PostgreSQL Hackers , Tatsuo Ishii , Thomas Munro Subject: Re: Implementing Incremental View Maintenance Message-Id: <20210802152834.ecbaba6e17d1957547c3a55d@sraoss.co.jp> In-Reply-To: <20210719092430.54f991cb313323eaafb56311@sraoss.co.jp> References: <20201001140627.1f8fafe595f545e026d110c6@sraoss.co.jp> <20210122165954.d6386b0e3a530a76890b8fff@sraoss.co.jp> <20210216103155.762054c9d38727c782f0c11b@sraoss.co.jp> <20210219110126.7152beb4fa4fea5b1133eef3@sraoss.co.jp> <45b2760d-7ee9-0f46-50bf-b15fd440b52f@dunslane.net> <20210309092049.db22add3e2863c2140290a90@sraoss.co.jp> <20210309172750.8fd722a08ed3a2423bb25ccb@sraoss.co.jp> <20210407182537.5fdb444f6d4510a60524baab@sraoss.co.jp> <345103.1618868431@sss.pgh.pa.us> <20210420095134.8463a06e517f207886e33028@sraoss.co.jp> <20210426154621.64e91e0f113c7fe1533a0306@sraoss.co.jp> <20210426160348.eb879618d865166e459f9e58@sraoss.co.jp> <20210507141416.dddc25efb2def1d759c590b6@sraoss.co.jp> <20210517133646.04c3d9ed232af22c0f290ebc@sraoss.co.jp> <20210719092430.54f991cb313323eaafb56311@sraoss.co.jp> X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.32; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu" X-Proofpoint-ORIG-GUID: EzpmtBPtPvVPUIXrIt2_p14S_x8imzYj X-Proofpoint-GUID: EzpmtBPtPvVPUIXrIt2_p14S_x8imzYj X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10063 signatures=668682 X-Proofpoint-Spam-Details: rule=spam_low_notspam policy=spam_low score=0 lowpriorityscore=0 suspectscore=0 bulkscore=0 mlxscore=0 phishscore=0 adultscore=0 impostorscore=0 malwarescore=0 priorityscore=1501 clxscore=1034 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108020045 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Hi hackers, On Mon, 19 Jul 2021 09:24:30 +0900 Yugo NAGATA wrote: > On Wed, 14 Jul 2021 21:22:37 +0530 > vignesh C wrote: > > The patch does not apply on Head anymore, could you rebase and post a > > patch. I'm changing the status to "Waiting for Author". > > Ok. I'll update the patch in a few days. Attached is the latest patch set to add support for Incremental Materialized View Maintenance (IVM) The patches are rebased to the master and also revised with some code cleaning. IVM is a way to make materialized views up-to-date in which only incremental changes are computed and applied on views rather than recomputing the contents from scratch as REFRESH MATERIALIZED VIEW does. IVM can update materialized views more efficiently than recomputation when only small part of the view need updates. The patch set implements a feature so that materialized views could be updated automatically and immediately when a base table is modified. Currently, our IVM implementation supports views which could contain tuple duplicates whose definition includes: - inner and outer joins including self-join - DISTINCT - some built-in aggregate functions (count, sum, agv, min, and max) - a part of subqueries -- simple subqueries in FROM clause -- EXISTS subqueries in WHERE clause - CTEs We hope the IVM feature would be adopted into pg15. However, the size of patch set has grown too large through supporting above features. Therefore, I think it is better to consider only a part of these features for the first release. Especially, I would like propose the following features for pg15. - inner joins including self-join - DISTINCT and views with tuple duplicates - some built-in aggregate functions (count, sum, agv, min, and max) By omitting outer-join, sub-queries, and CTE features, the patch size becomes less than half. I hope this will make a bit easer to review the IVM patch set. Here is a list of separated patches. - 0001: Add a new syntax: CREATE INCREMENTAL MATERIALIZED VIEW - 0002: Add a new column relisivm to pg_class - 0003: Add new deptype option 'm' in pg_depend - 0004: Change trigger.c to allow to prolong life span of tupestores containing Transition Tables generated via AFTER trigger - 0005: Add IVM supprot for pg_dump - 0006: Add IVM support for psql - 0007: Add the basic IVM future: This supports inner joins, DISTINCT, and tuple duplicates. - 0008: Add aggregates (count, sum, avg, min, max) support for IVM - 0009: Add regression tests for IVM - 0010: Add documentation for IVM We could split the patch furthermore if this would make reviews much easer. For example, I think 0007 could be split into the more basic part and the part for handling tuple duplicates. Moreover, 0008 could be split into "min/max" and other aggregates because handling min/max is a bit more complicated than others. I also attached IVM_extra.tar.gz that contains patches for sub-quereis, outer-join, CTE support, just for your information. Regards, Yugo Nagata -- Yugo NAGATA -- Yugo NAGATA --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: application/gzip; name="IVM_extra.tar.gz" Content-Disposition: attachment; filename="IVM_extra.tar.gz" Content-Transfer-Encoding: base64 H4sIAIKLB2EAA+w8/XPaSLL5VfwVs6l6OcDCRmCwSS65JTbJcufYWUyyubq6UslisLUWEisJJ77d /O/X3fOhkRDYzovfq6v3XAkfo5memf7unh7GH9+5/EuWeHtPHu2vDX8HBz18dw56bfNd/T1x9h1o bLf7vYMnbafjdHtPWO/xlpT/rdLMSxh7cru6jFvR5n53Pf8P/Rtr+t90ui0gjNMazmatdHXx24on AU/h43IZJ1kriFrQeXfpZf7Vw+ZAAvf7+xvo3+/u73cF/fsHB/1u/0m74/QdoH/7cbZc/Ps/Tv83 Sbxgs0Pf680vvMODwYBfdJwOd7qzrtPf7znt9kV71p3Nne7BYY+9iyN2zpfMOWDt9nP6xzpAzRqC ec7+Djhip96ll3nszxG9/5gmXpymu368++vyVe3Yy/hzNr1a2awzYH9dhTC847CO83z/4Hn7gO20 B+127Xx18Sv3s+fsH++H06OfGLAmc5w9p/dPBszJcuZkkjlZEDFgzlrtKE4SHsIkhV6rNIgu2ejT +Hx6zryEq2F8tlurDSP1RI64RWi//DSaMD/0VilnQcoS/jkJsoxHLIvZyXA6mgxPamb/N5Ozd7K/ zbxohsv5E1smsc/TlPlexK6gNeQsuwrSWhhcc+alzGNRnCy8kP0aB9EuG4ZpbLOrYDaDifw4XC2i lD0NbhYgokGapa4fr6LM/eS6T2u4DQ86znBF/hX3r1kwVzvx42gWZAFQC9aeelmQzgPYLPvlCgB7 7MJLeS3zLmA50DPzggjABBoP+bYX8SzAkTYtm9H07MYLV7yGGH/37iP2Wi1nhPEgSjPuzVg8Z7hm 0RswPI5wpbQem6XBYhlykzpF7BF9PEBEgUgJT7Mk8BFEyp7XWsy7vEw4MBhn81Uk23FkFGf5QEDN Jc+ueMI+B9mV3N4ujI4AXJFF1sZitzVsIh1xjXEU3gqYw9NjNvqyTKB7q1VjaeLvXXj+NY9me368 WADN0z0/4bBQD4SA/cGcfp/tFP42jlt42U3AP8MwGNftDkrjdspgJJwg8sPVjK/DuUI4jHXYTo11 2TwIYd8+sOUlEni/30YC8oRwWd9pQNMBm/GQi4ZWo1YDZpizVusyyJi3d9dWL+7qASw0419Yv9Pv DLwO9y92d9v+/rw76PmwV9A8/f19xOrdc9Vg+/eY78cfWetw3wb8w6vTZvA9u13yGZ8z4K6Vn9XW vv9eY9ZFHIfWlZe6wHMvajv5dyX/ujGIlKDqR9ZekwEsUAognYC3gPjdY6JfrnKaewDkBNqspgTx 28oL3RsvSRF8EGWWBZ1DfsPD1F0tX9TYVyH2LsqaIR8uijSYdOhRYzD7KgvCILs15GQeJ+xoOjwH 6s6DSHI2ogk/wEIQT/vdASIKTLPtOIip0RfuHxEup6g4hmn9PayNn2cog81liu82K/Q4zxYZa6bw atcYa+J/oULqzQYiQqgvXA21oo6ewdaBfzOWrZah0A3IuukuDRb6UekSUJslVZnrRASaqwjSWABj BxYB/w19HwbR9Sb9nhOHJP2n4cfx6VsaLzWVVrx6U69Ye5cV1J2X702skgAkfM6TBBsRhkZJJNRs blfSXMWDGmNxElwGsG+CAW4LKDcWAsYIO3s19rNgpZrcDqfvb+IEFXVdPiTINkNWY01Y0qm34GlD 0LzXJZr3O3aPpOPhYJiVY/Il7Gx5e0amvE79G8CTlmCV1qslyMoyca9BC0DX0af3E/dv49Nj93x0 MjqautPh5O1oiky8AzIEL7Dl8VwgSGAHRLBosWy1YNkB5F+MAyQy162yo5Iou9QPRRCktK630HoF c4AzcgJcApvbsX6H/ySlRzwMWTP0UTatCarQ6UU4ijJEDVgPag7IViZRDNtrQwu27TX1GktMWMV3 tCKJUe5SmwucXVYy+YIb+TToKJVYbzPbFQ2/mBYm4p5/VQ99mxkYSchvQGQIbFhgQRJiAZgBuYD2 Tpv3sixaLcT303jGsRfMeRSHgJHTDycn4hHhaGdHLt0C9MHjegmpDRbOgyTNYDkNOQNQ6ocEOUnj 648/mGhBDzDxQlolrDBGLKzkynL+E1uByda2p9aS7wp6gbQd6e+g95LsF9AL9QT13lOTvZ7a7JnY u7FUE5TYfeXqDAzVBc4aC++af/SSOuHJlli1xWDLGp9OD8/GxzZrOTaoHlB1wewsAIPeFsxgLEBC /sGcXtDQyggPONOU1AphvV5HzwZxr8bqWekPVY8b8ugSsZCjUCgmlJIG22FOcQxgP5mtlgY6GsUO c/D9uMSbVQkVFkozA+J8L6ue2RZdcENOHTbXkBC/4iu+wH/QRSAnl0m8WrJrfgveLsSC7IKTHiZI IWk3UKxltUCDjkhaUOf9Tupz0HVQfQ7Queii+hTGcHyzmILevuRJeha9Bs+brGMKJttfgQHVChXV qaB4BK82AyIy6bkRPSccZqK/JoQ4wSwFPFwswCuZzdwFX1zwpC4fgLwig9MXUrkW7ZxZCgGWBU4E Z7Qp6plxoYZfssl05J5/eP3zh9Hk75JBJYtYcqFa2l6ygvRJklnDFJ3IelEyFc8pwbKs+yFHAbC1 NOQtJnLUxvkXN4z9a8U/YtPi9QImvEZ7grRy2vsDG8JNx2njewWx6oh/AEvg8bOeC1Sb0ydPUX8O FqBUbYZOoFoC+OKgZwKf3cTBrFbtqtVzejfQ12xZWz06pt5fst9JTL7iNh82xGbFt9PxCaiKr+Qs bgD12QuvAR+CK59JgMhXXyUq++Aj7iMuO+g/QNtWOCaPb197U80lBRDH5MqToXrIVkkktoI7aKGn 0EKL/wtEtqsk4VEGgdosjv6kIzsG9rxFrLxLXffgFYGP06HcInR4H3pRA81JoRXdgEYD+ls84Qir PppMziY2tlhWHUyrD12x8ejseOS+GQ2nHyYj9/RsCiL1/v0ZyNZxQ/Zm0HuRXtafms5eMXjF6D0C v3wBu/BC2McCAnUM1slqLdDEBaDt/8UFbz5toJKDGNBKwV/1rwS+pt4lvTeEngKk+SBlbOoSCp4L CjrdPmouxzk8tJ3HomBBCTEhn6B/9f4LIb5wQZThIhikTXT8g642e/aMgdpsvVJRg9DIpGzGyryW aaVszb3JpQZoih2DrzQ+PZrqPEmk8gnS7VxLQ3wTJR+wf/BSh5eX6aPtuCrZ8li7Fhyi9yzjbvbH y8JWhb6ypENtMA4GftKvk5Gs44gYdsfptHuPpp/ITgomXw8H7vRnmVAKlrbISioa8kGlxnm41nls zVPYA1Eh9Sjd94OhtSu2wx68HUHZ/gDzOEDZTt/uOY+ouB6yNLaG6Y/Dkw+j8++EZ6k6rXt6b9p9 K8qU4cVheir3y6xKdUPhmXy+DcPKSyswsc1yp2HbJK2Wem7YCfogQt8s4VxNRO7ytqWAx4iel4he aDKb/Tz9xR2/PT0Dok2Gp29H7nT4+mSU43SvaSaXDJhiJqFdKjN3ZVVdzuFJm4Qau1qdA/HaJX+7 lGeQrTIsZxSXb5xvjfgQQFpNeIKq6KOXVAXUpSk7upXc2jmEgJiokc6WZpc8TdCxhYo24j/VSS/D MsJLqymiTr2OTr4Qi9YI6+0YU4mQH10yGIh7XgKVoWNDz1OOpvEPgRi71kNfFMBCr9Yrma15iWmb jvoKJFMPMfQuPqeWfPp8n5bCl5Qu1SyjEPX9q/qkP9AeaXAOdoNB/waTbiglI5MmFROooPjzt6il Mvwr6A4T5DE0RdbyqImRPGRXXkY+g8hJiRxnsHYUZ7hCCkVflY+gAjpL6gvp2BocptzbwaF9iFZi 0Le7vce0Ev9ts7ZuO84+TEcT9tez8el3sh86DWLwokQdiMhzMyuklEYg82RKhhA3L7TKHM/XDlCJ qCmSlIgdzyl9DevmCdMmIQw5JvqveMpFN1OTqllJyoSazJ1erfbWD1yU0GzWxKBwPFhINNusrW3a oWI8Q2S/GsiSAWEBYYCN0/izvX6QatIszG2GhGE1VRYYcCzbSnhGnMhOZDmwy/R2SZ6VmA0N/8n4 9G+PFgU8isLI1UX59H2k0EcYkwhMS4cyBUTfN256NAwVj5W/l2MtgwO0WTxNUf+YjtA2PbXJE1JY 2iJHRcu13Sfcqk1Npk05ivyaO7jdGdy+SJX72WIQhhi9zqUt6Dk9SnX099v2AE0B2Ap3mQQLDxzM a37rkkVP3XkSL8SJS+WJF2p+2CpSTzg6zMKcUcSTcTSPWRLH4vyVDq/YlIonPDyAobSmzVJ1dCg8 SiwSCW64fC6PYVVqL808eJ3xZXZVlxHyXhNeWJMdSSUqN4BpbDwfSIKLVcbBXcXiHuAzJkIxLF2g Og/ayi4si5dGCKgpz6Sri46mCKapaiaLEwHAQBTZd2iKkxko94tbjEEYnezvSoxDbIbWt9/bxzTh 90S59TrIFt4S1suaOSDiAH1Iv8QdKmZm8nAMN5cXfwA5cuIAGbSCphBre4AlFH8+uTgnumN7SSk2 MrcluN5Yed0AbqTRJY/LzQgCAzCkdvUCRKpzw54mo5PhdHx2SnvKDwe29GMq4YG58XwP8E1I4D1Q QvAxca+z0c8KgGCbd1ec5EUyGwpAdAdZb3Lh8w7verw72911vPZg3/cOOwf3qDfJAW0pN8k7EfN3 xSlDVx4yFMtL2LuPrjxtoGMPwBuxOmPNOJy5gPr0hXAqAPXkRY1OJ0Iy8bkokUCZTOmMSo2N+Oet Y/G5MZYOuIAz1HAY6hKmCAIAoAcAQAu28udUdRjT+badApSll12pVSgg1IaOIULbCIeVk2iq8gE3 JmpqjHIIgpV6NwDhgmM8Kg908aBbx+jsaxnd+izo4JCIRG/QIM9tpA5C1pVEldI7wQJD1NGyGb6T lS4NXD+2B/stSuVcOoKu1HJSkI1AgE0TL0o9MqrjGfuC4nIk2A2++vDVyO2x9aIcI6iufK5mVhTD vRhHV4DLS3jOE7BAYea5yEaVpT+bd2NAFMUCeCwrwfEocSNvwesk+fI5IGoefBHHblJBYD0BnUIb wEo8IpQMp+Ugn5RTrnRQX+IBuUSZT8TjD0wU43vHYIUSoFWE/g0homoKY9UCC7hHFEiiKggHYWoU 3QRJHKEnKJEGLcbmNnKROuxwgQFcnZlPS9xUyQcg5iZlwVj4ggz1DWghtGuFUKJxbeeBwIp8VoaG HH8M1Jtwn4MRBjaYwTcXNJ3NKtpBi4kxU1Rl0MGHeUC84IMYU9Gux2ylQOtOOY7TbIsg3wuXKJWP NM+aOKuJ8uweiMok4+p7pSIy1UFRIUD0HN5KUpeLFKa5XRG8VzJUFT1K5qhM1A00LZBUmNrDnrC1 h32hx/FqQ7BY8FmAGKRIi0de5PP6+7fumw+nR+jIuMPJ23PyZjKRbihbR8rbvjA6KNO8/kQZXvWk pZ8YNlU93Ck8FGYxh8hRn6gjFCNlUWi3hcFsyMOwd3wRJ7dHIlw6pzPgaVyH5foUahGS9geOODHZ H3SkN/5ANAUY9jF5hIvuhSxnIy/0ileVkm2vqXtwfRuV7ZzzzIxsqHRS1f2BdwF2oCVEh0k22yvV Jt5tnnVvmxXRnhtnYVPlU7LM8nPRLItu6+bYbKdSDKrWKSG1fNyMlwtytW+cyN5ji5tsh7FVw1iI 8K17YHd6wDG9fVUW+i2ChYHMmsIqH4KKeCaM46WI0Hg4b+HNCBm6FI4a1iWrUaxaLPYhAZNRC9IF C6r1WH0I4WImKj+IKGrCqqMK1OmB+GgqUUzQmHWFokfRWZCFogVtB8jK1d3GZ6BnxDMKcoEn5FWL QjaEPCkmSy6hQxiJKkZRdylySuVnqjhPVjOKx+5TuVcgjb5nUuG2i8r2gvjvqkQnUrMeUB0sC/7M CoWD2s4wUC2FokRCrqRPgUA4PMquXB+NV+5PBDr9t+HMXZGvMF4TTBV/2nLSFnMaecGaPG5VITim oR9QO1fgiqqqOX2aWyxdzU+UygW1FeT79gJVyygRVQArDu9MViuca6nTq/yIRuTghKgBZNqrrt7A yirZUizdEXOuLUUVfSHS7w2qwEr353S2eaPqOBoFAbzdFZKKkR+kzJDQU5bhC0uOMnxoQ9fmDm7u 0u5sHC+Y/I7hTBoV9qzgOD0z/SVmhhtqR2hQUaqlMJftKbiiawYVt3qHSS17sMbqy5iRrPhdAOaO K0AEozSd/L0uaStqdZ2+c2A7PUoEw4fBg82a6bvnFy7+94P9B8b64oZTsdKAka4B5eAKNWpqaylq iXY8lQVksvxSpwukYFDWiXilKo+gViWxotMFj5E139m0OjpfioHOoF8ZWQiRRs19V+0A5IWFD9kN E07dp0+fnrNxyiKOC/eS27/gNzpVA6/oKpNlxH8RjtzQ/20VJFx6giexf53WFWC6T6YK5QU7d3vI xcDO/QG6a/nlnW9mScrulvMba1a1qpCtVEVS6bkJbw2wAghGNpPFAfn5HlCbe4l/9W1p+YqgYs29 N5PwKqAigUMx0165jLxyxwMvNeVeS+51YNY89y+rg/O7vGB0K1ukj+mNNTf7WqK+JMMLvQlfhp7P 13OieESqAD3FkIj29JSeURhBW2NBlkqXR94zE3fKxMA9bcNVth4RLj1renMDWapSUT1USXihhkmP 5byjTwmMBF6ilXqZLnh9x0zZIKzKWNsk33oHpJ4cbRbmfDvxZBWAeP8u5FOQ7qAf6acC+eRAo6bi TgJqV+87kUUAq0hzqHRG+dEDxK1EMuOmx1cVug4cusPqDHoD+7BaJX5zMhUmMgqNGJayCY8RbeIb iMmPyITMI/yeVxTJu7yFe4GG3GQXoUsmWFiMN8VbrHgRNr+OVPSDG3LLEK3Tlg8Gdnf/++/ZkrNu vI219tzGHZAVzHNG+m4iXj+kQ2k06BWZI6U+BF5sVnQ/avJC5v0KriVPC1CCh/Kbh5Wl1zsPuOKo ImRptLbcNUXfurQ2IaP/w5cNqUFcqam+asjs4lXDu24ayrmLt3TKs1p011AHVxtvHFJZi5kpq/BC t100vOueoXnNsJKnpYaq4Ge9fnUMb9wmLHD2vW62C4GeRxIZSnXUjfuL+Hqe4b3d+lOC97TRkDe3 js5Gk6ORO/r0/mR8NJ66R8OTEyQQyds8ar0C4XPpF25UICvURE+4i512Bz4cPIKikFezZI6Dkejv NeXl9zvyvfqevkqDlu/rqwKxijvTOG6vfLRS21FzlmZSRUrFHZkFoCjkIocGvPz7IwUmGy5sSfVW vLBlJDXwR3eEvGBtiaxp1oWI+OMAG2vqlFugBqIOyME1JNdj7hWTVtLGqpqrO5ApsahAt15hXSXW puQyQwucJfHy7hWqu3lr4PILeUKsSx0KV8w31XC+juMQt1xAq2qk8kqs2NBYxS+IqbrqojRVo/UK c7HxUk6nCIoDtlf1UxutZnh6jHI8ea7UlHkBvzClnBFUUlqRpBPcGy9Lqr2i2v+BBBUgS3QUpIxv eCJEVVWv0uz6J08KvzKkNqpJbFBZzVEmbtknRZV4hBUD9ddnZyeVVikN/sXjeR3pAtoS0feWZ8de tlrUURM2dNERfVtLZRYr9YlCZ5MCgagNa0ILVCPzi4KOxPpJH3XlMi3+/jNL+s8mEKHgnvPfREop Eln7pSR1HZC8O/UzTGY1f8l73yalVZXW5JLBir0Ubfs/TofvRsfD6fBkdPrPF0aP3PxXdZFqX1em yuZ1iyYfbPihj/zBhM+xWd0tGeL6WJOWKZvwwDxejmfmVPTbKUsIAccZX4DhSUGQtE+5McwQJqAQ aezk8j+PXKN6XCiOeFmwEbrQvLL4vFjjn3gBkIKYVca0CRc8FrVK1tlU3P9fsX6vinVpNBQbIiEk azbWy6dNG79OFk0G/HWco+moTA4Bo/XKF+Ufj4Z6nPo73VcRwIUgStXvLlFuRB4vL4i1HvrbQhqR OssCCgxMWVbxi4BVv89T/OU6M+UiKRotwWvP5nWtqLRZ0i2NUjyXW9//mrkuRHfNgr3VIHPNpmGa sU4JqAhC1yBqRsJfSDo6+3A6XYtbkvizvK214OBRytuPuaLXnuT3j2Ak3IogRj5xpY9DWhTnPUtE 0KxT8TgaKyQiwhNNiPyCD7T9z+e6V2Qqp938UziGlG39KRyDgoX2wi/g7G389SqdT1AUqJy3lJ6p 7FP+sRwZ3WqWy8eoQ9fSpWG5SA8loyV13RXoOiUwJD10GaHk3JMUSGSTsTQlRdTDUD9hEqHjv9v7 0u40kmTR+Sp+RV696TZYgKBAaHHbc9U27dYdWfKV5J6ed889nAIKqWy2rgLZfjP9319E5FKZtUCB ACG7ON0W1JJLZGTsEQndcF6LFIMW5peRdy3jhOSqa6km1J50meiSHrcS8YZ5MShDzZFGpqgpwAv8 jOYW8v3pgFtxh85nHi3to0sJr6CZVk4a5AMxZWTVeU12CMZG7g0VmzLDAUfP52fobqAoIOT7plkj 5Rs4S68QRyjjCo0K4slzOXm5PLM0nnr7Hw6V0SRSPhp3+oQV+QK+h2orjOOofjvJs2evnoG05Hj2 ZOQVyvjagDxlbYfR4sDzWjv5QtjqjZIWzPoSmkAKAArAJZDPbhJpekVkSdm54Esdvkgoc/FJciAc No0a+wD9QqohnKgEG1rSjSjZkHcCXUZ0mKjLAFxqFo4Q/yptphJSZbQu9DaC74q0ul9gGkge3a6v UvcZn2fB3A1a0F4kCdwgEPcAyf8GBVxTPZXcSbMW3Edlj4WF/q7Ts6f9CZf21U28ZdqU9tDothex 8CRHEepGHmHMUVHl0hQfZ1jfixePFpON1iiTBgJptE7LbOUskMwEmfLRYjxxB5pgKoCezkQgV1z8 /FEJGmQg3RclQFV0i6oK+lqF0+DMuHPeB1IwxK0Pw5sOMbks8KR1eALGqMdE6A3zx06HXGy5ErTX /koPa6loe+ZVdCbJqpl6na7Sg2Piv5U4eF6PTLdWSU5kRs4ZDLEVRM6Zji16OxR1LpBEj9dZrnYO Wd7fCmTB1CgR3SELAB0fHxTR2G1ZIufzoYsssXgqEoG6U1EMFseJASU8qoNjJpbY9e8cP4RuInKo tKb0gjiCuJp8gkdBDBkWjTH5UUc5uVAM+6CM1dfTxggZrAoVgQJkqFWKNSshUOZBKzDb+2HOVbk6 gJsH+Rmf79zOXRASQ+ltkr5JKnYLNGAo6Jjh85CtQLuLpHlwhhi2Tge/8UFaLGFVh5ui0mNsNC/l 2fPHjIo1Wn/clKcH+RpB2FwAMbb+jjC54Q01npmoZ6LdvNqyvM2C7g9eLN5XGdTDIb6a+MI7kTzx /1ANboeCD0V4ap6XloRfBRI1eQVl1IlkuOouU4+gor9b4KW+CcFr8EEErx0c1QWCd/qOPWxNxy3s BMSauxZlKWi071e4KIBBt9CDRw9yd3Z+cN+a8CdhcXojraLAjyLhQaZIoqT56+n1r62r5rvL35og A/BaNtFdEOPeVrvhmnvRbaniUtQu3xVoB5j43JYW7AOfJHiuYwZXYYGK1B5KQQL8eJcH+fEQnNHQ IZs2rjGsOYpFdEPuKu77p70U55CPSwDkr8CQhPtQuOf5DlOxBEMRSBDecrrRWKISyJ89jvd52ZqU WHUvEb1TeiUaDypWq5e0WAcY3rAzGOPf3+x+PqCeoE3Q0OEf2NR4v1AIylOJgIhIIHW0GRXPS2DX Jiw95vKWICi6iOj6Z/cDBDRJiJEk8BknHLTnPCCSwKs1++DQsnrVw3LZah+0u43eccWOJoHPaChI Ap/xEO1IykurHXDZw/kCcvyQl4p9Z09+g+fOAqn/XRB1fOY3SfLv0l7jTFC8THpfKF6ZZz5HQ5Vf zHxrZpQzLIp4lQsr8z33po4VdE3U4nRy2gZtA6mQnJJsn2+Kh2+xoEeCr0IjI7tYhOT+HwfUoZ5Q lEG4eHd689tZ8x+tXynmscRyVhmwo5rLPfYRPWv9RM5/qtP5T3C5NAH1ogSLXKI4lRJS3NJoCtCl nKySsvXNPRJq5vlP9Ua1UrNC5z8dWJVqdv7TJj50/tPBUafdOW44x92OfXhcPaq3D7pWrVNt17ud yqHVOKgcHrWPurX1nf9knVQPTg6qs85/qqvznwA5GSInOSp4EBWJAwFyFgNLNGhryP1f3zRz/Iie crm87zm3WEtp3/kCYvXE6e5rhhdZ7aEMzeGBOQe1ai1y8o44agdHoRrz/+jHtgPXsZ1q7ZCf/APv WuHTdxoHx43w8TvHtXnH7xjdz51Le/F3BLfsOQftzmGt7ljlcqd93Kked2qVmCN6Fms7YKCLvUdH lRxZdFQJ/LGOqaYvd+2B7EAeORBYPZddXr1pXrGf/8lcC0g+jBeXoZJjeYvcWig2711dnp//fPr6 78CMYB6ySLhy8SGK6ceH5UEK+rn59uwCnn991TwF1ezsAr68a17cnJ4jE2lenZ2en/3f5huG7ARH EuPcY6fXcsR22QXJrfxRDRxP52rZIPxyn7LoVzxdNR9rs7Z4DJrBAwHKGNC+d3F5c/a6ecLQuNnv klDLz0GCvSBruAyjJkW2Gz/aXWZPJ6MBqpdomMzt/Xp2cQOtv462STa6SLsUFtfruR3XIXVPLSzT BBCQuOUiFlmcx6vSaqnZx8E0WO4i+whQYC6s90fcewmN4ZKX8H/+T9wHWqkS1tDRWTGfKjxhwU1r 5hM1PMFr5hN1uFmf9US+LpAW1ICzi+vm1Q06KC41jOFlhvOY4V8o5hvFBv6pFWvyDynkAYhXBeGU IJ4L4zlATvEIrkOKhZizEikfAXjOW885C3o4e0Hb+oKSAWPeU9X5T1nQljX3qQY81Xiy2FJL98gc bLG2C1tYA282ZiLUUYBQH96/Qd4UEIfr5g1DDlEV/OIje9lAA0LowXbwYIWJRz/Rr2rSw/x2xTKe tp4q8qQkNdW5q/HE8EtHnjewcLDMIYGEL2/eLX5E4xPQmwYxlLiHpVCCePStY8I2rWEjWENNrl2B vGotILC67NVLmA+djPr4squ1MeH1AeCtrVMfWHQl9pCa07uf1rAetaewHnVtPVT9pe1dGIJcV8At Ae517oLdXWix6rsPo93WunSyVehTVkAs580x9np19txri8899kY1HVAeoME+QLV9gM77gMVLqSWv Y1Xr2ao+8qqmU5UTNRWrovQaKZ3Ot6bMe4iMLPOU6pWYYB6JolqpnphPc2sZzeXQmgXGB7w6Z42W 39gzX33Axj7IyPVs+D5wVR+wNLVYZVJ3krh0tl9COtSCeqdcwAQPSZzcXcxHvT6c5Bai0vYKNJmN u0Nm8hyraC3Bc4zNEmdi0fdIgPEKf5XwoH/h+Kl/4WiXaJHQkIjqu64WhbDJODxCClAEIYB6yGtX AbmKqFVFla+ziwt5+pau8H24Prt4m3cLBTYZjFeAWcaIN4Ze/zi7+RXWynYRRvlZyMbIu1AoItJZ BXbVvPlwdYHpPRVMQmDt5BbaRgtV8kRVa7WYNrrUxjxrYt14UeI1oRRgWxWxzWAmBr6biKEjffGT wRoASSLkXifhIGQaP6vqZ9W8W1V3lchdrVjyp6V+KgpfrdSMn7WaaZqdSYkxyGFha18HcOr0GjqF D2IEYD0uRDJpnUeVV057O1jz7BsguQjpTVPaB2ODTdgQjwx2obgUtmQ48m3hSEocuAccuJc4kE+B XamRp8DuM/x5kvhjTiANUqyO5GRY81SxhtDki4EGDxPjTYh8eUJo8V0J8WE8fSpiOwvwnoHYzmOW edoIhpZiBkyOiQ1B4a3HleJxne0dHlWK8L1K5em1hSGITvRVqdC6ytMhWcpNpSnHAsi8ZWjhAfgf xn22JtwvhfFDw40XuRLhxT1ghEIGuCYoWElYzXKlwA04s7U91VoiPcR7ulNRuCDEWqH/4Z78Dwmx Mct2i9RX7zaZxN6c/nyuxpN3MXOmkJ4AJ+PKQ2jlxkOLE8FMEAZQVuJdTeEtB/tNVp5K2zQZZNkD VqsImCAWbcbQeIEjGKD8iiVDgEYwJBL0hcsKi6+7i3kVg/x9QV9/9vbq8sN7OdtZQQmz4w92U20F 6F8P75IhWiRwEAvA0M75u/AeOqlW0u2++C5BlJJd1h6yAx+ypvxrLfhaD74eLL3EA3cISwx/7S+P utQwDvzX/hKN51OLzR0YB4bUoEsfYtyc4uLCV9OteZq+rUX7rj1i3wer7rtu9B1vlqSELLLw+7O2 ghfhRdpNX+2Tj8kPwWb6mLiLPJ1oszztnBr/J+5xXz1uFfkLRf4K/1WXfyJR9KEtG+pHDFpm2DLX x6Smzl1eStHXDM9txK3417/iYr4+P71q5va+0GVJSmARmr83X39AQy97Jhb09el1E5f6Iqi29ZJV 2A1eefbs8u/PnrHmOTzy7NnFW/x+8UZo27m9nZD+HUGR5u+vm+9v2On5eYj5A2W4B3628+ECZwP3 YUAzG5vR1gAbQ3ka/jzjwEQVTACGvsOg4d+//pWdn168/XD6tsnG/fGt/0efY1yt9Nn+quEby/em /T77keGfQmpCiEdb+MjlQJGcFLi6KYbqYfSfz//Bu+WPtCGC3ebh0nmwwDA90jQFNsFVnwGIoIGX 8H74AVrxCT4A7b6EZp+IPCUgSoDbFrglEzh9fIhZHvJ1Yu4f4Z9JWGuU8hp3j8t/pJxP/1QjV0gY z8lob/2Z6BVqrx5/RbbDjN5rcVfyx1r4xOlvzfcAnhs2rs4lg0idOF1aP8wWg2IyXJeD9PpgX40G MwVEnRiq+MUhFUSCXP7d4J0zVgrkufzB97lK9cgqBVdkyweRMadat2iI6FLrJmUeBusWv+V8zUKz vs1WlbCrxV7ZIMnSHdUr3RG6TLbVgAyuRN+SV1YF7GplU3g8CbGOR2C11TngjsfkauwVE7jVOOAu yxKqa9oAwQqARgFsIVuB+BWYe6W+QsYd2TgRDdyTtt6Xs+0Oy63YtgmjK6L+M6A4216+HBRTQ8iK vxLF122H4mw71HJQTA2htWDe4aY2tL9WVEwgwcaVb2FD+2tFxdRQnHPl6UCxvu1QVP9s3YaWNvKP m9zQM3j2qrZ4WmX4eE3IGcB1g1t8M3BdH3JeXb6PNUvLe9zYmsbizQviydT1Vdi/VRr84xjCl3Ax prVQP8rMHn9HxGsHSzE9Y0dsqxn6seDx5C3DywFuRfbbVUlPW2msXRtK6iG3T90Qu5zZdTmwrYuz Px2L6gNsdRuhgFtrCF0R4EJX6mukgE/CWrkc/10R+Xt0S+Ta7I4bgNATtzKuCkJP06a4XUi1jfbC bYXQ9tgCU1j+Ysq/fNd2vqWsequStZ6oDS+NxW5ttHwlFru+05tsV4zqefOXmyxGdS1wC9nHHn9X rTL+K6GifRZ/uigg1xZIlMWW5pa0Tear0YKx344pMosbzeJGvx8rZxrz3YqiFlflYX86ltAnCNyn aRt9VNfQo9tG12YtzWyjIXhszEizjZbQ7dpk22gJXco2+l1ZQiNXYk4p+K7tno8ah/dE7Z7fYuyi 597ebZkp9Ors7a+ZLXRNgFuzMTQqRKfw0UWvJJ0+skmzZho1a7nJrUi6ewxLYxqgRC17GwTKkzP1 JWvaK5JYtt1Et4hBbm1C7VZY1rbM1JNq267KtPkYxrYnCO+5V+rrdJs8mo1uZcLEE7K/zbKkbdfs 12tb24rZP7YlbbnJr6jqwoZtYts613VYt8x5bHiuj23LSjPVFbGyzVqc8uvy967G5uMOh463XTYf 7biAzOazasA9vs1H38ZPyp6zKlL7ZGw1a+MtmR0ms8NkdhgFt0VCbtYVq/f9mVi+vczOVCz327J6 bGBmj2TRWNXMts5aET1h6kkaItY/jTXYGNanjG7UWrCqaWSWALQEYALcY5zVEErXys5q2CDcnnYe 3Bw7YZYHtyJAZnlwW5cHl27dsjMWlliib6ls16PmwW2uaty3mOI2Z28/YQPQloFy7pX6OtM1n3KO 2xwU/cZz3BZ20a8onPUxbFTbG47whLLVFgbiulJrnlD+2hbCbIsz2h4XZo8dO7TmnK5HQrwnmge3 vnzllVg5N17ta661Lqv2tTa4PVUrp76HMgtmEpDWVdolM07msiJdmeExMzx+N4bHtSHgt2VU3FxR 8idhC3yiuXRpZvYko8o2F9zzJGxva0PPJ2FF28Dst9gellV42pCVJrNabcRqtfnKTHPNL1llpjUC 7nGy9NK79rY+b2/OVNaVxrWtmXypgPKdV11aW3rnlllhFrC5bA4lnn4m32a8hN9g3t5swM29Ul+d J2B77DJLsuUnZIVZLmBh/XPNIqm+pck/RpLgFs41q1a01sln1Yo2ZQfZfLWiuep8Vq1ojYDbtB0k nW7/2CaOVZGWJ2O++N4LEWWmicw0sVzEwrrKyz9ls8PmTlTasLlgXSVJ1moJWFstjyzW4hGBv1Y9 ewOjzmrvZLV3NqHXkmP/MYrvhL3QWfWdTQJuY3rtCpK9Z58kmMoUpoVlP75a/RjZFKsC5FMuyPO4 K5Amn2WTK/DkTBFrpyTZMfQpzB4rA/Y3dQz92kwFIVBmh2U9WXjPvfJtHpa1dfLf0zFMpYeQFX8l iq/bDsX1lrqeA6G1YN7GKpysN4hmk5t1/UfDr7c+98Nm/zTTpr7rYJx53G5VSjpLCdZ10en1plZt EIrry6NejTF24zWC5tsUsyJB6wPct2OM1TdWZmhNAtITrhu09TbUtaViZObRzDyqrmTm0adsrnvK Jcy/UVB+RwbLb706+cONkRuA0JMyNG4uXvRJGBGfaF2lNDNbV8pWFlf4bRqgMhPdY5noNl8Qab6p KauItE7IPU5JpIf4jx7fApeskSw3uXVlE21r3uEmgfLkLF7JSum6qmttmaVqEbvU2pSszMC0JC3L 4u+y+LsNF3LammCUDZR2ehKzX68Zaitm/9hGp+Um/4TLP23hXLOIs7VOPiv/tDGzz+brP803XmQF oNYJucc3++g7+UmZdNZWNWhbzTXfe5mozBSTmWK2zDSQxfpsHSifgHFkA+dvPZLh42meLLbJmW2d wWJdVdU2bItY/zSy2JasZtYmjAFkBHiMmllhfTWrmbVJwD2lNK05tsEsTSsFkLI0rfWlaaVagKzU 1TILkh3mnsYIkwaQ39Jh7mu2JczZydnhY8sBbu6Vb/LwsY3KP0/IvLP47DccQpoda7aFyVRbgwzZ 0WffxFwfO/ZlHp9YpQr4XadprS8CcDXmt41XSZpvRcqqJK0PcE/D/LZd+VePYP1Zm66Xmc2y6kaZ RSyziG29YSc78+67OPNuZWLCt2WBepoBRt/IcXqPukDZkXvfavjQtlkUMnvKOuwpmy9pM98ukJW0 WSfkHie3Kb2Z+vENKMkyb5qprCv5ZVvznzbpXnpyJgxpwllXUtyWGSMWMD1sDiWeslkh1ebKAm2y QJvFSftjeso3kBG1NVEBWTjMd7LQWajLNzHXxza7pJlqVuZlaVOIZgT57E7uxIQ7fXvqO4uZQeKV +HjVPZRpFNXc+TDw2ytWebL2jbWCZCHDhcLKABeFFGRpf3U3qNxp9eCv8iGzXOnD+zc4WbFNrps3 7J69ZFZFEkL88SJXSh6VC3cZ7pF7Bs3JrQHXSGKtwBcaVWVVFpB0EJgHkSTIbE2e1GqnWRfTrK9Z +n+o3SJ+1roOtQiar0uBXNA6sdik0k5uKwtYrJY8PaZ6Fj+TuJFbob+bncl8gSZ+JnEjj1uLrTyu Jf3i6FPZhqNWkhcjGClTcjAwVAk44LI/N9+eXcBfIZ/cnP58rhh33gWBZlII7s4T9VAsMYfKW3rx iBpVLHC2Q7z2nX5Pk7EXkKirReZZMQKkZ6WUID1m+/C0FCHxvRc5tkJBmq1JkJ4vNOb4UoUE7vVA bAE5mwZAyIhMGhrKBTofDoC7CvAbtzziN26pnfdtxcUK0g909rdgGrO/pZvk4t/y1e0ItP5W4Ck8 mcRSDvDbAUJp+0z+8+C98PZaf+7nHOFk3oxSzuPRq7XPnAdTPBko+By/5AWQ5kIaAwLnBSXYQCU0 JAS2wKUBsKj0VGT3UoiaMSmcUJHB1ORXC78CvKoV8cWqLCeHoVVpOsjfFwzL5duryw/vJZwCrk+8 uis4NTD1Fn3bTWb6qWw40L9mxJFWHFZDMw631Nb0pXmgTDUdj0fehPnuYNqf2ENnNPXZxG73Hda5 s4e3jo+zWUroWrHNLjNc+guKUDPNHjLqcBlbTqLJ8h7GUa2kM1XGo3mtotAcET5A83+c3fyKk3ER JPlUchu7at58uLo4u3jLgCzAy37Cy6aBzHhNAph2y+OQNf61FnytB18PlqZyA3cIVA7+2l8eldrB OPBf+4uODAolOL0jfEX56SE2vnm2vkWt1A8pJsU0PEp0lOGGqqbbS2mAaEkgRhiG3Fpd2h2zJMno jkp4RzMkzdhOqcFQWwMYljWishS48QARODVMDlYNk/pDZQmUINyxkhzYqEe81LcHjhApMlEiEyUi biHF2asPYO2ea817mxsfom/OI3vQGMvXivVUgsHew8CYBM4k8IZ8itzQkHvYWYAaBXoAQWCBfMSA Npy9e/cb7DXAcnfoDm/Z1Acq0HV67hB2xeTr2MkxIUX953+yUsM6KlZrbO/g0Doo1g8YXAtDtTX4 iq+90Ier9VkywzFcn7a40HacbmqhKW8X2wVNcrJxz7WxvqsYBuGKzWyt4oS82mZt3G/wyss27rdS 8+rq8gooxeWHG5lsER4YUg2NIvS/cpoA/xMBjRATmuhrmMjSM0SSgoyA4T6EEeNmiEKbT6igg8Eu f4xAAWAjtg08AUzKnHfcOBeecCjQBq89Q5XVu3UmfdefABuCfwSqsXvb86F9osT0PNzouhN3NPRT 8x9z+ZdZ+T0JgckdzP+PqeN9lYCAKY8+LwMGwXUWnziPTVqCD4e2wcfip7Sw+HANRDPvFrYeEvNw q2MPSVQQTQ+hDX/iuZ0J602HnYegFVEV0Os+sVegQgPPYJ9g/5Sqhe3FtxTAWBWuceC8Pr1uInm5 YJ9eVdgNfqmy5jlcrLDmxZtvDFImLgqCieY4GILvDlDQdv6YuouJ0ithZ6/WDbrLmDli2wHXWBav EADFT8WPcUAowpaLsrS5SMROAfU+vfy4bRCJpWV6SGocWg6n/X7Jcz46gJUgpo09pwtqzsRJT9ZW JzGJwX5iZ9cMzV1rBbAAzAIQeRAOrhRCxDSIZ3zEL+tls6sAFKCmfXvrObeoMEckwgft7fv4nY1O nU9L8NLAGCrbWitwF4EKeW/aRvergeGiuJlP1BY2xmsVIEZevJaxNXu1+fvZ9c21glk1BDPWtpQC 9ZEUqE/r3c4KciEIRRVKVOPlfve/+hNnAD/708Ewx1IBEyfp3g8qVcMR8bH4BU2UJjhBq5czNnp6 uPZI1oV6XVgXDmv4Ba4tNIF6zAROTibOFyqJ8KUFP9c2/sVG2giP1O36zrCbxxEX1GA3BvySjkEC 74iOaaLyyDNlFERLwEq5byKvLwaQmgmQ8C7WLX7yqfAGlULJT6Cm1VhBM28kU+OFAbU0PiYbZmZR 6WSTzermtAkaBohhIA/XlaTmZJI6iVA8O2sxNDoI7SsF3E8Rch6GbVQG+R6hrRECaR1PuQRiBQ5D KxBjitJK2sTZojRfGb6vUT11Q8xrNUyncVA8Rp5zDMzHQp4ju6NpAdmLznWxjhfE4aM0VKPdGpDL KoZErHz8D+Z2xwvRQbQnXX64uMk/L4Q3bRAAsXHiGCgAgWUI9SvHx2Z5b7bnbBKu1UoKuBrSbbJw G6WGoLTalIZ58NToIIwcsP7i8kbzKTB72NVEFWFmlgsGxKTLJiMQde4cL6UKIhehuvJFmPNYRzzW prXqwFrpiohAyNUtlM4SBlMeLhDY1BdDWCudlPeSebBco0G+cHICY9NIWngAK2IBR1bxEFjAUbUO fzUO8AYWAmZ2g9riA3paEEqmEBO1fFIY2/Xpu/fnTXb9z+ub5rv8AdqW1LD1B1bFKhENPruwKp81 AhgheIsJyFVdDXpeZJ3pwGl1QdnIA/n5DVh9PhAFSCsCtAjLEy8W3K6NWODmHzYIuq7j6XxILbUA USbEhTh/NAApbyy4UsQ+sBCAdAEu4MNn5/B8Xm7Qn1gtLC7z+IDDSg2FKNhKtYNilfaSGW7w4eIM CxNOYA4DCrJ50GCtOQSF93Z6fh48E1YEXmirRo/vAxNpXl3D0/vN3183398Eo13FMqJWrciK3Jzz 5IjFcNzSlennJoKrrqPQ4kg8GGvMZOGBLoXVzmA8+SpMDYxsDSFBgHD87Ld3CyKHTkcTLSmJnT9s Tr+AEMJDsfevfz29igkDWWwujdmIHnSnTS0yhlXxgsWw8TAZG2fOJAYf1zYjxm1dE4UEITcKzEMa 2yZ39gRJgjfhePmshbN8tuB66koeBb/vUjOtzmg6nLRau8kIK4YxxOBN453VyEKHVq3YQALeqITU YefL2HN8H5mOFi9mD3XfCHQ4dwMRsEXs35K7ggOxhXaN1r3dnzo+SeH8a76qy0IJHa0J56MDs2Kw X40zBsVXNl4S2wlVRj02BkoN3JZTbX/sdNyeSykDpv2Hx+fY47FjezIyRyON5cV2fs0yVKPV6aaP F61Emx+5nuf0HM8jrVGDYmD8Bujj8EV49ec7t3PH7jDxGqTCvnPv9JnvdKaeO4FnKTzzwzXImGe/ yJYQi+zuwB3q2ezeCNraVbd2WXfk8Ek7X2C8ReZ/csdj2JW5rtvrwRhuYTfa+77X2Z8A296HXYo7 eN//o7+vwQYDThEEZbjO2gs9nuMR1bVq1TluHzd67XL5oN2od2vdo9oxq1YqjXqdvAuLjSK3t7e3 6EiQeFWPGki88E+1cozES48exkBmLVM1GkvpuZrt0TJDZc2kPKldY8T41Ec6KJC4kNsT4bILEQxa P7+l2g35NVe8eQJlR3AQ0T1nJBVgJbKVuLFp5ln2MVTWKBgfp2P5ahEzP/OwLPinVqzJPxVzIGsa R1sfB/KFuU9V5z9lQVvW3KcQBRsrn6RI7wvgjBl+uLQStT+yl41K9MF28GCFKVcZ/qomPcxvVyzj aWvVE9I3aNgSlHcxHuwlZv81CF/iHm5rZqOVjy4gAHsr2djWAjvbZa9eshoFsS2zyR8wyto66c+i E9pDJAwCLh4wLd0dqzTbbZvfQ9DXiuDvvNZir1dn91LbSC/1SC9pKXwiMbMqivRJYjGfd817iFja PF6wEob3ba+uQWnN+gcYWqv7tqldWbx1IaIsR5MgZ8X6JWfk3MS4IGcji1W0lkAWA5BxrIrvjiQI iij1VcIPm4wDIq5lUcZa5LWrNp5xAxQuSghjggGQ+MpggIJQj0Uuoh0tFBAGMiMpDhMSi5YVzSps J7fQNlqokqBarcXkNLajmYlx0kg9KS8xdmFNmIZjIGasr9ghr2+ai4sonYnDM1fXmWa21j2BMwht hQcJbRGI2ASReIDYheJSEPsW4ZQSDvcAh3sJh3wKCKcGYIHS3Z8qDEMmwo2i3tOGHIHqiwGKh3Ec c/RfvnnuE14c4jf68qA1E+1sGPHU6U+7aO/CvB+VGi6t9jML7KCBzrLIQFezjotVq3YYttDF1dKI oLdRSyPRRBcEzPsR9NKG6anasDE3fTWHj8kPwUTV3dklElieygbV+D9xj+u1s/kLRf4K/1WXfyLW JzES9bbZjxr2Lx9A5T2jYBZZc0BgyTWTYfJ//SvaWV6fn16BPPGFLr8QEATka/7efP0BJQ32TMYI qNxPkvwxWuAlE0mgz55d/v3ZM54J+uzZxVv8LtNB87m9nXivYLDSwgGvOfDFvXsgWVVoQDn4g+Ib 8Y3NaGuAjWHtBvjzjIMT97wADH2HQcO/f/0rOz+9ePvh9G2TjfvjW/+PvpTFomdt9qb9PvuR4Z+N nrQ594SF0ANxp0Wu+gzLVYxpdvEO7VySUO2UNZwtGVe3Y6UnN8Z1ECn4sdJzEFNNaflzB5ed0ENO 8Us1pYecdpdyUqs68y2uu1WdsbZg2wudE7QsmJY8RSzVVJY8tWvBthc6JWtZMC15LlWqqSx5MlTa qaQuYZaCxfJYHGnVXwXDVR6CbeS8jzK4jAVnLDhjwRkLzlhwxoJNFtx3epPt0nKDog/bo+WmGNNC hT+19c5YbMZiMxabsdiMxX6rLNZzb++2jMdenb39deuYbJpBZVw247IZl824bMZlMy5rcll3OHS8 7eKyWrjK9nDZNIPKuGzGZTMum3HZjMtmXFZwWbQTP0ZQVMjyuhVBUasYU8ZiMxabsdiMxWYsNmOx JovduEd2Ljt7BI/sKsaUsdiMxWYsNmOxGYvNWKzJYjfvkZ3Lzx7DI7uSQWVcNuOyGZfNuGzGZTMu a3LZzXtk5zK0x/DIrmRQGZfNuGzGZTMum3HZjMsKLktK7GO4ZMNa4Vb4ZFcyqIzLZlw247IZl824 bMZlQ1x2417Z+QztEdyyKxlUxmUzLptx2YzLZlw247IhLrt5x+x8jvYYntnVjCpjtBmjzRhtxmgz Rpsx2hCj3bxvdj5Lewzn7GpGlTHajNFmjDZjtBmjzRitYLTEYB/DOxvmV1vhnV3JoDIum3HZjMtm XDbjshmXDXHZjXtn5zO0R/DOrmRQGZfNuGzGZTMum3HZjMuGuOzmvbPzOdpjeGdXM6qM0WaMNmO0 GaPNGO13ymg1FktH4PGhdfr21HcWY7LxbGzJAsF8GPjtFavEcs+1drcQW1whO1wbG9wc+1sT29sQ Z1gHR1gHJ9gQB1gH5V8HxV8befSdfk+jkQtQxGqReVYMkfKslFTKY7YPT0syhe9FCeGaeklP/2gA q6KAsrE10MBZTa+UiMzqaHkysmSrc3bjokBZcO9Mx+ORNwEhZjDtT+yhM5r6bGK3+w7r3NnDW8dn o+Fym2vF/H/tAsZSIsU/zm5+he48F1vLp9pe7Kp58+Hq4uziLasUivCyn/CyydSN1+TYVkh05Uy6 NJhZeys6gYR3NC627tEviPaI7O5YITkb9djkzmG+PXAE9mdYnwbrqw9Ae8+15r3NmUb0zXk4Co2x fK1YX/umWQTt5Ov4gwEOnr179xvrjIYT2x26w1sGypzHuk7PHTpdNvk6dnLs5+bbs4sXOZb7z/9k JeuwVqwes71qo3JYrDUYXAtPozX4ii/iGyzojuVKphLp+mw4mkja73RzpbTobRfbiHyyYxtRr41B OaJ/Wgab2Zr/UF5tszaiHbzyso1oh2N6DX0uPRjcBCVAeYbYCI0jSkQhwvsu6CO2yx8jA4ZpCOSB J9hLRkMM6d73rvP5GXJH79aZ9F1/AmQE/hFLyO5tz2fwFwkJPQ83uu7EHQ391PTDhO0yYF2up+LH 4qe0vX24hu2Udwsx1okohDr2EFdXwmgIZNWfeG5nwnrTYechwCHEK7L8J/aKVSuwgdknWLdStbBB qPEhvD69biLyXLBPryrsBr9UWfMcLlZY8+LNUuMJQ1ZsD5TG+l9RWkPm5fwxdRdjTyvZv68WAhn2 WfxU/BjXbxHWLLoX58KHnQJUP738mIyBuoUsDgeH036/5DkfHUBBIL1jz+m6HXvipEfG1ZFCMdhP 7OyaXQC3Xgy0Kx0DbSbaSx/xi9ri9u2t59wCfCLUmhskl5GUEC3u45HCnw7yn5bYx+zt1eWH98jF ZVuBmtP+Y+p4X1c7/kVhn0/kTstRo5Wvf/P3s+ubazXOamicrG0pJvmRmOSngpRm5O7yv/oTZwA/ +9PBMMdSzQR7cO8Hlao2F2RJXwbQoDmXFws2aZlNhiHDZ0P9vGTPGtXKs0U7qMeM+eRk4nwhT+6X 1sOn0Aj34HZ9Z9jNY0+F5E5K+qII9Ke9oHHnkWdSStwMsBckHkReX2zgtTSw57K6fCqMcJI0/gQs vsYA2dIJh3GLkyz5zdqWEZlwMRAchNZO9fUpsrXCXUXp3wtjo0mNJeWIxIAOQwOKkfq0oIU4sS9Q lej9F1w3OT4uNlA1OagUD1EzWQhKh3OXaUGwH6VZ93ZrgNLCwxf5eCEsQ4Hx8sPFTf55IYwDin25 D0c9EInnj8og+8lUP4qaICHY5Ig7SMut5LCqKx/WnMc64rE2jb4Doy+QfqxtpcGU23wClWQxUKdi Mti7Zw+7o0G+cHLiDid859SqNdgxsHMaB4vvnKpO4N4AGOCtG2DruE0Bz2gkC87FpFhR6fzm9Ofz 5vXpu/fnTXb9z+ub5rv8QUVB9LMLE/wcAJLZnrOUei+HozO/50XWmQ6cVhdYVx5w8DcgQ/mAGhEv BAiHSdqiGNqIhUD+YYOg6xMJpkCelpDicqg/GjhsNOagixhGFltInagGBOfsHJ7PS4z8idXCbEZg Za1ahK0KaHlUKwJOwDXNngQz+HBxhjFqE5jDwBlOHjhYa84O4r2dnp8Hz4QZ6AuELYo3ahsImQY1 PsdHUZ9rAFGMXAg9LF2qeW7ihuo6OgW+/oOxFJedwXjyVYhijGQxAUG73x99lnrJ2W/vFgSkvn0j Yif1/AvQ7g/v30AL+9e/nl7FGOAW67Exe+mC7hbch9ZhMqBn9hOAmk8YgDxRUA7ZAqAfoaqwyZ09 QYT2Jhz2z1o4imcLQkOXPMits0vNtDqj6XDSau3G77Y6l54OKwdF4NtpmQDvsqZzVNSc3cI+/vmo 72wBCmFXX3LFeX8ttEC17u3+1PGpa/41Xy2kV1zD7VgxS62a1bfOHnFtWq9Rj41hLwPt4fvaHzsd t+fC1oF1lQoMJwFkprXHY8f2pIFW23nlxfCyZhkSzOqEKomv+IBwQn2+czt37A69/8BV+86902e+ 05l67gSUMXIwfLgGznP2i5wxAtbugj6ICiDLWeXqYbmay/0l+3wHH2AXLefLxLP3761aqVKp1kqn 3W7p9U2zJPZ6yR2W4KnyGJ1aS/VRgU+jUce/1cODiv4XvzYOrfpfqvUqfjts1Bp/qViVSr32F1ZZ 8VxjP1Ok34z95ev0dlQaJj837/4T/fzijQbsyLGPGo1eu3ro9HpOp9EDNKjUuweHjUqvYR1Ve93D g1r9wGHvQOq8dsYMJMVK5YT+Y7BY1Rw2c8L+CTBiFzZIqTb7aUh//9P37JHvlzuj8sfxq9wbEP5O 2M3dtMisY/Zf0z68blVZ9fjEOjipgPRYOa5UctfTNprZT9j/vD+9ef0rA9Rk1dp+9eB/GSAnuQBl EArQZpR4ctfcvQG3fEECuyOH8yzJuJUE7aMRSwleXLj7nAvsu747cPu21wdhayT9Jv60XUKm4Tp+ OZf72emN4DWbfAfIGwfQAXbidIsojvFxYMvQ+b1DrUJbnZHnOf4YfXzD25wwkUGLeM8ZEvmGbxPP AbGCt+Dz+AL8q54us19g/JM71885Q+jPHfah41ZnQoIrqgID9/853v64bw+Brbd9pw/ALHfYZ2jG +UJgg27G03bf7ZTR6JdjvtfZb9udT9Dgfmc0GIDS6e93cCCODYvHxOff7KDC9mI/peR2BvYE4RQ0 A+2AyqTeNF+bMYF/M1Zj8nl32OlPu472vPpWvpPdsAPsJsfqrOf2AdA8dgOA1qgDsHxYGdSb8nuF IjoEu9ARv1Aq5HJdt9cD5nrrgiCwPw9A7XlP5EDNdb5AL/WDTrVzfFArlw96dr3m1HrVY7hcadTr ZICd21cOZpSiP5QSjxu14hHbwz9cI3tN98/uBzeee3vreP7l8GeQHW4Q9/yWh3KC7+T/m6Sj5394 X4vsYtR12PMh/Ftkl26XicWEr0V25eTYzk4HhY+bFr10ghd2/kX/7shm6M9LJpstMGwNRDN85hyk qdcO7Jjn/Q5JM3gx3ShpeHk+vgJ1UnqF/hjYPzBWfZye03e7fhGQv9UfdcgBgd3gHrY7d/l+p8jE +7CJcEQF/sC/+J+d1wDe0ZC6b34Ze+w57jWYUOR6gfV7rgdafr8jO9k59RHN8mf+aR5eoy6osyIj eBTUg+lmHUAx1FroZ4r5/0nQ5v+2ofNPMmykWq3VuS4PbLlIYSOdO6fziWRwoGDkjYd90vps9z85 Xl7HkfgHkQajf+O5+FLgC61Wn4n1lxcFnXiOARIUjYI39p8ztyfJuUN8AOc2mXpD5ngeEMXn+/xJ eAzRQ60n+4+X7OLsvMDv7jieg1Qw37y6urwqios7O3lopAOTwOuvL980W7+AiP/hqtm6uLxpXX94 //7y6qb5phC8gL0O/Nv8blxQCoaDAY3yyNgBPOGrYhJE6GF9gJrbfaBXXWIjuwWyMhrDv7PvgVP8 99Tu0ww+nJ8LwIWnwBadAl/mwwYpkdXDo2LVWt8yLzIwFoZtxPggjTSrgTffCoBauIbaXHxEpr3Q avhXtAvde0dQiPA67C063b3wdD3ZQ2yk03KT3ONQhUkartbIdOkpikl6CeJBv9/C761Rr0VqZF7S WkaUtyKxVSej+EKBI5dVOyArtVWvcSv12nArYS8sgXTaOqB8xYwAAGlmW3gJAoLB27+Dx/O7qeML xBrOagJIXyy2xLbE5HQFIAmxW6qtlyCATh3FCmr1OtGI2oFVtCprXUd9aovFgyVMVDA5DjvB4fbo 2p4SW0I8/IQeEIx/Ab4vRJqAYrg+cPILEN4V1x/Cj8La6AauP/bAfoi6ARZG2UCa4IMuqKkpnAF8 Iargt6bjvT2BnzNRIySg5O9HIE6iHCPw4MWMHkolsZxSTtkRgotYxRtClybIfiCCcrw9rBD5qR1b 6yU/SdRnceKzZtrzcNLzYMqzI3eHQXpAMV8dNGnxDw8tbh0/rguiBdjRGnvuwPa+tj45X1v2ZDIc +a2eNxq0aCZ5XVspMpIan+NiAwogjLlawJCJ3uDyAFViQiYvMn/EsYYBLwQpu+P4xMEFL4dFI+bK EcufgNLW6jrjyV1eCCDQprATcLUfzQ6BcYDkEMFkWYKyQgTrPejLQ8c7G/ZGzBuNJrRyy+ou+C6t lNi1ntMjhwR7+ZLi4sVOdYdTSfewy/IYBABsn4ZIAwjME/kf8RGiLYRVSIbNqTAS07nAsv8c/mEg l436qP4zsXoMVo/B6nlue4qmHFxBMrmQ+dvHkHgy51PDZVgrJ/QGb9V3Jj5fLYQqN52jvcafAKCp AQ1LuLNtCMjeBXxuf2VXiPKoz5dTWAkC80eC0q4eEDaCxlGlfXjQq3Vr5bLd6dUPnHqj1khhIwga mmEiCB7iW+VAbBXF3z3ns+dOHL4tWgAj2DjOdNwF4tIi723sVuHgL/Id8vvvv5+wM6AJDm4FWLW/ 4S8ia7xIOBpqhrd/4/vitPPH1PWcK97vOWiofl60jaJIkfXsvu9keyV5r6A/qIr0Ncak4dHKIOn6 F4+7siqHyBkty6pyzigX3PkCzQaSoOSN5mrrnBLIIvJDmGkhUFyVwU4D+mp01jWq3bHilHQKrlrj pImPaSvBug3sT06LUIFvLtLyX+gPlV6NYW3GXusT0Ad4o/n7+6vW388u3rS4p7B1c3r1tnnzIpkW zbCptlM+KGhT57h3aFeOu9UK0Ka67VSOK73GDNo0o8EojZrxMGLu0THi7dExt0ghaFwY/2jUl7pB qzvoa6IcQjHhqTi5j8ujgTQa+y6JNC1MH4dNnqIv4/m4Xs8IrM9JHsBgVdEIDUYjBzq9es5JQxy1 Cg8jaCGu78jdqJQbao+8IW7va4v7joXoFDM4nWhojYjLgoLLViaj1unFP2c0ozBBGq7qRMLgb0MI 90uBO8dQyABpIADEieIuQHRa3CbUA8I17HA3zS3IdUMu8g7hJz50/eHn//7QvPqnj03tGyuY26N/ l1nIHEMrOgif085kxkKJvzF7f7Z/pD3/GbHj69V2/bhtNRpWudyudI+q7e5R147u+NltBZt99nNc d6txk0MNjdBwBSboeEO+J8hZ3ZkI6QTWEQgHrA4wf7HOnE/pRgUmpXl3eG/3zybOwA/dJ9x+jva9 0edrESzB5Q20OQ9ZmBydkKCR29MGlttbarfuIQ5SF8Ac9/GfP6Z2X/QArEL0IFCY3Jctzo5M4rNs 5EbE/39A/n+4XOqOMGpxiNgMWlAJRIsSba8SKnQl6RD9WpobGTDb/1+rNKqNkP//wKpVMv//Jj7k /68f16r1ttU7Pq5220eHjeP28VG9Vj+2j+vtQ6dq1ZxqxbGs+qr8/794bpFVLfYOIM/9/wcnlYOT 2tEs//+B8v8DcjITOUmTC5CzqNz1SAhA60GCneOe7nK5vO/fDvr7QNeFu7SlS2otUo7wCeGorjfC Hm9oBbonYsZbmoLmo16J+aC7uxbxm+NorLBf2jo6DjumrZmOaXMkc+fUXvAFwQQO7Han0wMkscvl erfWawA29CyTCSzYMOcIC75ETu1a0Tpme8fo09bD3//HCMb735hwvP/BqDfQKmTk2/+yn4Dg9u2O g2T5FSlKrSFFPorPT6j6g240eKUusZ9AWLdf8Txy+lxjCHZ76vYxViomUNtHexcoSZSlWGT2/W2R DRBLB/aXgkqgstmvp7+dXbzVGuaEvsyoTAP/XCqTmjB6Y2Rib4TmQcxK1T06J8ihglHjLBCaOKNX 2g19jtpVOU39ktb7vMx2tMhoVhpplYHRmi2Gst7L5t2fxt7o1rMHNMLh7auVJsXn9mbnU5OaesKW SN4ncoRUKpIpakxuPzI7824Y/j/tayu1jhWMelpKYU/LehYoMTE/t7dMJnzMyqVxIqVatUdftplp /atanpRpwrm9VGn/cj0uY0aMVpXAopK4BCutHJDbW7J0wAom8iBUWhMJWKz0gdlaUAdhy3AvoU6C XEIx5wXqPKwEOVc7sZjiC+uY34NwdhUoexqE7kZyscLDXSUapqg2sSAtiSs3IZdsXskMXJvJCDja HUw1skiPsjDXaT3Jj0ocUhfRkCsxx0E+eyEejRrML8WR24upxTF/0u2vc6bMjLV9ECJqV02NJdJC Ke79UsLL8saHYTAt6TQUehRMm+fRMmGiNHWrEzU2vT/V8E/xo02lQGpKnb5PrrWQ/19Hn517xyuG 9D0g2YMYmqi1ouCtXQqvAEcKQK9ymfx33GofwYMTfXWW3cCJZT50QrpInY8f+5MXstSHjj2JSBjB wAg8NIdyNPcvCu3VwSW59kdIEUpZ/GNF8LjgydCCvmjQWR8oEos+mPx2+aoPub10ZR9WBEITdl85 zYnJ6V/bnhPe/ZgM/xgde/F01NyeXuTjx1vYkwfrBV5gbRkRU5rc2UNS1dYGwsT6KA/ASaNACvD6 lMDTbxGwtAu6VKeuxnDYCJj1HDpKbe+OUiTQxfGamLEsalPV8+20IaKp9cEW1qA9aWEN+Gtc41o9 q3VS//lFgQBBVlEV6OGYJRcqildhyUvJHQ+RvJQFfHEpS/0OlEhtZTXSoQz4cZhETiRApvIs50vg Bgr7NdQd6U45PqzVrJ5z3CiX65XjdrvSq1WPZrlTghbi/CbBXZ56cVzFfL89+tI4Rh/JjTtwTtAN ddQoW/UDBlJjvlI5qTbKcKWQYz8B+PrOq2u1RL9hZusbLC3sirI8AIErzb3w0z5/BXe2vn1fTz0s sdD/WmSfyZrEuEuH/WR3vNHw6+AVVjH+aV/+QucAirL8IDgSgovUGVa24U0GhEcThH38Y/fNJBZ7 POZpu4hAXTV6rPyAvQpS8rAh4thI+eGNiRH7BmXSpmBk7s4cPW9v7hSCnQlffaczqb1S6/dfOJhg ZfR1OQvgK7YZTglDZchnyqNflQJhOJqS9IufR6gH9sJwSd+JBIfY3rgZRRhu3OK6vOI65VxrgBE0 AzH0q74wUqEy2RJBrC5/cEDpaI1BfXzuCEsFyj2NpCjy+Yv0OgfWVL+ozWKmby6GnAWMNA27Zins 0GWahSYrkQerjanriNH8NbuPtUGMhjujAXTn+oh3Q5Eu5lPJ+88jrdqIP4P5J9mX0s7tNN6DGHIY qgkFaaGx5VqY+bm5c2AOWvO4XNQFptojRfjkfOUWUd+xvc4dLuBkOu4rlTDSZALtoOQAovr8XAx4 dUC9eQ5FPXNXgdPruR0XaNKMZn97F7S6RrjfhNTdFE5BLi3yjAjMxjUbtOkdNL/jFpj2JzRj+u0O x9OJKGjgfLERj4s40Alu/ldYJBsnw3+ZjaoyYr4cYJsv4BohowZGAlUwMiFF+su7TsIApLj3KJgM OCWMTXg69NFtCj6ald7sc4YxeNVjCGxmqxoDvxBDqE22si/4ivhWM9gzj2o8OCxaFkhl9SqIZ5V0 YllMYyXBtS5Jdn3rDIlTxopme0qsk1CJlw1uFMviArHBdnk5lVveD/AbJaAkULyTiJClWYy4IKWw lWvJGrKS1EQMjbdimuLUe6jvRDZgSLogl4hu+0zD+mMgFcv6b1IxeY5NSXbsqDoVdKRd0XXhyGbW 35xrR2FCYkgGv+4DkDmH8fxuDjFMoulnF0kUnYUc0Mn0SptxnH0hPWDJ0vFQiKYAppRRAjqkXjq9 eBO8EZFT1GOXV8q7Uy6XC6ENox7TYttCz4VbliK9WOT0UF4hRZ4pqK2aMQgb9tzVijVrb0KomDWg MOB6SbQ3Ij6SXBJj09qEGKAJiNGVt9X22ARwpbEyViwyVIeNjEvfE8FpToGyTaFdJv3lJj5BXBcb 2QLSyx4yTSFn4Hf5VXBLJJiacKG1oHN7ro8GYvyNzh7EbNbPyc2xxq1EOh6+EkU97Z6AQW/bdlh8 SIvg2yxsSyXgaq2uQ4pNIbzpAA7bnBn7R7i8eUTeeNSaq5H8H4vyf4xEn4eWAZ2Z/1MF5acarv9Z Pag2svyfTXwo/6ftOO3KYe+w2zuq1Y5qh43KwZFjHXYqx4eVyhGsUMeyjqyGtc76n9WjE6wJmZz/ Y6n8nwA5I2VAz4aMKsKMetpTaIzvcnshbX3Y9F2nP7ElURh1OlOq8kx2bHolR618BjHQQZugeptn 4jjo/+RZOthT1x7e9gMDYZFbc1y/mCPrD6bwDbtYAZTbDzkNomCyO4cbNEuBgRbLlg6fTRidIVnO 5ajYCHFUle6EBli3Q1NBU2xXmBVBYLgdeaBdDHBQ57aH1tsf2f+9G01zeVilwwL7n+r/lpkoXkpm dEpYd4Y2prcWmayeSNofzPYrGURBwdjFh3Paw+zWs8d3u/igmk5RgNNzxkB5yTTHL5h2U7KSE5W2 sdisgoMtqnBgNdVRl8pyl9kNfxUPO+BNwTTl3Zx6HumpCWpsvut60DySetlP+yt3NvJsqzKIYZ27 HL8rqrXa/c60b0eKwAaI4ZdBhaZhcB4DLXWLuRAGaEsMXeJreC8vB1QIRiSeR8tf20HnYHjMZV5t ZuDY0sz6GSsLfOWIyTEZEVKOICda5CAdfpU94Ow4xvLXBLSLaO/+jBUYbZ9PFB+JzIda01rinXZF 8RXeFOAqoBdrSks2e6ehCyDkO70cAnlYSuSb+I3wIW/gKya9Ab7m7iaTsX+yv+86jvNl3Ae0LePX 8si73ZcZgPt1y6o2Duq5nOnWu7Pvk4W63CKndM73QsxrLoUNPbfA2Y2rPb5wobP6FjkXL1Vl4X+z au2YxdQBjisf/G92dHicUIBYT8+MpFVWK5WDSMHf2iYK/lY69V7t+KBTOy6Xg+K/6yr4e1THer/w Lz8TAk/27To9xssI5CK/scQA5r7viDJcWAhF/aaVxIV8oT8ll19ddIfhwjEvsEgPr9zj9phL55SQ KiFC0BQGUbUezM3feY79tNAVieVexTXRLF5tiUKw0NtkZ0evDAcX/5xdso0fmFGt8FqnlSNeMUKv sxH7+qziHisuJRc0T8MRhYARfS6HqJWESvBcOX0uB4idAb+1Jn52JwN77DvQfvqiZ8klzwAljIIi fkuSoZaif/nLMa9pMBrHg0oRmJYkbAiUPNaThlu9IcyaV8bZfy4KfFSOeYGPqqjeFy6OTO96Tp/q G+N3VewYBg6CJGK6+k7Fpop8QKIGsgHx3HwMwL1SSqhvGKrFwV6yf1HFKlG4Sv25ODsvssqfuMke 1pDenmqUqQpzCYjJUfJHDen+FOFC1hFtDQtPcFpTGWCt9p+HFY3gH17N6KVRQ0WWipKVt6PFAhVV KqiiTQm1NJeoprmuGowzi7emLMKYtmpr+LFwoU6+5IdVvuRHleLhOqu6/hmqwgqcQdOmtBZ4u1ol By7daIxiJjawH39ksVPWarAprOLs5TOaizjPaYmix7ygmvYMSE4oOFFhAPFMD9C2Fb5u1EYOytCX XlHzBbNRxdCiPQal8EW9VbOYnZphwC7lNlA7htPiHaDFWLpOUuaYinUSoP8RT9OBlgdbLHmPLbPJ 1r7LtG02P1U3YbuJ1sJg7AHvBa2mRdCCAdgceczTCBgxQtGCvtyoDSCMQfLOq+vFmcW2R2Osta2a +zMeM1RjEXz4zQZkgDvYe+RwAgPpLHWVOGVvNKVyc8RtNIgENQatIkd3ruloBydo3e9opXl3nk/6 jj4OS4c0ISMekwAPlV5hwbsig8FrWBg0irOi0wEsXB36kXZpVOt6ZVvsGp4tvYJ/hiOkGdi2/Am0 Rd4kEcq4T1e0naINckeCkNPm4LpWi1ouavjbn8YWpYaeOsdbopDGjHrEwUaIIeQxGyoddsSS76I8 114tCO3jvjO8ndzx76g78jJx+fBwiizMLQoFytaWQEpazcUXc81rGSzlEtnlM8msZOxYw29ICghI BS1lHAmxV43efEN7Yuk693tyOyxWIZ2yBGpYZ69RL1oH6xICRf9o6+MF9rUTguRFRvo/EmUk58HV Ai+rXwpOdZHPiU0KWh5sJUyNbtEZ5IVATjy7bl1+uGletfBuzHuFkFAYkS1VIuzKapMbpdgJv2iE PLV7tYVsN0cKliomH18D/unSw/QWVJPuJSgzpsQQI/WjEIWnkw67+Zi73JPD0d1QQIRQgXewRD1s WWRzpoFgFg1IPrXBoDol44IgACCkGXtfyqUuATqQ4viW5xGzVoW01Doecrut5xOJk6KTt8FDziWa b/5Y+UEia9kqMeQitC8icDPMHgoGUWtzYPowHgybPfjdWeiDb7jDT/SqqDasYTlHyAY3m9QPrGK9 viaEFCkAQu1if/IytLk9rFMcp6mzkjh8ApAFy+6Nxuh7HHlBbkybnLoy0wed14BKd3rqDxpYNLNt bm+unTe3h4zz0u3uoI48HCEdwPliOd3nfac3oZLlxlUq/a8uX9GhdDs79CzpWb5+mT8cXJc9US4E 8m+8GFPFnx8RcOaj38FmbUBOPKVkzB1jf+OWJFKpUNNE3ZrPqcD+/W+mS/WjcekViEx+AU9es2i3 mQuDPeHEAf/wWQkDNXXS9jFFze4HjeETARjwEd9B8JpPaCDRdBn4JU8FUJ1oDc56PuiSXtCgCM/j 5Rv4mjdaFWBqD/zWcDpoDZxB2wFtSRsaQaaKcAs/pA9IPYUQxAcJ6/R25PCC5ysJ4MYBjcbQDYiy iJlId/II+CLTplRAfR1dH+q51mjcswduH4tE4+MFdR5g0I1guEEvuEVmdBL/bmjMf/L6zlRjfJYn BB9Yx0kiVbtyXO/YR9ZhuRycKrKmk0QODok2wp/6se58dAA1cuz/UMamwy6a/2i9aZ7fnLaaF1cX p++aDDjo5xYF5OwGj12evwk/Nup31WOKIt443qBJsRL4k1+CfilXkQ4zpgidu9Fnogci1IOf+QJv SqtDJLKFIh6oNVs/XH06xANgTKalglUEGTWmTUQSY+zenF3hSQ3NX36BP0Vx8ewi9vLFpbgACKRN kDBMm3YwYTGT4cgbCP6LdYFw4jTIQNwrs9PIUzCFtn0LU4P71BxmeTpUVmjY1UEFF7QkbXyM0pu1 bFAZAYSsHIgQtaZCcrpTOt3YHk64DZbzfwqi0RNKWVP1CJtoRCfZ29SSsiWQjyAYSRjwwrlNoBfc hB+A+oLcD/w7Dl8zy8jTgzSPNHdG0ytJPeuPj22MPfHJ982/Cyi6wxj0oheDpd3hqEovI64aEUs0 Ph7JQO9xnNCx4ex+oAXbvMUeFHKQDAKTDYd78YHMgh32skPV61+g7ISgg+/Yltwi8phtjFejcBsx cQ0sBAP5Ppk5FHrR3sMAsgAi/B2y26lwAONNYSqkhG8en4wnP2FzzxhXjLR2AOnx+Dq9La0pedc8 lZsZnQt4x8CXwK+cyBiVIKhhS4MxP+ULWGyFHHO6j3/i2UMfw6QAHfFUJW8yHeffgGR45XQc9x4Q 7DlmzfMzdrhEB/uzyAPi4LkOkVcXBB8pmx6RE5d8uDUtwOEK8eam3SfLD3tO2xxpqp8P3QGZm3vS aYPIcABn6LW8CT/YCYU4ChhoDu9dbzQc0Fr/Ia5ET/mIHiTVpSojnUkLSEvACv1QfMN7PAznmo7H ec5PwOEKkWg9ZjXYczpFRwc9YVWqwAnZLp8wCg4BicRRD2JPQPo4EZEh5uuDaX/ijvtfW4SU8Sdl 4Z2q8cvirT4nHNbaJFQRR29FJxcLCLrF0YZkAK21xGVRSmVovKqRYjzQqa9whIhsXtrZ5pwfpQZp 7g+MxOTcPv/ut5YI+qDTM8RRYxJ+gJytsT25M4+c4Xq6uZ+68KsFQkSRxVwHGYS/E2yw57Al4V6H vxNznd6hvXfMT9SiU2mCrRf41gCprvhZWPg7FivkPAoRUkGZCgIW4RPYeXwsnlnHj5N6jkLSJLgY 8wRKW/oToVknTNqYM9p3zDnQyoOC1RLFc4BdeeJoMLjUp1NEuaVo4ffm4F4YVCBstHzYw1zhI/Sm 5vIU1yQ6ELS/yK5BEx/ekv7YnvZo0srCE7qHM9cuAfXiBwOmGwNWjE09AjqmjR+CYx3JQ3D0PjyH 9oct+rm3+1Ogotfvz2jB5DaZjvmJBvxp2m6NOkNFDddfjRpeQz36/cTjp2TCdxoyfy2CchxdcfA4 e/kby2Vw9oFfSQTXAJPUhTb4cDccTOIKB1WoZ7lQ1Cm0Ijs1iCeP+WzJgGaB+TNJp0lHuAULOF8M 8yD7kFpS2qE4Or6I6kUTx8MD5KHUGx4g4HIYbWXIutDPNbSkfgA3+fKOxqT/cugzZnSASj2aVAxE R92exwUKG1r1iB/MeyjOBUeVxx0MQMQOsbn8+7etXz5cvL45u7xonV69vUbrGLcDkYzyQv0MEAev EXzI4iQiVUvaJc6eoAcMvuElwePuvqBoOtujq0HzLRGXB9C5Ba5sl15NYMk4d8LvaML5n8r/4tty OXaC5RABrIxrslw0TBQbpclRHoYF8t3RwuCCDgP2+avt3wlhDwVPCuENMdedHc5dESoxKLjzPCKF aGCUMcIU3E+VqILoEREsTHEMPEotkPIAj4ScJ+ZcP+Rnjh+AMMvPJF0MR+SO8EEnARrTUWbWf7Pr 5usPV2c3/2xdNa9vrs5e3zTftC7fN69OsQE6dpFeQqM+mY9xes7ntx9eY/74OQUSBAeRkg5ItlWe LeNGwtf080jjDuUUp5FGRXEVJdNCUcmU1YvMPHY0LqCSXIo//shMLyg9aTpAufszvGqB+ykIVUHn 0Z/CPmY8T+3EIUaCUC54vymEayf+CglSOCEQojIvBxU0nrpFT+4Lw9MBdxM1rOoSW2QnDH6C/wtT LNrZ0eSixHtAVjUhZ+e5pvjgoAVAjaB5fYeIe4pVmPeInLKQcKR2nyCuB41jDoq6tQxx1XyzuuQm 7/xLyUb6GM3gJn1mml9GRYIGEaF8wEcVzFjABB4sjrPMiKNjEbEPNCbYpr/EhJQW45L7BOL5ZRlw ysMj1YxgP0URXwYQKGVGeXdPfZQ9jFghJeXjPq1K5yk31gb+ghZ6yjSFQLSXqAhGrhQDtQpfDOEh 9DRLB1Q8sCiVv0j7omHuTnYARWf3Q/yUe/Lw+GaZZMe3smZ6Q71CU/yEnhdoeNrIAmUuUN/25rwe Gl58I0wyjx8NTehHXQFius1DTuvamYhyJpSVyA3LnIRNRmw88iclvoBMKGKSeh1bJA81MBR7qQ2A MdU8m/0uBFNRRTLimuXnbsNH1y116dtUI4vMVBoDHcmAURRERU23U9EvcVph5HrR3Gnsb1EkZCdM OyuZkxWA0GtMGc6Lq/8StrHKUfEYYHx03BCZUTw1huvEaOCKtYYlWB7Ms0OlnPcWdHzx9QImANuo T478Ha6Edfojny6iCQTPOA/ECKBAbOw70+6IdSYupWc2L65EqSxv9Bl9a23HozVDChHI5vkf/YlH nXCtM3SHYL4rygVPyuL4ix8AL3YR6vqdInV9ckJikvYYTHVXHjZj3zr5SfnLwB0WGCYNOvkfpvyN k5MvbpcHNbLLK/GWeBaowA9TKsw9KXfgt3wD6eVPcKuwywEKoOEL/wURkP7pUFoQX0GhNRxXDlez gihW5F2yzTIXBqJTaXqo9CpkSynAg3t7/CB1tpMAcbbLPlzA/mSn5+dsFw9mTXpyl5mnkvzg7xIJ m/c0dEFbBgCcD5AjX2Cje/xTEGv473+zZ61n9OeH7rPQTzp3BBdcNmWMYEfMnyMu5VTxJSJ5johF C+3CuFx5dMvtFln0FY78f0rJ5OiAh04c14Xa97D1wzgJQ+w10oheyNtakAh8JWc5XnY8p1d6JUiN HkwVc7dIYixfjzyoKV53Os7vEvB4pI7oSRwb3Grbnuc6ni68wR5/3XdAfOu5Tr8r8rj9u9G032Wi Fgjm6xnVh3A+nEzzeaLCC22e4RHGbpc0XW56rDZQgLIqlWMRI7W0id/Q22Fpeu4XWneUglwescx2 UiM+iV/pqZtgScl0bCdpw+zoe2P9m0PAbsaWEKCTUA7tBAu4EVn2rMPD4kGF4m6j4lgoyMkwGWtB QvLF0OMyAEkwYRk9LX7+KA1RPLZIuRFfB0V/1XkUVAF35JHmdSPddkmOZlSFy2Y0kYslbYldk0ek 1RmMhR00sI6PqxL1tGuWiDBC19p9FSvXh4JM8nnyez8viGyScbVQEPuER9HQm1aaNy3zTaHjQq+v 2L0R+lOqvghu/hS6SffE94oI/tgT8Qw7GgjaPIEVZg0ds3wIOOLyT5QfoJYmXpNWTt5TND/8PyfI oBBGiS69CFRltvc32bEGzS/tWePrp7mBg3Q3ftHu9/Xk7j0jGamaYIIi0/m0P5ErvOPKyBsqLfyS xZtOC2yMYYydvA9oO+rFPcSjk3k7lNmj+YHVyOXtiG9ZPUIyHfyDgRno5KUdQmB65gcVVoRkHtpJ otoHhVpoQRk7elQGRWTALxVqocUjwIrvj7wgxqLtTD47zjBUsXuHq7dyTeI8nnGEI8iLKYSmiVoP NRbk9iBL4+GvMAzUChzuXh95GEHT5rqKoPyjHm8nGo9RZgogQ6fj+L7tifI92B/GCVElFmoTKdXY G+FTVGkZW6RRlXlNa9rrQSjLMBTLQoEEXGvCfTMR3WL7GPYgR8iDKWJjhgLIkiyJ08/TAIrM3OMG nlFMhRYqSG+EAHw+Gn3i8+NBJhTcJMNkyjz+Z2JxtBEhGPAEv1ylIgx0lzdm43kWLqw2htoCOXR4 SEY1KFskCu1jnZeRw6NyhQHE5vCbqJqw2CBXxvwpleeBFiYivdLxmGqcw9SK3gmApqUz4lgIDNI+ yun1pKrnDgrrSTiBUUgdBFYjNsU2olOIiGODL5m+Etyig+Zidyhj+vSRWdrIhOFHjM2K5DXumIOr STsOrSravNlZz9wGEgERupYMiObsgp9RwleJVpIawLpBQoC01XKW2WnfH6mjLfCcBF+cstIfjcby 3bbToVoxfKWpM7g/xL7EMps9l8WLmmUsytQQQrg0IjpdGo+lJe6spyMBjLUoCjJR/SO1RIb9Dbm3 S6INPA9LJjk1vGoprm1Y4gSAcZJXzmCEay+jhATVFOUlaQj8WANLh4coQDkcqXYEYMI7pywfEPnp GqbUYHwwVrFjQ7lGAmeQWwlRJMCdWigtLjR/S5t/aPqouQVdogLCh9MSPsYOr4+U1x5CV0ItlB1i tiE1I+OliaVbAHkm5p8aUZOshUd/yaDstw7lEvAgLllJiTNPTnId11NhWQIHDBkhPrFT5QbqG1kk vojMzR2u6EeojOBrZv4ypzhBuhwTWct541ohQol+AcbZGt+2VLkoKhwFLypPwSn8zgcSEoy0i0+j ikAtGKEBKFDAH1CF8thO6RUmGiuLmcx/lm4gTHFQDiFKcYZmPk6Hn2IpmgqAj021Dj3Mc8dFek5i Fq0pMJhZ1rJL4otcCs8bC1tEnND9UUlSmNLT4+7zVl4YDcTJaeFGIs/EqvziKZnEJrxhgcivxNI/ Nbk9RrYKwnZXJB2uVjjUFQLcETDWBUPiSO4XahK/hFyWTLWatiScxuI5xEX+bFFFLF45vYLajyTq 74jsfFDutYcQ3UTXaA+SPhhFZWWwrGQMqOny456Q6gi5C2Ym2R4KWup4HF7aTitvLFoA5mr7/nTg KEnRFsslq+HtKb4g/EDepNVzJp27PM0i7AomWq48uM3zv59dvGnBX+mYpgnQ+ATTiKg0eFEgJz4n mYNQgEmN45OajIZ8DPrDSokRznxxOWAEslIJvy5i618yI2BdDFMsMbepYsdVEmupO9w36DFi4WXH EqKUiaPWXAuQ1PqXz6EvtIfAUBcUHpiCIeYj0RXuefVbXBN4qaVtRDGDowLpM9xSNVFKDmwdsghR wXpSLRB2XIvBJoxKfCjrjJ0OZQjwYn4TpYv4E2es44mKEyiyXukVVunqy5Ia/1LiJBcbZuhski3J ZFsJv1BkqnhaKCjAg/IKMn/D7jk+CBeLaEqDncxwESKATIQR/D55lWVud7DKKtsb1uljKP07vJ48 uSyF4vqx9KoP3FpmIDLu1E/zmoevvUhCP6FdxGAJlcIkJPk8Ci4ryUa6ngFR+GLPXpa+WBdP/P2o JfKKMZwN9UK4mGTL69wKLEUqFi7jyXv2P7sTlIA+aqEgAY5Rti4l1OOp4ydCJBSD1cvXiJH1lcKq C/xGQ1dnb39N0ZI3v6VfABkXHdLDeqSqAieRB7pOz4aFEzec/uhWJOCy3SkmXvPsDW6KA/iesB/Q SaJDXO6dWbsH74Xa9pzO6HZIIgFx1qB1rp7loXmeQ31j34pdR9tXl0pMnFMCicDqofPZwOz2V/UC 4ZKB32XU6QitXZ9LPO49+uc9rniFla6ei4ujleKlV3GPEObeiUboKghBOBRhw4FuRkNlfDEGgUSW yuB9doIDyJ5rRXrtHgZ/8L5xCsL8hL3ESzyriOLXzJ87z5V1UjOAInPaed6HNp73qfSSTH0lsVik LOWSk1xzZH15PRreYx1qCbLTize8XrTbcSda6hdmb5MO5Tm8UA2t5Yg3wq1o7q3Lo2C4LReA/g9N yhFsTtpURJmSnLTcRQraBpYd5AECJJzIUEMvk+s1cTjLXAj1vHMPcCFHQSuYl59HDhXztFJFOjYM D+tlw64h+T6fl+wFf4lyivHvktSEyY4tCVLzZSXSoKOr35fJZXqGUkDztXWNT9OVLfL2AOMdl/BX 4rekE7MQm5A6gDtHUxLAz87Rw8UxVVxQKxJYz80X/iafP+FfhKpL+qNc0p2IBqQJM6D0VYviTtRo ZoqzUnKpKgtIxFSm4BsgpDysR8JcmJSobWkfI/OlbCow9Ej6I7cEsEg6b4+HpEuDF/v9999PDIHf ngTdi1zHoOZRjJ3MNOFoaMDlr3grllqSBCsOvw/cToWRaXq7suaYwr0y5gRX9Z1jsqPAz8KvGysq sCFnGmjh32rSguooIc2oVixKWDFNWGlwQuGD2n98RCAKUbtFrlULpiILs8vmuOFPcCy3j1Y+7x64 mcj6xVeDmgspkYDRGSafXSzXavRtYKZue4zFSNmaiZhJNlIDwfAWGQpo0XXrKUIkQD0D9/hawGDT KZz8JfF4nM6o3UxUHI2nDNU1xfDD7yuED4Q++DP+KlsRqC8aMWrk6Nwqvk2xicwbBvXWt69pbaJQ OqUMBwbUFLakpPBPJb994OGFyjP0zKdQwykoIW1QFbR8eHnWAuWuo1RXVs59cWazkKCQkMpTFlRS sjykQL70+W7kO8o3ZZzsoM4zEIKduqEddZB3ym45oquMhgEd9+nEg7Z2gsHIo+bEOQSFMt9nMm9d nm0XPveBp+p3SLop84IqFHlL1huPEqxl9CZelo4+keCvDDu6wEjlo/cemKIZiIpm/VeMXUbyh/ZI 7xOL1BEgJU/MObo4Ya9akXFXnTDPx9DuWH+EbsAVdEaYcEV0cAwLi9nqxsBfcM1GxA3PIwx/5gxf 81ADR6iCggYQjpR4AoaKpRUoyhuKRehcjDNyDtgSYRbvmgz5Ek0Hkc4Ix7ENB2yQhLsAaiGwYaj6 OHzXAJUgb5MQ4EMA1U3dOi2aET+uyNGVkcaBXnby04OOokWAY80TvufMzJnQ9lR8PtWmJH0NBrHq TGe+T/mbLNDpkLFc0olFXHVR0UfcXo2/SJ9gPO8Zf4fdS1InBFURU8gC1vkLqFOvbVQUe0NSBoOs KZEzJfNsxniHruRFWDQiCn8G0IxUppaIlWz+/v6qRRZmHjnXujm9etsU5tveUDQp+85rdlzNMbJL YWS7hYIoQv/6snn1utmCts/PXp/dtF6fol5W4sJfb1h6Zd/eqgoAhvl1P0AWZTIJZ1LxuPagfFMk zVzyV6m7Se+OdKrInIhCxKqicElnAT/w0ExKWNvV8in4cPEYqoEzwYIoaD+0VaycZpLTRiz9Fjx+ 7POdCx3k8/Iqxac5X5RrTJAb7saU8g432BfYK1HOnGiF9ALqdvvYXLKw70G5vrjzIbAqasVugjiJ uAQz5ZoICBcfCGIIjEv2dO30e2cTZ/Ael8HxTqVz9IKCHors5uzN5dkbxJOiFtwq/bZcPtp/zj1f kQN1bSzNUuLbQEg0Niw+8PtwdLGYjbJJzz6EwLC4azx5J1R1Oy/JSMGINi5ES2ILwXqOJ1mXQf/D n3gYECiduTw4XgYe8+dkPEEs5NWL8P2m+fuNAvOb5i+nH85hg16e8zVs0a2KysXQy1P/aYqpAddO MnoisBGquwFW5bhj5AOeTES5+IoK4GwKJm+gTC6y8QlBUMQ6aW5KblAUrQLdA5rvjqn45lDmMaMk afMmKAIN+IzuahFEVsrm+JM7fV+jeSl/dnFTR5gIeKC2BYDTfmmYql0VehEges0qmI/DlbfO5A1I 4wPhe9Pv8wM9tAtIG4XXhTtQeVEUpMiX3utRfzoY5jlpLzIkrWh14wii0Wr0hhcZN4/1hurYEEGR 42Egi5oIfQTzmjD5gUfQ3In1I0oXWhQevf5f15cXP7MRMUMhSaXlJh/90bDVnrr9bou/n5KzLAgf Ps0koIjGOiPHAxme+IpsRvivYIK0W2imakvRCHEnyTHGDfr6RvVj+757O+Q8uTPqC4Or0Rc9x+kF gkkjGYEJksak0MZINJSROAEdYntMw2GVydCiopTIzFotvWSnsormtLge1VpMGEVwk2iPejVZow3z 8IjoGBxIKE8CCWcDI06KI444wy0roVF0kSw0UmKoPAIsONJRGMvlCXlcE9aPMDQ7VOqmil0VxzGa vfF2/JEMy+sIuz3VUAvkHF8dN8jE0ZUwrt4Uy3oGz/AhjagM5D4VWDBa6DpjGW0rxjvyudVKek9w QELANmPGQ4I0rwW6VEUfEb0PjCwpAkS5kKWPeFb4BzwSCfowQz6wC9n1Trg0o8a3wr1HXNVJ4Qgz Aw8Sww40pS/ZsU8SSRjKYX++IcBGJ/hnHGgTJregF15NbrZs+VGtRTi4RTMhJE1TeOyNSQbA0Tyn uM7KGSxgYdxmRpUD9jLwZr8QgouAR1yj5KtO3yq5h6VKHrsigcSZPHFvnROnGaWYOQJo8YnPmvbM 7RbRwhb1bRuebVVkVaV/U0lVyreyGrw611EjyDxMXY0rdE7ct1KLiwC62lJc/DTJIHPvDZ4CLWsH vYje4uICldBq0RO0WPVaHRerVqlVRN2rVYNcZjQO3OHA/kL9o7UsVC2rIHKS4+psRQprYXKZ/K41 W2S8KpZ5TKNQaM/48dGm2Tts9OBnJQeKOE+1gNH+h6Cvd7Z/ensrwssTal0Jc3Ok6koxWmS3aNSp YuJQEVFCQypTUkbi+ejI4E0sFtAUP/kdUYMt9GRBJpX+Syz+AVUvqVXgL0/yXsfi8w9fGqrihktL NQt+DOFkkYVrFTBB2UQeLIxNFS/OG7XKImWOo61rVpc35iHOq0CGhLpiycgQGbHKWiUM+IfzjBKC ho4q+y/l70h5ibIsblGrHdOK1iuVWtFqHKi871UVgBPcBdsLl8qP3QxKw4jdfvrB3Tb5hLiJF8Vm fl9qCzzq2oid5n61aKkNHvyEZYzD1m2+gMK6nSpDk3ubtq+mHRf2E2n/XqSUGjapx0GRtCy8Xtoj IG38MR1NeNgOKVMtt+sMJ/jVy1MgJwbuj21Q1in1WivGcSHvaBkXBfOUwJ3E4h36Ky9C0Q+dYrxD SIvp6lBQF3p44N/WmMSiEHRExjZGFMTdDs6FirsrnEaJb9Oq0A3DuReyM/Il9h10Ne3umpes4Jqe ZatiipTb6D9i/UbxWSeh2iUaCERYUeiBAAjx93UwxD+hSi7STZkElCYLSDrjFs8DWkEikCr2lSoX SD09uG8FL8zYOLuD+92izLQKtYGW0pSt4KORdqggI0juVK1CDZYu8AdElOOE5CcZ47iTEOWoZzYl pDaFsSzBxr5A4pIWaoTcSgYYqZicripd7vDW92W/sdF9QaaaCPOTfU2wPFVF642rORTYFNhb0Zap QpBUn2h0ajuYlii5cCR7mkccEXeVDQtzD3ldRc6lzMMLc0IVagHbf6Q4VFkPaNIjk1yR0Wf7Qa/S 3MRHLGOc+HnwPJauG8RR8YeCuE7ZcCjelDz6MeGm82KigsWZFXMXrT4SkCBQUX/wsQoIQ/IYRvsd STOLuyoEaJ+7PCigi0Jv0b0hT+rhwS0k5/kTjDYTKdZiiWSVLuUOisnWA9lPcDM9sU7wplBmHKbv GWl5IQ9SdOYadcbiLj/4jE9deYlCFWPNF2jDFxkLSFKR6aRFtSLYz+nFm91Yj1NO9yBhIjziKFmB kbTjD75FeMZD8/ez65trkXzj87fEuxTFqfK4pC/RdXw9i5pwOHQyFUDPH4lGMArUcymwBI+0wpOU gCaU2a+jz7DDvSKdG4JPidNdRHYYFlQBQdWR2WM81FAOnNLTATfQodnt6lsuwL5QnhDL86pVUR5S ejWEf31Rq0rxsVXxpJQcSTowh+jBDJyX3NfAEQWwqWpJNDQsQXIR+XmYSauya8SsYl1BiRChnHpO zzh8FVUTqTC67OMqB73y0Lsx3nmdelBxH+WSF6CObiVhUQH6wfK6u6X0iuoNnZzAwnRAbgMglGHt j/hGCyAvhsl4odrLq7z2PGzqXR0QxpQAKj+jR43ZWuhqT+ryvs5BRkRZVVCeio0yzjIx8tRQemqN JTldLI6pNTYCmcZhcTIuVCnM56Nw1uVBAndAsSQcNbkW59EeTaQUGE83SYiXARnhPGGDd2ixGPIo w4hkGNlDsEV+w9OwgviuIII0GnAXnFBdxOguM/k/mZJHwILE/OyaXVze8Hj3JJ6mABZPnSWuXV6Z KJhk3fKR72HNVbX/wuK6VBmDLANdx4ydnKpOzoe3e3Zx3by6YWcXN5c4zfwPsIh8z3FbTqsgSoQF 9Q8pOEAUFZP3dnbz4u4bIEZnF/AFW4IWT6/N9lS1xOBVngEKl6iUGElbqk9+D0EvyJzsp6oaGtwz 2UJB2L53DDuIxhawCLl4JHyVE0TxPU6R1/FCb8joS+Pq9Aw9UjBdasusSnQZ4HrCEvAFKgY16n1i 3PkqlynKCeu7s0vtbcdCFtj9Q9Zy44uLm+/6/Rkwa6eTl8sp+qvQAXp49/LvLb60fHsabFy+zXo2 cNPuCZXQY0ZTMtJEDwqIN1wqw12sqXS+4U6ZrR/Vcvdohz1wAx23wiXa4L4zC5xunwzfUyuUZGH7 5HxFt4Cfyjr3YCOc3JCrsrwZ04u3nMkJprO8zRQ2+0Jrm47JrBG1NOjZIErwJHApgXMFghoXkNKI aQtIabOFtEUsDcaazBDTIlJatCm5douYLHai5dwXGWPC+HQxUQmRmaaSaSpbo6kkyBOLaSrzBODd NyAa3jRDEmYgXmJ1dCFY7pJsfHahxEkpFsNfEkDDqoEu0NEFY5Nqop+kCkKIC/005IUii4iP2MSC kiGf9MMlQxncYxocfYrwKZVYzipXD8vVXO4v2Sf7ZJ/sk32yT/bJPtkn+2Sf7JN9sk/2yT7ZJ/tk n+yTfbJP9sk+2Sf7ZJ/sk32yT/bJPtkn+3xnn/8PVGG4yADoAwA= --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0001-Add-a-syntax-to-create-Incrementally-Maintainabl.patch" Content-Disposition: attachment; filename="v23-0001-Add-a-syntax-to-create-Incrementally-Maintainabl.patch" Content-Transfer-Encoding: 7bit From 058ac1e8c9ff85ee1100e7996ddd71ee8086d0d1 Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Fri, 20 Dec 2019 10:05:02 +0900 Subject: [PATCH v23 01/15] Add a syntax to create Incrementally Maintainable Materialized Views Allow to create Incrementally Maintainable Materialized View (IMMV) by using INCREMENTAL option in CREATE MATERIALIZED VIEW command as follow: CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....; --- src/backend/parser/gram.y | 32 +++++++++++++++++++++----------- src/include/nodes/primnodes.h | 1 + src/include/parser/kwlist.h | 1 + 3 files changed, 23 insertions(+), 11 deletions(-) diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 10da5c5c51..9c2382fed7 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -440,6 +440,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type OptTempTableName %type into_clause create_as_target create_mv_target +%type incremental %type createfunc_opt_item common_func_opt_item dostmt_opt_item %type func_arg func_arg_with_default table_func_column aggr_arg @@ -669,7 +670,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); HANDLER HAVING HEADER_P HOLD HOUR_P IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE - INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P + INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION @@ -4257,30 +4258,32 @@ opt_with_data: *****************************************************************************/ CreateMatViewStmt: - CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data + CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data { CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt); - ctas->query = $7; - ctas->into = $5; + ctas->query = $8; + ctas->into = $6; ctas->objtype = OBJECT_MATVIEW; ctas->is_select_into = false; ctas->if_not_exists = false; /* cram additional flags into the IntoClause */ - $5->rel->relpersistence = $2; - $5->skipData = !($8); + $6->rel->relpersistence = $2; + $6->skipData = !($9); + $6->ivm = $3; $$ = (Node *) ctas; } - | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data + | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data { CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt); - ctas->query = $10; - ctas->into = $8; + ctas->query = $11; + ctas->into = $9; ctas->objtype = OBJECT_MATVIEW; ctas->is_select_into = false; ctas->if_not_exists = true; /* cram additional flags into the IntoClause */ - $8->rel->relpersistence = $2; - $8->skipData = !($11); + $9->rel->relpersistence = $2; + $9->skipData = !($12); + $9->ivm = $3; $$ = (Node *) ctas; } ; @@ -4297,9 +4300,14 @@ create_mv_target: $$->tableSpaceName = $5; $$->viewQuery = NULL; /* filled at analysis time */ $$->skipData = false; /* might get changed later */ + $$->ivm = false; } ; +incremental: INCREMENTAL { $$ = true; } + | /*EMPTY*/ { $$ = false; } + ; + OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; } | /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; } ; @@ -15584,6 +15592,7 @@ unreserved_keyword: | INCLUDE | INCLUDING | INCREMENT + | INCREMENTAL | INDEX | INDEXES | INHERIT @@ -16136,6 +16145,7 @@ bare_label_keyword: | INCLUDE | INCLUDING | INCREMENT + | INCREMENTAL | INDEX | INDEXES | INHERIT diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index c04282f91f..a0af94eb49 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -117,6 +117,7 @@ typedef struct IntoClause char *tableSpaceName; /* table space to use, or NULL */ Node *viewQuery; /* materialized view's SELECT query */ bool skipData; /* true for WITH NO DATA */ + bool ivm; /* true for WITH IVM */ } IntoClause; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f836acf876..2cafb4e7fe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -205,6 +205,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL) -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0002-Add-relisivm-column-to-pg_class-system-catalog.patch" Content-Disposition: attachment; filename="v23-0002-Add-relisivm-column-to-pg_class-system-catalog.patch" Content-Transfer-Encoding: 7bit From 906915d5215de7d3271e69d81408e875d5713bf8 Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Fri, 20 Dec 2019 10:07:23 +0900 Subject: [PATCH v23 02/15] Add relisivm column to pg_class system catalog If this boolean column is true, a relations is Incrementally Maintainable Materialized View (IMMV). This is set when IMMV is created. --- src/backend/catalog/heap.c | 1 + src/backend/catalog/index.c | 1 + src/backend/utils/cache/lsyscache.c | 24 ++++++++++++++++++++++++ src/backend/utils/cache/relcache.c | 2 ++ src/include/catalog/pg_class.h | 3 +++ src/include/utils/lsyscache.h | 1 + src/include/utils/rel.h | 2 ++ 7 files changed, 34 insertions(+) diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 83746d3fd9..ade3586779 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -963,6 +963,7 @@ InsertPgClassTuple(Relation pg_class_desc, values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite); values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid); values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid); + values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm); if (relacl != (Datum) 0) values[Anum_pg_class_relacl - 1] = relacl; else diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 26bfa74ce7..763f442a1d 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -961,6 +961,7 @@ index_create(Relation heapRelation, indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner; indexRelation->rd_rel->relam = accessMethodObjectId; indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid); + indexRelation->rd_rel->relisivm = false; /* * store index's pg_class entry diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c index 6bba5f8ec4..8d5214fb90 100644 --- a/src/backend/utils/cache/lsyscache.c +++ b/src/backend/utils/cache/lsyscache.c @@ -2013,6 +2013,30 @@ get_rel_relispartition(Oid relid) return false; } +/* + * get_rel_relisivm + * + * Returns the relisivm flag associated with a given relation. + */ +bool +get_rel_relisivm(Oid relid) +{ + HeapTuple tp; + + tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid)); + if (HeapTupleIsValid(tp)) + { + Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp); + bool result; + + result = reltup->relisivm; + ReleaseSysCache(tp); + return result; + } + else + return false; +} + /* * get_rel_tablespace * diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 13d9994af3..5b130f0ed2 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -1865,6 +1865,8 @@ formrdesc(const char *relationName, Oid relationReltype, /* ... and they're always populated, too */ relation->rd_rel->relispopulated = true; + /* ... and they're always no ivm, too */ + relation->rd_rel->relisivm = false; relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING; relation->rd_rel->relpages = 0; diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h index fef9945ed8..2058978b89 100644 --- a/src/include/catalog/pg_class.h +++ b/src/include/catalog/pg_class.h @@ -119,6 +119,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat /* is relation a partition? */ bool relispartition BKI_DEFAULT(f); + /* is relation a matview with ivm? */ + bool relisivm BKI_DEFAULT(f); + /* link to original rel during table rewrite; otherwise 0 */ Oid relrewrite BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class); diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h index 77871aaefc..09346aaa17 100644 --- a/src/include/utils/lsyscache.h +++ b/src/include/utils/lsyscache.h @@ -137,6 +137,7 @@ extern Oid get_rel_namespace(Oid relid); extern Oid get_rel_type_id(Oid relid); extern char get_rel_relkind(Oid relid); extern bool get_rel_relispartition(Oid relid); +extern bool get_rel_relisivm(Oid relid); extern Oid get_rel_tablespace(Oid relid); extern char get_rel_persistence(Oid relid); extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index b4faa1c123..b5028f3449 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -649,6 +649,8 @@ RelationGetSmgr(Relation rel) */ #define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated) +#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm) + /* * RelationIsAccessibleInLogicalDecoding * True if we need to log enough information to have access via -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch" Content-Disposition: attachment; filename="v23-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch" Content-Transfer-Encoding: 7bit From 245223e82c4d0d6fa611bfd1a59536f9608585b1 Mon Sep 17 00:00:00 2001 From: Takuma Hoshiai Date: Fri, 17 Jan 2020 16:04:14 +0900 Subject: [PATCH v23 03/15] Add new deptype option 'm' in pg_depend system catalog The deptype option 'm' mean specific database obejects referenced Incrementally Maintainable Materialized View(IMMV). If set NO DATA flag to IMVM, these database objects must be dropped. --- src/backend/catalog/dependency.c | 2 ++ src/include/catalog/dependency.h | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 76b65e39c4..288ad44d4e 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -596,6 +596,7 @@ findDependentObjects(const ObjectAddress *object, case DEPENDENCY_NORMAL: case DEPENDENCY_AUTO: case DEPENDENCY_AUTO_EXTENSION: + case DEPENDENCY_IMMV: /* no problem */ break; @@ -913,6 +914,7 @@ findDependentObjects(const ObjectAddress *object, subflags = DEPFLAG_AUTO; break; case DEPENDENCY_INTERNAL: + case DEPENDENCY_IMMV: subflags = DEPFLAG_INTERNAL; break; case DEPENDENCY_PARTITION_PRI: diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 2885f35ccd..d89b5dd774 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -36,7 +36,8 @@ typedef enum DependencyType DEPENDENCY_PARTITION_PRI = 'P', DEPENDENCY_PARTITION_SEC = 'S', DEPENDENCY_EXTENSION = 'e', - DEPENDENCY_AUTO_EXTENSION = 'x' + DEPENDENCY_AUTO_EXTENSION = 'x', + DEPENDENCY_IMMV = 'm' } DependencyType; /* -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0004-Allow-to-prolong-life-span-of-transition-tables-.patch" Content-Disposition: attachment; filename="v23-0004-Allow-to-prolong-life-span-of-transition-tables-.patch" Content-Transfer-Encoding: 7bit From 3a5edb458d8c79d3462e1c0ca9d438489b7baa00 Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Fri, 20 Dec 2019 10:09:45 +0900 Subject: [PATCH v23 04/15] Allow to prolong life span of transition tables until transaction end Originally, tuplestores of AFTER trigger's transition tables were freed for each query depth. For our IVM implementation, we would like to prolong life of the tuplestores because we have to preserve them for a whole query assuming that some base tables might be changed in some trigger functions. --- src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++- src/include/commands/trigger.h | 2 + 2 files changed, 80 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index fc0a4b2fa7..049007a484 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList * end of the list, so it is relatively easy to discard them. The event * list chunks themselves are stored in event_cxt. * + * prolonged_tuplestored is a list of transition table tuplestores whose + * life are prolonged to the end of the outmost query instead of each nested + * query. + * * query_depth is the current depth of nested AfterTriggerBeginQuery calls * (-1 when the stack is empty). * @@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData SetConstraintState state; /* the active S C state */ AfterTriggerEventList events; /* deferred-event list */ MemoryContext event_cxt; /* memory context for events, if any */ + List *prolonged_tuplestores; /* list of prolonged tuplestores */ /* per-query-level data: */ AfterTriggersQueryData *query_stack; /* array of structs shown below */ @@ -3670,6 +3675,7 @@ struct AfterTriggersTableData bool closed; /* true when no longer OK to add tuples */ bool before_trig_done; /* did we already queue BS triggers? */ bool after_trig_done; /* did we already queue AS triggers? */ + bool prolonged; /* are transition tables prolonged? */ AfterTriggerEventList after_trig_events; /* if so, saved list pointer */ Tuplestorestate *old_tuplestore; /* "old" transition table, if any */ Tuplestorestate *new_tuplestore; /* "new" transition table, if any */ @@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events, } +/* + * SetTransitionTablePreserved + * + * Prolong lifespan of transition tables corresponding specified relid and + * command type to the end of the outmost query instead of each nested query. + * This enables to use nested AFTER trigger's transition tables from outer + * query's triggers. Currently, only immediate incremental view maintenance + * uses this. + */ +void +SetTransitionTablePreserved(Oid relid, CmdType cmdType) +{ + AfterTriggersTableData *table; + AfterTriggersQueryData *qs; + bool found = false; + ListCell *lc; + + /* Check state, like AfterTriggerSaveEvent. */ + if (afterTriggers.query_depth < 0) + elog(ERROR, "SetTransitionTablePreserved() called outside of query"); + + qs = &afterTriggers.query_stack[afterTriggers.query_depth]; + + foreach(lc, qs->tables) + { + table = (AfterTriggersTableData *) lfirst(lc); + if (table->relid == relid && table->cmdType == cmdType && + table->closed) + { + table->prolonged = true; + found = true; + } + } + + if (!found) + elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType); +} + + /* * GetAfterTriggersTableData * @@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void) */ afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */ afterTriggers.query_depth = -1; + afterTriggers.prolonged_tuplestores = NIL; /* * Verify that there is no leftover state remaining. If these assertions @@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs) ts = table->old_tuplestore; table->old_tuplestore = NULL; if (ts) - tuplestore_end(ts); + { + if (table->prolonged && afterTriggers.query_depth > 0) + { + MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext); + afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts); + MemoryContextSwitchTo(oldcxt); + } + else + tuplestore_end(ts); + } ts = table->new_tuplestore; table->new_tuplestore = NULL; if (ts) - tuplestore_end(ts); + { + if (table->prolonged && afterTriggers.query_depth > 0) + { + MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext); + afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts); + MemoryContextSwitchTo(oldcxt); + } + else + tuplestore_end(ts); + } if (table->storeslot) ExecDropSingleTupleTableSlot(table->storeslot); } @@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs) */ qs->tables = NIL; list_free_deep(tables); + + /* Release prolonged tuplestores at the end of the outmost query */ + if (afterTriggers.query_depth == 0) + { + foreach(lc, afterTriggers.prolonged_tuplestores) + { + ts = (Tuplestorestate *) lfirst(lc); + if (ts) + tuplestore_end(ts); + } + afterTriggers.prolonged_tuplestores = NIL; + } } diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h index 9ef7f6d768..0cfb0b3c5c 100644 --- a/src/include/commands/trigger.h +++ b/src/include/commands/trigger.h @@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit); extern void AfterTriggerSetState(ConstraintsSetStmt *stmt); extern bool AfterTriggerPendingOnRel(Oid relid); +extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType); + /* * in utils/adt/ri_triggers.c -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch" Content-Disposition: attachment; filename="v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch" Content-Transfer-Encoding: 7bit From ba59f8e8c1c1e19876eeb885bc2febe59356e602 Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Wed, 11 Nov 2020 17:01:25 +0900 Subject: [PATCH v23 05/15] Add Incremental View Maintenance support to pg_dump Support CREATE INCREMENTAL MATERIALIZED VIEW syntax. --- src/bin/pg_dump/pg_dump.c | 42 ++++++++++++++++++++++++-------- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 15 ++++++++++++ 3 files changed, 48 insertions(+), 10 deletions(-) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 90ac445bcd..17e0a2193d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -6268,6 +6268,7 @@ getTables(Archive *fout, int *numTables) int i_ispartition; int i_partbound; int i_amname; + int i_isivm; /* * Find all the tables and table-like objects. @@ -6299,6 +6300,7 @@ getTables(Archive *fout, int *numTables) char *ispartition = "false"; char *partbound = "NULL"; char *relhasoids = "c.relhasoids"; + char *isivm = "false"; PQExpBuffer acl_subquery = createPQExpBuffer(); PQExpBuffer racl_subquery = createPQExpBuffer(); @@ -6326,6 +6328,10 @@ getTables(Archive *fout, int *numTables) if (fout->remoteVersion >= 120000) relhasoids = "'f'::bool"; + /* The information about incremental view maintenance */ + if (fout->remoteVersion >= 150000) + isivm = "c.relisivm"; + /* * Left join to pick up dependency info linking sequences to their * owning column, if any (note this dependency is AUTO as of 8.2) @@ -6384,7 +6390,8 @@ getTables(Archive *fout, int *numTables) "AS changed_acl, " "%s AS partkeydef, " "%s AS ispartition, " - "%s AS partbound " + "%s AS partbound, " + "%s AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6413,6 +6420,7 @@ getTables(Archive *fout, int *numTables) partkeydef, ispartition, partbound, + isivm, RELKIND_SEQUENCE, RELKIND_PARTITIONED_TABLE, RELKIND_RELATION, RELKIND_SEQUENCE, @@ -6466,7 +6474,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6519,7 +6528,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6572,7 +6582,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6623,7 +6634,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6672,7 +6684,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6720,7 +6733,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6768,7 +6782,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6815,7 +6830,8 @@ getTables(Archive *fout, int *numTables) "NULL AS changed_acl, " "NULL AS partkeydef, " "false AS ispartition, " - "NULL AS partbound " + "NULL AS partbound, " + "false AS isivm " "FROM pg_class c " "LEFT JOIN pg_depend d ON " "(c.relkind = '%c' AND " @@ -6887,6 +6903,7 @@ getTables(Archive *fout, int *numTables) i_ispartition = PQfnumber(res, "ispartition"); i_partbound = PQfnumber(res, "partbound"); i_amname = PQfnumber(res, "amname"); + i_isivm = PQfnumber(res, "isivm"); if (dopt->lockWaitTimeout) { @@ -7000,6 +7017,9 @@ getTables(Archive *fout, int *numTables) /* foreign server */ tblinfo[i].foreign_server = atooid(PQgetvalue(res, i, i_foreignserver)); + /* Incremental view maintenance information */ + tblinfo[i].isivm = (strcmp(PQgetvalue(res, i, i_isivm), "t") == 0); + /* * Read-lock target tables to make sure they aren't DROPPED or altered * in schema before we get around to dumping them. @@ -15990,9 +16010,11 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) binary_upgrade_set_pg_class_oids(fout, q, tbinfo->dobj.catId.oid, false); - appendPQExpBuffer(q, "CREATE %s%s %s", + appendPQExpBuffer(q, "CREATE %s%s%s %s", tbinfo->relpersistence == RELPERSISTENCE_UNLOGGED ? "UNLOGGED " : "", + tbinfo->relkind == RELKIND_MATVIEW && tbinfo->isivm ? + "INCREMENTAL " : "", reltypename, qualrelname); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index f5e170e0db..ae8f24bb78 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -298,6 +298,7 @@ typedef struct _tableInfo bool dummy_view; /* view's real definition must be postponed */ bool postponed_def; /* matview must be postponed into post-data */ bool ispartition; /* is table a partition? */ + bool isivm; /* is incrementally maintainable materialized view? */ /* * These fields are computed only if we decide the table is interesting diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index c5d8915be8..9d1776a506 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2160,6 +2160,21 @@ my %tests = ( { exclude_dump_test_schema => 1, no_toast_compression => 1, }, }, + 'CREATE MATERIALIZED VIEW matview_ivm' => { + create_order => 21, + create_sql => 'CREATE INCREMENTAL MATERIALIZED VIEW dump_test.matview_ivm (col1) AS + SELECT col1 FROM dump_test.test_table;', + regexp => qr/^ + \QCREATE INCREMENTAL MATERIALIZED VIEW dump_test.matview_ivm AS\E + \n\s+\QSELECT test_table.col1\E + \n\s+\QFROM dump_test.test_table\E + \n\s+\QWITH NO DATA;\E + /xm, + like => + { %full_runs, %dump_test_schema_runs, section_pre_data => 1, }, + unlike => { exclude_dump_test_schema => 1, }, + }, + 'CREATE POLICY p1 ON test_table' => { create_order => 22, create_sql => 'CREATE POLICY p1 ON dump_test.test_table -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0006-Add-Incremental-View-Maintenance-support-to-psql.patch" Content-Disposition: attachment; filename="v23-0006-Add-Incremental-View-Maintenance-support-to-psql.patch" Content-Transfer-Encoding: 7bit From b100526d02a873fe3d4ee25ddd4d6e0b7346a99a Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Fri, 20 Dec 2019 10:21:54 +0900 Subject: [PATCH v23 06/15] Add Incremental View Maintenance support to psql Add tab completion and meta-command output for IVM. --- src/bin/psql/describe.c | 32 +++++++++++++++++++++++++++++++- src/bin/psql/tab-complete.c | 14 +++++++++----- 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index ba658f731b..0518c88b5f 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1657,6 +1657,7 @@ describeOneTableDetails(const char *schemaname, char relpersistence; char relreplident; char *relam; + bool isivm; } tableinfo; bool show_column_details = false; @@ -1669,7 +1670,26 @@ describeOneTableDetails(const char *schemaname, initPQExpBuffer(&tmpbuf); /* Get general table info */ - if (pset.sversion >= 120000) + if (pset.sversion >= 150000) + { + printfPQExpBuffer(&buf, + "SELECT c.relchecks, c.relkind, c.relhasindex, c.relhasrules, " + "c.relhastriggers, c.relrowsecurity, c.relforcerowsecurity, " + "false AS relhasoids, c.relispartition, %s, c.reltablespace, " + "CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, " + "c.relpersistence, c.relreplident, am.amname, " + "c.relisivm\n" + "FROM pg_catalog.pg_class c\n " + "LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)\n" + "LEFT JOIN pg_catalog.pg_am am ON (c.relam = am.oid)\n" + "WHERE c.oid = '%s';", + (verbose ? + "pg_catalog.array_to_string(c.reloptions || " + "array(select 'toast.' || x from pg_catalog.unnest(tc.reloptions) x), ', ')\n" + : "''"), + oid); + } + else if (pset.sversion >= 120000) { printfPQExpBuffer(&buf, "SELECT c.relchecks, c.relkind, c.relhasindex, c.relhasrules, " @@ -1853,6 +1873,10 @@ describeOneTableDetails(const char *schemaname, (char *) NULL : pg_strdup(PQgetvalue(res, 0, 14)); else tableinfo.relam = NULL; + if (pset.sversion >= 150000) + tableinfo.isivm = strcmp(PQgetvalue(res, 0, 15), "t") == 0; + else + tableinfo.isivm = false; PQclear(res); res = NULL; @@ -3630,6 +3654,12 @@ describeOneTableDetails(const char *schemaname, printfPQExpBuffer(&buf, _("Access method: %s"), tableinfo.relam); printTableAddFooter(&cont, buf.data); } + + /* Incremental view maintance info */ + if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm) + { + printTableAddFooter(&cont, _("Incremental view maintenance: yes")); + } } /* reloptions, if verbose */ diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index d6bf725971..9fb97596dc 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -1055,6 +1055,7 @@ static const pgsql_thing_t words_after_create[] = { {"FOREIGN TABLE", NULL, NULL, NULL}, {"FUNCTION", NULL, NULL, Query_for_list_of_functions}, {"GROUP", Query_for_list_of_roles}, + {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews, THING_NO_DROP | THING_NO_ALTER}, {"INDEX", NULL, NULL, &Query_for_list_of_indexes}, {"LANGUAGE", Query_for_list_of_languages}, {"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP}, @@ -2698,7 +2699,7 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW"); /* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */ else if (TailMatches("CREATE", "UNLOGGED")) - COMPLETE_WITH("TABLE", "MATERIALIZED VIEW"); + COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW"); /* Complete PARTITION BY with RANGE ( or LIST ( or ... */ else if (TailMatches("PARTITION", "BY")) COMPLETE_WITH("RANGE (", "LIST (", "HASH ("); @@ -2997,13 +2998,16 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("SELECT"); /* CREATE MATERIALIZED VIEW */ - else if (Matches("CREATE", "MATERIALIZED")) + else if (Matches("CREATE", "MATERIALIZED") || + Matches("CREATE", "INCREMENTAL", "MATERIALIZED")) COMPLETE_WITH("VIEW"); - /* Complete CREATE MATERIALIZED VIEW with AS */ - else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny)) + /* Complete CREATE MATERIALIZED VIEW with AS */ + else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) || + Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny)) COMPLETE_WITH("AS"); /* Complete "CREATE MATERIALIZED VIEW AS with "SELECT" */ - else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS")) + else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") || + Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS")) COMPLETE_WITH("SELECT"); /* CREATE EVENT TRIGGER */ -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0007-Add-Incremental-View-Maintenance-support.patch" Content-Disposition: attachment; filename="v23-0007-Add-Incremental-View-Maintenance-support.patch" Content-Transfer-Encoding: 7bit From 7117de103841d4a3f1065b2d1a8b3b86c3ca56eb Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Tue, 22 Dec 2020 18:40:18 +0900 Subject: [PATCH v23 07/15] Add Incremental View Maintenance support In this implementation, AFTER triggers are used to collecting tuplestores containing transition table contents. When multiple tables are changed, multiple AFTER triggers are invoked, then the final AFTER trigger performs actual update of the matview. In addition, BEFORE triggers are also used to handle global information for view maintenance. To calculate view deltas, we need both pre-state and post-state of base tables. Post-update states are available in AFTER trigger, and pre-update states can be calculated by filtering inserted tuples using cmin/xmin system columns, and append deleted tuples which are contained in a old transition table. This patch also allows self-join, simultaneous updates of more than one base tables, and multiple updates of the same base table. DISTINCT and tuple duplicates in views are supported. When IMMV is created with DISTINCT, multiplicity of tuples is counted and stored in "__ivm_count__" column, which is a hidden column of IMMV. The value in__ivm_count__ is updated when IMMV is maintained incrementally. A tuple in IMMV can be removed if and only if the count becomes zero. --- src/backend/access/transam/xact.c | 5 + src/backend/commands/createas.c | 697 ++++++++++++ src/backend/commands/indexcmds.c | 40 + src/backend/commands/matview.c | 1549 ++++++++++++++++++++++++++- src/backend/commands/tablecmds.c | 9 + src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/parser/parse_relation.c | 18 +- src/backend/rewrite/rewriteDefine.c | 3 +- src/include/catalog/pg_proc.dat | 8 + src/include/commands/createas.h | 5 + src/include/commands/matview.h | 8 + src/include/nodes/parsenodes.h | 2 + 15 files changed, 2308 insertions(+), 40 deletions(-) diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 441445927e..d67a8bc906 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -35,6 +35,7 @@ #include "catalog/pg_enum.h" #include "catalog/storage.h" #include "commands/async.h" +#include "commands/matview.h" #include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/spi.h" @@ -2705,6 +2706,7 @@ AbortTransaction(void) AtAbort_Notify(); AtEOXact_RelationMap(false, is_parallel_worker); AtAbort_Twophase(); + AtAbort_IVM(); /* * Advertise the fact that we aborted in pg_xact (assuming that we got as @@ -4948,6 +4950,9 @@ AbortSubTransaction(void) AbortBufferIO(); UnlockBuffers(); + /* Clean up hash entries for incremental view maintenance */ + AtAbort_IVM(); + /* Reset WAL record construction state */ XLogResetInsertion(); diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index 0982851715..3fc2af1c4a 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -32,24 +32,41 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/namespace.h" +#include "catalog/index.h" +#include "catalog/pg_constraint.h" +#include "catalog/pg_inherits.h" +#include "catalog/pg_trigger.h" +#include "catalog/pg_type.h" #include "catalog/toasting.h" #include "commands/createas.h" +#include "commands/defrem.h" #include "commands/matview.h" #include "commands/prepare.h" #include "commands/tablecmds.h" +#include "commands/tablespace.h" +#include "commands/trigger.h" #include "commands/view.h" #include "miscadmin.h" +#include "optimizer/clauses.h" +#include "optimizer/optimizer.h" #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" +#include "parser/parser.h" +#include "parser/parsetree.h" #include "parser/parse_clause.h" +#include "parser/parse_func.h" +#include "parser/parse_type.h" #include "rewrite/rewriteHandler.h" +#include "rewrite/rewriteManip.h" #include "storage/smgr.h" #include "tcop/tcopprot.h" #include "utils/builtins.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/rel.h" #include "utils/rls.h" #include "utils/snapmgr.h" +#include "utils/syscache.h" typedef struct { @@ -73,6 +90,12 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self); static void intorel_shutdown(DestReceiver *self); static void intorel_destroy(DestReceiver *self); +static void CreateIvmTriggersOnBaseTables_recurse(Query *qry, Node *node, Oid matviewOid, Relids *relids, bool ex_lock); +static void CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 timing, bool ex_lock); +static void check_ivm_restriction(Node *node); +static bool check_ivm_restriction_walker(Node *node, void *context); +static void CreateIndexOnIMMV(Query *query, Relation matviewRel); +static Bitmapset *get_primary_key_attnos_from_query(Query *qry, List **constraintList); /* * create_ctas_internal @@ -108,6 +131,8 @@ create_ctas_internal(List *attrList, IntoClause *into) create->oncommit = into->onCommit; create->tablespacename = into->tableSpaceName; create->if_not_exists = false; + /* Using Materialized view only */ + create->ivm = into->ivm; create->accessMethod = into->accessMethod; /* @@ -238,6 +263,7 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt, List *rewritten; PlannedStmt *plan; QueryDesc *queryDesc; + Query *query_immv = NULL; /* Check if the relation exists or not */ if (CreateTableAsRelExists(stmt)) @@ -282,6 +308,22 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt, save_nestlevel = NewGUCNestLevel(); } + if (is_matview && into->ivm) + { + /* check if the query is supported in IMMV definition */ + if (contain_mutable_functions((Node *) query)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("mutable function is not supported on incrementally maintainable materialized view"), + errhint("functions must be marked IMMUTABLE"))); + + check_ivm_restriction((Node *) query); + + /* For IMMV, we need to rewrite matview query */ + query = rewriteQueryForIMMV(query, into->colNames); + query_immv = copyObject(query); + } + if (into->skipData) { /* @@ -358,11 +400,74 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt, /* Restore userid and security context */ SetUserIdAndSecContext(save_userid, save_sec_context); + + if (into->ivm) + { + Oid matviewOid = address.objectId; + Relation matviewRel = table_open(matviewOid, NoLock); + + /* + * Mark relisivm field, if it's a matview and into->ivm is true. + */ + SetMatViewIVMState(matviewRel, true); + + /* Create an index on incremental maintainable materialized view, if possible */ + CreateIndexOnIMMV((Query *) into->viewQuery, matviewRel); + + /* Create triggers on incremental maintainable materialized view */ + if (!into->skipData) + { + Assert(query_immv != NULL); + CreateIvmTriggersOnBaseTables(query_immv, matviewOid, true); + } + table_close(matviewRel, NoLock); + } } return address; } +/* + * rewriteQueryForIMMV -- rewrite view definition query for IMMV + * + * count(*) is added for counting distinct tuples in views. + */ +Query * +rewriteQueryForIMMV(Query *query, List *colNames) +{ + Query *rewritten; + + TargetEntry *tle; + Node *node; + ParseState *pstate = make_parsestate(NULL); + FuncCall *fn; + + rewritten = copyObject(query); + pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET; + + /* Convert DISTINCT to GROUP BY. count(*) will be added afterward. */ + if (rewritten->distinctClause) + rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false); + + /* Add count(*) for counting distinct tuples in views */ + if (rewritten->distinctClause) + { + fn = makeFuncCall(list_make1(makeString("count")), NIL, COERCE_EXPLICIT_CALL, -1); + fn->agg_star = true; + + node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1); + + tle = makeTargetEntry((Expr *) node, + list_length(rewritten->targetList) + 1, + pstrdup("__ivm_count__"), + false); + rewritten->targetList = lappend(rewritten->targetList, tle); + rewritten->hasAggs = true; + } + + return rewritten; +} + /* * GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS * @@ -623,3 +728,595 @@ intorel_destroy(DestReceiver *self) { pfree(self); } + +/* + * CreateIvmTriggersOnBaseTables -- create IVM triggers on all base tables + */ +void +CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid, bool is_create) +{ + Relids relids = NULL; + bool ex_lock = false; + Index first_rtindex = is_create ? 1 : PRS2_NEW_VARNO + 1; + RangeTblEntry *rte; + + /* Immediately return if we don't have any base tables. */ + if (list_length(qry->rtable) < first_rtindex) + return; + + /* + * If the view has more than one base tables, we need an exclusive lock + * on the view so that the view would be maintained serially to avoid + * the inconsistency that occurs when two base tables are modified in + * concurrent transactions. However, if the view has only one table, + * we can use a weaker lock. + * + * The type of lock should be determined here, because if we check the + * view definition at maintenance time, we need to acquire a weaker lock, + * and upgrading the lock level after this increases probability of + * deadlock. + */ + + rte = list_nth(qry->rtable, first_rtindex - 1); + if (list_length(qry->rtable) > first_rtindex || + rte->rtekind != RTE_RELATION) + ex_lock = true; + + CreateIvmTriggersOnBaseTables_recurse(qry, (Node *)qry, matviewOid, &relids, ex_lock); + + bms_free(relids); +} + +static void +CreateIvmTriggersOnBaseTables_recurse(Query *qry, Node *node, Oid matviewOid, Relids *relids, bool ex_lock) +{ + /* This can recurse, so check for excessive recursion */ + check_stack_depth(); + + if (node == NULL) + return; + + switch (nodeTag(node)) + { + case T_Query: + { + Query *query = (Query *) node; + + CreateIvmTriggersOnBaseTables_recurse(qry, (Node *)query->jointree, matviewOid, relids, ex_lock); + } + break; + + case T_RangeTblRef: + { + int rti = ((RangeTblRef *) node)->rtindex; + RangeTblEntry *rte = rt_fetch(rti, qry->rtable); + + if (rte->rtekind == RTE_RELATION) + { + if (!bms_is_member(rte->relid, *relids)) + { + CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_BEFORE, ex_lock); + CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_BEFORE, ex_lock); + CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_BEFORE, ex_lock); + CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_AFTER, ex_lock); + CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_AFTER, ex_lock); + CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_AFTER, ex_lock); + + *relids = bms_add_member(*relids, rte->relid); + } + } + } + break; + + case T_FromExpr: + { + FromExpr *f = (FromExpr *) node; + ListCell *l; + + foreach(l, f->fromlist) + CreateIvmTriggersOnBaseTables_recurse(qry, lfirst(l), matviewOid, relids, ex_lock); + } + break; + + case T_JoinExpr: + { + JoinExpr *j = (JoinExpr *) node; + + CreateIvmTriggersOnBaseTables_recurse(qry, j->larg, matviewOid, relids, ex_lock); + CreateIvmTriggersOnBaseTables_recurse(qry, j->rarg, matviewOid, relids, ex_lock); + } + break; + + default: + elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node)); + } +} + +/* + * CreateIvmTrigger -- create IVM trigger on a base table + */ +static void +CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 timing, bool ex_lock) +{ + ObjectAddress refaddr; + ObjectAddress address; + CreateTrigStmt *ivm_trigger; + List *transitionRels = NIL; + + Assert(timing == TRIGGER_TYPE_BEFORE || timing == TRIGGER_TYPE_AFTER); + + refaddr.classId = RelationRelationId; + refaddr.objectId = viewOid; + refaddr.objectSubId = 0; + + ivm_trigger = makeNode(CreateTrigStmt); + ivm_trigger->relation = NULL; + ivm_trigger->row = false; + + ivm_trigger->timing = timing; + ivm_trigger->events = type; + + switch (type) + { + case TRIGGER_TYPE_INSERT: + ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_ins_before" : "IVM_trigger_ins_after"); + break; + case TRIGGER_TYPE_DELETE: + ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_del_before" : "IVM_trigger_del_after"); + break; + case TRIGGER_TYPE_UPDATE: + ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_upd_before" : "IVM_trigger_upd_after"); + break; + default: + elog(ERROR, "unsupported trigger type"); + } + + if (timing == TRIGGER_TYPE_AFTER) + { + if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE) + { + TriggerTransition *n = makeNode(TriggerTransition); + n->name = "__ivm_newtable"; + n->isNew = true; + n->isTable = true; + + transitionRels = lappend(transitionRels, n); + } + if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE) + { + TriggerTransition *n = makeNode(TriggerTransition); + n->name = "__ivm_oldtable"; + n->isNew = false; + n->isTable = true; + + transitionRels = lappend(transitionRels, n); + } + } + + ivm_trigger->funcname = + (timing == TRIGGER_TYPE_BEFORE ? SystemFuncName("IVM_immediate_before") : SystemFuncName("IVM_immediate_maintenance")); + + ivm_trigger->columns = NIL; + ivm_trigger->transitionRels = transitionRels; + ivm_trigger->whenClause = NULL; + ivm_trigger->isconstraint = false; + ivm_trigger->deferrable = false; + ivm_trigger->initdeferred = false; + ivm_trigger->constrrel = NULL; + ivm_trigger->args = list_make2( + makeString(DatumGetPointer(DirectFunctionCall1(oidout, ObjectIdGetDatum(viewOid)))), + makeString(DatumGetPointer(DirectFunctionCall1(boolout, BoolGetDatum(ex_lock)))) + ); + + address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid, + InvalidOid, InvalidOid, InvalidOid, NULL, true, false); + + recordDependencyOn(&address, &refaddr, DEPENDENCY_IMMV); + + /* Make changes-so-far visible */ + CommandCounterIncrement(); +} + +/* + * check_ivm_restriction --- look for specify nodes in the query tree + */ +static void +check_ivm_restriction(Node *node) +{ + check_ivm_restriction_walker(node, NULL); +} + +static bool +check_ivm_restriction_walker(Node *node, void *context) +{ + /* This can recurse, so check for excessive recursion */ + check_stack_depth(); + + if (node == NULL) + return false; + + /* + * We currently don't support Sub-Query. + */ + if (IsA(node, SubPlan) || IsA(node, SubLink)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("subquery is not supported on incrementally maintainable materialized view"))); + + switch (nodeTag(node)) + { + case T_Query: + { + Query *qry = (Query *)node; + ListCell *lc; + List *vars; + + /* if contained CTE, return error */ + if (qry->cteList != NIL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("CTE is not supported on incrementally maintainable materialized view"))); + if (qry->havingQual != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg(" HAVING clause is not supported on incrementally maintainable materialized view"))); + if (qry->sortClause != NIL) /* There is a possibility that we don't need to return an error */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ORDER BY clause is not supported on incrementally maintainable materialized view"))); + if (qry->limitOffset != NULL || qry->limitCount != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view"))); + if (qry->hasDistinctOn) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("DISTINCT ON is not supported on incrementally maintainable materialized view"))); + if (qry->hasWindowFuncs) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("window functions are not supported on incrementally maintainable materialized view"))); + if (qry->groupingSets != NIL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("GROUPING SETS, ROLLUP, or CUBE clauses is not supported on incrementally maintainable materialized view"))); + if (qry->setOperations != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("UNION/INTERSECT/EXCEPT statements are not supported on incrementally maintainable materialized view"))); + if (list_length(qry->targetList) == 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("empty target list is not supported on incrementally maintainable materialized view"))); + if (qry->rowMarks != NIL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("FOR UPDATE/SHARE clause is not supported on incrementally maintainable materialized view"))); + + /* system column restrictions */ + vars = pull_vars_of_level((Node *) qry, 0); + foreach(lc, vars) + { + if (IsA(lfirst(lc), Var)) + { + Var *var = (Var *) lfirst(lc); + /* if system column, return error */ + if (var->varattno < 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("system column is not supported on incrementally maintainable materialized view"))); + } + } + + /* restrictions for rtable */ + foreach(lc, qry->rtable) + { + RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc); + + if (rte->subquery) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("subquery is not supported on incrementally maintainable materialized view"))); + + if (rte->tablesample != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("TABLESAMPLE clause is not supported on incrementally maintainable materialized view"))); + + if (rte->relkind == RELKIND_RELATION && find_inheritance_children(rte->relid, NoLock) != NIL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("inheritance parent is not supported on incrementally maintainable materialized view"))); + + if (rte->relkind == RELKIND_VIEW || + rte->relkind == RELKIND_MATVIEW) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("VIEW or MATERIALIZED VIEW is not supported on incrementally maintainable materialized view"))); + + if (rte->rtekind == RTE_VALUES) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("VALUES is not supported on incrementally maintainable materialized view"))); + + } + + query_tree_walker(qry, check_ivm_restriction_walker, NULL, QTW_IGNORE_RANGE_TABLE); + + break; + } + case T_TargetEntry: + { + TargetEntry *tle = (TargetEntry *)node; + if (isIvmName(tle->resname)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column name %s is not supported on incrementally maintainable materialized view", tle->resname))); + + expression_tree_walker(node, check_ivm_restriction_walker, NULL); + break; + } + case T_JoinExpr: + { + JoinExpr *joinexpr = (JoinExpr *)node; + + if (joinexpr->jointype > JOIN_INNER) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("OUTER JOIN is not supported on incrementally maintainable materialized view"))); + + expression_tree_walker(node, check_ivm_restriction_walker, NULL); + } + break; + expression_tree_walker(node, check_ivm_restriction_walker, NULL); + break; + } + return false; +} + +/* + * CreateindexOnIMMV + * + * Create a unique index on incremental maintainable materialized view. + * If the view definition query has a GROUP BY clause, the index is created + * on the columns of GROUP BY expressions. Otherwise, if the view contains + * all primary key attritubes of its base tables in the target list, the index + * is created on these attritubes. In other cases, no index is created. + */ +static void +CreateIndexOnIMMV(Query *query, Relation matviewRel) +{ + Query *qry = (Query *) copyObject(query); + ListCell *lc; + IndexStmt *index; + ObjectAddress address; + List *constraintList = NIL; + char idxname[NAMEDATALEN]; + + snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel)); + + index = makeNode(IndexStmt); + + index->unique = true; + index->primary = false; + index->isconstraint = false; + index->deferrable = false; + index->initdeferred = false; + index->idxname = idxname; + index->relation = + makeRangeVar(get_namespace_name(RelationGetNamespace(matviewRel)), + pstrdup(RelationGetRelationName(matviewRel)), + -1); + index->accessMethod = DEFAULT_INDEX_TYPE; + index->options = NIL; + index->tableSpace = get_tablespace_name(matviewRel->rd_rel->reltablespace); + index->whereClause = NULL; + index->indexParams = NIL; + index->indexIncludingParams = NIL; + index->excludeOpNames = NIL; + index->idxcomment = NULL; + index->indexOid = InvalidOid; + index->oldNode = InvalidOid; + index->oldCreateSubid = InvalidSubTransactionId; + index->oldFirstRelfilenodeSubid = InvalidSubTransactionId; + index->transformed = true; + index->concurrent = false; + index->if_not_exists = false; + + if (qry->distinctClause) + { + /* create unique constraint on all columns */ + index->isconstraint = true; + foreach(lc, qry->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1); + IndexElem *iparam; + + iparam = makeNode(IndexElem); + iparam->name = pstrdup(NameStr(attr->attname)); + iparam->expr = NULL; + iparam->indexcolname = NULL; + iparam->collation = NIL; + iparam->opclass = NIL; + iparam->opclassopts = NIL; + iparam->ordering = SORTBY_DEFAULT; + iparam->nulls_ordering = SORTBY_NULLS_DEFAULT; + index->indexParams = lappend(index->indexParams, iparam); + } + } + else + { + Bitmapset *key_attnos; + + /* create index on the base tables' primary key columns */ + key_attnos = get_primary_key_attnos_from_query(qry, &constraintList); + if (key_attnos) + { + foreach(lc, qry->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1); + + if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos)) + { + IndexElem *iparam; + + iparam = makeNode(IndexElem); + iparam->name = pstrdup(NameStr(attr->attname)); + iparam->expr = NULL; + iparam->indexcolname = NULL; + iparam->collation = NIL; + iparam->opclass = NIL; + iparam->opclassopts = NIL; + iparam->ordering = SORTBY_DEFAULT; + iparam->nulls_ordering = SORTBY_NULLS_DEFAULT; + index->indexParams = lappend(index->indexParams, iparam); + } + } + + } + else + { + /* create no index, just notice that an appropriate index is necessary for efficient, IVM */ + ereport(NOTICE, + (errmsg("could not create an index on materialized view \"%s\" automatically", + RelationGetRelationName(matviewRel)), + errhint("Create an index on the materialized view for efficient incremental maintenance."))); + return; + } + } + + address = DefineIndex(RelationGetRelid(matviewRel), + index, + InvalidOid, + InvalidOid, + InvalidOid, + false, true, false, false, true); + + ereport(NOTICE, + (errmsg("created index \"%s\" on materialized view \"%s\"", + idxname, RelationGetRelationName(matviewRel)))); + + + /* + * Make dependencies so that the index is dropped if any base tables's + * primary key is dropped. + */ + foreach(lc, constraintList) + { + Oid constraintOid = lfirst_oid(lc); + ObjectAddress refaddr; + + refaddr.classId = ConstraintRelationId; + refaddr.objectId = constraintOid; + refaddr.objectSubId = 0; + + recordDependencyOn(&address, &refaddr, DEPENDENCY_NORMAL); + } +} + + +/* + * get_primary_key_attnos_from_query + * + * Identify the columns in base tables' primary keys in the target list. + * + * Returns a Bitmapset of the column attnos of the primary key's columns of + * tables that used in the query. The attnos are offset by + * FirstLowInvalidHeapAttributeNumber as same as get_primary_key_attnos. + * + * If any table has no primary key or any primary key's columns is not in + * the target list, return NULL. We also return NULL if any pkey constraint + * is deferrable. + * + * constraintList is set to a list of the OIDs of the pkey constraints. + */ +static Bitmapset * +get_primary_key_attnos_from_query(Query *query, List **constraintList) +{ + List *key_attnos_list = NIL; + ListCell *lc; + int i; + Bitmapset *keys = NULL; + Relids rels_in_from; + PlannerInfo root; + + + /* + * Collect primary key attributes from all tables used in query. The key attributes + * sets for each table are stored in key_attnos_list in order by RTE index. + */ + i = 1; + foreach(lc, query->rtable) + { + RangeTblEntry *r = (RangeTblEntry*) lfirst(lc); + Bitmapset *key_attnos; + bool has_pkey = true; + + /* for tables, call get_primary_key_attnos */ + if (r->rtekind == RTE_RELATION) + { + Oid constraintOid; + key_attnos = get_primary_key_attnos(r->relid, false, &constraintOid); + *constraintList = lappend_oid(*constraintList, constraintOid); + has_pkey = (key_attnos != NULL); + } + /* for other RTEs, store NULL into key_attnos_list */ + else + key_attnos = NULL; + + /* + * If any table or subquery has no primary key or its pkey constraint is deferrable, + * we cannot get key attributes for this query, so return NULL. + */ + if (!has_pkey) + return NULL; + + key_attnos_list = lappend(key_attnos_list, key_attnos); + } + + /* Collect key attributes appearing in the target list */ + i = 1; + foreach(lc, query->targetList) + { + TargetEntry *tle = (TargetEntry *) flatten_join_alias_vars(query, lfirst(lc)); + + if (IsA(tle->expr, Var)) + { + Var *var = (Var*) tle->expr; + Bitmapset *attnos = list_nth(key_attnos_list, var->varno - 1); + + /* check if this attribute is from a base table's primary key */ + if (bms_is_member(var->varattno - FirstLowInvalidHeapAttributeNumber, attnos)) + { + /* + * Remove found key attributes from key_attnos_list, and add this + * to the result list. + */ + bms_del_member(attnos, var->varattno - FirstLowInvalidHeapAttributeNumber); + keys = bms_add_member(keys, i - FirstLowInvalidHeapAttributeNumber); + } + } + i++; + } + + /* Collect relations appearing in the FROM clause */ + rels_in_from = pull_varnos_of_level(&root, (Node *)query->jointree, 0); + + /* + * Check if all key attributes of relations in FROM are appearing in the target + * list. If an attribute remains in key_attnos_list in spite of the table is used + * in FROM clause, the target is missing this key attribute, so we return NULL. + */ + i = 1; + foreach(lc, key_attnos_list) + { + Bitmapset *bms = (Bitmapset *)lfirst(lc); + if (!bms_is_empty(bms) && bms_is_member(i, rels_in_from)) + return NULL; + i++; + } + + return keys; +} diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index c14ca27c5e..5581c21298 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -36,6 +36,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/matview.h" #include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" @@ -1054,6 +1055,45 @@ DefineIndex(Oid relationId, safe_index = indexInfo->ii_Expressions == NIL && indexInfo->ii_Predicate == NIL; + /* + * We disallow unique indexes on IVM columns of IMMVs. + */ + if (RelationIsIVM(rel) && stmt->unique) + { + for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) + { + AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i]; + if (attno > 0) + { + char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname); + if (name && isIvmName(name)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("unique index creation on IVM columns is not supported"))); + } + } + + if (indexInfo->ii_Expressions) + { + Bitmapset *indexattrs = NULL; + int varno = -1; + + pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs); + + while ((varno = bms_next_member(indexattrs, varno)) >= 0) + { + int attno = varno + FirstLowInvalidHeapAttributeNumber; + char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname); + if (name && isIvmName(name)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("unique index creation on IVM columns is not supported"))); + } + + } + } + + /* * Report index creation if appropriate (delay this till after most of the * error checks) diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index 25bbd8a5c1..f1f46e2e14 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -25,26 +25,47 @@ #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" +#include "catalog/pg_collation.h" +#include "catalog/pg_depend.h" +#include "catalog/pg_type.h" #include "catalog/pg_opclass.h" #include "catalog/pg_operator.h" #include "commands/cluster.h" +#include "commands/defrem.h" #include "commands/matview.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" +#include "commands/createas.h" #include "executor/executor.h" #include "executor/spi.h" +#include "executor/tstoreReceiver.h" #include "miscadmin.h" +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/optimizer.h" +#include "parser/analyze.h" +#include "parser/parser.h" +#include "parser/parsetree.h" +#include "parser/parse_clause.h" +#include "parser/parse_coerce.h" +#include "parser/parse_collate.h" +#include "parser/parse_func.h" #include "parser/parse_relation.h" +#include "parser/parse_type.h" #include "pgstat.h" #include "rewrite/rewriteHandler.h" +#include "rewrite/rewriteManip.h" +#include "rewrite/rowsecurity.h" #include "storage/lmgr.h" #include "storage/smgr.h" #include "tcop/tcopprot.h" #include "utils/builtins.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/rel.h" #include "utils/snapmgr.h" #include "utils/syscache.h" +#include "utils/typcache.h" typedef struct @@ -58,6 +79,50 @@ typedef struct BulkInsertState bistate; /* bulk insert state */ } DR_transientrel; +#define MV_INIT_QUERYHASHSIZE 16 + +/* + * MV_TriggerHashEntry + * + * Hash entry for base tables on which IVM trigger is invoked + */ +typedef struct MV_TriggerHashEntry +{ + Oid matview_id; /* OID of the materialized view */ + int before_trig_count; /* count of before triggers invoked */ + int after_trig_count; /* count of after triggers invoked */ + + TransactionId xid; /* Transaction id before the first table is modified*/ + CommandId cid; /* Command id before the first table is modified */ + + List *tables; /* List of MV_TriggerTable */ + bool has_old; /* tuples are deleted from any table? */ + bool has_new; /* tuples are inserted into any table? */ +} MV_TriggerHashEntry; + +/* + * MV_TriggerTable + * + * IVM related data for tables on which the trigger is invoked. + */ +typedef struct MV_TriggerTable +{ + Oid table_id; /* OID of the modified table */ + List *old_tuplestores; /* tuplestores for deleted tuples */ + List *new_tuplestores; /* tuplestores for inserted tuples */ + List *old_rtes; /* RTEs of ENRs for old_tuplestores*/ + List *new_rtes; /* RTEs of ENRs for new_tuplestores */ + + List *rte_indexes; /* List of RTE index of the modified table */ + RangeTblEntry *original_rte; /* the original RTE saved before rewriting query */ +} MV_TriggerTable; + +static HTAB *mv_trigger_info = NULL; + +/* ENR name for materialized view delta */ +#define NEW_DELTA_ENRNAME "new_delta" +#define OLD_DELTA_ENRNAME "old_delta" + static int matview_maintenance_depth = 0; static void transientrel_startup(DestReceiver *self, int operation, TupleDesc typeinfo); @@ -65,7 +130,9 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self); static void transientrel_shutdown(DestReceiver *self); static void transientrel_destroy(DestReceiver *self); static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query, - const char *queryString); + QueryEnvironment *queryEnv, + TupleDesc *resultTupleDesc, + const char *queryString); static char *make_temptable_name_n(char *tempname, int n); static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner, int save_sec_context); @@ -73,6 +140,45 @@ static void refresh_by_heap_swap(Oid matviewOid, Oid OIDNewHeap, char relpersist static bool is_usable_unique_index(Relation indexRel); static void OpenMatViewIncrementalMaintenance(void); static void CloseMatViewIncrementalMaintenance(void); +static Query *get_matview_query(Relation matviewRel); + +static Query *rewrite_query_for_preupdate_state(Query *query, List *tables, + TransactionId xid, CommandId cid, + ParseState *pstate); +static void register_delta_ENRs(ParseState *pstate, Query *query, List *tables); +static char *make_delta_enr_name(const char *prefix, Oid relid, int count); +static RangeTblEntry *get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table, + TransactionId xid, CommandId cid, + QueryEnvironment *queryEnv); +static RangeTblEntry *union_ENRs(RangeTblEntry *rte, Oid relid, List *enr_rtes, const char *prefix, + QueryEnvironment *queryEnv); +static Query *rewrite_query_for_distinct(Query *query, ParseState *pstate); + +static void calc_delta(MV_TriggerTable *table, int rte_index, Query *query, + DestReceiver *dest_old, DestReceiver *dest_new, + TupleDesc *tupdesc_old, TupleDesc *tupdesc_new, + QueryEnvironment *queryEnv); +static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte_index); + +static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores, + TupleDesc tupdesc_old, TupleDesc tupdesc_new, + Query *query, bool use_count, char *count_colname); +static void apply_old_delta(const char *matviewname, const char *deltaname_old, + List *keys); +static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old, + List *keys, const char *count_colname); +static void apply_new_delta(const char *matviewname, const char *deltaname_new, + StringInfo target_list); +static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new, + List *keys, StringInfo target_list, const char* count_colname); +static char *get_matching_condition_string(List *keys); +static void generate_equal(StringInfo querybuf, Oid opttype, + const char *leftop, const char *rightop); + +static void mv_InitHashTables(void); +static void clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry); + +static List *get_securityQuals(Oid relId, int rt_index, Query *query); /* * SetMatViewPopulatedState @@ -114,6 +220,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate) CommandCounterIncrement(); } +/* + * SetMatViewIVMState + * Mark a materialized view as IVM, or not. + * + * NOTE: caller must be holding an appropriate lock on the relation. + */ +void +SetMatViewIVMState(Relation relation, bool newstate) +{ + Relation pgrel; + HeapTuple tuple; + + Assert(relation->rd_rel->relkind == RELKIND_MATVIEW); + + /* + * Update relation's pg_class entry. Crucial side-effect: other backends + * (and this one too!) are sent SI message to make them rebuild relcache + * entries. + */ + pgrel = table_open(RelationRelationId, RowExclusiveLock); + tuple = SearchSysCacheCopy1(RELOID, + ObjectIdGetDatum(RelationGetRelid(relation))); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for relation %u", + RelationGetRelid(relation)); + + ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate; + + CatalogTupleUpdate(pgrel, &tuple->t_self, tuple); + + heap_freetuple(tuple); + table_close(pgrel, RowExclusiveLock); + + /* + * Advance command counter to make the updated pg_class row locally + * visible. + */ + CommandCounterIncrement(); +} + /* * ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command * @@ -140,8 +286,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, { Oid matviewOid; Relation matviewRel; - RewriteRule *rule; - List *actions; Query *dataQuery; Oid tableSpace; Oid relowner; @@ -155,6 +299,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, int save_sec_context; int save_nestlevel; ObjectAddress address; + bool oldPopulated; /* Determine strength of lock needed. */ concurrent = stmt->concurrent; @@ -167,6 +312,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, lockmode, 0, RangeVarCallbackOwnsTable, NULL); matviewRel = table_open(matviewOid, NoLock); + oldPopulated = RelationIsPopulated(matviewRel); /* Make sure it is a materialized view. */ if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW) @@ -187,32 +333,12 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("CONCURRENTLY and WITH NO DATA options cannot be used together"))); - /* - * Check that everything is correct for a refresh. Problems at this point - * are internal errors, so elog is sufficient. - */ - if (matviewRel->rd_rel->relhasrules == false || - matviewRel->rd_rules->numLocks < 1) - elog(ERROR, - "materialized view \"%s\" is missing rewrite information", - RelationGetRelationName(matviewRel)); - - if (matviewRel->rd_rules->numLocks > 1) - elog(ERROR, - "materialized view \"%s\" has too many rules", - RelationGetRelationName(matviewRel)); - rule = matviewRel->rd_rules->rules[0]; - if (rule->event != CMD_SELECT || !(rule->isInstead)) - elog(ERROR, - "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule", - RelationGetRelationName(matviewRel)); + dataQuery = get_matview_query(matviewRel); - actions = rule->actions; - if (list_length(actions) != 1) - elog(ERROR, - "the rule for materialized view \"%s\" is not a single action", - RelationGetRelationName(matviewRel)); + /* For IMMV, we need to rewrite matview query */ + if (!stmt->skipData && RelationIsIVM(matviewRel)) + dataQuery = rewriteQueryForIMMV(dataQuery,NIL); /* * Check that there is a unique index with no WHERE clause on one or more @@ -247,12 +373,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, errhint("Create a unique index with no WHERE clause on one or more columns of the materialized view."))); } - /* - * The stored query was rewritten at the time of the MV definition, but - * has not been scribbled on by the planner. - */ - dataQuery = linitial_node(Query, actions); - /* * Check for active uses of the relation in the current transaction, such * as open scans. @@ -293,6 +413,52 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, relpersistence = matviewRel->rd_rel->relpersistence; } + /* delete immv triggers */ + if (RelationIsIVM(matviewRel) && stmt->skipData ) + { + /* use deleted trigger */ + Relation depRel; + ScanKeyData key; + SysScanDesc scan; + HeapTuple tup; + ObjectAddresses *immv_triggers; + + immv_triggers = new_object_addresses(); + + /* + * We save some cycles by opening pg_depend just once and passing the + * Relation pointer down to all the recursive deletion steps. + */ + depRel = table_open(DependRelationId, RowExclusiveLock); + + ScanKeyInit(&key, + Anum_pg_depend_refobjid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(matviewOid)); + + scan = systable_beginscan(depRel, DependReferenceIndexId, true, + NULL, 1, &key); + while ((tup = systable_getnext(scan)) != NULL) + { + ObjectAddress obj; + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->deptype == DEPENDENCY_IMMV) + { + obj.classId = foundDep->classid; + obj.objectId = foundDep->objid; + obj.objectSubId = foundDep->refobjsubid; + add_exact_object_address(&obj, immv_triggers); + } + } + systable_endscan(scan); + + performMultipleDeletions(immv_triggers, DROP_RESTRICT, PERFORM_DELETION_INTERNAL); + + table_close(depRel, RowExclusiveLock); + free_object_addresses(immv_triggers); + } + /* * Create the transient table that will receive the regenerated data. Lock * it against access by any other process until commit (by which time it @@ -311,7 +477,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, /* Generate the data, if wanted. */ if (!stmt->skipData) - processed = refresh_matview_datafill(dest, dataQuery, queryString); + processed = refresh_matview_datafill(dest, dataQuery, NULL, NULL, queryString); /* Make the matview match the newly generated data. */ if (concurrent) @@ -346,6 +512,9 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, pgstat_count_heap_insert(matviewRel, processed); } + if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated) + CreateIvmTriggersOnBaseTables(dataQuery, matviewOid, false); + table_close(matviewRel, NoLock); /* Roll back any GUC changes */ @@ -380,6 +549,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, */ static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query, + QueryEnvironment *queryEnv, + TupleDesc *resultTupleDesc, const char *queryString) { List *rewritten; @@ -416,7 +587,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query, /* Create a QueryDesc, redirecting output to our tuple receiver */ queryDesc = CreateQueryDesc(plan, queryString, GetActiveSnapshot(), InvalidSnapshot, - dest, NULL, NULL, 0); + dest, NULL, queryEnv ? queryEnv: NULL, 0); /* call ExecutorStart to prepare the plan for execution */ ExecutorStart(queryDesc, 0); @@ -426,6 +597,9 @@ refresh_matview_datafill(DestReceiver *dest, Query *query, processed = queryDesc->estate->es_processed; + if (resultTupleDesc) + *resultTupleDesc = CreateTupleDescCopy(queryDesc->tupDesc); + /* and clean up */ ExecutorFinish(queryDesc); ExecutorEnd(queryDesc); @@ -933,3 +1107,1308 @@ CloseMatViewIncrementalMaintenance(void) matview_maintenance_depth--; Assert(matview_maintenance_depth >= 0); } + +/* + * get_matview_query - get the Query from a matview's _RETURN rule. + */ +static Query * +get_matview_query(Relation matviewRel) +{ + RewriteRule *rule; + List * actions; + + /* + * Check that everything is correct for a refresh. Problems at this point + * are internal errors, so elog is sufficient. + */ + if (matviewRel->rd_rel->relhasrules == false || + matviewRel->rd_rules->numLocks < 1) + elog(ERROR, + "materialized view \"%s\" is missing rewrite information", + RelationGetRelationName(matviewRel)); + + if (matviewRel->rd_rules->numLocks > 1) + elog(ERROR, + "materialized view \"%s\" has too many rules", + RelationGetRelationName(matviewRel)); + + rule = matviewRel->rd_rules->rules[0]; + if (rule->event != CMD_SELECT || !(rule->isInstead)) + elog(ERROR, + "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule", + RelationGetRelationName(matviewRel)); + + actions = rule->actions; + if (list_length(actions) != 1) + elog(ERROR, + "the rule for materialized view \"%s\" is not a single action", + RelationGetRelationName(matviewRel)); + + /* + * The stored query was rewritten at the time of the MV definition, but + * has not been scribbled on by the planner. + */ + return linitial_node(Query, actions); +} + + +/* ---------------------------------------------------- + * Incremental View Maintenance routines + * --------------------------------------------------- + */ + +/* + * IVM_immediate_before + * + * IVM trigger function invoked before base table is modified. If this is + * invoked firstly in the same statement, we save the transaction id and the + * command id at that time. + */ +Datum +IVM_immediate_before(PG_FUNCTION_ARGS) +{ + TriggerData *trigdata = (TriggerData *) fcinfo->context; + char *matviewOid_text = trigdata->tg_trigger->tgargs[0]; + char *ex_lock_text = trigdata->tg_trigger->tgargs[1]; + Oid matviewOid; + MV_TriggerHashEntry *entry; + bool found; + bool ex_lock; + + matviewOid = DatumGetObjectId(DirectFunctionCall1(oidin, CStringGetDatum(matviewOid_text))); + ex_lock = DatumGetBool(DirectFunctionCall1(boolin, CStringGetDatum(ex_lock_text))); + + /* If the view has more than one tables, we have to use an exclusive lock. */ + if (ex_lock) + { + /* + * Wait for concurrent transactions which update this materialized view at + * READ COMMITED. This is needed to see changes committed in other + * transactions. No wait and raise an error at REPEATABLE READ or + * SERIALIZABLE to prevent update anomalies of matviews. + * XXX: dead-lock is possible here. + */ + if (!IsolationUsesXactSnapshot()) + LockRelationOid(matviewOid, ExclusiveLock); + else if (!ConditionalLockRelationOid(matviewOid, ExclusiveLock)) + { + /* try to throw error by name; relation could be deleted... */ + char *relname = get_rel_name(matviewOid); + + if (!relname) + ereport(ERROR, + (errcode(ERRCODE_LOCK_NOT_AVAILABLE), + errmsg("could not obtain lock on materialized view during incremental maintenance"))); + + ereport(ERROR, + (errcode(ERRCODE_LOCK_NOT_AVAILABLE), + errmsg("could not obtain lock on materialized view \"%s\" during incremental maintenance", + relname))); + } + } + else + LockRelationOid(matviewOid, RowExclusiveLock); + + /* + * On the first call initialize the hashtable + */ + if (!mv_trigger_info) + mv_InitHashTables(); + + entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info, + (void *) &matviewOid, + HASH_ENTER, &found); + + /* On the first BEFORE to update the view, initialize trigger data */ + if (!found) + { + Snapshot snapshot = GetActiveSnapshot(); + + entry->matview_id = matviewOid; + entry->before_trig_count = 0; + entry->after_trig_count = 0; + entry->xid = GetCurrentTransactionId(); + entry->cid = snapshot->curcid; + entry->tables = NIL; + entry->has_old = false; + entry->has_new = false; + } + + entry->before_trig_count++; + + return PointerGetDatum(NULL); +} + +/* + * IVM_immediate_before + * + * IVM trigger function invoked after base table is modified. + * For each table, tuplestores of transition tables are collected. + * and after the last modification + */ +Datum +IVM_immediate_maintenance(PG_FUNCTION_ARGS) +{ + TriggerData *trigdata = (TriggerData *) fcinfo->context; + Relation rel; + Oid relid; + Oid matviewOid; + Query *query; + Query *rewritten = NULL; + char *matviewOid_text = trigdata->tg_trigger->tgargs[0]; + Relation matviewRel; + int old_depth = matview_maintenance_depth; + + Oid relowner; + Tuplestorestate *old_tuplestore = NULL; + Tuplestorestate *new_tuplestore = NULL; + DestReceiver *dest_new = NULL, *dest_old = NULL; + Oid save_userid; + int save_sec_context; + int save_nestlevel; + + MV_TriggerHashEntry *entry; + MV_TriggerTable *table; + bool found; + + ParseState *pstate; + QueryEnvironment *queryEnv = create_queryEnv(); + MemoryContext oldcxt; + ListCell *lc; + int i; + + + /* Create a ParseState for rewriting the view definition query */ + pstate = make_parsestate(NULL); + pstate->p_queryEnv = queryEnv; + pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET; + + rel = trigdata->tg_relation; + relid = rel->rd_id; + + matviewOid = DatumGetObjectId(DirectFunctionCall1(oidin, CStringGetDatum(matviewOid_text))); + + /* + * On the first call initialize the hashtable + */ + if (!mv_trigger_info) + mv_InitHashTables(); + + /* get the entry for this materialized view */ + entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info, + (void *) &matviewOid, + HASH_FIND, &found); + Assert (found && entry != NULL); + entry->after_trig_count++; + + /* search the entry for the modified table and create new entry if not found */ + found = false; + foreach(lc, entry->tables) + { + table = (MV_TriggerTable *) lfirst(lc); + if (table->table_id == relid) + { + found = true; + break; + } + } + if (!found) + { + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + table = (MV_TriggerTable *) palloc0(sizeof(MV_TriggerTable)); + table->table_id = relid; + table->old_tuplestores = NIL; + table->new_tuplestores = NIL; + table->old_rtes = NIL; + table->new_rtes = NIL; + table->rte_indexes = NIL; + entry->tables = lappend(entry->tables, table); + + MemoryContextSwitchTo(oldcxt); + } + + /* Save the transition tables and make a request to not free immediately */ + if (trigdata->tg_oldtable) + { + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + table->old_tuplestores = lappend(table->old_tuplestores, trigdata->tg_oldtable); + entry->has_old = true; + MemoryContextSwitchTo(oldcxt); + } + if (trigdata->tg_newtable) + { + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + table->new_tuplestores = lappend(table->new_tuplestores, trigdata->tg_newtable); + entry->has_new = true; + MemoryContextSwitchTo(oldcxt); + } + if (entry->has_new || entry->has_old) + { + CmdType cmd; + + if (TRIGGER_FIRED_BY_INSERT(trigdata->tg_event)) + cmd = CMD_INSERT; + else if (TRIGGER_FIRED_BY_DELETE(trigdata->tg_event)) + cmd = CMD_DELETE; + else if (TRIGGER_FIRED_BY_UPDATE(trigdata->tg_event)) + cmd = CMD_UPDATE; + else + elog(ERROR,"unsupported trigger type"); + + /* Prolong lifespan of transition tables to the end of the last AFTER trigger */ + SetTransitionTablePreserved(relid, cmd); + } + + + /* If this is not the last AFTER trigger call, immediately exit. */ + Assert (entry->before_trig_count >= entry->after_trig_count); + if (entry->before_trig_count != entry->after_trig_count) + return PointerGetDatum(NULL); + + /* + * If this is the last AFTER trigger call, continue and update the view. + */ + + /* + * Advance command counter to make the updated base table row locally + * visible. + */ + CommandCounterIncrement(); + + matviewRel = table_open(matviewOid, NoLock); + + /* get view query*/ + query = get_matview_query(matviewRel); + + /* Make sure it is a materialized view. */ + Assert(matviewRel->rd_rel->relkind == RELKIND_MATVIEW); + + /* + * Get and push the latast snapshot to see any changes which is committed + * during waiting in other transactions at READ COMMITTED level. + */ + PushActiveSnapshot(GetTransactionSnapshot()); + + /* + * Check for active uses of the relation in the current transaction, such + * as open scans. + * + * NB: We count on this to protect us against problems with refreshing the + * data using TABLE_INSERT_FROZEN. + */ + CheckTableNotInUse(matviewRel, "refresh a materialized view incrementally"); + + /* + * Switch to the owner's userid, so that any functions are run as that + * user. Also arrange to make GUC variable changes local to this command. + * We will switch modes when we are about to execute user code. + */ + relowner = matviewRel->rd_rel->relowner; + GetUserIdAndSecContext(&save_userid, &save_sec_context); + SetUserIdAndSecContext(relowner, + save_sec_context | SECURITY_RESTRICTED_OPERATION); + save_nestlevel = NewGUCNestLevel(); + + /* + * rewrite query for calculating deltas + */ + + rewritten = copyObject(query); + + /* Replace resnames in a target list with materialized view's attnames */ + i = 0; + foreach (lc, rewritten->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i); + char *resname = NameStr(attr->attname); + + tle->resname = pstrdup(resname); + i++; + } + + /* Set all tables in the query to pre-update state */ + rewritten = rewrite_query_for_preupdate_state(rewritten, entry->tables, + entry->xid, entry->cid, + pstate); + /* Rewrite for DISTINCT clause */ + rewritten = rewrite_query_for_distinct(rewritten, pstate); + + /* Create tuplestores to store view deltas */ + if (entry->has_old) + { + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + old_tuplestore = tuplestore_begin_heap(false, false, work_mem); + dest_old = CreateDestReceiver(DestTuplestore); + SetTuplestoreDestReceiverParams(dest_old, + old_tuplestore, + TopTransactionContext, + false, + NULL, + NULL); + + MemoryContextSwitchTo(oldcxt); + } + if (entry->has_new) + { + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + new_tuplestore = tuplestore_begin_heap(false, false, work_mem); + dest_new = CreateDestReceiver(DestTuplestore); + SetTuplestoreDestReceiverParams(dest_new, + new_tuplestore, + TopTransactionContext, + false, + NULL, + NULL); + MemoryContextSwitchTo(oldcxt); + } + + /* for all modified tables */ + foreach(lc, entry->tables) + { + ListCell *lc2; + + table = (MV_TriggerTable *) lfirst(lc); + + /* loop for self-join */ + foreach(lc2, table->rte_indexes) + { + int rte_index = lfirst_int(lc2); + TupleDesc tupdesc_old; + TupleDesc tupdesc_new; + bool use_count = false; + char *count_colname = NULL; + + count_colname = pstrdup("__ivm_count__"); + + if (query->distinctClause) + use_count = true; + + /* calculate delta tables */ + calc_delta(table, rte_index, rewritten, dest_old, dest_new, + &tupdesc_old, &tupdesc_new, queryEnv); + + /* Set the table in the query to post-update state */ + rewritten = rewrite_query_for_postupdate_state(rewritten, table, rte_index); + + PG_TRY(); + { + /* apply the delta tables to the materialized view */ + apply_delta(matviewOid, old_tuplestore, new_tuplestore, + tupdesc_old, tupdesc_new, query, use_count, + count_colname); + } + PG_CATCH(); + { + matview_maintenance_depth = old_depth; + PG_RE_THROW(); + } + PG_END_TRY(); + + /* clear view delta tuplestores */ + if (old_tuplestore) + tuplestore_clear(old_tuplestore); + if (new_tuplestore) + tuplestore_clear(new_tuplestore); + } + } + + /* Clean up hash entry and delete tuplestores */ + clean_up_IVM_hash_entry(entry); + if (old_tuplestore) + { + dest_old->rDestroy(dest_old); + tuplestore_end(old_tuplestore); + } + if (new_tuplestore) + { + dest_new->rDestroy(dest_new); + tuplestore_end(new_tuplestore); + } + + /* Pop the original snapshot. */ + PopActiveSnapshot(); + + table_close(matviewRel, NoLock); + + /* Roll back any GUC changes */ + AtEOXact_GUC(false, save_nestlevel); + + /* Restore userid and security context */ + SetUserIdAndSecContext(save_userid, save_sec_context); + + return PointerGetDatum(NULL); +} + +/* + * rewrite_query_for_preupdate_state + * + * Rewrite the query so that base tables' RTEs will represent "pre-update" + * state of tables. This is necessary to calculate view delta after multiple + * tables are modified. xid and cid are the transaction id and command id + * before the first table was modified. + */ +static Query* +rewrite_query_for_preupdate_state(Query *query, List *tables, + TransactionId xid, CommandId cid, + ParseState *pstate) +{ + ListCell *lc; + int num_rte = list_length(query->rtable); + int i; + + + /* register delta ENRs */ + register_delta_ENRs(pstate, query, tables); + + /* XXX: Is necessary? Is this right timing? */ + AcquireRewriteLocks(query, true, false); + + i = 1; + foreach(lc, query->rtable) + { + RangeTblEntry *r = (RangeTblEntry*) lfirst(lc); + + ListCell *lc2; + foreach(lc2, tables) + { + MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc2); + /* + * if the modified table is found then replace the original RTE with + * "pre-state" RTE and append its index to the list. + */ + if (r->relid == table->table_id) + { + lfirst(lc) = get_prestate_rte(r, table, xid, cid, pstate->p_queryEnv); + table->rte_indexes = lappend_int(table->rte_indexes, i); + break; + } + } + + /* finish the loop if we processed all RTE included in the original query */ + if (i++ >= num_rte) + break; + } + + return query; +} + +/* + * register_delta_ENRs + * + * For all modified tables, make ENRs for their transition tables + * and register them to the queryEnv. ENR's RTEs are also appended + * into the list in query tree. + */ +static void +register_delta_ENRs(ParseState *pstate, Query *query, List *tables) +{ + QueryEnvironment *queryEnv = pstate->p_queryEnv; + ListCell *lc; + RangeTblEntry *rte; + + foreach(lc, tables) + { + MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc); + ListCell *lc2; + int count; + + count = 0; + foreach(lc2, table->old_tuplestores) + { + Tuplestorestate *oldtable = (Tuplestorestate *) lfirst(lc2); + EphemeralNamedRelation enr = + palloc(sizeof(EphemeralNamedRelationData)); + ParseNamespaceItem *nsitem; + + enr->md.name = make_delta_enr_name("old", table->table_id, count); + enr->md.reliddesc = table->table_id; + enr->md.tupdesc = NULL; + enr->md.enrtype = ENR_NAMED_TUPLESTORE; + enr->md.enrtuples = tuplestore_tuple_count(oldtable); + enr->reldata = oldtable; + register_ENR(queryEnv, enr); + + nsitem = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true); + rte = nsitem->p_rte; + /* if base table has RLS, set security condition to enr */ + rte->securityQuals = get_securityQuals(table->table_id, list_length(query->rtable) + 1, query); + + query->rtable = lappend(query->rtable, rte); + table->old_rtes = lappend(table->old_rtes, rte); + + count++; + } + + count = 0; + foreach(lc2, table->new_tuplestores) + { + Tuplestorestate *newtable = (Tuplestorestate *) lfirst(lc2); + EphemeralNamedRelation enr = + palloc(sizeof(EphemeralNamedRelationData)); + ParseNamespaceItem *nsitem; + + enr->md.name = make_delta_enr_name("new", table->table_id, count); + enr->md.reliddesc = table->table_id; + enr->md.tupdesc = NULL; + enr->md.enrtype = ENR_NAMED_TUPLESTORE; + enr->md.enrtuples = tuplestore_tuple_count(newtable); + enr->reldata = newtable; + register_ENR(queryEnv, enr); + + nsitem = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true); + rte = nsitem->p_rte; + /* if base table has RLS, set security condition to enr*/ + rte->securityQuals = get_securityQuals(table->table_id, list_length(query->rtable) + 1, query); + + query->rtable = lappend(query->rtable, rte); + table->new_rtes = lappend(table->new_rtes, rte); + + count++; + } + } +} + +/* + * get_prestate_rte + * + * Rewrite RTE of the modified table to a subquery which represents + * "pre-state" table. The original RTE is saved in table->rte_original. + */ +static RangeTblEntry* +get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table, + TransactionId xid, CommandId cid, + QueryEnvironment *queryEnv) +{ + StringInfoData str; + RawStmt *raw; + Query *sub; + Relation rel; + ParseState *pstate; + char *relname; + int i; + + pstate = make_parsestate(NULL); + pstate->p_queryEnv = queryEnv; + pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET; + + /* + * We can use NoLock here since AcquireRewriteLocks should + * have locked the rel already. + */ + rel = table_open(table->table_id, NoLock); + relname = quote_qualified_identifier( + get_namespace_name(RelationGetNamespace(rel)), + RelationGetRelationName(rel)); + table_close(rel, NoLock); + + initStringInfo(&str); + appendStringInfo(&str, + "SELECT t.* FROM %s t" + " WHERE (age(t.xmin) - age(%u::text::xid) > 0) OR" + " (t.xmin = %u AND t.cmin::text::int < %u)", + relname, xid, xid, cid); + + for (i = 0; i < list_length(table->old_tuplestores); i++) + { + appendStringInfo(&str, " UNION ALL "); + appendStringInfo(&str," SELECT * FROM %s", + make_delta_enr_name("old", table->table_id, i)); + } + + raw = (RawStmt*)linitial(raw_parser(str.data, RAW_PARSE_DEFAULT)); + sub = transformStmt(pstate, raw->stmt); + + /* If this query has setOperations, RTEs in rtables has a subquery which contains ENR */ + if (sub->setOperations != NULL) + { + ListCell *lc; + + /* add securityQuals for tuplestores */ + foreach (lc, sub->rtable) + { + RangeTblEntry *rte; + RangeTblEntry *sub_rte; + + rte = (RangeTblEntry *)lfirst(lc); + Assert(rte->subquery != NULL); + + sub_rte = (RangeTblEntry *)linitial(rte->subquery->rtable); + if (sub_rte->rtekind == RTE_NAMEDTUPLESTORE) + /* rt_index is always 1, bacause subquery has enr_rte only */ + sub_rte->securityQuals = get_securityQuals(sub_rte->relid, 1, sub); + } + } + + /* save the original RTE */ + table->original_rte = copyObject(rte); + + rte->rtekind = RTE_SUBQUERY; + rte->subquery = sub; + rte->security_barrier = false; + /* Clear fields that should not be set in a subquery RTE */ + rte->relid = InvalidOid; + rte->relkind = 0; + rte->rellockmode = 0; + rte->tablesample = NULL; + rte->inh = false; /* must not be set for a subquery */ + + rte->requiredPerms = 0; /* no permission check on subquery itself */ + rte->checkAsUser = InvalidOid; + rte->selectedCols = NULL; + rte->insertedCols = NULL; + rte->updatedCols = NULL; + rte->extraUpdatedCols = NULL; + + return rte; +} + +/* + * make_delta_enr_name + * + * Make a name for ENR of a transition table from the base table's oid. + * prefix will be "new" or "old" depending on its transition table kind.. + */ +static char* +make_delta_enr_name(const char *prefix, Oid relid, int count) +{ + char buf[NAMEDATALEN]; + char *name; + + snprintf(buf, NAMEDATALEN, "__ivm_%s_%u_%u", prefix, relid, count); + name = pstrdup(buf); + + return name; +} + +/* + * union_ENRs + * + * Make a single table delta by unionning all transition tables of the modified table + * whose RTE is specified by + */ +static RangeTblEntry* +union_ENRs(RangeTblEntry *rte, Oid relid, List *enr_rtes, const char *prefix, + QueryEnvironment *queryEnv) +{ + StringInfoData str; + ParseState *pstate; + RawStmt *raw; + Query *sub; + int i; + RangeTblEntry *enr_rte; + + /* Create a ParseState for rewriting the view definition query */ + pstate = make_parsestate(NULL); + pstate->p_queryEnv = queryEnv; + pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET; + + initStringInfo(&str); + + for (i = 0; i < list_length(enr_rtes); i++) + { + if (i > 0) + appendStringInfo(&str, " UNION ALL "); + + appendStringInfo(&str, + " SELECT * FROM %s", + make_delta_enr_name(prefix, relid, i)); + } + + raw = (RawStmt*)linitial(raw_parser(str.data, RAW_PARSE_DEFAULT)); + sub = transformStmt(pstate, raw->stmt); + + rte->rtekind = RTE_SUBQUERY; + rte->subquery = sub; + rte->security_barrier = false; + /* Clear fields that should not be set in a subquery RTE */ + rte->relid = InvalidOid; + rte->relkind = 0; + rte->rellockmode = 0; + rte->tablesample = NULL; + rte->inh = false; /* must not be set for a subquery */ + + rte->requiredPerms = 0; /* no permission check on subquery itself */ + rte->checkAsUser = InvalidOid; + rte->selectedCols = NULL; + rte->insertedCols = NULL; + rte->updatedCols = NULL; + rte->extraUpdatedCols = NULL; + /* if base table has RLS, set security condition to enr*/ + enr_rte = (RangeTblEntry *)linitial(sub->rtable); + /* rt_index is always 1, bacause subquery has enr_rte only */ + enr_rte->securityQuals = get_securityQuals(relid, 1, sub); + + return rte; +} + +/* + * rewrite_query_for_distinct + * + * Rewrite query for counting DISTINCT clause. + */ +static Query * +rewrite_query_for_distinct(Query *query, ParseState *pstate) +{ + TargetEntry *tle_count; + FuncCall *fn; + Node *node; + + /* Add count(*) for counting distinct tuples in views */ + fn = makeFuncCall(list_make1(makeString("count")), NIL, COERCE_EXPLICIT_CALL, -1); + fn->agg_star = true; + if (!query->groupClause && !query->hasAggs) + query->groupClause = transformDistinctClause(NULL, &query->targetList, query->sortClause, false); + + node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1); + + tle_count = makeTargetEntry((Expr *) node, + list_length(query->targetList) + 1, + pstrdup("__ivm_count__"), + false); + query->targetList = lappend(query->targetList, tle_count); + query->hasAggs = true; + + return query; +} + +/* + * calc_delta + * + * Calculate view deltas generated under the modification of a table specified + * by the RTE index. + */ +static void +calc_delta(MV_TriggerTable *table, int rte_index, Query *query, + DestReceiver *dest_old, DestReceiver *dest_new, + TupleDesc *tupdesc_old, TupleDesc *tupdesc_new, + QueryEnvironment *queryEnv) +{ + ListCell *lc = list_nth_cell(query->rtable, rte_index - 1); + RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc); + + /* Generate old delta */ + if (list_length(table->old_rtes) > 0) + { + /* Replace the modified table with the old delta table and calculate the old view delta. */ + lfirst(lc) = union_ENRs(rte, table->table_id, table->old_rtes, "old", queryEnv); + refresh_matview_datafill(dest_old, query, queryEnv, tupdesc_old, ""); + } + + /* Generate new delta */ + if (list_length(table->new_rtes) > 0) + { + /* Replace the modified table with the new delta table and calculate the new view delta*/ + lfirst(lc) = union_ENRs(rte, table->table_id, table->new_rtes, "new", queryEnv); + refresh_matview_datafill(dest_new, query, queryEnv, tupdesc_new, ""); + } +} + +/* + * rewrite_query_for_postupdate_state + * + * Rewrite the query so that the specified base table's RTEs will represent + * "post-update" state of tables. This is called after the view delta + * calculation due to changes on this table finishes. + */ +static Query* +rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte_index) +{ + ListCell *lc = list_nth_cell(query->rtable, rte_index - 1); + + /* Retore the original RTE */ + lfirst(lc) = table->original_rte; + + return query; +} + +/* + * apply_delta + * + * Apply deltas to the materialized view. In outer join cases, this requires + * the view maintenance graph. + */ +static void +apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores, + TupleDesc tupdesc_old, TupleDesc tupdesc_new, + Query *query, bool use_count, char *count_colname) +{ + StringInfoData querybuf; + StringInfoData target_list_buf; + Relation matviewRel; + char *matviewname; + ListCell *lc; + int i; + List *keys = NIL; + + + /* + * get names of the materialized view and delta tables + */ + + matviewRel = table_open(matviewOid, NoLock); + matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)), + RelationGetRelationName(matviewRel)); + + /* + * Build parts of the maintenance queries + */ + + initStringInfo(&querybuf); + initStringInfo(&target_list_buf); + + /* build string of target list */ + for (i = 0; i < matviewRel->rd_att->natts; i++) + { + Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i); + char *resname = NameStr(attr->attname); + + if (i != 0) + appendStringInfo(&target_list_buf, ", "); + appendStringInfo(&target_list_buf, "%s", quote_qualified_identifier(NULL, resname)); + } + + i = 0; + foreach (lc, query->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i); + char *resname = NameStr(attr->attname); + + i++; + + if (tle->resjunk) + continue; + + keys = lappend(keys, resname); + } + + /* Start maintaining the materialized view. */ + OpenMatViewIncrementalMaintenance(); + + /* Open SPI context. */ + if (SPI_connect() != SPI_OK_CONNECT) + elog(ERROR, "SPI_connect failed"); + + /* For tuple deletion */ + if (old_tuplestores && tuplestore_tuple_count(old_tuplestores) > 0) + { + EphemeralNamedRelation enr = palloc(sizeof(EphemeralNamedRelationData)); + int rc; + + /* convert tuplestores to ENR, and register for SPI */ + enr->md.name = pstrdup(OLD_DELTA_ENRNAME); + enr->md.reliddesc = InvalidOid; + enr->md.tupdesc = tupdesc_old; + enr->md.enrtype = ENR_NAMED_TUPLESTORE; + enr->md.enrtuples = tuplestore_tuple_count(old_tuplestores); + enr->reldata = old_tuplestores; + + rc = SPI_register_relation(enr); + if (rc != SPI_OK_REL_REGISTER) + elog(ERROR, "SPI_register failed"); + + if (use_count) + /* apply old delta and get rows to be recalculated */ + apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME, + keys, count_colname); + else + apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys); + + } + /* For tuple insertion */ + if (new_tuplestores && tuplestore_tuple_count(new_tuplestores) > 0) + { + EphemeralNamedRelation enr = palloc(sizeof(EphemeralNamedRelationData)); + int rc; + + /* convert tuplestores to ENR, and register for SPI */ + enr->md.name = pstrdup(NEW_DELTA_ENRNAME); + enr->md.reliddesc = InvalidOid; + enr->md.tupdesc = tupdesc_new;; + enr->md.enrtype = ENR_NAMED_TUPLESTORE; + enr->md.enrtuples = tuplestore_tuple_count(new_tuplestores); + enr->reldata = new_tuplestores; + + rc = SPI_register_relation(enr); + if (rc != SPI_OK_REL_REGISTER) + elog(ERROR, "SPI_register failed"); + + /* apply new delta */ + if (use_count) + apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME, + keys, &target_list_buf, count_colname); + else + apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf); + } + + /* We're done maintaining the materialized view. */ + CloseMatViewIncrementalMaintenance(); + + table_close(matviewRel, NoLock); + + /* Close SPI context. */ + if (SPI_finish() != SPI_OK_FINISH) + elog(ERROR, "SPI_finish failed"); +} + +/* + * apply_old_delta_with_count + * + * Execute a query for applying a delta table given by deltname_old + * which contains tuples to be deleted from to a materialized view given by + * matviewname. This is used when counting is required, that is, the view + * has aggregate or distinct. + */ +static void +apply_old_delta_with_count(const char *matviewname, const char *deltaname_old, + List *keys, const char *count_colname) +{ + StringInfoData querybuf; + char *match_cond; + + /* build WHERE condition for searching tuples to be deleted */ + match_cond = get_matching_condition_string(keys); + + /* Search for matching tuples from the view and update or delete if found. */ + initStringInfo(&querybuf); + appendStringInfo(&querybuf, + "WITH t AS (" /* collecting tid of target tuples in the view */ + "SELECT diff.%s, " /* count column */ + "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, " + "mv.ctid " + "FROM %s AS mv, %s AS diff " + "WHERE %s" /* tuple matching condition */ + "), updt AS (" /* update a tuple if this is not to be deleted */ + "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s " + "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt " + "), dlt AS (" /* delete a tuple if this is to be deleted */ + "DELETE FROM %s AS mv USING t " + "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt" + ")", + count_colname, + count_colname, count_colname, + matviewname, deltaname_old, + match_cond, + matviewname, count_colname, count_colname, count_colname, + matviewname); + + if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE) + elog(ERROR, "SPI_exec failed: %s", querybuf.data); +} + +/* + * apply_old_delta + * + * Execute a query for applying a delta table given by deltname_old + * which contains tuples to be deleted from to a materialized view given by + * matviewname. This is used when counting is not required. + */ +static void +apply_old_delta(const char *matviewname, const char *deltaname_old, + List *keys) +{ + StringInfoData querybuf; + StringInfoData keysbuf; + char *match_cond; + ListCell *lc; + + /* build WHERE condition for searching tuples to be deleted */ + match_cond = get_matching_condition_string(keys); + + /* build string of keys list */ + initStringInfo(&keysbuf); + foreach (lc, keys) + { + Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc); + char *resname = NameStr(attr->attname); + appendStringInfo(&keysbuf, "%s", quote_qualified_identifier("mv", resname)); + if (lnext(keys, lc)) + appendStringInfo(&keysbuf, ", "); + } + + /* Search for matching tuples from the view and update or delete if found. */ + initStringInfo(&querybuf); + appendStringInfo(&querybuf, + "DELETE FROM %s WHERE ctid IN (" + "SELECT tid FROM (SELECT row_number() over (partition by %s) AS \"__ivm_row_number__\"," + "mv.ctid AS tid," + "diff.\"__ivm_count__\"" + "FROM %s AS mv, %s AS diff " + "WHERE %s) v " + "WHERE v.\"__ivm_row_number__\" OPERATOR(pg_catalog.<=) v.\"__ivm_count__\")", + matviewname, + keysbuf.data, + matviewname, deltaname_old, + match_cond); + + if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE) + elog(ERROR, "SPI_exec failed: %s", querybuf.data); +} + +/* + * apply_new_delta_with_count + * + * Execute a query for applying a delta table given by deltname_new + * which contains tuples to be inserted into a materialized view given by + * matviewname. This is used when counting is required, that is, the view + * has aggregate or distinct. Also, when a table in EXISTS sub queries + * is modified. + */ +static void +apply_new_delta_with_count(const char *matviewname, const char* deltaname_new, + List *keys, StringInfo target_list, const char* count_colname) +{ + StringInfoData querybuf; + StringInfoData returning_keys; + ListCell *lc; + char *match_cond = ""; + + /* build WHERE condition for searching tuples to be updated */ + match_cond = get_matching_condition_string(keys); + + /* build string of keys list */ + initStringInfo(&returning_keys); + if (keys) + { + foreach (lc, keys) + { + Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc); + char *resname = NameStr(attr->attname); + appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname)); + if (lnext(keys, lc)) + appendStringInfo(&returning_keys, ", "); + } + } + else + appendStringInfo(&returning_keys, "NULL"); + + /* Search for matching tuples from the view and update if found or insert if not. */ + initStringInfo(&querybuf); + appendStringInfo(&querybuf, + "WITH updt AS (" /* update a tuple if this exists in the view */ + "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s " + "FROM %s AS diff " + "WHERE %s " /* tuple matching condition */ + "RETURNING %s" /* returning keys of updated tuples */ + ") INSERT INTO %s (%s)" /* insert a new tuple if this doesn't existw */ + "SELECT %s FROM %s AS diff " + "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);", + matviewname, count_colname, count_colname, count_colname, + deltaname_new, + match_cond, + returning_keys.data, + matviewname, target_list->data, + target_list->data, deltaname_new, + match_cond); + + if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT) + elog(ERROR, "SPI_exec failed: %s", querybuf.data); +} + +/* + * apply_new_delta + * + * Execute a query for applying a delta table given by deltname_new + * which contains tuples to be inserted into a materialized view given by + * matviewname. This is used when counting is not required. + */ +static void +apply_new_delta(const char *matviewname, const char *deltaname_new, + StringInfo target_list) +{ + StringInfoData querybuf; + + /* Search for matching tuples from the view and update or delete if found. */ + initStringInfo(&querybuf); + appendStringInfo(&querybuf, + "INSERT INTO %s (%s) SELECT %s FROM (" + "SELECT diff.*, generate_series(1, diff.\"__ivm_count__\") " + "FROM %s AS diff) AS v", + matviewname, target_list->data, target_list->data, + deltaname_new); + + if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT) + elog(ERROR, "SPI_exec failed: %s", querybuf.data); +} + +/* + * get_matching_condition_string + * + * Build a predicate string for looking for a tuple with given keys. + */ +static char * +get_matching_condition_string(List *keys) +{ + StringInfoData match_cond; + ListCell *lc; + + /* If there is no key columns, the condition is always true. */ + if (keys == NIL) + return "true"; + + initStringInfo(&match_cond); + foreach (lc, keys) + { + Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc); + char *resname = NameStr(attr->attname); + char *mv_resname = quote_qualified_identifier("mv", resname); + char *diff_resname = quote_qualified_identifier("diff", resname); + Oid typid = attr->atttypid; + + /* Considering NULL values, we can not use simple = operator. */ + appendStringInfo(&match_cond, "("); + generate_equal(&match_cond, typid, mv_resname, diff_resname); + appendStringInfo(&match_cond, " OR (%s IS NULL AND %s IS NULL))", + mv_resname, diff_resname); + + if (lnext(keys, lc)) + appendStringInfo(&match_cond, " AND "); + } + + return match_cond.data; +} + +/* + * generate_equals + * + * Generate an equality clause using given operands' default equality + * operator. + */ +static void +generate_equal(StringInfo querybuf, Oid opttype, + const char *leftop, const char *rightop) +{ + TypeCacheEntry *typentry; + + typentry = lookup_type_cache(opttype, TYPECACHE_EQ_OPR); + if (!OidIsValid(typentry->eq_opr)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_FUNCTION), + errmsg("could not identify an equality operator for type %s", + format_type_be(opttype)))); + + generate_operator_clause(querybuf, + leftop, opttype, + typentry->eq_opr, + rightop, opttype); +} + +/* + * mv_InitHashTables + */ +static void +mv_InitHashTables(void) +{ + HASHCTL ctl; + + memset(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(Oid); + ctl.entrysize = sizeof(MV_TriggerHashEntry); + mv_trigger_info = hash_create("MV trigger info", + MV_INIT_QUERYHASHSIZE, + &ctl, HASH_ELEM | HASH_BLOBS); +} + +/* + * AtAbort_IVM + * + * Clean up hash entries for all materialized views. This is called at + * transaction abort. + */ +void +AtAbort_IVM() +{ + HASH_SEQ_STATUS seq; + MV_TriggerHashEntry *entry; + + if (mv_trigger_info) + { + hash_seq_init(&seq, mv_trigger_info); + while ((entry = hash_seq_search(&seq)) != NULL) + clean_up_IVM_hash_entry(entry); + } +} + +/* + * clean_up_IVM_hash_entry + * + * Clean up tuple stores and hash entries for a materialized view after its + * maintenance finished. + */ +static void +clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry) +{ + bool found; + ListCell *lc; + + foreach(lc, entry->tables) + { + MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc); + + list_free(table->old_tuplestores); + list_free(table->new_tuplestores); + } + list_free(entry->tables); + + hash_search(mv_trigger_info, (void *) &entry->matview_id, HASH_REMOVE, &found); +} + +/* + * isIvmName + * + * Check if this is a IVM hidden column from the name. + */ +bool +isIvmName(const char *s) +{ + if (s) + return (strncmp(s, "__ivm_", 6) == 0); + return false; +} + +/* + * get_securityQuals + * + * Get row security policy on a relation. + * This is used by IVM for copying RLS from base table to enr. + */ +static List * +get_securityQuals(Oid relId, int rt_index, Query *query) +{ + ParseState *pstate; + Relation rel; + ParseNamespaceItem *nsitem; + RangeTblEntry *rte; + List *securityQuals; + List *withCheckOptions; + bool hasRowSecurity; + bool hasSubLinks; + + securityQuals = NIL; + pstate = make_parsestate(NULL); + + rel = table_open(relId, NoLock); + nsitem = addRangeTableEntryForRelation(pstate, rel, AccessShareLock, NULL, false, false); + rte = nsitem->p_rte; + + get_row_security_policies(query, rte, rt_index, + &securityQuals, &withCheckOptions, + &hasRowSecurity, &hasSubLinks); + + /* + * Make sure the query is marked correctly if row level security + * applies, or if the new quals had sublinks. + */ + if (hasRowSecurity) + query->hasRowSecurity = true; + if (hasSubLinks) + query->hasSubLinks = true; + + table_close(rel, NoLock); + + return securityQuals; +} diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index a16e749506..9aad4792e5 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -50,6 +50,7 @@ #include "commands/cluster.h" #include "commands/comment.h" #include "commands/defrem.h" +#include "commands/matview.h" #include "commands/event_trigger.h" #include "commands/policy.h" #include "commands/sequence.h" @@ -3437,6 +3438,14 @@ renameatt_internal(Oid myrelid, targetrelation = relation_open(myrelid, AccessExclusiveLock); renameatt_check(myrelid, RelationGetForm(targetrelation), recursing); + /* + * Don't rename IVM columns. + */ + if (RelationIsIVM(targetrelation) && isIvmName(oldattname)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("IVM column can not be renamed"))); + /* * if the 'recurse' flag is set then we are supposed to rename this * attribute in all classes that inherit from 'relname' (as well as in diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 29020c908e..afda022a60 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2467,6 +2467,7 @@ _copyRangeTblEntry(const RangeTblEntry *from) COPY_SCALAR_FIELD(relkind); COPY_SCALAR_FIELD(rellockmode); COPY_NODE_FIELD(tablesample); + COPY_SCALAR_FIELD(relisivm); COPY_NODE_FIELD(subquery); COPY_SCALAR_FIELD(security_barrier); COPY_SCALAR_FIELD(jointype); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 8a1762000c..32345b7f41 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2730,6 +2730,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b) COMPARE_SCALAR_FIELD(relkind); COMPARE_SCALAR_FIELD(rellockmode); COMPARE_NODE_FIELD(tablesample); + COMPARE_SCALAR_FIELD(relisivm); COMPARE_NODE_FIELD(subquery); COMPARE_SCALAR_FIELD(security_barrier); COMPARE_SCALAR_FIELD(jointype); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 48202d2232..fd87a6153a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -3251,6 +3251,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node) WRITE_CHAR_FIELD(relkind); WRITE_INT_FIELD(rellockmode); WRITE_NODE_FIELD(tablesample); + WRITE_BOOL_FIELD(relisivm); break; case RTE_SUBQUERY: WRITE_NODE_FIELD(subquery); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 77d082d8b4..cd85f6910a 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1442,6 +1442,7 @@ _readRangeTblEntry(void) READ_CHAR_FIELD(relkind); READ_INT_FIELD(rellockmode); READ_NODE_FIELD(tablesample); + READ_BOOL_FIELD(relisivm); break; case RTE_SUBQUERY: READ_NODE_FIELD(subquery); diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c index 7465919044..5246c78ff8 100644 --- a/src/backend/parser/parse_relation.c +++ b/src/backend/parser/parse_relation.c @@ -36,6 +36,7 @@ #include "utils/rel.h" #include "utils/syscache.h" #include "utils/varlena.h" +#include "commands/matview.h" /* @@ -79,7 +80,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset, int rtindex, int sublevels_up, int location, bool include_dropped, - List **colnames, List **colvars); + List **colnames, List **colvars, bool is_ivm); static int specialAttNum(const char *attname); static bool isQueryUsingTempRelation_walker(Node *node, void *context); @@ -1433,6 +1434,7 @@ addRangeTableEntry(ParseState *pstate, rte->relid = RelationGetRelid(rel); rte->relkind = rel->rd_rel->relkind; rte->rellockmode = lockmode; + rte->relisivm = rel->rd_rel->relisivm; /* * Build the list of effective column names using user-supplied aliases @@ -1521,6 +1523,7 @@ addRangeTableEntryForRelation(ParseState *pstate, rte->relid = RelationGetRelid(rel); rte->relkind = rel->rd_rel->relkind; rte->rellockmode = lockmode; + rte->relisivm = rel->rd_rel->relisivm; /* * Build the list of effective column names using user-supplied aliases @@ -2676,7 +2679,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up, expandTupleDesc(tupdesc, rte->eref, rtfunc->funccolcount, atts_done, rtindex, sublevels_up, location, - include_dropped, colnames, colvars); + include_dropped, colnames, colvars, false); } else if (functypclass == TYPEFUNC_SCALAR) { @@ -2944,7 +2947,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up, expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0, rtindex, sublevels_up, location, include_dropped, - colnames, colvars); + colnames, colvars, RelationIsIVM(rel)); relation_close(rel, AccessShareLock); } @@ -2961,7 +2964,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset, int rtindex, int sublevels_up, int location, bool include_dropped, - List **colnames, List **colvars) + List **colnames, List **colvars, bool is_ivm) { ListCell *aliascell; int varattno; @@ -2974,6 +2977,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset, { Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno); + if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled()) + continue; + if (attr->attisdropped) { if (include_dropped) @@ -3127,6 +3133,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem, Var *varnode = (Var *) lfirst(var); TargetEntry *te; + /* if transform * into columnlist with IMMV, remove IVM columns */ + if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled()) + continue; + te = makeTargetEntry((Expr *) varnode, (AttrNumber) pstate->p_next_resno++, label, diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c index 6589345ac4..02f33d404b 100644 --- a/src/backend/rewrite/rewriteDefine.c +++ b/src/backend/rewrite/rewriteDefine.c @@ -776,7 +776,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect, attr->atttypmod)))); } - if (i != resultDesc->natts) + /* No check for materialized views since this could have special columns for IVM */ + if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts) ereport(ERROR, (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), isSelect ? diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 8cd0252082..c2841ab604 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11625,4 +11625,12 @@ prorettype => 'bytea', proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_send' }, +# IVM +{ oid => '786', descr => 'ivm trigger (before)', + proname => 'IVM_immediate_before', provolatile => 'v', prorettype => 'trigger', + proargtypes => '', prosrc => 'IVM_immediate_before' }, +{ oid => '787', descr => 'ivm trigger (after)', + proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger', + proargtypes => '', prosrc => 'IVM_immediate_maintenance' }, + ] diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h index ad5054d116..a57ce463e1 100644 --- a/src/include/commands/createas.h +++ b/src/include/commands/createas.h @@ -16,6 +16,7 @@ #include "catalog/objectaddress.h" #include "nodes/params.h" +#include "nodes/pathnodes.h" #include "parser/parse_node.h" #include "tcop/dest.h" #include "utils/queryenvironment.h" @@ -25,6 +26,10 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st ParamListInfo params, QueryEnvironment *queryEnv, QueryCompletion *qc); +extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid, bool is_create); + +extern Query *rewriteQueryForIMMV(Query *query, List *colNames); + extern int GetIntoRelEFlags(IntoClause *intoClause); extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause); diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h index 214b1c1df6..13a5722f17 100644 --- a/src/include/commands/matview.h +++ b/src/include/commands/matview.h @@ -15,6 +15,7 @@ #define MATVIEW_H #include "catalog/objectaddress.h" +#include "fmgr.h" #include "nodes/params.h" #include "nodes/parsenodes.h" #include "tcop/dest.h" @@ -23,6 +24,8 @@ extern void SetMatViewPopulatedState(Relation relation, bool newstate); +extern void SetMatViewIVMState(Relation relation, bool newstate); + extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, ParamListInfo params, QueryCompletion *qc); @@ -30,4 +33,9 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid); extern bool MatViewIncrementalMaintenanceIsEnabled(void); +extern Datum IVM_immediate_before(PG_FUNCTION_ARGS); +extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS); +extern void AtAbort_IVM(void); +extern bool isIvmName(const char *s); + #endif /* MATVIEW_H */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 947660a4b0..d10ce8780b 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -1023,6 +1023,7 @@ typedef struct RangeTblEntry char relkind; /* relation kind (see pg_class.relkind) */ int rellockmode; /* lock level that query requires on the rel */ struct TableSampleClause *tablesample; /* sampling info, or NULL */ + bool relisivm; /* * Fields valid for a subquery RTE (else NULL): @@ -2180,6 +2181,7 @@ typedef struct CreateStmt char *tablespacename; /* table space to use, or NULL */ char *accessMethod; /* table access method */ bool if_not_exists; /* just do nothing if it already exists? */ + bool ivm; /* incremental view maintenance is used by materialized view */ } CreateStmt; /* ---------- -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0008-Add-aggregates-support-in-IVM.patch" Content-Disposition: attachment; filename="v23-0008-Add-aggregates-support-in-IVM.patch" Content-Transfer-Encoding: quoted-printable From 7f00d0bbf0601f391c5709296061c743cc9d4200 Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Mon, 2 Aug 2021 14:59:27 +0900 Subject: [PATCH v23 08/15] Add aggregates support in IVM MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit Currently, count, sum, avg, min and max are supported. As a restriction, expressions specified in GROUP BY must appear in the target list because tuples to be updated in IMMV are identified by using this group keys. However, in the case of aggregates without GROUP BY, there is only one tuple in the view, so keys are not uses to identify tuples. When creating a IMMV, in addition to __ivm_count column, some hidden columns for each aggregate are added to the target list. For example, names of these hidden columns are ivm_count_avg and ivm_sum_avg for the average function, and so on. In the case of views without aggregate functions, only the number of tuple multiplicities in __ivm_count__ column are updated at incremental maintenance. On the other hand, in the case of view with aggregates, the aggregated values and related hidden columns are also updated. The way of update depends the kind of aggregate function.=E3=80=80Specifically, sum and count are updated by simply adding or subtracting delta value calculated from delta tables. avg is updated by using values of sum and count stored in views as hidden columns and deltas calculated from delta tables. In min or max cases, it becomes more complicated. For an example of min, when tuples are inserted, the smaller value between the current min value in the view and the value calculated from the new delta table is used. When tuples are deleted, if the current min value in the view equals to the min in the old delta table, we need re-computation the latest min value from base tables. Otherwise, the current value in the view remains. As to sum, avg, min, and max (any aggregate functions except to count), NULL in input values is ignored, and this returns a null value when no rows are selected. To support this specification, the number of not-NULL input values is counted and stored in views as a hidden column. In the case of count(), count(x) returns zero when no rows are selected, and count(*) doesn't ignore NULL input. These specification are also supported. --- src/backend/commands/createas.c | 294 ++++++++- src/backend/commands/matview.c | 1016 ++++++++++++++++++++++++++++++- src/include/commands/createas.h | 1 + 3 files changed, 1280 insertions(+), 31 deletions(-) diff --git a/src/backend/commands/createas.c b/src/backend/commands/createa= s.c index 3fc2af1c4a..aebbb7290e 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -63,6 +63,7 @@ #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/regproc.h" #include "utils/rel.h" #include "utils/rls.h" #include "utils/snapmgr.h" @@ -80,6 +81,11 @@ typedef struct BulkInsertState bistate; /* bulk insert state */ } DR_intorel; =20 +typedef struct +{ + bool has_agg; +} check_ivm_restriction_context; + /* utility functions for CTAS definition creation */ static ObjectAddress create_ctas_internal(List *attrList, IntoClause *into= ); static ObjectAddress create_ctas_nodata(List *tlist, IntoClause *into); @@ -93,9 +99,10 @@ static void intorel_destroy(DestReceiver *self); static void CreateIvmTriggersOnBaseTables_recurse(Query *qry, Node *node, = Oid matviewOid, Relids *relids, bool ex_lock); static void CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 ti= ming, bool ex_lock); static void check_ivm_restriction(Node *node); -static bool check_ivm_restriction_walker(Node *node, void *context); +static bool check_ivm_restriction_walker(Node *node, check_ivm_restriction= _context *context); static void CreateIndexOnIMMV(Query *query, Relation matviewRel); static Bitmapset *get_primary_key_attnos_from_query(Query *qry, List **con= straintList); +static bool check_aggregate_supports_ivm(Oid aggfnoid); =20 /* * create_ctas_internal @@ -431,6 +438,7 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt= *stmt, * rewriteQueryForIMMV -- rewrite view definition query for IMMV * * count(*) is added for counting distinct tuples in views. + * Also, additional hidden columns are added for aggregate values. */ Query * rewriteQueryForIMMV(Query *query, List *colNames) @@ -445,12 +453,45 @@ rewriteQueryForIMMV(Query *query, List *colNames) rewritten =3D copyObject(query); pstate->p_expr_kind =3D EXPR_KIND_SELECT_TARGET; =20 + /* group keys must be in targetlist */ + if (rewritten->groupClause) + { + ListCell *lc; + foreach(lc, rewritten->groupClause) + { + SortGroupClause *scl =3D (SortGroupClause *) lfirst(lc); + TargetEntry *tle =3D get_sortgroupclause_tle(scl, rewritten->targetList= ); + + if (tle->resjunk) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("GROUP BY expression not appeared in select list is not supp= orted on incrementally maintainable materialized view"))); + } + } /* Convert DISTINCT to GROUP BY. count(*) will be added afterward. */ - if (rewritten->distinctClause) + else if (!rewritten->hasAggs && rewritten->distinctClause) rewritten->groupClause =3D transformDistinctClause(NULL, &rewritten->tar= getList, rewritten->sortClause, false); =20 + /* Add additional columns for aggregate values */ + if (rewritten->hasAggs) + { + ListCell *lc; + List *aggs =3D NIL; + AttrNumber next_resno =3D list_length(rewritten->targetList) + 1; + + foreach(lc, rewritten->targetList) + { + TargetEntry *tle =3D (TargetEntry *) lfirst(lc); + char *resname =3D (colNames =3D=3D NIL ? tle->resname : strVal(list_nth= (colNames, tle->resno - 1))); + + if (IsA(tle->expr, Aggref)) + makeIvmAggColumn(pstate, (Aggref *)tle->expr, resname, &next_resno, &a= ggs); + } + rewritten->targetList =3D list_concat(rewritten->targetList, aggs); + } + /* Add count(*) for counting distinct tuples in views */ - if (rewritten->distinctClause) + if (rewritten->distinctClause || rewritten->hasAggs) { fn =3D makeFuncCall(list_make1(makeString("count")), NIL, COERCE_EXPLICI= T_CALL, -1); fn->agg_star =3D true; @@ -468,6 +509,91 @@ rewriteQueryForIMMV(Query *query, List *colNames) return rewritten; } =20 +/* + * makeIvmAggColumn -- make additional aggregate columns for IVM + * + * For an aggregate column specified by aggref, additional aggregate colum= ns + * are added, which are used to calculate the new aggregate value in IMMV. + * An additional aggregate columns has a name based on resname + * (ex. ivm_count_resname), and resno specified by next_resno. The created + * columns are returned to aggs, and the resno for the next column is also + * returned to next_resno. + * + * Currently, an additional count() is created for aggref other than count. + * In addition, sum() is created for avg aggregate column. + */ +void +makeIvmAggColumn(ParseState *pstate, Aggref *aggref, char *resname, AttrNu= mber *next_resno, List **aggs) +{ + TargetEntry *tle_count; + Node *node; + FuncCall *fn; + Const *dmy_arg =3D makeConst(INT4OID, + -1, + InvalidOid, + sizeof(int32), + Int32GetDatum(1), + false, + true); /* pass by value */ + const char *aggname =3D get_func_name(aggref->aggfnoid); + + /* + * For aggregate functions except to count, add count() func with the sam= e arg parameters. + * This count result is used for determining if the aggregate value shoul= d be NULL or not. + * Also, add sum() func for avg because we need to calculate an average v= alue as sum/count. + * + * XXX: If there are same expressions explicitly in the target list, we c= an use this instead + * of adding new duplicated one. + */ + if (strcmp(aggname, "count") !=3D 0) + { + fn =3D makeFuncCall(list_make1(makeString("count")), NIL, COERCE_EXPLICI= T_CALL, -1); + + /* Make a Func with a dummy arg, and then override this by the original = agg's args. */ + node =3D ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NU= LL, fn, false, -1); + ((Aggref *)node)->args =3D aggref->args; + + tle_count =3D makeTargetEntry((Expr *) node, + *next_resno, + pstrdup(makeObjectName("__ivm_count",resname, "_")), + false); + *aggs =3D lappend(*aggs, tle_count); + (*next_resno)++; + } + if (strcmp(aggname, "avg") =3D=3D 0) + { + List *dmy_args =3D NIL; + ListCell *lc; + foreach(lc, aggref->aggargtypes) + { + Oid typeid =3D lfirst_oid(lc); + Type type =3D typeidType(typeid); + + Const *con =3D makeConst(typeid, + -1, + typeTypeCollation(type), + typeLen(type), + (Datum) 0, + true, + typeByVal(type)); + dmy_args =3D lappend(dmy_args, con); + ReleaseSysCache(type); + } + fn =3D makeFuncCall(list_make1(makeString("sum")), NIL, COERCE_EXPLICIT_= CALL, -1); + + /* Make a Func with dummy args, and then override this by the original a= gg's args. */ + node =3D ParseFuncOrColumn(pstate, fn->funcname, dmy_args, NULL, fn, fal= se, -1); + ((Aggref *)node)->args =3D aggref->args; + + tle_count =3D makeTargetEntry((Expr *) node, + *next_resno, + pstrdup(makeObjectName("__ivm_sum",resname, "_")), + false); + *aggs =3D lappend(*aggs, tle_count); + (*next_resno)++; + } +} + /* * GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS * @@ -923,11 +1049,13 @@ CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type= , int16 timing, bool ex_lock static void check_ivm_restriction(Node *node) { - check_ivm_restriction_walker(node, NULL); + check_ivm_restriction_context context =3D {false}; + + check_ivm_restriction_walker(node, &context); } =20 static bool -check_ivm_restriction_walker(Node *node, void *context) +check_ivm_restriction_walker(Node *node, check_ivm_restriction_context *co= ntext) { /* This can recurse, so check for excessive recursion */ check_stack_depth(); @@ -1008,6 +1136,8 @@ check_ivm_restriction_walker(Node *node, void *contex= t) } } =20 + context->has_agg |=3D qry->hasAggs; + /* restrictions for rtable */ foreach(lc, qry->rtable) { @@ -1041,7 +1171,7 @@ check_ivm_restriction_walker(Node *node, void *contex= t) =20 } =20 - query_tree_walker(qry, check_ivm_restriction_walker, NULL, QTW_IGNORE_= RANGE_TABLE); + query_tree_walker(qry, check_ivm_restriction_walker, (void *) context,= QTW_IGNORE_RANGE_TABLE); =20 break; } @@ -1052,8 +1182,12 @@ check_ivm_restriction_walker(Node *node, void *conte= xt) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("column name %s is not supported on incrementally maintain= able materialized view", tle->resname))); + if (context->has_agg && !IsA(tle->expr, Aggref) && contain_aggs_of_lev= el((Node *) tle->expr, 0)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("expression containing an aggregate in it is not supported = on incrementally maintainable materialized view"))); =20 - expression_tree_walker(node, check_ivm_restriction_walker, NULL); + expression_tree_walker(node, check_ivm_restriction_walker, (void *) co= ntext); break; } case T_JoinExpr: @@ -1065,15 +1199,128 @@ check_ivm_restriction_walker(Node *node, void *con= text) (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("OUTER JOIN is not supported on incrementally maintainable= materialized view"))); =20 - expression_tree_walker(node, check_ivm_restriction_walker, NULL); + expression_tree_walker(node, check_ivm_restriction_walker, (void *) co= ntext); + break; } - break; - expression_tree_walker(node, check_ivm_restriction_walker, NULL); + case T_Aggref: + { + /* Check if this supports IVM */ + Aggref *aggref =3D (Aggref *) node; + const char *aggname =3D format_procedure(aggref->aggfnoid); + + if (aggref->aggfilter !=3D NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("aggregate function with FILTER clause is not supported on = incrementally maintainable materialized view"))); + + if (aggref->aggdistinct !=3D NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("aggregate function with DISTINCT arguments is not supporte= d on incrementally maintainable materialized view"))); + + if (aggref->aggorder !=3D NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("aggregate function with ORDER clause is not supported on i= ncrementally maintainable materialized view"))); + + if (!check_aggregate_supports_ivm(aggref->aggfnoid)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("aggregate function %s is not supported on incrementally ma= intainable materialized view", aggname))); + break; + } + default: + expression_tree_walker(node, check_ivm_restriction_walker, (void *) con= text); break; } return false; } =20 +/* + * check_aggregate_supports_ivm + * + * Check if the given aggregate function is supporting IVM + */ +static bool +check_aggregate_supports_ivm(Oid aggfnoid) +{ + switch (aggfnoid) + { + /* count */ + case F_COUNT_ANY: + case F_COUNT_: + + /* sum */ + case F_SUM_INT8: + case F_SUM_INT4: + case F_SUM_INT2: + case F_SUM_FLOAT4: + case F_SUM_FLOAT8: + case F_SUM_MONEY: + case F_SUM_INTERVAL: + case F_SUM_NUMERIC: + + /* avg */ + case F_AVG_INT8: + case F_AVG_INT4: + case F_AVG_INT2: + case F_AVG_NUMERIC: + case F_AVG_FLOAT4: + case F_AVG_FLOAT8: + case F_AVG_INTERVAL: + + /* min */ + case F_MIN_ANYARRAY: + case F_MIN_INT8: + case F_MIN_INT4: + case F_MIN_INT2: + case F_MIN_OID: + case F_MIN_FLOAT4: + case F_MIN_FLOAT8: + case F_MIN_DATE: + case F_MIN_TIME: + case F_MIN_TIMETZ: + case F_MIN_MONEY: + case F_MIN_TIMESTAMP: + case F_MIN_TIMESTAMPTZ: + case F_MIN_INTERVAL: + case F_MIN_TEXT: + case F_MIN_NUMERIC: + case F_MIN_BPCHAR: + case F_MIN_TID: + case F_MIN_ANYENUM: + case F_MIN_INET: + case F_MIN_PG_LSN: + + /* max */ + case F_MAX_ANYARRAY: + case F_MAX_INT8: + case F_MAX_INT4: + case F_MAX_INT2: + case F_MAX_OID: + case F_MAX_FLOAT4: + case F_MAX_FLOAT8: + case F_MAX_DATE: + case F_MAX_TIME: + case F_MAX_TIMETZ: + case F_MAX_MONEY: + case F_MAX_TIMESTAMP: + case F_MAX_TIMESTAMPTZ: + case F_MAX_INTERVAL: + case F_MAX_TEXT: + case F_MAX_NUMERIC: + case F_MAX_BPCHAR: + case F_MAX_TID: + case F_MAX_ANYENUM: + case F_MAX_INET: + case F_MAX_PG_LSN: + return true; + + default: + return false; + } +} + /* * CreateindexOnIMMV * @@ -1123,7 +1370,32 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel) index->concurrent =3D false; index->if_not_exists =3D false; =20 - if (qry->distinctClause) + + if (qry->groupClause) + { + /* create unique constraint on GROUP BY expression columns */ + foreach(lc, qry->groupClause) + { + SortGroupClause *scl =3D (SortGroupClause *) lfirst(lc); + TargetEntry *tle =3D get_sortgroupclause_tle(scl, qry->targetList); + Form_pg_attribute attr =3D TupleDescAttr(matviewRel->rd_att, tle->resno= - 1); + IndexElem *iparam; + + iparam =3D makeNode(IndexElem); + iparam->name =3D pstrdup(NameStr(attr->attname)); + iparam->expr =3D NULL; + iparam->indexcolname =3D NULL; + iparam->collation =3D NIL; + iparam->opclass =3D NIL; + iparam->opclassopts =3D NIL; + iparam->ordering =3D SORTBY_DEFAULT; + iparam->nulls_ordering =3D SORTBY_NULLS_DEFAULT; + index->indexParams =3D lappend(index->indexParams, iparam); + } + + index->isconstraint =3D true; + } + else if (qry->distinctClause) { /* create unique constraint on all columns */ index->isconstraint =3D true; diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index f1f46e2e14..bce2e3ae3d 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -81,6 +81,32 @@ typedef struct =20 #define MV_INIT_QUERYHASHSIZE 16 =20 +/* MV query type codes */ +#define MV_PLAN_RECALC 1 +#define MV_PLAN_SET_VALUE 2 + +/* + * MI_QueryKey + * + * The key identifying a prepared SPI plan in our query hashtable + */ +typedef struct MV_QueryKey +{ + Oid matview_id; /* OID of materialized view */ + int32 query_type; /* query type ID, see MV_PLAN_XXX above */ +} MV_QueryKey; + +/* + * MV_QueryHashEntry + * + * Hash entry for cached plans used to maintain materialized views. + */ +typedef struct MV_QueryHashEntry +{ + MV_QueryKey key; + SPIPlanPtr plan; +} MV_QueryHashEntry; + /* * MV_TriggerHashEntry * @@ -117,8 +143,16 @@ typedef struct MV_TriggerTable RangeTblEntry *original_rte; /* the original RTE saved before rewriting q= uery */ } MV_TriggerTable; =20 +static HTAB *mv_query_cache =3D NULL; static HTAB *mv_trigger_info =3D NULL; =20 +/* kind of IVM operation for the view */ +typedef enum +{ + IVM_ADD, + IVM_SUB +} IvmOp; + /* ENR name for materialized view delta */ #define NEW_DELTA_ENRNAME "new_delta" #define OLD_DELTA_ENRNAME "old_delta" @@ -152,7 +186,7 @@ static RangeTblEntry *get_prestate_rte(RangeTblEntry *r= te, MV_TriggerTable *tabl QueryEnvironment *queryEnv); static RangeTblEntry *union_ENRs(RangeTblEntry *rte, Oid relid, List *enr_= rtes, const char *prefix, QueryEnvironment *queryEnv); -static Query *rewrite_query_for_distinct(Query *query, ParseState *pstate); +static Query *rewrite_query_for_distinct_and_aggregates(Query *query, Pars= eState *pstate); =20 static void calc_delta(MV_TriggerTable *table, int rte_index, Query *query, DestReceiver *dest_old, DestReceiver *dest_new, @@ -163,19 +197,48 @@ static Query *rewrite_query_for_postupdate_state(Quer= y *query, MV_TriggerTable * static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, = Tuplestorestate *new_tuplestores, TupleDesc tupdesc_old, TupleDesc tupdesc_new, Query *query, bool use_count, char *count_colname); +static void append_set_clause_for_count(const char *resname, StringInfo bu= f_old, + StringInfo buf_new,StringInfo aggs_list); +static void append_set_clause_for_sum(const char *resname, StringInfo buf_= old, + StringInfo buf_new, StringInfo aggs_list); +static void append_set_clause_for_avg(const char *resname, StringInfo buf_= old, + StringInfo buf_new, StringInfo aggs_list, + const char *aggtype); +static void append_set_clause_for_minmax(const char *resname, StringInfo b= uf_old, + StringInfo buf_new, StringInfo aggs_list, + bool is_min); +static char *get_operation_string(IvmOp op, const char *col, const char *a= rg1, const char *arg2, + const char* count_col, const char *castType); +static char *get_null_condition_string(IvmOp op, const char *arg1, const c= har *arg2, + const char* count_col); static void apply_old_delta(const char *matviewname, const char *deltaname= _old, List *keys); static void apply_old_delta_with_count(const char *matviewname, const char= *deltaname_old, - List *keys, const char *count_colname); + List *keys, StringInfo aggs_list, StringInfo aggs_set, + List *minmax_list, List *is_min_list, + const char *count_colname, + SPITupleTable **tuptable_recalc, uint64 *num_recalc); static void apply_new_delta(const char *matviewname, const char *deltaname= _new, StringInfo target_list); static void apply_new_delta_with_count(const char *matviewname, const char= * deltaname_new, - List *keys, StringInfo target_list, const char* count_colname); + List *keys, StringInfo target_list, StringInfo aggs_set, + const char* count_colname); static char *get_matching_condition_string(List *keys); +static char *get_returning_string(List *minmax_list, List *is_min_list, Li= st *keys); +static char *get_minmax_recalc_condition_string(List *minmax_list, List *i= s_min_list); +static char *get_select_for_recalc_string(List *keys); +static void recalc_and_set_values(SPITupleTable *tuptable_recalc, int64 nu= m_tuples, + List *namelist, List *keys, Relation matviewRel); +static SPIPlanPtr get_plan_for_recalc(Oid matviewOid, List *namelist, List= *keys, Oid *keyTypes); +static SPIPlanPtr get_plan_for_set_values(Oid matviewOid, char *matviewnam= e, List *namelist, + Oid *valTypes); static void generate_equal(StringInfo querybuf, Oid opttype, const char *leftop, const char *rightop); =20 static void mv_InitHashTables(void); +static SPIPlanPtr mv_FetchPreparedPlan(MV_QueryKey *key); +static void mv_HashPreparedPlan(MV_QueryKey *key, SPIPlanPtr plan); +static void mv_BuildQueryKey(MV_QueryKey *key, Oid matview_id, int32 query= _type); static void clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry); =20 static List *get_securityQuals(Oid relId, int rt_index, Query *query); @@ -1432,8 +1495,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS) rewritten =3D rewrite_query_for_preupdate_state(rewritten, entry->tables, entry->xid, entry->cid, pstate); - /* Rewrite for DISTINCT clause */ - rewritten =3D rewrite_query_for_distinct(rewritten, pstate); + /* Rewrite for DISTINCT clause and aggregates functions */ + rewritten =3D rewrite_query_for_distinct_and_aggregates(rewritten, pstate= ); =20 /* Create tuplestores to store view deltas */ if (entry->has_old) @@ -1484,7 +1547,7 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS) =20 count_colname =3D pstrdup("__ivm_count__"); =20 - if (query->distinctClause) + if (query->hasAggs || query->distinctClause) use_count =3D true; =20 /* calculate delta tables */ @@ -1846,17 +1909,34 @@ union_ENRs(RangeTblEntry *rte, Oid relid, List *enr= _rtes, const char *prefix, } =20 /* - * rewrite_query_for_distinct + * rewrite_query_for_distinct_and_aggregates * - * Rewrite query for counting DISTINCT clause. + * Rewrite query for counting DISTINCT clause and aggregate functions. */ static Query * -rewrite_query_for_distinct(Query *query, ParseState *pstate) +rewrite_query_for_distinct_and_aggregates(Query *query, ParseState *pstate) { TargetEntry *tle_count; FuncCall *fn; Node *node; =20 + /* For aggregate views */ + if (query->hasAggs) + { + ListCell *lc; + List *aggs =3D NIL; + AttrNumber next_resno =3D list_length(query->targetList) + 1; + + foreach(lc, query->targetList) + { + TargetEntry *tle =3D (TargetEntry *) lfirst(lc); + + if (IsA(tle->expr, Aggref)) + makeIvmAggColumn(pstate, (Aggref *)tle->expr, tle->resname, &next_resn= o, &aggs); + } + query->targetList =3D list_concat(query->targetList, aggs); + } + /* Add count(*) for counting distinct tuples in views */ fn =3D makeFuncCall(list_make1(makeString("count")), NIL, COERCE_EXPLICIT= _CALL, -1); fn->agg_star =3D true; @@ -1925,6 +2005,8 @@ rewrite_query_for_postupdate_state(Query *query, MV_T= riggerTable *table, int rte return query; } =20 +#define IVM_colname(type, col) makeObjectName("__ivm_" type, col, "_") + /* * apply_delta * @@ -1938,11 +2020,16 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tu= plestores, Tuplestorestate *n { StringInfoData querybuf; StringInfoData target_list_buf; + StringInfo aggs_list_buf =3D NULL; + StringInfo aggs_set_old =3D NULL; + StringInfo aggs_set_new =3D NULL; Relation matviewRel; char *matviewname; ListCell *lc; int i; List *keys =3D NIL; + List *minmax_list =3D NIL; + List *is_min_list =3D NIL; =20 =20 /* @@ -1960,6 +2047,15 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tup= lestores, Tuplestorestate *n initStringInfo(&querybuf); initStringInfo(&target_list_buf); =20 + if (query->hasAggs) + { + if (old_tuplestores && tuplestore_tuple_count(old_tuplestores) > 0) + aggs_set_old =3D makeStringInfo(); + if (new_tuplestores && tuplestore_tuple_count(new_tuplestores) > 0) + aggs_set_new =3D makeStringInfo(); + aggs_list_buf =3D makeStringInfo(); + } + /* build string of target list */ for (i =3D 0; i < matviewRel->rd_att->natts; i++) { @@ -1983,7 +2079,65 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tup= lestores, Tuplestorestate *n if (tle->resjunk) continue; =20 - keys =3D lappend(keys, resname); + /* + * For views without aggregates, all attributes are used as keys to iden= tify a + * tuple in a view. + */ + if (!query->hasAggs) + keys =3D lappend(keys, attr); + + /* For views with aggregates, we need to build SET clause for updating a= ggregate + * values. */ + if (query->hasAggs && IsA(tle->expr, Aggref)) + { + Aggref *aggref =3D (Aggref *) tle->expr; + const char *aggname =3D get_func_name(aggref->aggfnoid); + + /* + * We can use function names here because it is already checked if these + * can be used in IMMV by its OID at the definition time. + */ + + /* count */ + if (!strcmp(aggname, "count")) + append_set_clause_for_count(resname, aggs_set_old, aggs_set_new, aggs_= list_buf); + + /* sum */ + else if (!strcmp(aggname, "sum")) + append_set_clause_for_sum(resname, aggs_set_old, aggs_set_new, aggs_li= st_buf); + + /* avg */ + else if (!strcmp(aggname, "avg")) + append_set_clause_for_avg(resname, aggs_set_old, aggs_set_new, aggs_li= st_buf, + format_type_be(aggref->aggtype)); + + /* min/max */ + else if (!strcmp(aggname, "min") || !strcmp(aggname, "max")) + { + bool is_min =3D (!strcmp(aggname, "min")); + + append_set_clause_for_minmax(resname, aggs_set_old, aggs_set_new, aggs= _list_buf, is_min); + + /* make a resname list of min and max aggregates */ + minmax_list =3D lappend(minmax_list, resname); + is_min_list =3D lappend_int(is_min_list, is_min); + } + else + elog(ERROR, "unsupported aggregate function: %s", aggname); + } + } + + /* If we have GROUP BY clause, we use its entries as keys. */ + if (query->hasAggs && query->groupClause) + { + foreach (lc, query->groupClause) + { + SortGroupClause *sgcl =3D (SortGroupClause *) lfirst(lc); + TargetEntry *tle =3D get_sortgroupclause_tle(sgcl, query->targetList); + Form_pg_attribute attr =3D TupleDescAttr(matviewRel->rd_att, tle->resno= - 1); + + keys =3D lappend(keys, attr); + } } =20 /* Start maintaining the materialized view. */ @@ -1997,6 +2151,8 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tupl= estores, Tuplestorestate *n if (old_tuplestores && tuplestore_tuple_count(old_tuplestores) > 0) { EphemeralNamedRelation enr =3D palloc(sizeof(EphemeralNamedRelationData)= ); + SPITupleTable *tuptable_recalc =3D NULL; + uint64 num_recalc; int rc; =20 /* convert tuplestores to ENR, and register for SPI */ @@ -2014,10 +2170,19 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tu= plestores, Tuplestorestate *n if (use_count) /* apply old delta and get rows to be recalculated */ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME, - keys, count_colname); + keys, aggs_list_buf, aggs_set_old, + minmax_list, is_min_list, + count_colname, &tuptable_recalc, &num_recalc); else apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys); =20 + /* + * If we have min or max, we might have to recalculate aggregate values = from base tables + * on some tuples. TIDs and keys such tuples are returned as a result of= the above query. + */ + if (minmax_list && tuptable_recalc) + recalc_and_set_values(tuptable_recalc, num_recalc, minmax_list, keys, m= atviewRel); + } /* For tuple insertion */ if (new_tuplestores && tuplestore_tuple_count(new_tuplestores) > 0) @@ -2040,7 +2205,7 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tupl= estores, Tuplestorestate *n /* apply new delta */ if (use_count) apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME, - keys, &target_list_buf, count_colname); + keys, aggs_set_new, &target_list_buf, count_colname); else apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf); } @@ -2055,49 +2220,410 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_t= uplestores, Tuplestorestate *n elog(ERROR, "SPI_finish failed"); } =20 +/* + * append_set_clause_for_count + * + * Append SET clause string for count aggregation to given buffers. + * Also, append resnames required for calculating the aggregate value. + */ +static void +append_set_clause_for_count(const char *resname, StringInfo buf_old, + StringInfo buf_new,StringInfo aggs_list) +{ + /* For tuple deletion */ + if (buf_old) + { + /* resname =3D mv.resname - t.resname */ + appendStringInfo(buf_old, + ", %s =3D %s", + quote_qualified_identifier(NULL, resname), + get_operation_string(IVM_SUB, resname, "mv", "t", NULL, NULL)); + } + /* For tuple insertion */ + if (buf_new) + { + /* resname =3D mv.resname + diff.resname */ + appendStringInfo(buf_new, + ", %s =3D %s", + quote_qualified_identifier(NULL, resname), + get_operation_string(IVM_ADD, resname, "mv", "diff", NULL, NULL)); + } + + appendStringInfo(aggs_list, ", %s", + quote_qualified_identifier("diff", resname) + ); +} + +/* + * append_set_clause_for_sum + * + * Append SET clause string for sum aggregation to given buffers. + * Also, append resnames required for calculating the aggregate value. + */ +static void +append_set_clause_for_sum(const char *resname, StringInfo buf_old, + StringInfo buf_new, StringInfo aggs_list) +{ + char *count_col =3D IVM_colname("count", resname); + + /* For tuple deletion */ + if (buf_old) + { + /* sum =3D mv.sum - t.sum */ + appendStringInfo(buf_old, + ", %s =3D %s", + quote_qualified_identifier(NULL, resname), + get_operation_string(IVM_SUB, resname, "mv", "t", count_col, NULL) + ); + /* count =3D mv.count - t.count */ + appendStringInfo(buf_old, + ", %s =3D %s", + quote_qualified_identifier(NULL, count_col), + get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL) + ); + } + /* For tuple insertion */ + if (buf_new) + { + /* sum =3D mv.sum + diff.sum */ + appendStringInfo(buf_new, + ", %s =3D %s", + quote_qualified_identifier(NULL, resname), + get_operation_string(IVM_ADD, resname, "mv", "diff", count_col, NULL) + ); + /* count =3D mv.count + diff.count */ + appendStringInfo(buf_new, + ", %s =3D %s", + quote_qualified_identifier(NULL, count_col), + get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL) + ); + } + + appendStringInfo(aggs_list, ", %s, %s", + quote_qualified_identifier("diff", resname), + quote_qualified_identifier("diff", IVM_colname("count", resname)) + ); +} + +/* + * append_set_clause_for_avg + * + * Append SET clause string for avg aggregation to given buffers. + * Also, append resnames required for calculating the aggregate value. + */ +static void +append_set_clause_for_avg(const char *resname, StringInfo buf_old, + StringInfo buf_new, StringInfo aggs_list, + const char *aggtype) +{ + char *sum_col =3D IVM_colname("sum", resname); + char *count_col =3D IVM_colname("count", resname); + + /* For tuple deletion */ + if (buf_old) + { + /* avg =3D (mv.sum - t.sum)::aggtype / (mv.count - t.count) */ + appendStringInfo(buf_old, + ", %s =3D %s OPERATOR(pg_catalog./) %s", + quote_qualified_identifier(NULL, resname), + get_operation_string(IVM_SUB, sum_col, "mv", "t", count_col, aggtype), + get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL) + ); + /* sum =3D mv.sum - t.sum */ + appendStringInfo(buf_old, + ", %s =3D %s", + quote_qualified_identifier(NULL, sum_col), + get_operation_string(IVM_SUB, sum_col, "mv", "t", count_col, NULL) + ); + /* count =3D mv.count - t.count */ + appendStringInfo(buf_old, + ", %s =3D %s", + quote_qualified_identifier(NULL, count_col), + get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL) + ); + + } + /* For tuple insertion */ + if (buf_new) + { + /* avg =3D (mv.sum + diff.sum)::aggtype / (mv.count + diff.count) */ + appendStringInfo(buf_new, + ", %s =3D %s OPERATOR(pg_catalog./) %s", + quote_qualified_identifier(NULL, resname), + get_operation_string(IVM_ADD, sum_col, "mv", "diff", count_col, aggtype= ), + get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL) + ); + /* sum =3D mv.sum + diff.sum */ + appendStringInfo(buf_new, + ", %s =3D %s", + quote_qualified_identifier(NULL, sum_col), + get_operation_string(IVM_ADD, sum_col, "mv", "diff", count_col, NULL) + ); + /* count =3D mv.count + diff.count */ + appendStringInfo(buf_new, + ", %s =3D %s", + quote_qualified_identifier(NULL, count_col), + get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL) + ); + } + + appendStringInfo(aggs_list, ", %s, %s, %s", + quote_qualified_identifier("diff", resname), + quote_qualified_identifier("diff", IVM_colname("sum", resname)), + quote_qualified_identifier("diff", IVM_colname("count", resname)) + ); +} + +/* + * append_set_clause_for_minmax + * + * Append SET clause string for min or max aggregation to given buffers. + * Also, append resnames required for calculating the aggregate value. + * is_min is true if this is min, false if not. + */ +static void +append_set_clause_for_minmax(const char *resname, StringInfo buf_old, + StringInfo buf_new, StringInfo aggs_list, + bool is_min) +{ + char *count_col =3D IVM_colname("count", resname); + + /* For tuple deletion */ + if (buf_old) + { + /* + * If the new value doesn't became NULL then use the value remaining + * in the view although this will be recomputated afterwords. + */ + appendStringInfo(buf_old, + ", %s =3D CASE WHEN %s THEN NULL ELSE %s END", + quote_qualified_identifier(NULL, resname), + get_null_condition_string(IVM_SUB, "mv", "t", count_col), + quote_qualified_identifier("mv", resname) + ); + /* count =3D mv.count - t.count */ + appendStringInfo(buf_old, + ", %s =3D %s", + quote_qualified_identifier(NULL, count_col), + get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL) + ); + } + /* For tuple insertion */ + if (buf_new) + { + /* + * min =3D LEAST(mv.min, diff.min) + * max =3D GREATEST(mv.max, diff.max) + */ + appendStringInfo(buf_new, + ", %s =3D CASE WHEN %s THEN NULL ELSE %s(%s,%s) END", + quote_qualified_identifier(NULL, resname), + get_null_condition_string(IVM_ADD, "mv", "diff", count_col), + + is_min ? "LEAST" : "GREATEST", + quote_qualified_identifier("mv", resname), + quote_qualified_identifier("diff", resname) + ); + /* count =3D mv.count + diff.count */ + appendStringInfo(buf_new, + ", %s =3D %s", + quote_qualified_identifier(NULL, count_col), + get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL) + ); + } + + appendStringInfo(aggs_list, ", %s, %s", + quote_qualified_identifier("diff", resname), + quote_qualified_identifier("diff", IVM_colname("count", resname)) + ); +} + +/* + * get_operation_string + * + * Build a string to calculate the new aggregate values. + */ +static char * +get_operation_string(IvmOp op, const char *col, const char *arg1, const ch= ar *arg2, + const char* count_col, const char *castType) +{ + StringInfoData buf; + StringInfoData castString; + char *col1 =3D quote_qualified_identifier(arg1, col); + char *col2 =3D quote_qualified_identifier(arg2, col); + char op_char =3D (op =3D=3D IVM_SUB ? '-' : '+'); + + initStringInfo(&buf); + initStringInfo(&castString); + + if (castType) + appendStringInfo(&castString, "::%s", castType); + + if (!count_col) + { + /* + * If the attributes don't have count columns then calc the result + * by using the operator simply. + */ + appendStringInfo(&buf, "(%s OPERATOR(pg_catalog.%c) %s)%s", + col1, op_char, col2, castString.data); + } + else + { + /* + * If the attributes have count columns then consider the condition + * where the result becomes NULL. + */ + char *null_cond =3D get_null_condition_string(op, arg1, arg2, count_col); + + appendStringInfo(&buf, + "(CASE WHEN %s THEN NULL " + "WHEN %s IS NULL THEN %s " + "WHEN %s IS NULL THEN %s " + "ELSE (%s OPERATOR(pg_catalog.%c) %s)%s END)", + null_cond, + col1, col2, + col2, col1, + col1, op_char, col2, castString.data + ); + } + + return buf.data; +} + +/* + * get_null_condition_string + * + * Build a predicate string for CASE clause to check if an aggregate value + * will became NULL after the given operation is applied. + */ +static char * +get_null_condition_string(IvmOp op, const char *arg1, const char *arg2, + const char* count_col) +{ + StringInfoData null_cond; + initStringInfo(&null_cond); + + switch (op) + { + case IVM_ADD: + appendStringInfo(&null_cond, + "%s OPERATOR(pg_catalog.=3D) 0 AND %s OPERATOR(pg_catalog.=3D) 0", + quote_qualified_identifier(arg1, count_col), + quote_qualified_identifier(arg2, count_col) + ); + break; + case IVM_SUB: + appendStringInfo(&null_cond, + "%s OPERATOR(pg_catalog.=3D) %s", + quote_qualified_identifier(arg1, count_col), + quote_qualified_identifier(arg2, count_col) + ); + break; + default: + elog(ERROR,"unknown operation"); + } + + return null_cond.data; +} + + /* * apply_old_delta_with_count * * Execute a query for applying a delta table given by deltname_old * which contains tuples to be deleted from to a materialized view given by * matviewname. This is used when counting is required, that is, the view - * has aggregate or distinct. + * has aggregate or distinct. Also, when a table in EXISTS sub queries + * is modified. + * + * If the view desn't have aggregates or has GROUP BY, this requires a keys + * list to identify a tuple in the view. If the view has aggregates, this + * requires strings representing resnames of aggregates and SET clause for + * updating aggregate values. + * + * If the view has min or max aggregate, this requires a list of resnames = of + * min/max aggregates and a list of boolean which represents which entries= in + * minmax_list is min. These are necessary to check if we need to recalcul= ate + * min or max aggregate values. In this case, this query returns TID and k= eys + * of tuples which need to be recalculated. This result and the number of= rows + * are stored in tuptables and num_recalc repectedly. + * */ static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_= old, - List *keys, const char *count_colname) + List *keys, StringInfo aggs_list, StringInfo aggs_set, + List *minmax_list, List *is_min_list, + const char *count_colname, + SPITupleTable **tuptable_recalc, uint64 *num_recalc) { StringInfoData querybuf; char *match_cond; + char *updt_returning =3D ""; + char *select_for_recalc =3D "SELECT"; + bool agg_without_groupby =3D (list_length(keys) =3D=3D 0); + + Assert(tuptable_recalc !=3D NULL); + Assert(num_recalc !=3D NULL); =20 /* build WHERE condition for searching tuples to be deleted */ match_cond =3D get_matching_condition_string(keys); =20 + /* + * We need a special RETURNING clause and SELECT statement for min/max to + * check which tuple needs re-calculation from base tables. + */ + if (minmax_list) + { + updt_returning =3D get_returning_string(minmax_list, is_min_list, keys); + select_for_recalc =3D get_select_for_recalc_string(keys); + } + /* Search for matching tuples from the view and update or delete if found= . */ initStringInfo(&querybuf); appendStringInfo(&querybuf, "WITH t AS (" /* collecting tid of target tuples in the view */ "SELECT diff.%s, " /* count column */ - "(diff.%s OPERATOR(pg_catalog.=3D) mv.%s) AS for_dlt, " + "(diff.%s OPERATOR(pg_catalog.=3D) mv.%s AND %s) AS for_dlt, " "mv.ctid " + "%s " /* aggregate columns */ "FROM %s AS mv, %s AS diff " "WHERE %s" /* tuple matching condition */ "), updt AS (" /* update a tuple if this is not to be deleted */ "UPDATE %s AS mv SET %s =3D mv.%s OPERATOR(pg_catalog.-) t.%s " + "%s" /* SET clauses for aggregates */ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=3D) t.ctid AND NOT for_dl= t " + "%s" /* RETURNING clause for recalc infomation */ "), dlt AS (" /* delete a tuple if this is to be deleted */ "DELETE FROM %s AS mv USING t " "WHERE mv.ctid OPERATOR(pg_catalog.=3D) t.ctid AND for_dlt" - ")", + ") %s", /* SELECT returning which tuples need to be recalculate= d */ count_colname, - count_colname, count_colname, + count_colname, count_colname, (agg_without_groupby ? "false" : "true"= ), + (aggs_list !=3D NULL ? aggs_list->data : ""), matviewname, deltaname_old, match_cond, matviewname, count_colname, count_colname, count_colname, - matviewname); + (aggs_set !=3D NULL ? aggs_set->data : ""), + updt_returning, + matviewname, + select_for_recalc); =20 - if (SPI_exec(querybuf.data, 0) !=3D SPI_OK_DELETE) + if (SPI_exec(querybuf.data, 0) !=3D SPI_OK_SELECT) elog(ERROR, "SPI_exec failed: %s", querybuf.data); + + + /* Return tuples to be recalculated. */ + if (minmax_list) + { + *tuptable_recalc =3D SPI_tuptable; + *num_recalc =3D SPI_processed; + } + else + { + *tuptable_recalc =3D NULL; + *num_recalc =3D 0; + } } =20 /* @@ -2157,10 +2683,15 @@ apply_old_delta(const char *matviewname, const char= *deltaname_old, * matviewname. This is used when counting is required, that is, the view * has aggregate or distinct. Also, when a table in EXISTS sub queries * is modified. + * + * If the view desn't have aggregates or has GROUP BY, this requires a keys + * list to identify a tuple in the view. If the view has aggregates, this + * requires strings representing SET clause for updating aggregate values. */ static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_= new, - List *keys, StringInfo target_list, const char* count_colname) + List *keys, StringInfo aggs_set, StringInfo target_list, + const char* count_colname) { StringInfoData querybuf; StringInfoData returning_keys; @@ -2191,6 +2722,7 @@ apply_new_delta_with_count(const char *matviewname, c= onst char* deltaname_new, appendStringInfo(&querybuf, "WITH updt AS (" /* update a tuple if this exists in the view */ "UPDATE %s AS mv SET %s =3D mv.%s OPERATOR(pg_catalog.+) diff.%s " + "%s " /* SET clauses for aggregates */ "FROM %s AS diff " "WHERE %s " /* tuple matching condition */ "RETURNING %s" /* returning keys of updated tuples */ @@ -2198,6 +2730,7 @@ apply_new_delta_with_count(const char *matviewname, c= onst char* deltaname_new, "SELECT %s FROM %s AS diff " "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);", matviewname, count_colname, count_colname, count_colname, + (aggs_set !=3D NULL ? aggs_set->data : ""), deltaname_new, match_cond, returning_keys.data, @@ -2272,6 +2805,349 @@ get_matching_condition_string(List *keys) return match_cond.data; } =20 +/* + * get_returning_string + * + * Build a string for RETURNING clause of UPDATE used in apply_old_delta_w= ith_count. + * This clause returns ctid and a boolean value that indicates if we need = to + * recalculate min or max value, for each updated row. + */ +static char * +get_returning_string(List *minmax_list, List *is_min_list, List *keys) +{ + StringInfoData returning; + char *recalc_cond; + ListCell *lc; + + Assert(minmax_list !=3D NIL && is_min_list !=3D NIL); + recalc_cond =3D get_minmax_recalc_condition_string(minmax_list, is_min_li= st); + + initStringInfo(&returning); + + appendStringInfo(&returning, "RETURNING mv.ctid AS tid, (%s) AS recalc", = recalc_cond); + foreach (lc, keys) + { + Form_pg_attribute attr =3D (Form_pg_attribute) lfirst(lc); + char *resname =3D NameStr(attr->attname); + appendStringInfo(&returning, ", %s", quote_qualified_identifier("mv", re= sname)); + } + + return returning.data; +} + +/* + * get_minmax_recalc_condition_string + * + * Build a predicate string for checking if any min/max aggregate + * value needs to be recalculated. + */ +static char * +get_minmax_recalc_condition_string(List *minmax_list, List *is_min_list) +{ + StringInfoData recalc_cond; + ListCell *lc1, *lc2; + + initStringInfo(&recalc_cond); + + Assert (list_length(minmax_list) =3D=3D list_length(is_min_list)); + + forboth (lc1, minmax_list, lc2, is_min_list) + { + char *resname =3D (char *) lfirst(lc1); + bool is_min =3D (bool) lfirst_int(lc2); + char *op_str =3D (is_min ? ">=3D" : "<=3D"); + + appendStringInfo(&recalc_cond, "%s OPERATOR(pg_catalog.%s) %s", + quote_qualified_identifier("mv", resname), + op_str, + quote_qualified_identifier("t", resname) + ); + + if (lnext(minmax_list, lc1)) + appendStringInfo(&recalc_cond, " OR "); + } + + return recalc_cond.data; +} + +/* + * get_select_for_recalc_string + * + * Build a query to return tid and keys of tuples which need + * recalculation. This is used as the result of the query + * built by apply_old_delta. + */ +static char * +get_select_for_recalc_string(List *keys) +{ + StringInfoData qry; + ListCell *lc; + + initStringInfo(&qry); + + appendStringInfo(&qry, "SELECT tid"); + foreach (lc, keys) + { + Form_pg_attribute attr =3D (Form_pg_attribute) lfirst(lc); + appendStringInfo(&qry, ", %s", NameStr(attr->attname)); + } + + appendStringInfo(&qry, " FROM updt WHERE recalc"); + + return qry.data; +} + +/* + * recalc_and_set_values + * + * Recalculate tuples in a materialized from base tables and update these. + * The tuples which needs recalculation are specified by keys, and resnames + * of columns to be updated are specified by namelist. TIDs and key values + * are given by tuples in tuptable_recalc. Its first attribute must be TID + * and key values must be following this. + */ +static void +recalc_and_set_values(SPITupleTable *tuptable_recalc, int64 num_tuples, + List *namelist, List *keys, Relation matviewRel) +{ + TupleDesc tupdesc_recalc =3D tuptable_recalc->tupdesc; + Oid *keyTypes =3D NULL, *types =3D NULL; + char *keyNulls =3D NULL, *nulls =3D NULL; + Datum *keyVals =3D NULL, *vals =3D NULL; + int num_vals =3D list_length(namelist); + int num_keys =3D list_length(keys); + uint64 i; + Oid matviewOid; + char *matviewname; + + matviewOid =3D RelationGetRelid(matviewRel); + matviewname =3D quote_qualified_identifier(get_namespace_name(RelationGet= Namespace(matviewRel)), + RelationGetRelationName(matviewRel)); + + /* If we have keys, initialize arrays for them. */ + if (keys) + { + keyTypes =3D palloc(sizeof(Oid) * num_keys); + keyNulls =3D palloc(sizeof(char) * num_keys); + keyVals =3D palloc(sizeof(Datum) * num_keys); + /* a tuple contains keys to be recalculated and ctid to be updated*/ + Assert(tupdesc_recalc->natts =3D=3D num_keys + 1); + + /* Types of key attributes */ + for (i =3D 0; i < num_keys; i++) + keyTypes[i] =3D TupleDescAttr(tupdesc_recalc, i + 1)->atttypid; + } + + /* allocate memory for all attribute names and tid */ + types =3D palloc(sizeof(Oid) * (num_vals + 1)); + nulls =3D palloc(sizeof(char) * (num_vals + 1)); + vals =3D palloc(sizeof(Datum) * (num_vals + 1)); + + /* For each tuple which needs recalculation */ + for (i =3D 0; i < num_tuples; i++) + { + int j; + bool isnull; + SPIPlanPtr plan; + SPITupleTable *tuptable_newvals; + TupleDesc tupdesc_newvals; + + /* Set group key values as parameters if needed. */ + if (keys) + { + for (j =3D 0; j < num_keys; j++) + { + keyVals[j] =3D SPI_getbinval(tuptable_recalc->vals[i], tupdesc_recalc,= j + 2, &isnull); + if (isnull) + keyNulls[j] =3D 'n'; + else + keyNulls[j] =3D ' '; + } + } + + /* + * Get recalculated values from base tables. The result must be + * only one tuple thich contains the new values for specified keys. + */ + plan =3D get_plan_for_recalc(matviewOid, namelist, keys, keyTypes); + if (SPI_execute_plan(plan, keyVals, keyNulls, false, 0) !=3D SPI_OK_SELE= CT) + elog(ERROR, "SPI_execute_plan"); + if (SPI_processed !=3D 1) + elog(ERROR, "SPI_execute_plan returned zero or more than one rows"); + + tuptable_newvals =3D SPI_tuptable; + tupdesc_newvals =3D tuptable_newvals->tupdesc; + + Assert(tupdesc_newvals->natts =3D=3D num_vals); + + /* Set the new values as parameters */ + for (j =3D 0; j < tupdesc_newvals->natts; j++) + { + if (i =3D=3D 0) + types[j] =3D TupleDescAttr(tupdesc_newvals, j)->atttypid; + + vals[j] =3D SPI_getbinval(tuptable_newvals->vals[0], tupdesc_newvals, j= + 1, &isnull); + if (isnull) + nulls[j] =3D 'n'; + else + nulls[j] =3D ' '; + } + /* Set TID of the view tuple to be updated as a parameter */ + types[j] =3D TIDOID; + vals[j] =3D SPI_getbinval(tuptable_recalc->vals[i], tupdesc_recalc, 1, &= isnull); + nulls[j] =3D ' '; + + /* Update the view tuple to the new values */ + plan =3D get_plan_for_set_values(matviewOid, matviewname, namelist, type= s); + if (SPI_execute_plan(plan, vals, nulls, false, 0) !=3D SPI_OK_UPDATE) + elog(ERROR, "SPI_execute_plan"); + } +} + + +/* + * get_plan_for_recalc + * + * Create or fetch a plan for recalculating value in the view's target list + * from base tables using the definition query of materialized view specif= ied + * by matviewOid. namelist is a list of resnames of values to be recalcula= ted. + * + * keys is a list of keys to identify tuples to be recalculated if this is= not + * empty. KeyTypes is an array of types of keys. + */ +static SPIPlanPtr +get_plan_for_recalc(Oid matviewOid, List *namelist, List *keys, Oid *keyTy= pes) +{ + MV_QueryKey hash_key; + SPIPlanPtr plan; + + /* Fetch or prepare a saved plan for the recalculation */ + mv_BuildQueryKey(&hash_key, matviewOid, MV_PLAN_RECALC); + if ((plan =3D mv_FetchPreparedPlan(&hash_key)) =3D=3D NULL) + { + ListCell *lc; + StringInfoData str; + char *viewdef; + + /* get view definition of matview */ + viewdef =3D text_to_cstring((text *) DatumGetPointer( + DirectFunctionCall1(pg_get_viewdef, ObjectIdGetDatum(matviewOid)))); + /* get rid of trailing semi-colon */ + viewdef[strlen(viewdef)-1] =3D '\0'; + + /* + * Build a query string for recalculating values. This is like + * + * SELECT x1, x2, x3, ... FROM ( ... view definition query ...) mv + * WHERE (key1, key2, ...) =3D ($1, $2, ...); + */ + + initStringInfo(&str); + appendStringInfo(&str, "SELECT "); + foreach (lc, namelist) + { + appendStringInfo(&str, "%s", (char *) lfirst(lc)); + if (lnext(namelist, lc)) + appendStringInfoString(&str, ", "); + } + appendStringInfo(&str, " FROM (%s) mv", viewdef); + + if (keys) + { + int i =3D 1; + char paramname[16]; + + appendStringInfo(&str, " WHERE ("); + foreach (lc, keys) + { + Form_pg_attribute attr =3D (Form_pg_attribute) lfirst(lc); + char *resname =3D NameStr(attr->attname); + Oid typid =3D attr->atttypid; + + sprintf(paramname, "$%d", i); + appendStringInfo(&str, "("); + generate_equal(&str, typid, resname, paramname); + appendStringInfo(&str, " OR (%s IS NULL AND %s IS NULL))", + resname, paramname); + + if (lnext(keys, lc)) + appendStringInfoString(&str, " AND "); + i++; + } + appendStringInfo(&str, ")"); + } + else + keyTypes =3D NULL; + + plan =3D SPI_prepare(str.data, list_length(keys), keyTypes); + if (plan =3D=3D NULL) + elog(ERROR, "SPI_prepare returned %s for %s", SPI_result_code_string(SP= I_result), str.data); + + SPI_keepplan(plan); + mv_HashPreparedPlan(&hash_key, plan); + } + + return plan; +} + +/* + * get_plan_for_set_values + * + * Create or fetch a plan for applying new values calculated by + * get_plan_for_recalc to a materialized view specified by matviewOid. + * matviewname is the name of the view. namelist is a list of resnames + * of attributes to be updated, and valTypes is an array of types of the + * values. + */ +static SPIPlanPtr +get_plan_for_set_values(Oid matviewOid, char *matviewname, List *namelist, + Oid *valTypes) +{ + MV_QueryKey key; + SPIPlanPtr plan; + + /* Fetch or prepare a saved plan for the real check */ + mv_BuildQueryKey(&key, matviewOid, MV_PLAN_SET_VALUE); + if ((plan =3D mv_FetchPreparedPlan(&key)) =3D=3D NULL) + { + ListCell *lc; + StringInfoData str; + int i; + + /* + * Build a query string for applying min/max values. This is like + * + * UPDATE matviewname AS mv + * SET (x1, x2, x3, x4) =3D ($1, $2, $3, $4) + * WHERE ctid =3D $5; + */ + + initStringInfo(&str); + appendStringInfo(&str, "UPDATE %s AS mv SET (", matviewname); + foreach (lc, namelist) + { + appendStringInfo(&str, "%s", (char *) lfirst(lc)); + if (lnext(namelist, lc)) + appendStringInfoString(&str, ", "); + } + appendStringInfo(&str, ") =3D ROW("); + + for (i =3D 1; i <=3D list_length(namelist); i++) + appendStringInfo(&str, "%s$%d", (i=3D=3D1 ? "" : ", "), i); + + appendStringInfo(&str, ") WHERE ctid OPERATOR(pg_catalog.=3D) $%d", i); + + plan =3D SPI_prepare(str.data, list_length(namelist) + 1, valTypes); + if (plan =3D=3D NULL) + elog(ERROR, "SPI_prepare returned %s for %s", SPI_result_code_string(SP= I_result), str.data); + + SPI_keepplan(plan); + mv_HashPreparedPlan(&key, plan); + } + + return plan; +} + /* * generate_equals * @@ -2305,6 +3181,13 @@ mv_InitHashTables(void) { HASHCTL ctl; =20 + memset(&ctl, 0, sizeof(ctl)); + ctl.keysize =3D sizeof(MV_QueryKey); + ctl.entrysize =3D sizeof(MV_QueryHashEntry); + mv_query_cache =3D hash_create("MV query cache", + MV_INIT_QUERYHASHSIZE, + &ctl, HASH_ELEM | HASH_BLOBS); + memset(&ctl, 0, sizeof(ctl)); ctl.keysize =3D sizeof(Oid); ctl.entrysize =3D sizeof(MV_TriggerHashEntry); @@ -2313,6 +3196,99 @@ mv_InitHashTables(void) &ctl, HASH_ELEM | HASH_BLOBS); } =20 +/* + * mv_FetchPreparedPlan + */ +static SPIPlanPtr +mv_FetchPreparedPlan(MV_QueryKey *key) +{ + MV_QueryHashEntry *entry; + SPIPlanPtr plan; + + /* + * On the first call initialize the hashtable + */ + if (!mv_query_cache) + mv_InitHashTables(); + + /* + * Lookup for the key + */ + entry =3D (MV_QueryHashEntry *) hash_search(mv_query_cache, + (void *) key, + HASH_FIND, NULL); + if (entry =3D=3D NULL) + return NULL; + + /* + * Check whether the plan is still valid. If it isn't, we don't want to + * simply rely on plancache.c to regenerate it; rather we should start + * from scratch and rebuild the query text too. This is to cover cases + * such as table/column renames. We depend on the plancache machinery to + * detect possible invalidations, though. + * + * CAUTION: this check is only trustworthy if the caller has already + * locked both materialized views and base tables. + */ + plan =3D entry->plan; + if (plan && SPI_plan_is_valid(plan)) + return plan; + + /* + * Otherwise we might as well flush the cached plan now, to free a little + * memory space before we make a new one. + */ + entry->plan =3D NULL; + if (plan) + SPI_freeplan(plan); + + return NULL; +} + +/* + * mv_HashPreparedPlan + * + * Add another plan to our private SPI query plan hashtable. + */ +static void +mv_HashPreparedPlan(MV_QueryKey *key, SPIPlanPtr plan) +{ + MV_QueryHashEntry *entry; + bool found; + + /* + * On the first call initialize the hashtable + */ + if (!mv_query_cache) + mv_InitHashTables(); + + /* + * Add the new plan. We might be overwriting an entry previously found + * invalid by mv_FetchPreparedPlan. + */ + entry =3D (MV_QueryHashEntry *) hash_search(mv_query_cache, + (void *) key, + HASH_ENTER, &found); + Assert(!found || entry->plan =3D=3D NULL); + entry->plan =3D plan; +} + +/* + * mv_BuildQueryKey + * + * Construct a hashtable key for a prepared SPI plan for IVM. + */ +static void +mv_BuildQueryKey(MV_QueryKey *key, Oid matview_id, int32 query_type) +{ + /* + * We assume struct MV_QueryKey contains no padding bytes, else we'd need + * to use memset to clear them. + */ + key->matview_id =3D matview_id; + key->query_type =3D query_type; +} + /* * AtAbort_IVM * diff --git a/src/include/commands/createas.h b/src/include/commands/createa= s.h index a57ce463e1..702b097079 100644 --- a/src/include/commands/createas.h +++ b/src/include/commands/createas.h @@ -29,6 +29,7 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate= , CreateTableAsStmt *st extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid, bool= is_create); =20 extern Query *rewriteQueryForIMMV(Query *query, List *colNames); +extern void makeIvmAggColumn(ParseState *pstate, Aggref *aggref, char *res= name, AttrNumber *next_resno, List **aggs); =20 extern int GetIntoRelEFlags(IntoClause *intoClause); =20 --=20 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0009-Add-regression-tests-for-Incremental-View-Mainte.patch" Content-Disposition: attachment; filename="v23-0009-Add-regression-tests-for-Incremental-View-Mainte.patch" Content-Transfer-Encoding: 7bit From 59ef91e832577adc23d679f397b5921d2163357b Mon Sep 17 00:00:00 2001 From: Takuma Hoshiai Date: Wed, 10 Mar 2021 11:11:13 +0900 Subject: [PATCH v23 09/15] Add regression tests for Incremental View Maintenance --- .../regress/expected/incremental_matview.out | 799 ++++++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/sql/incremental_matview.sql | 400 +++++++++ 3 files changed, 1200 insertions(+), 1 deletion(-) create mode 100644 src/test/regress/expected/incremental_matview.out create mode 100644 src/test/regress/sql/incremental_matview.sql diff --git a/src/test/regress/expected/incremental_matview.out b/src/test/regress/expected/incremental_matview.out new file mode 100644 index 0000000000..fe5bc734e2 --- /dev/null +++ b/src/test/regress/expected/incremental_matview.out @@ -0,0 +1,799 @@ +-- create a table to use as a basis for views and materialized views in various combinations +CREATE TABLE mv_base_a (i int, j int); +INSERT INTO mv_base_a VALUES + (1,10), + (2,20), + (3,30), + (4,40), + (5,50); +CREATE TABLE mv_base_b (i int, k int); +INSERT INTO mv_base_b VALUES + (1,101), + (2,102), + (3,103), + (4,104); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i) WITH NO DATA; +NOTICE: could not create an index on materialized view "mv_ivm_1" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; +ERROR: materialized view "mv_ivm_1" has not been populated +HINT: Use the REFRESH MATERIALIZED VIEW command. +REFRESH MATERIALIZED VIEW mv_ivm_1; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; + i | j | k +---+----+----- + 1 | 10 | 101 + 2 | 20 | 102 + 3 | 30 | 103 + 4 | 40 | 104 +(4 rows) + +-- immediate maintenance +BEGIN; +INSERT INTO mv_base_b VALUES(5,105); +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; + i | j | k +---+----+----- + 1 | 10 | 101 + 2 | 20 | 102 + 3 | 30 | 103 + 4 | 40 | 104 + 5 | 50 | 105 +(5 rows) + +UPDATE mv_base_a SET j = 0 WHERE i = 1; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; + i | j | k +---+----+----- + 1 | 0 | 101 + 2 | 20 | 102 + 3 | 30 | 103 + 4 | 40 | 104 + 5 | 50 | 105 +(5 rows) + +DELETE FROM mv_base_b WHERE (i,k) = (5,105); +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; + i | j | k +---+----+----- + 1 | 0 | 101 + 2 | 20 | 102 + 3 | 30 | 103 + 4 | 40 | 104 +(4 rows) + +ROLLBACK; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; + i | j | k +---+----+----- + 1 | 10 | 101 + 2 | 20 | 102 + 3 | 30 | 103 + 4 | 40 | 104 +(4 rows) + +-- rename of IVM columns +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_rename AS SELECT DISTINCT * FROM mv_base_a; +NOTICE: created index "mv_ivm_rename_index" on materialized view "mv_ivm_rename" +ALTER MATERIALIZED VIEW mv_ivm_rename RENAME COLUMN __ivm_count__ TO xxx; +ERROR: IVM column can not be renamed +DROP MATERIALIZED VIEW mv_ivm_rename; +-- unique index on IVM columns +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_unique AS SELECT DISTINCT * FROM mv_base_a; +NOTICE: created index "mv_ivm_unique_index" on materialized view "mv_ivm_unique" +CREATE UNIQUE INDEX ON mv_ivm_unique(__ivm_count__); +ERROR: unique index creation on IVM columns is not supported +CREATE UNIQUE INDEX ON mv_ivm_unique((__ivm_count__)); +ERROR: unique index creation on IVM columns is not supported +CREATE UNIQUE INDEX ON mv_ivm_unique((__ivm_count__ + 1)); +ERROR: unique index creation on IVM columns is not supported +DROP MATERIALIZED VIEW mv_ivm_unique; +-- some query syntax +BEGIN; +CREATE FUNCTION ivm_func() RETURNS int LANGUAGE 'sql' + AS 'SELECT 1' IMMUTABLE; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_func AS SELECT * FROM ivm_func(); +NOTICE: could not create an index on materialized view "mv_ivm_func" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_no_tbl AS SELECT 1; +NOTICE: could not create an index on materialized view "mv_ivm_no_tbl" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +ROLLBACK; +-- result of materialized view have DISTINCT clause or the duplicate result. +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a; +NOTICE: could not create an index on materialized view "mv_ivm_duplicate" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a; +NOTICE: created index "mv_ivm_distinct_index" on materialized view "mv_ivm_distinct" +INSERT INTO mv_base_a VALUES(6,20); +SELECT * FROM mv_ivm_duplicate ORDER BY 1; + j +---- + 10 + 20 + 20 + 30 + 40 + 50 +(6 rows) + +SELECT * FROM mv_ivm_distinct ORDER BY 1; + j +---- + 10 + 20 + 30 + 40 + 50 +(5 rows) + +DELETE FROM mv_base_a WHERE (i,j) = (2,20); +SELECT * FROM mv_ivm_duplicate ORDER BY 1; + j +---- + 10 + 20 + 30 + 40 + 50 +(5 rows) + +SELECT * FROM mv_ivm_distinct ORDER BY 1; + j +---- + 10 + 20 + 30 + 40 + 50 +(5 rows) + +ROLLBACK; +-- support SUM(), COUNT() and AVG() aggregate functions +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i), AVG(j) FROM mv_base_a GROUP BY i; +NOTICE: created index "mv_ivm_agg_index" on materialized view "mv_ivm_agg" +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; + i | sum | count | avg +---+-----+-------+--------------------- + 1 | 10 | 1 | 10.0000000000000000 + 2 | 20 | 1 | 20.0000000000000000 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +INSERT INTO mv_base_a VALUES(2,100); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; + i | sum | count | avg +---+-----+-------+--------------------- + 1 | 10 | 1 | 10.0000000000000000 + 2 | 120 | 2 | 60.0000000000000000 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; + i | sum | count | avg +---+-----+-------+---------------------- + 1 | 10 | 1 | 10.0000000000000000 + 2 | 220 | 2 | 110.0000000000000000 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +DELETE FROM mv_base_a WHERE (i,j) = (2,200); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; + i | sum | count | avg +---+-----+-------+--------------------- + 1 | 10 | 1 | 10.0000000000000000 + 2 | 20 | 1 | 20.0000000000000000 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +ROLLBACK; +-- support COUNT(*) aggregate function +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(*) FROM mv_base_a GROUP BY i; +NOTICE: created index "mv_ivm_agg_index" on materialized view "mv_ivm_agg" +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3; + i | sum | count +---+-----+------- + 1 | 10 | 1 + 2 | 20 | 1 + 3 | 30 | 1 + 4 | 40 | 1 + 5 | 50 | 1 +(5 rows) + +INSERT INTO mv_base_a VALUES(2,100); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3; + i | sum | count +---+-----+------- + 1 | 10 | 1 + 2 | 120 | 2 + 3 | 30 | 1 + 4 | 40 | 1 + 5 | 50 | 1 +(5 rows) + +ROLLBACK; +-- support aggregate functions without GROUP clause +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j), COUNT(j), AVG(j) FROM mv_base_a; +NOTICE: could not create an index on materialized view "mv_ivm_group" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv_ivm_group ORDER BY 1; + sum | count | avg +-----+-------+--------------------- + 150 | 5 | 30.0000000000000000 +(1 row) + +INSERT INTO mv_base_a VALUES(6,60); +SELECT * FROM mv_ivm_group ORDER BY 1; + sum | count | avg +-----+-------+--------------------- + 210 | 6 | 35.0000000000000000 +(1 row) + +DELETE FROM mv_base_a; +SELECT * FROM mv_ivm_group ORDER BY 1; + sum | count | avg +-----+-------+----- + | 0 | +(1 row) + +ROLLBACK; +-- resolved issue: When use AVG() function and values is indivisible, result of AVG() is incorrect. +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i; +NOTICE: created index "mv_ivm_avg_bug_index" on materialized view "mv_ivm_avg_bug" +SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3; + i | sum | count | avg +---+-----+-------+--------------------- + 1 | 10 | 1 | 10.0000000000000000 + 2 | 20 | 1 | 20.0000000000000000 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +INSERT INTO mv_base_a VALUES + (1,0), + (1,0), + (2,30), + (2,30); +SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3; + i | sum | count | avg +---+-----+-------+--------------------- + 1 | 10 | 3 | 3.3333333333333333 + 2 | 80 | 3 | 26.6666666666666667 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +DELETE FROM mv_base_a WHERE (i,j) = (1,0); +DELETE FROM mv_base_a WHERE (i,j) = (2,30); +SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3; + i | sum | count | avg +---+-----+-------+--------------------- + 1 | 10 | 1 | 10.0000000000000000 + 2 | 20 | 1 | 20.0000000000000000 + 3 | 30 | 1 | 30.0000000000000000 + 4 | 40 | 1 | 40.0000000000000000 + 5 | 50 | 1 | 50.0000000000000000 +(5 rows) + +ROLLBACK; +-- support MIN(), MAX() aggregate functions +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT i, MIN(j), MAX(j) FROM mv_base_a GROUP BY i; +NOTICE: created index "mv_ivm_min_max_index" on materialized view "mv_ivm_min_max" +SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3; + i | min | max +---+-----+----- + 1 | 10 | 10 + 2 | 20 | 20 + 3 | 30 | 30 + 4 | 40 | 40 + 5 | 50 | 50 +(5 rows) + +INSERT INTO mv_base_a VALUES + (1,11), (1,12), + (2,21), (2,22), + (3,31), (3,32), + (4,41), (4,42), + (5,51), (5,52); +SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3; + i | min | max +---+-----+----- + 1 | 10 | 12 + 2 | 20 | 22 + 3 | 30 | 32 + 4 | 40 | 42 + 5 | 50 | 52 +(5 rows) + +DELETE FROM mv_base_a WHERE (i,j) IN ((1,10), (2,21), (3,32)); +SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3; + i | min | max +---+-----+----- + 1 | 11 | 12 + 2 | 20 | 22 + 3 | 30 | 31 + 4 | 40 | 42 + 5 | 50 | 52 +(5 rows) + +ROLLBACK; +-- support MIN(), MAX() aggregate functions without GROUP clause +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT MIN(j), MAX(j) FROM mv_base_a; +NOTICE: could not create an index on materialized view "mv_ivm_min_max" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv_ivm_min_max; + min | max +-----+----- + 10 | 50 +(1 row) + +INSERT INTO mv_base_a VALUES + (0,0), (6,60), (7,70); +SELECT * FROM mv_ivm_min_max; + min | max +-----+----- + 0 | 70 +(1 row) + +DELETE FROM mv_base_a WHERE (i,j) IN ((0,0), (7,70)); +SELECT * FROM mv_ivm_min_max; + min | max +-----+----- + 10 | 60 +(1 row) + +DELETE FROM mv_base_a; +SELECT * FROM mv_ivm_min_max; + min | max +-----+----- + | +(1 row) + +ROLLBACK; +-- support self join view and multiple change on the same table +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (1, 10), (2, 20), (3, 30); +CREATE INCREMENTAL MATERIALIZED VIEW mv_self(v1, v2) AS + SELECT t1.v, t2.v FROM base_t AS t1 JOIN base_t AS t2 ON t1.i = t2.i; +NOTICE: could not create an index on materialized view "mv_self" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv_self ORDER BY v1; + v1 | v2 +----+---- + 10 | 10 + 20 | 20 + 30 | 30 +(3 rows) + +INSERT INTO base_t VALUES (4,40); +DELETE FROM base_t WHERE i = 1; +UPDATE base_t SET v = v*10 WHERE i=2; +SELECT * FROM mv_self ORDER BY v1; + v1 | v2 +-----+----- + 30 | 30 + 40 | 40 + 200 | 200 +(3 rows) + +WITH + ins_t1 AS (INSERT INTO base_t VALUES (5,50) RETURNING 1), + ins_t2 AS (INSERT INTO base_t VALUES (6,60) RETURNING 1), + upd_t AS (UPDATE base_t SET v = v + 100 RETURNING 1), + dlt_t AS (DELETE FROM base_t WHERE i IN (4,5) RETURNING 1) +SELECT NULL; + ?column? +---------- + +(1 row) + +SELECT * FROM mv_self ORDER BY v1; + v1 | v2 +-----+----- + 50 | 50 + 60 | 60 + 130 | 130 + 300 | 300 +(4 rows) + +ROLLBACK; +-- support simultaneous table changes +BEGIN; +CREATE TABLE base_r (i int, v int); +CREATE TABLE base_s (i int, v int); +INSERT INTO base_r VALUES (1, 10), (2, 20), (3, 30); +INSERT INTO base_s VALUES (1, 100), (2, 200), (3, 300); +CREATE INCREMENTAL MATERIALIZED VIEW mv(v1, v2) AS + SELECT r.v, s.v FROM base_r AS r JOIN base_s AS s USING(i); +NOTICE: could not create an index on materialized view "mv" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv ORDER BY v1; + v1 | v2 +----+----- + 10 | 100 + 20 | 200 + 30 | 300 +(3 rows) + +WITH + ins_r AS (INSERT INTO base_r VALUES (1,11) RETURNING 1), + ins_r2 AS (INSERT INTO base_r VALUES (3,33) RETURNING 1), + ins_s AS (INSERT INTO base_s VALUES (2,222) RETURNING 1), + upd_r AS (UPDATE base_r SET v = v + 1000 WHERE i = 2 RETURNING 1), + dlt_s AS (DELETE FROM base_s WHERE i = 3 RETURNING 1) +SELECT NULL; + ?column? +---------- + +(1 row) + +SELECT * FROM mv ORDER BY v1; + v1 | v2 +------+----- + 10 | 100 + 11 | 100 + 1020 | 200 + 1020 | 222 +(4 rows) + +ROLLBACK; +-- support foreign reference constraints +BEGIN; +CREATE TABLE ri1 (i int PRIMARY KEY); +CREATE TABLE ri2 (i int PRIMARY KEY REFERENCES ri1(i) ON UPDATE CASCADE ON DELETE CASCADE, v int); +INSERT INTO ri1 VALUES (1),(2),(3); +INSERT INTO ri2 VALUES (1),(2),(3); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ri(i1, i2) AS + SELECT ri1.i, ri2.i FROM ri1 JOIN ri2 USING(i); +NOTICE: created index "mv_ri_index" on materialized view "mv_ri" +SELECT * FROM mv_ri ORDER BY i1; + i1 | i2 +----+---- + 1 | 1 + 2 | 2 + 3 | 3 +(3 rows) + +UPDATE ri1 SET i=10 where i=1; +DELETE FROM ri1 WHERE i=2; +SELECT * FROM mv_ri ORDER BY i2; + i1 | i2 +----+---- + 3 | 3 + 10 | 10 +(2 rows) + +ROLLBACK; +-- views including NULL +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (1,10),(2, NULL); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT * FROM base_t; +NOTICE: could not create an index on materialized view "mv" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv ORDER BY i; + i | v +---+---- + 1 | 10 + 2 | +(2 rows) + +UPDATE base_t SET v = 20 WHERE i = 2; +SELECT * FROM mv ORDER BY i; + i | v +---+---- + 1 | 10 + 2 | 20 +(2 rows) + +ROLLBACK; +BEGIN; +CREATE TABLE base_t (i int); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT * FROM base_t; +NOTICE: could not create an index on materialized view "mv" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT * FROM mv ORDER BY i; + i +--- +(0 rows) + +INSERT INTO base_t VALUES (1),(NULL); +SELECT * FROM mv ORDER BY i; + i +--- + 1 + +(2 rows) + +ROLLBACK; +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (NULL, 1), (NULL, 2), (1, 10), (1, 20); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT i, sum(v) FROM base_t GROUP BY i; +NOTICE: created index "mv_index" on materialized view "mv" +SELECT * FROM mv ORDER BY i; + i | sum +---+----- + 1 | 30 + | 3 +(2 rows) + +UPDATE base_t SET v = v * 10; +SELECT * FROM mv ORDER BY i; + i | sum +---+----- + 1 | 300 + | 30 +(2 rows) + +ROLLBACK; +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (NULL, 1), (NULL, 2), (NULL, 3), (NULL, 4), (NULL, 5); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT i, min(v), max(v) FROM base_t GROUP BY i; +NOTICE: created index "mv_index" on materialized view "mv" +SELECT * FROM mv ORDER BY i; + i | min | max +---+-----+----- + | 1 | 5 +(1 row) + +DELETE FROM base_t WHERE v = 1; +SELECT * FROM mv ORDER BY i; + i | min | max +---+-----+----- + | 2 | 5 +(1 row) + +DELETE FROM base_t WHERE v = 3; +SELECT * FROM mv ORDER BY i; + i | min | max +---+-----+----- + | 2 | 5 +(1 row) + +DELETE FROM base_t WHERE v = 5; +SELECT * FROM mv ORDER BY i; + i | min | max +---+-----+----- + | 2 | 4 +(1 row) + +ROLLBACK; +-- IMMV containing user defined type +BEGIN; +CREATE TYPE mytype; +CREATE FUNCTION mytype_in(cstring) + RETURNS mytype AS 'int4in' + LANGUAGE INTERNAL STRICT IMMUTABLE; +NOTICE: return type mytype is only a shell +CREATE FUNCTION mytype_out(mytype) + RETURNS cstring AS 'int4out' + LANGUAGE INTERNAL STRICT IMMUTABLE; +NOTICE: argument type mytype is only a shell +CREATE TYPE mytype ( + LIKE = int4, + INPUT = mytype_in, + OUTPUT = mytype_out +); +CREATE FUNCTION mytype_eq(mytype, mytype) + RETURNS bool AS 'int4eq' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE FUNCTION mytype_lt(mytype, mytype) + RETURNS bool AS 'int4lt' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE FUNCTION mytype_cmp(mytype, mytype) + RETURNS integer AS 'btint4cmp' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE OPERATOR = ( + leftarg = mytype, rightarg = mytype, + procedure = mytype_eq); +CREATE OPERATOR < ( + leftarg = mytype, rightarg = mytype, + procedure = mytype_lt); +CREATE OPERATOR CLASS mytype_ops + DEFAULT FOR TYPE mytype USING btree AS + OPERATOR 1 <, + OPERATOR 3 = , + FUNCTION 1 mytype_cmp(mytype,mytype); +CREATE TABLE t_mytype (x mytype); +CREATE INCREMENTAL MATERIALIZED VIEW mv_mytype AS + SELECT * FROM t_mytype; +NOTICE: could not create an index on materialized view "mv_mytype" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +INSERT INTO t_mytype VALUES ('1'::mytype); +SELECT * FROM mv_mytype; + x +--- + 1 +(1 row) + +ROLLBACK; +-- outer join is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv(a,b) AS SELECT a.i, b.i FROM mv_base_a a LEFT JOIN mv_base_b b ON a.i=b.i; +ERROR: OUTER JOIN is not supported on incrementally maintainable materialized view +-- CTE is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv AS + WITH b AS ( SELECT * FROM mv_base_b) SELECT a.i,a.j FROM mv_base_a a, b WHERE a.i = b.i; +ERROR: CTE is not supported on incrementally maintainable materialized view +-- contain system column +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm01 AS SELECT i,j,xmin FROM mv_base_a; +ERROR: system column is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610'; +ERROR: system column is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT i,j,xmin::text AS x_min FROM mv_base_a; +ERROR: system column is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm06 AS SELECT i,j,xidsend(xmin) AS x_min FROM mv_base_a; +ERROR: system column is not supported on incrementally maintainable materialized view +-- contain subquery +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 ); +ERROR: subquery is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT a.i,a.j FROM mv_base_a a, (SELECT * FROM mv_base_b) b WHERE a.i = b.i; +ERROR: subquery is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a; +ERROR: subquery is not supported on incrementally maintainable materialized view +-- contain ORDER BY +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i) ORDER BY i,j,k; +ERROR: ORDER BY clause is not supported on incrementally maintainable materialized view +-- contain HAVING +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i) GROUP BY i,j,k HAVING SUM(i) > 5; +ERROR: HAVING clause is not supported on incrementally maintainable materialized view +-- contain view or materialized view +CREATE VIEW b_view AS SELECT i,k FROM mv_base_b; +CREATE MATERIALIZED VIEW b_mview AS SELECT i,k FROM mv_base_b; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT a.i,a.j FROM mv_base_a a,b_view b WHERE a.i = b.i; +ERROR: VIEW or MATERIALIZED VIEW is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT a.i,a.j FROM mv_base_a a,b_mview b WHERE a.i = b.i; +ERROR: VIEW or MATERIALIZED VIEW is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm09 AS SELECT a.i,a.j FROM mv_base_a a, (SELECT i, COUNT(*) FROM mv_base_b GROUP BY i) b WHERE a.i = b.i; +ERROR: subquery is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm10 AS SELECT a.i,a.j FROM mv_base_a a WHERE EXISTS(SELECT 1 FROM mv_base_b b WHERE a.i = b.i) OR a.i > 5; +ERROR: subquery is not supported on incrementally maintainable materialized view +-- contain mutable functions +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm12 AS SELECT i,j FROM mv_base_a WHERE i = random()::int; +ERROR: mutable function is not supported on incrementally maintainable materialized view +HINT: functions must be marked IMMUTABLE +-- LIMIT/OFFSET is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm13 AS SELECT i,j FROM mv_base_a LIMIT 10 OFFSET 5; +ERROR: LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view +-- DISTINCT ON is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm14 AS SELECT DISTINCT ON(i) i, j FROM mv_base_a; +ERROR: DISTINCT ON is not supported on incrementally maintainable materialized view +-- TABLESAMPLE clause is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm15 AS SELECT i, j FROM mv_base_a TABLESAMPLE SYSTEM(50); +ERROR: TABLESAMPLE clause is not supported on incrementally maintainable materialized view +-- window functions are not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm16 AS SELECT *, cume_dist() OVER (ORDER BY i) AS rank FROM mv_base_a; +ERROR: window functions are not supported on incrementally maintainable materialized view +-- aggregate function with some options is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm17 AS SELECT COUNT(*) FILTER(WHERE i < 3) FROM mv_base_a; +ERROR: aggregate function with FILTER clause is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm18 AS SELECT COUNT(DISTINCT i) FROM mv_base_a; +ERROR: aggregate function with DISTINCT arguments is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm19 AS SELECT array_agg(j ORDER BY i DESC) FROM mv_base_a; +ERROR: aggregate function with ORDER clause is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm20 AS SELECT i,SUM(j) FROM mv_base_a GROUP BY GROUPING SETS((i),()); +ERROR: GROUPING SETS, ROLLUP, or CUBE clauses is not supported on incrementally maintainable materialized view +-- inheritance parent is not supported +BEGIN; +CREATE TABLE parent (i int, v int); +CREATE TABLE child_a(options text) INHERITS(parent); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm21 AS SELECT * FROM parent; +ERROR: inheritance parent is not supported on incrementally maintainable materialized view +ROLLBACK; +-- UNION statement is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm22 AS SELECT i,j FROM mv_base_a UNION ALL SELECT i,k FROM mv_base_b;; +ERROR: UNION/INTERSECT/EXCEPT statements are not supported on incrementally maintainable materialized view +-- empty target list is not allowed with IVM +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm25 AS SELECT FROM mv_base_a; +ERROR: empty target list is not supported on incrementally maintainable materialized view +-- FOR UPDATE/SHARE is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm26 AS SELECT i,j FROM mv_base_a FOR UPDATE; +ERROR: FOR UPDATE/SHARE clause is not supported on incrementally maintainable materialized view +-- tartget list cannot contain ivm column that start with '__ivm' +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm28 AS SELECT i AS "__ivm_count__" FROM mv_base_a; +ERROR: column name __ivm_count__ is not supported on incrementally maintainable materialized view +-- expressions specified in GROUP BY must appear in the target list. +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm29 AS SELECT COUNT(i) FROM mv_base_a GROUP BY i; +ERROR: GROUP BY expression not appeared in select list is not supported on incrementally maintainable materialized view +-- experssions containing an aggregate is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm30 AS SELECT sum(i)*0.5 FROM mv_base_a; +ERROR: expression containing an aggregate in it is not supported on incrementally maintainable materialized view +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm31 AS SELECT sum(i)/sum(j) FROM mv_base_a; +ERROR: expression containing an aggregate in it is not supported on incrementally maintainable materialized view +-- VALUES is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_only_values1 AS values(1); +ERROR: VALUES is not supported on incrementally maintainable materialized view +-- base table which has row level security +DROP USER IF EXISTS ivm_admin; +NOTICE: role "ivm_admin" does not exist, skipping +DROP USER IF EXISTS ivm_user; +NOTICE: role "ivm_user" does not exist, skipping +CREATE USER ivm_admin; +CREATE USER ivm_user; +SET SESSION AUTHORIZATION ivm_admin; +CREATE TABLE rls_tbl(id int, data text, owner name); +INSERT INTO rls_tbl VALUES + (1,'foo','ivm_user'), + (2,'bar','postgres'); +CREATE TABLE num_tbl(id int, num text); +INSERT INTO num_tbl VALUES + (1,'one'), + (2,'two'), + (3,'three'), + (4,'four'); +CREATE POLICY rls_tbl_policy ON rls_tbl FOR SELECT TO PUBLIC USING(owner = current_user); +CREATE POLICY rls_tbl_policy2 ON rls_tbl FOR INSERT TO PUBLIC WITH CHECK(current_user LIKE 'ivm_%'); +ALTER TABLE rls_tbl ENABLE ROW LEVEL SECURITY; +GRANT ALL on rls_tbl TO PUBLIC; +GRANT ALL on num_tbl TO PUBLIC; +SET SESSION AUTHORIZATION ivm_user; +CREATE INCREMENTAL MATERIALIZED VIEW ivm_rls AS SELECT * FROM rls_tbl; +NOTICE: could not create an index on materialized view "ivm_rls" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +SELECT id, data, owner FROM ivm_rls ORDER BY 1,2,3; + id | data | owner +----+------+---------- + 1 | foo | ivm_user +(1 row) + +INSERT INTO rls_tbl VALUES + (3,'baz','ivm_user'), + (4,'qux','postgres'); +SELECT id, data, owner FROM ivm_rls ORDER BY 1,2,3; + id | data | owner +----+------+---------- + 1 | foo | ivm_user + 3 | baz | ivm_user +(2 rows) + +CREATE INCREMENTAL MATERIALIZED VIEW ivm_rls2 AS SELECT * FROM rls_tbl JOIN num_tbl USING(id); +NOTICE: could not create an index on materialized view "ivm_rls2" automatically +HINT: Create an index on the materialized view for efficient incremental maintenance. +RESET SESSION AUTHORIZATION; +WITH + x AS (UPDATE rls_tbl SET data = data || '_2' where id in (3,4)), + y AS (UPDATE num_tbl SET num = num || '_2' where id in (3,4)) +SELECT; +-- +(1 row) + +SELECT * FROM ivm_rls2 ORDER BY 1,2,3; + id | data | owner | num +----+-------+----------+--------- + 1 | foo | ivm_user | one + 3 | baz_2 | ivm_user | three_2 +(2 rows) + +DROP TABLE rls_tbl CASCADE; +NOTICE: drop cascades to 2 other objects +DETAIL: drop cascades to materialized view ivm_rls +drop cascades to materialized view ivm_rls2 +DROP TABLE num_tbl CASCADE; +DROP USER ivm_user; +DROP USER ivm_admin; +DROP TABLE mv_base_b CASCADE; +NOTICE: drop cascades to 3 other objects +DETAIL: drop cascades to materialized view mv_ivm_1 +drop cascades to view b_view +drop cascades to materialized view b_mview +DROP TABLE mv_base_a CASCADE; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 7be89178f0..b0350f47dd 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -86,7 +86,7 @@ test: brin_bloom brin_multi # ---------- # Another group of parallel tests # ---------- -test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tid tidscan tidrangescan collate.icu.utf8 incremental_sort +test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tid tidscan tidrangescan collate.icu.utf8 incremental_sort incremental_matview # rules cannot run concurrently with any test that creates # a view or rule in the public schema diff --git a/src/test/regress/sql/incremental_matview.sql b/src/test/regress/sql/incremental_matview.sql new file mode 100644 index 0000000000..311e9b96fb --- /dev/null +++ b/src/test/regress/sql/incremental_matview.sql @@ -0,0 +1,400 @@ +-- create a table to use as a basis for views and materialized views in various combinations +CREATE TABLE mv_base_a (i int, j int); +INSERT INTO mv_base_a VALUES + (1,10), + (2,20), + (3,30), + (4,40), + (5,50); +CREATE TABLE mv_base_b (i int, k int); +INSERT INTO mv_base_b VALUES + (1,101), + (2,102), + (3,103), + (4,104); + +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i) WITH NO DATA; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; +REFRESH MATERIALIZED VIEW mv_ivm_1; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; +-- immediate maintenance +BEGIN; +INSERT INTO mv_base_b VALUES(5,105); +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; +UPDATE mv_base_a SET j = 0 WHERE i = 1; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; +DELETE FROM mv_base_b WHERE (i,k) = (5,105); +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; +ROLLBACK; +SELECT * FROM mv_ivm_1 ORDER BY 1,2,3; + +-- rename of IVM columns +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_rename AS SELECT DISTINCT * FROM mv_base_a; +ALTER MATERIALIZED VIEW mv_ivm_rename RENAME COLUMN __ivm_count__ TO xxx; +DROP MATERIALIZED VIEW mv_ivm_rename; + +-- unique index on IVM columns +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_unique AS SELECT DISTINCT * FROM mv_base_a; +CREATE UNIQUE INDEX ON mv_ivm_unique(__ivm_count__); +CREATE UNIQUE INDEX ON mv_ivm_unique((__ivm_count__)); +CREATE UNIQUE INDEX ON mv_ivm_unique((__ivm_count__ + 1)); +DROP MATERIALIZED VIEW mv_ivm_unique; + +-- some query syntax +BEGIN; +CREATE FUNCTION ivm_func() RETURNS int LANGUAGE 'sql' + AS 'SELECT 1' IMMUTABLE; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_func AS SELECT * FROM ivm_func(); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_no_tbl AS SELECT 1; +ROLLBACK; + +-- result of materialized view have DISTINCT clause or the duplicate result. +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a; +INSERT INTO mv_base_a VALUES(6,20); +SELECT * FROM mv_ivm_duplicate ORDER BY 1; +SELECT * FROM mv_ivm_distinct ORDER BY 1; +DELETE FROM mv_base_a WHERE (i,j) = (2,20); +SELECT * FROM mv_ivm_duplicate ORDER BY 1; +SELECT * FROM mv_ivm_distinct ORDER BY 1; +ROLLBACK; + +-- support SUM(), COUNT() and AVG() aggregate functions +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i), AVG(j) FROM mv_base_a GROUP BY i; +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; +INSERT INTO mv_base_a VALUES(2,100); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; +UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; +DELETE FROM mv_base_a WHERE (i,j) = (2,200); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4; +ROLLBACK; + +-- support COUNT(*) aggregate function +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(*) FROM mv_base_a GROUP BY i; +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3; +INSERT INTO mv_base_a VALUES(2,100); +SELECT * FROM mv_ivm_agg ORDER BY 1,2,3; +ROLLBACK; + +-- support aggregate functions without GROUP clause +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j), COUNT(j), AVG(j) FROM mv_base_a; +SELECT * FROM mv_ivm_group ORDER BY 1; +INSERT INTO mv_base_a VALUES(6,60); +SELECT * FROM mv_ivm_group ORDER BY 1; +DELETE FROM mv_base_a; +SELECT * FROM mv_ivm_group ORDER BY 1; +ROLLBACK; + +-- resolved issue: When use AVG() function and values is indivisible, result of AVG() is incorrect. +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i; +SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3; +INSERT INTO mv_base_a VALUES + (1,0), + (1,0), + (2,30), + (2,30); +SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3; +DELETE FROM mv_base_a WHERE (i,j) = (1,0); +DELETE FROM mv_base_a WHERE (i,j) = (2,30); +SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3; +ROLLBACK; + +-- support MIN(), MAX() aggregate functions +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT i, MIN(j), MAX(j) FROM mv_base_a GROUP BY i; +SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3; +INSERT INTO mv_base_a VALUES + (1,11), (1,12), + (2,21), (2,22), + (3,31), (3,32), + (4,41), (4,42), + (5,51), (5,52); +SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3; +DELETE FROM mv_base_a WHERE (i,j) IN ((1,10), (2,21), (3,32)); +SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3; +ROLLBACK; + +-- support MIN(), MAX() aggregate functions without GROUP clause +BEGIN; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT MIN(j), MAX(j) FROM mv_base_a; +SELECT * FROM mv_ivm_min_max; +INSERT INTO mv_base_a VALUES + (0,0), (6,60), (7,70); +SELECT * FROM mv_ivm_min_max; +DELETE FROM mv_base_a WHERE (i,j) IN ((0,0), (7,70)); +SELECT * FROM mv_ivm_min_max; +DELETE FROM mv_base_a; +SELECT * FROM mv_ivm_min_max; +ROLLBACK; + +-- support self join view and multiple change on the same table +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (1, 10), (2, 20), (3, 30); +CREATE INCREMENTAL MATERIALIZED VIEW mv_self(v1, v2) AS + SELECT t1.v, t2.v FROM base_t AS t1 JOIN base_t AS t2 ON t1.i = t2.i; +SELECT * FROM mv_self ORDER BY v1; +INSERT INTO base_t VALUES (4,40); +DELETE FROM base_t WHERE i = 1; +UPDATE base_t SET v = v*10 WHERE i=2; +SELECT * FROM mv_self ORDER BY v1; +WITH + ins_t1 AS (INSERT INTO base_t VALUES (5,50) RETURNING 1), + ins_t2 AS (INSERT INTO base_t VALUES (6,60) RETURNING 1), + upd_t AS (UPDATE base_t SET v = v + 100 RETURNING 1), + dlt_t AS (DELETE FROM base_t WHERE i IN (4,5) RETURNING 1) +SELECT NULL; +SELECT * FROM mv_self ORDER BY v1; +ROLLBACK; + +-- support simultaneous table changes +BEGIN; +CREATE TABLE base_r (i int, v int); +CREATE TABLE base_s (i int, v int); +INSERT INTO base_r VALUES (1, 10), (2, 20), (3, 30); +INSERT INTO base_s VALUES (1, 100), (2, 200), (3, 300); +CREATE INCREMENTAL MATERIALIZED VIEW mv(v1, v2) AS + SELECT r.v, s.v FROM base_r AS r JOIN base_s AS s USING(i); +SELECT * FROM mv ORDER BY v1; +WITH + ins_r AS (INSERT INTO base_r VALUES (1,11) RETURNING 1), + ins_r2 AS (INSERT INTO base_r VALUES (3,33) RETURNING 1), + ins_s AS (INSERT INTO base_s VALUES (2,222) RETURNING 1), + upd_r AS (UPDATE base_r SET v = v + 1000 WHERE i = 2 RETURNING 1), + dlt_s AS (DELETE FROM base_s WHERE i = 3 RETURNING 1) +SELECT NULL; +SELECT * FROM mv ORDER BY v1; +ROLLBACK; + +-- support foreign reference constraints +BEGIN; +CREATE TABLE ri1 (i int PRIMARY KEY); +CREATE TABLE ri2 (i int PRIMARY KEY REFERENCES ri1(i) ON UPDATE CASCADE ON DELETE CASCADE, v int); +INSERT INTO ri1 VALUES (1),(2),(3); +INSERT INTO ri2 VALUES (1),(2),(3); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ri(i1, i2) AS + SELECT ri1.i, ri2.i FROM ri1 JOIN ri2 USING(i); +SELECT * FROM mv_ri ORDER BY i1; +UPDATE ri1 SET i=10 where i=1; +DELETE FROM ri1 WHERE i=2; +SELECT * FROM mv_ri ORDER BY i2; +ROLLBACK; + +-- views including NULL +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (1,10),(2, NULL); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT * FROM base_t; +SELECT * FROM mv ORDER BY i; +UPDATE base_t SET v = 20 WHERE i = 2; +SELECT * FROM mv ORDER BY i; +ROLLBACK; + +BEGIN; +CREATE TABLE base_t (i int); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT * FROM base_t; +SELECT * FROM mv ORDER BY i; +INSERT INTO base_t VALUES (1),(NULL); +SELECT * FROM mv ORDER BY i; +ROLLBACK; + +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (NULL, 1), (NULL, 2), (1, 10), (1, 20); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT i, sum(v) FROM base_t GROUP BY i; +SELECT * FROM mv ORDER BY i; +UPDATE base_t SET v = v * 10; +SELECT * FROM mv ORDER BY i; +ROLLBACK; + +BEGIN; +CREATE TABLE base_t (i int, v int); +INSERT INTO base_t VALUES (NULL, 1), (NULL, 2), (NULL, 3), (NULL, 4), (NULL, 5); +CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT i, min(v), max(v) FROM base_t GROUP BY i; +SELECT * FROM mv ORDER BY i; +DELETE FROM base_t WHERE v = 1; +SELECT * FROM mv ORDER BY i; +DELETE FROM base_t WHERE v = 3; +SELECT * FROM mv ORDER BY i; +DELETE FROM base_t WHERE v = 5; +SELECT * FROM mv ORDER BY i; +ROLLBACK; + +-- IMMV containing user defined type +BEGIN; + +CREATE TYPE mytype; +CREATE FUNCTION mytype_in(cstring) + RETURNS mytype AS 'int4in' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE FUNCTION mytype_out(mytype) + RETURNS cstring AS 'int4out' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE TYPE mytype ( + LIKE = int4, + INPUT = mytype_in, + OUTPUT = mytype_out +); + +CREATE FUNCTION mytype_eq(mytype, mytype) + RETURNS bool AS 'int4eq' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE FUNCTION mytype_lt(mytype, mytype) + RETURNS bool AS 'int4lt' + LANGUAGE INTERNAL STRICT IMMUTABLE; +CREATE FUNCTION mytype_cmp(mytype, mytype) + RETURNS integer AS 'btint4cmp' + LANGUAGE INTERNAL STRICT IMMUTABLE; + +CREATE OPERATOR = ( + leftarg = mytype, rightarg = mytype, + procedure = mytype_eq); +CREATE OPERATOR < ( + leftarg = mytype, rightarg = mytype, + procedure = mytype_lt); + +CREATE OPERATOR CLASS mytype_ops + DEFAULT FOR TYPE mytype USING btree AS + OPERATOR 1 <, + OPERATOR 3 = , + FUNCTION 1 mytype_cmp(mytype,mytype); + +CREATE TABLE t_mytype (x mytype); +CREATE INCREMENTAL MATERIALIZED VIEW mv_mytype AS + SELECT * FROM t_mytype; +INSERT INTO t_mytype VALUES ('1'::mytype); +SELECT * FROM mv_mytype; + +ROLLBACK; + +-- outer join is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv(a,b) AS SELECT a.i, b.i FROM mv_base_a a LEFT JOIN mv_base_b b ON a.i=b.i; +-- CTE is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv AS + WITH b AS ( SELECT * FROM mv_base_b) SELECT a.i,a.j FROM mv_base_a a, b WHERE a.i = b.i; +-- contain system column +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm01 AS SELECT i,j,xmin FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610'; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT i,j,xmin::text AS x_min FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm06 AS SELECT i,j,xidsend(xmin) AS x_min FROM mv_base_a; +-- contain subquery +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 ); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT a.i,a.j FROM mv_base_a a, (SELECT * FROM mv_base_b) b WHERE a.i = b.i; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a; +-- contain ORDER BY +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i) ORDER BY i,j,k; +-- contain HAVING +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i) GROUP BY i,j,k HAVING SUM(i) > 5; + +-- contain view or materialized view +CREATE VIEW b_view AS SELECT i,k FROM mv_base_b; +CREATE MATERIALIZED VIEW b_mview AS SELECT i,k FROM mv_base_b; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT a.i,a.j FROM mv_base_a a,b_view b WHERE a.i = b.i; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT a.i,a.j FROM mv_base_a a,b_mview b WHERE a.i = b.i; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm09 AS SELECT a.i,a.j FROM mv_base_a a, (SELECT i, COUNT(*) FROM mv_base_b GROUP BY i) b WHERE a.i = b.i; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm10 AS SELECT a.i,a.j FROM mv_base_a a WHERE EXISTS(SELECT 1 FROM mv_base_b b WHERE a.i = b.i) OR a.i > 5; + +-- contain mutable functions +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm12 AS SELECT i,j FROM mv_base_a WHERE i = random()::int; + +-- LIMIT/OFFSET is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm13 AS SELECT i,j FROM mv_base_a LIMIT 10 OFFSET 5; + +-- DISTINCT ON is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm14 AS SELECT DISTINCT ON(i) i, j FROM mv_base_a; + +-- TABLESAMPLE clause is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm15 AS SELECT i, j FROM mv_base_a TABLESAMPLE SYSTEM(50); + +-- window functions are not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm16 AS SELECT *, cume_dist() OVER (ORDER BY i) AS rank FROM mv_base_a; + +-- aggregate function with some options is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm17 AS SELECT COUNT(*) FILTER(WHERE i < 3) FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm18 AS SELECT COUNT(DISTINCT i) FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm19 AS SELECT array_agg(j ORDER BY i DESC) FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm20 AS SELECT i,SUM(j) FROM mv_base_a GROUP BY GROUPING SETS((i),()); + +-- inheritance parent is not supported +BEGIN; +CREATE TABLE parent (i int, v int); +CREATE TABLE child_a(options text) INHERITS(parent); +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm21 AS SELECT * FROM parent; +ROLLBACK; + +-- UNION statement is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm22 AS SELECT i,j FROM mv_base_a UNION ALL SELECT i,k FROM mv_base_b;; + +-- empty target list is not allowed with IVM +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm25 AS SELECT FROM mv_base_a; + +-- FOR UPDATE/SHARE is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm26 AS SELECT i,j FROM mv_base_a FOR UPDATE; + +-- tartget list cannot contain ivm column that start with '__ivm' +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm28 AS SELECT i AS "__ivm_count__" FROM mv_base_a; + +-- expressions specified in GROUP BY must appear in the target list. +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm29 AS SELECT COUNT(i) FROM mv_base_a GROUP BY i; + +-- experssions containing an aggregate is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm30 AS SELECT sum(i)*0.5 FROM mv_base_a; +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm31 AS SELECT sum(i)/sum(j) FROM mv_base_a; + +-- VALUES is not supported +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_only_values1 AS values(1); + +-- base table which has row level security +DROP USER IF EXISTS ivm_admin; +DROP USER IF EXISTS ivm_user; +CREATE USER ivm_admin; +CREATE USER ivm_user; +SET SESSION AUTHORIZATION ivm_admin; + +CREATE TABLE rls_tbl(id int, data text, owner name); +INSERT INTO rls_tbl VALUES + (1,'foo','ivm_user'), + (2,'bar','postgres'); +CREATE TABLE num_tbl(id int, num text); +INSERT INTO num_tbl VALUES + (1,'one'), + (2,'two'), + (3,'three'), + (4,'four'); +CREATE POLICY rls_tbl_policy ON rls_tbl FOR SELECT TO PUBLIC USING(owner = current_user); +CREATE POLICY rls_tbl_policy2 ON rls_tbl FOR INSERT TO PUBLIC WITH CHECK(current_user LIKE 'ivm_%'); +ALTER TABLE rls_tbl ENABLE ROW LEVEL SECURITY; +GRANT ALL on rls_tbl TO PUBLIC; +GRANT ALL on num_tbl TO PUBLIC; + +SET SESSION AUTHORIZATION ivm_user; + +CREATE INCREMENTAL MATERIALIZED VIEW ivm_rls AS SELECT * FROM rls_tbl; +SELECT id, data, owner FROM ivm_rls ORDER BY 1,2,3; +INSERT INTO rls_tbl VALUES + (3,'baz','ivm_user'), + (4,'qux','postgres'); +SELECT id, data, owner FROM ivm_rls ORDER BY 1,2,3; +CREATE INCREMENTAL MATERIALIZED VIEW ivm_rls2 AS SELECT * FROM rls_tbl JOIN num_tbl USING(id); + +RESET SESSION AUTHORIZATION; + +WITH + x AS (UPDATE rls_tbl SET data = data || '_2' where id in (3,4)), + y AS (UPDATE num_tbl SET num = num || '_2' where id in (3,4)) +SELECT; +SELECT * FROM ivm_rls2 ORDER BY 1,2,3; + +DROP TABLE rls_tbl CASCADE; +DROP TABLE num_tbl CASCADE; +DROP USER ivm_user; +DROP USER ivm_admin; + +DROP TABLE mv_base_b CASCADE; +DROP TABLE mv_base_a CASCADE; -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu Content-Type: text/x-diff; name="v23-0010-Add-documentations-about-Incremental-View-Mainte.patch" Content-Disposition: attachment; filename="v23-0010-Add-documentations-about-Incremental-View-Mainte.patch" Content-Transfer-Encoding: 7bit From f37f3a26721cf2577731b05f294de11762098060 Mon Sep 17 00:00:00 2001 From: Yugo Nagata Date: Fri, 20 Dec 2019 10:25:34 +0900 Subject: [PATCH v23 10/15] Add documentations about Incremental View Maintenance --- doc/src/sgml/catalogs.sgml | 24 +- .../sgml/ref/create_materialized_view.sgml | 133 +++++- .../sgml/ref/refresh_materialized_view.sgml | 6 +- doc/src/sgml/rules.sgml | 443 ++++++++++++++++++ 4 files changed, 601 insertions(+), 5 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 2b2c70a26e..b1ebdae501 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -2183,6 +2183,15 @@ SCRAM-SHA-256$<iteration count>:&l + + + relisivm bool + + + True if materialized view enables incremental view maintenance + + + relrewrite oid @@ -3465,6 +3474,18 @@ SCRAM-SHA-256$<iteration count>:&l + + + DEPENDENCY_IMMV (m) + + + The dependent object was created as part of creation of the Materialized + View with Incremental View Maintenance reference, and is really just a + part of its internal implementation. The dependent object must not be + dropped unless the materialized view is also dropped. + + + Other dependency flavors might be needed in future. @@ -3848,7 +3869,8 @@ SCRAM-SHA-256$<iteration count>:&l extrelocatable bool - True if extension can be relocated to another schema + True for materialized views which are enabled for incremental + view maintenance (IVM). diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml index d8c48252f4..5abccf14ba 100644 --- a/doc/src/sgml/ref/create_materialized_view.sgml +++ b/doc/src/sgml/ref/create_materialized_view.sgml @@ -21,7 +21,7 @@ PostgreSQL documentation -CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] table_name +CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] table_name [ (column_name [, ...] ) ] [ USING method ] [ WITH ( storage_parameter [= value] [, ... ] ) ] @@ -60,6 +60,134 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] table_name Parameters + + INCREMENTAL + + + If specified, some triggers are automatically created so that the rows + of the materialized view are immediately updated when base tables of the + materialized view are updated. In general, this allows faster update of + the materialized view at a price of slower update of the base tables + because the triggers will be invoked. We call this form of materialized + view as "Incrementally Maintainable Materialized View" (IMMV). + + + When IMMV is defined, a unique index is created on the view + automatically if possible. If the view definition query has a GROUP BY clause, + a unique index is created on the columns of GROUP BY expressions. Also, if the + view has DISTINCT clause, a unique index is created on all columns in the target + list. Otherwise, if the view contains all primary key attritubes of its base + tables in the target list, a unique index is created on these attritubes. In + other cases, no index is created. + + + There are restrictions of query definitions allowed to use this + option. The following are supported in query definitions for IMMV: + + + + + Inner joins (including self-joins). + + + + + + Some built-in aggregate functions (count, sum, avg, min, max) without a HAVING + clause. + + + + + Unsupported queries with this option include the following: + + + + + Outer joins. + + + + + + + Sub-queries. + + + + + + + Aggregate functions other than built-in count, sum, avg, min and max. + + + + + Aggregate functions with a HAVING clause. + + + + + DISTINCT ON, WINDOW, VALUES, LIMIT and OFFSET clause. + + + + + Other restrictions include: + + + + + IMMVs must be based on simple base tables. It's not supported to + create them on top of views or materialized views. + + + + + + When the TRUNCATE command is executed on a base table, + no changes are made to the materialized view. + + + + + + It is not supported to include system columns in an IMMV. + +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610'; +ERROR: system column is not supported with IVM + + + + + + + Non-immutable functions are not supported. + +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm12 AS SELECT i,j FROM mv_base_a WHERE i = random()::int; +ERROR: functions in IMMV must be marked IMMUTABLE + + + + + + + IMMVs do not support expressions that contains aggregates + + + + + + Logical replication does not support IMMVs. + + + + + + + + + IF NOT EXISTS @@ -153,7 +281,8 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] table_name This clause specifies whether or not the materialized view should be populated at creation time. If not, the materialized view will be flagged as unscannable and cannot be queried until REFRESH - MATERIALIZED VIEW is used. + MATERIALIZED VIEW is used. Also, if the view is IMMV, + triggers for maintaining the view are not created. diff --git a/doc/src/sgml/ref/refresh_materialized_view.sgml b/doc/src/sgml/ref/refresh_materialized_view.sgml index 3bf8884447..dc81853057 100644 --- a/doc/src/sgml/ref/refresh_materialized_view.sgml +++ b/doc/src/sgml/ref/refresh_materialized_view.sgml @@ -35,9 +35,11 @@ REFRESH MATERIALIZED VIEW [ CONCURRENTLY ] nameWITH DATA is specified (or defaults) the backing query is executed to provide the new data, and the materialized view is left in a - scannable state. If WITH NO DATA is specified no new + scannable state. Also, if the view is an incrementally maintainable materialized + view (IMMV) and was unpopulated, triggers for maintaining + the view are created. If WITH NO DATA is specified no new data is generated and the materialized view is left in an unscannable - state. + state. If the view is IMMV, the triggers are dropped. CONCURRENTLY and WITH NO DATA may not diff --git a/doc/src/sgml/rules.sgml b/doc/src/sgml/rules.sgml index 5024e4ff70..597332fe96 100644 --- a/doc/src/sgml/rules.sgml +++ b/doc/src/sgml/rules.sgml @@ -1093,6 +1093,449 @@ SELECT word FROM words ORDER BY word <-> 'caterpiler' LIMIT 10; + +Incremental View Maintenance + + + incremental view maintenance + + + + materialized view + incremental view maintenance + + + + view + incremental view maintenance + + + +Overview + + + Incremental View Maintenance (IVM) is a way to make + materialized views up-to-date in which only incremental changes are computed + and applied on views rather than recomputing the contents from scratch as + REFRESH MATERIALIZED VIEW does. IVM + can update materialized views more efficiently than recomputation when only + small parts of the view are changed. + + + + There are two approaches with regard to timing of view maintenance: + immediate and deferred. In immediate maintenance, views are updated in the + same transaction that its base table is modified. In deferred maintenance, + views are updated after the transaction is committed, for example, when the + view is accessed, as a response to user command like REFRESH + MATERIALIZED VIEW, or periodically in background, and so on. + PostgreSQL currently implements only a kind of + immediate maintenance, in which materialized views are updated immediately + in AFTER triggers when a base table is modified. + + + + To create materialized views supporting IVM, use the + CREATE INCREMENTAL MATERIALIZED VIEW, for example: + +CREATE INCREMENTAL MATERIALIZED VIEW mymatview AS SELECT * FROM mytab; + + When a materialized view is created with the INCREMENTAL + keyword, some triggers are automatically created so that the view's contents are + immediately updated when its base tables are modified. We call this form + of materialized view an Incrementally Maintainable Materialized View + (IMMV). + +postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m AS SELECT * FROM t0; +NOTICE: could not create an index on materialized view "m" automatically +HINT: Create an index on the materialized view for effcient incremental maintenance. +SELECT 3 +postgres=# SELECT * FROM m; + i +--- + 1 + 2 + 3 +(3 rows) + +postgres=# INSERT INTO t0 VALUES (4); +INSERT 0 1 +postgres=# SELECT * FROM m; -- automatically updated + i +--- + 1 + 2 + 3 + 4 +(4 rows) + + + + + Some IMMVs have hidden columns which are added + automatically when a materialized view is created. Their name starts + with __ivm_ and they contain information required + for maintaining the IMMV. Such columns are not visible + when the IMMV is accessed by SELECT * + but are visible if the column name is explicitly specified in the target + list. We can also see the hidden columns in \d + meta-commands of psql commands. + + + + In general, IMMVs allow faster updates of materialized + views at the price of slower updates to their base tables. Updates of + IMMV is slower because triggers will be invoked and the + view is updated in triggers per modification statement. + + + + For example, suppose a normal materialized view defined as below: + + +test=# CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm AS + SELECT a.aid, b.bid, a.abalance, b.bbalance + FROM pgbench_accounts a JOIN pgbench_branches b USING(bid); +SELECT 10000000 + + + + Updating a tuple in a base table of this materialized view is rapid but the + REFRESH MATERIALIZED VIEW command on this view takes a long time: + + +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +UPDATE 1 +Time: 0.990 ms + +test=# REFRESH MATERIALIZED VIEW mv_normal ; +REFRESH MATERIALIZED VIEW +Time: 33533.952 ms (00:33.534) + + + + + On the other hand, after creating IMMV with the same view + definition as below: + + +test=# CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm AS + SELECT a.aid, b.bid, a.abalance, b.bbalance + FROM pgbench_accounts a JOIN pgbench_branches b USING(bid); +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +NOTICE: created index "mv_ivm_index" on materialized view "mv_ivm" + + + updating a tuple in a base table takes more than the normal view, + but its content is updated automatically and this is faster than the + REFRESH MATERIALIZED VIEW command. + + +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +UPDATE 1 +Time: 13.068 ms + + + + + + Appropriate indexes on IMMVs are necessary for + efficient IVM because it looks for tuples to be + updated in IMMV. If there are no indexes, it + will take a long time. + + + + Therefore, when IMMV is defined, a unique index is created on the view + automatically if possible. If the view definition query has a GROUP BY clause, a unique + index is created on the columns of GROUP BY expressions. Also, if the view has DISTINCT + clause, a unique index is created on all columns in the target list. Otherwise, if the + view contains all primary key attritubes of its base tables in the target list, a unique + index is created on these attritubes. In other cases, no index is created. + + + + In the previous example, a unique index "mv_ivm_index" is created on aid and bid + columns of materialized view "mv_ivm", and this enables the rapid update of the view. + Dropping this index make updating the view take a loger time. + +test=# DROP INDEX mv_ivm_index; +DROP INDEX +Time: 67.081 ms + +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +UPDATE 1 +Time: 16386.245 ms (00:16.386) + + + + + + IVM is effective when we want to keep a materialized + view up-to-date and small fraction of a base table is modified + infrequently. Due to the overhead of immediate maintenance, IVM + is not effective when a base table is modified frequently. Also, when a + large part of a base table is modified or large data is inserted into a + base table, IVM is not effective and the cost of + maintenance can be larger than the REFRESH MATERIALIZED VIEW + command. In such situation, we can use REFRESH MATERIALIZED VIEW + and specify WITH NO DATA to disable immediate + maintenance before modifying a base table. After a base table modification, + execute the REFRESH MATERIALIZED VIEW (with WITH DATA) + command to refresh the view data and enable immediate maintenance. + + + + + +Supported View Definitions and Restrictions + + + Currently, we can create IMMVs using inner joins, and some + aggregates. However, several restrictions apply to the definition of IMMV. + + + +Joins + + Inner joins including self-join are supported. Outer joins are not supported. + + + + +Aggregates + + Supported aggregate functions are count, sum, + avg, min, and max. + Currently, only built-in aggregate functions are supported and user defined + aggregates cannot be used. When a base table is modified, the new aggregated + values are incrementally calculated using the old aggregated values and values + of related hidden columns stored in IMMV. + + + + Note that for min or max, the new values + could be re-calculated from base tables with regard to the affected groups when a + tuple containing the current minimal or maximal values are deleted from a base table. + Therefore, it can takes a long time to update an IMMV containing + these functions. + + + + Also note that using sum or avg on + real (float4) type or double precision + (float8) type in IMMV is unsafe. This is + because aggregated values in IMMV can become different from + results calculated from base tables due to the limited precision of these types. + To avoid this problem, use the numeric type instead. + + + + Restrictions on Aggregates + + There are the following restrictions: + + + + If we have a GROUP BY clause, expressions specified in + GROUP BY must appear in the target list. This is + how tuples to be updated in the IMMV are identified. + These attributes are used as scan keys for searching tuples in the + IMMV, so indexes on them are required for efficient + IVM. + + + + + + HAVING clause cannot be used. + + + + + + + + +Other General Restrictions + + There are other restrictions which generally apply to IMMV: + + + + Sub-queries cannot be used. + + + + + + CTEs cannot be used. + + + + + + Window functions cannot be used. + + + + + + IMMVs must be based on simple base tables. It's not + supported to create them on top of views or materialized views. + + + + + + LIMIT and OFFSET clauses cannot be used. + + + + + + IMMVs cannot contain system columns. + + + + + + IMMVs cannot contain non-immutable functions. + + + + + + UNION/INTERSECT/EXCEPT clauses cannnot be used. + + + + + + DISTINCT ON clauses cannot be used. + + + + + + TABLESAMPLE parameter cannot be used. + + + + + + inheritance parent tables cannnot be used. + + + + + + VALUES clause cannnot be used. + + + + + + GROUPING SETS and FILTER clauses cannot be used. + + + + + + FOR UPDATE/SHARE cannot be used. + + + + + + targetlist cannot contain columns whose name start with __ivm_. + + + + + + targetlist cannot contain expressions which contain an aggregate in it. + + + + + + Logical replication is not supported, that is, even when a base table + at a publisher node is modified, IMMVs at subscriber + nodes are not updated. + + + + + + When the TRUNCATE command is executed on a base table, + nothing is changed on the IMMV. + + + + + + + + + + +<literal>DISTINCT</literal> + + + PostgreSQL supports IMMV with + DISTINCT. For example, suppose a IMMV + defined with DISTINCT on a base table containing duplicate + tuples. When tuples are deleted from the base table, a tuple in the view is + deleted if and only if the multiplicity of the tuple becomes zero. Moreover, + when tuples are inserted into the base table, a tuple is inserted into the + view only if the same tuple doesn't already exist in it. + + + + Physically, an IMMV defined with DISTINCT + contains tuples after eliminating duplicates, and the multiplicity of each tuple + is stored in a hidden column named __ivm_count__. + + + + +Concurrent Transactions + + Suppose an IMMV is defined on two base tables and each + table was modified in different a concurrent transaction simultaneously. + In the transaction which was committed first, IMMV can + be updated considering only the change which happened in this transaction. + On the other hand, in order to update the view correctly in the transaction + which was committed later, we need to know the changes occurred in + both transactions. For this reason, ExclusiveLock + is held on an IMMV immediately after a base table is + modified in READ COMMITTED mode to make sure that + the IMMV is updated in the latter transaction after + the former transaction is committed. In REPEATABLE READ + or SERIALIZABLE mode, an error is raised immediately + if lock acquisition fails because any changes which occurred in + other transactions are not be visible in these modes and + IMMV cannot be updated correctly in such situations. + However, as an exception if the view has only one base table, + the lock held on thew view is RowExclusiveLock. + + + + +Row Level Security + + If some base tables have row level security policy, rows that are not visible + to the materialized view's owner are excluded from the result. In addition, such + rows are excluded as well when views are incrementally maintained. However, if a + new policy is defined or policies are changed after the materialized view was created, + the new policy will not be applied to the view contents. To apply the new policy, + you need to refresh materialized views. + + + + + Rules on <command>INSERT</command>, <command>UPDATE</command>, and <command>DELETE</command> -- 2.17.1 --Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu--