Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fwbcm-0007yy-Of for pgsql-hackers@arkaria.postgresql.org; Sun, 02 Sep 2018 23:17:45 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fwbck-000226-Q2 for pgsql-hackers@arkaria.postgresql.org; Sun, 02 Sep 2018 23:17:42 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fwbcj-00021b-Jr for pgsql-hackers@lists.postgresql.org; Sun, 02 Sep 2018 23:17:42 +0000 Received: from mail-wm0-x234.google.com ([2a00:1450:400c:c09::234]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fwbcS-0003Wj-6r for pgsql-hackers@postgresql.org; Sun, 02 Sep 2018 23:17:40 +0000 Received: by mail-wm0-x234.google.com with SMTP id t25-v6so9946416wmi.3 for ; Sun, 02 Sep 2018 16:17:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=cnGuWKLyXfPPZJMjDAk2c/CslteSTBgcnAmFll3u2V0=; b=wtyETDsCBdRD+b4b945lPKd4VktpFCGQ/R4Sfp/jBtOLOwGJRAXUUPpr16K70eivfC p2+bu/btzBeQ+8Tst3gqvDGmQsE7MgFFNDp9bZXk10EqDVDTjPzWmOI12nA5O1YwHxD5 is2+KsS/f4kLrDQpK/VSkTIp1Vl9RMTnrv9WCX1hH9IkgwrA0rHTG5SNiYdz8XuMmQrn B4hAHEF/iR5fzBkEbw+kSZ3YsC0zDv1ammSjZ8RpFgdcTKUn4kiTDL5NuD+qPXoXNxp5 YMasIMhtTnQ7P6k54k9O+mrkdcKdZT0D/UANt3MGkoF2NngJK2N4GzVAkOx7WWMc9393 UIOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=cnGuWKLyXfPPZJMjDAk2c/CslteSTBgcnAmFll3u2V0=; b=G2934oBg8et1Fg7mfePkkelPCoKJtj4IXnUwE71rdmPXig6MGrjytXjH4gibE75s5/ oa68c/ynnjmiOmQlBoXbhGogthr7FRgJD7Gcj6vavZmBPJ441phB/EvgvgLLIf3clRDv MnOPmsz65uBO0nYor4Rn++2pjmW1He1KsN8evPcbganT2rZPVnWR2WttWz8D8th1FuJt RkWPpXqrKnwzua1dTVl+Q35AJhZl4G55URwvqNhQHzmfWepTtiCsf/pqy0U7OksFvhqi QpHtxCn+xHDbnn56HEFs1SG42fiG6fc1CGvNYpndfTaSiPCitedDOk/pV0wIxEAAxLjT iU2A== X-Gm-Message-State: APzg51CVIVXh79ui02H9jXa5jfD40/PjM8fitbHNVIXzhTHD6i1v+24y TsQ3DAfeguAkM37vzPyU2Kb6HLnw48SDwcy7Rmlqlm4SdwnM+MlmcrCARLGppHxJGUwaVnU0ZRS RfoPx2bK/vUL3hkM2s7oqEK7TcWZilxg8AsHMtqrA54sDsL6yQ6VuNljL7CMiOvlHWyk4tidDDI J7jgNmpBKo3og= X-Google-Smtp-Source: ANB0VdbfJ45J+fx4PRlD9cBb0uFI/gINDlZYmv0VqfbFmeiDgAT7F0/PLvOP4NP6tefDReota6bG4A== X-Received: by 2002:a1c:7915:: with SMTP id l21-v6mr3829249wme.136.1535930241318; Sun, 02 Sep 2018 16:17:21 -0700 (PDT) Received: from [10.137.2.19] (ip-86-49-246-121.net.upcbroadband.cz. [86.49.246.121]) by smtp.gmail.com with ESMTPSA id e14-v6sm11162407wrv.44.2018.09.02.16.17.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 16:17:19 -0700 (PDT) Subject: Re: [HACKERS] PATCH: multivariate histograms and MCV lists From: Tomas Vondra To: Dean Rasheed Cc: Bruce Momjian , Alvaro Herrera , Andres Freund , Thomas Munro , Mark Dilger , Adrien Nayrat , Pg Hackers References: <49348328-1a02-7e45-e50f-e1d4d0243c5c@2ndquadrant.com> <20180407141201.GA10394@momjian.us> <4416a567-4452-c5fd-7117-5cc7827b5fad@2ndquadrant.com> <80bc98b2-1c43-1fe8-b316-d447bb68cff2@2ndquadrant.com> <341ca23f-144d-130d-1b45-ddddf581797e@2ndquadrant.com> Message-ID: <8ac8bd94-478d-215d-e6bd-339f1f20a74c@2ndquadrant.com> Date: Mon, 3 Sep 2018 01:17:15 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <341ca23f-144d-130d-1b45-ddddf581797e@2ndquadrant.com> Content-Type: multipart/mixed; boundary="------------0E1BE69E814B19BC8B911192" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------0E1BE69E814B19BC8B911192 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Attached is an updated version of the patch series, adopting a couple of improvements - both for MCV lists and histograms. MCV --- For the MCV list part, I've adopted the approach proposed by Dean, using base selectivity and using it to correct the non-MCV part. I agree the simplicity of the approach is a nice feature, and it seems to produce better estimates. I'm not sure I understand the approach perfectly, but I've tried to add comments explaining how it works etc. I've also changed how we build the MCV lists, particularly how we decide how many / which items store in the MCV list. In the previous version I've adopted the same algorithm we use for per-column MCV lists, but in certain cases that turned out to be too restrictive. Consider for example a table with multiple perfectly correlated columns, with very few combinations. That is, something like this: CREATE TABLE t (a int, b int); INSERT INTO t SELECT mod(i,50), mod(i,50) FROM generate_series(1,1e6) s(i); CREATE STATISTICS s (mcv) ON a,b FROM t; Now, the data distribution is very simple - uniform, with 50 distinct combinations, each representing 2% of data (and the random sample should be pretty close to that). In these cases, analyze_mcv_list decides it does not need any MCV list, because the frequency for each value is pretty much 1/ndistinct. For single column that's reasonable, but for multiple correlated columns it's rather problematic. We might use the same ndistinct approach (assuming we have the ndistinct coefficients), but that still does not allow us to decide which combinations are "valid" with respect to the data. For example we can't decide (1,10) does not appear in the data. So I'm not entirely sure adopting the same algorithm analyze_mcv_list algorithm both for single-column and multi-column stats. It may make sense to keep more items in the multi-column case for reasons that are not really valid for a single single-column. For now I've added a trivial condition to simply keep all the groups when possible. This probably needs more thought. BTW Dean's patch also modified how the maximum number of items on a MCV list is determined - instead of the shaky defaults I used before, it derives the size from attstattarget values for the columns, keeping the maximum value. That seems to make sense, so I've kept this. histograms ---------- For histograms, I've made the two improvements I mentioned previously. Firstly, simple equality conditions (of the form "var = const") are estimated using as 1/ndistinct (possibly using ndistinct coefficients when available), and then used only as "conditions" (in the "conditional probability" sense) when estimating the rest of the clauses using the histogram. That is P(clauses) is split into two parts P(clauses) = P(equalities) * P(remaining|clauses) where the first part is estimated as 1/ndistinct, the second part is estimated using histogram. I'm sure this needs more thought, particularly when combining MCV and histogram estimates. But in general it seems to work quite nicely. The second improvement is about estimating what fraction of a bucket matches the conditions. Instead of using the rough 1/2-bucket estimate, I've adopted the convert_to_scalar approach, computing a geometric mean for all the clauses (at a bucket level). I'm not entirely sure the geometric mean is the right approach (or better than simply using 1/2 the bucket) because multiplying the per-clause frequencies is mostly equal to assumption of independence at the bucket level. Which is rather incompatible with the purpose of multi-column statistics, which are meant to be used exactly when the columns are not independent. measurements ------------ I think we need to maintain a set of tests (dataset + query), so that we can compare impact of various changes in the algorithm. So far we've used mostly ad-hoc queries, often created as counter-examples, and that does not seem very practical. So I'm attaching a simple SQL script that I consider an initial version of that. It has a couple of synthetic data sets, and queries estimated with and without extended statistics. I'm also attaching a spreadsheet with results for (a) the original version of the patch series, as submitted on 6/24, (b) the new version attached here and (c) the new version using the per-bucket estimates directly, without the geometric mean. Overall, the new versions seem to perform better than the version from 6/24, and also compared to only per-column statistics. There are cases where extended statistic produce over-estimates, but I find it somewhat natural due to lower resolution of the multi-column stats. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services --------------0E1BE69E814B19BC8B911192 Content-Type: application/vnd.oasis.opendocument.spreadsheet; name="stats.ods" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="stats.ods" UEsDBBQAAAgAAA6sIk2FbDmKLgAAAC4AAAAIAAAAbWltZXR5cGVhcHBsaWNhdGlvbi92bmQu b2FzaXMub3BlbmRvY3VtZW50LnNwcmVhZHNoZWV0UEsDBBQAAAgAAA6sIk1sB2Xv8yoAAPMq AAAYAAAAVGh1bWJuYWlscy90aHVtYm5haWwucG5niVBORw0KGgoAAAANSUhEUgAAANcAAAD/ CAMAAABVXLPcAAAAZlBMVEUAAAAMDAwTExMcHBwiIiIsLCwzMzM7OztDQ0NNTU1TU1NcXFxj Y2Nra2tzc3N7e3uDg4OLi4uTk5Obm5ujo6Orq6uzs7O9vb3Dw8PLy8vT09Pc3Nzi4uLq6urz 8/P+/v4AAAD///8X2GC3AAAqSElEQVR42u1diYKrOK7t7plbFfC+r5j//8snG0ggkIW6y3T3 u6lKVcLiBdk6kjBHf4z/wpcfxz/+xs0zQ/2rrt9F+af3K9oyRp+KD3GMQ7LDONg0knf75f6m /Qo0oLHTVHHbjcgQR0Ye0ED+6fJSbuxHNDISCoJ+mYJHLi75Hy8vJwZU+yVUrPKyIx7QcEn/ eHmNRqTRjj4rg0ebInw2Mjj3T5fX8sqYmn+hnv/6OP7F/cr/Ulz2/3h5DZt/q/qmVynJ7w4+ LqJ+En8LeSUdcpcG7UeLBwCfVX3FQNesGbjIyZto/Rg02BF6dLfDvAbrQofUpTCmoYCWz2xT foGSB2/BFoFzoxt+lbxQ1EM/6EiiY1r5sKqPeqUdD44brz7SZ+iLjP1IPGhxsghHRjTiyHMP GKag+cJkuikfR+Ec0jiihAaqyq+SlyMczAQueh/kwEhc7blwBsIiXnmvKvKSTOTnyBKjfJFJ IhKwuF6esfZLcaLu5PXJmXXKqSBGFmn+ZfNLZ5L70mdkA3PB83WPg4/wSyU0rPXL0/hZeNAq 2QWSlb8UlHipRWAPn8Rdv3B0qfYLbBBUDvv1c+QVwLyzwavkikparca/H7QpBTZllaMf64+2 ISYNvVnaV5R3eYAiTIzK5ySTKWGrUJQZc8oJKopjKH8D/HK/8eu78cu7X/cjjjd/qSz35DT1 przK8PUR5x6Ow1rqEMtkX2VQmyVu60kpges8rHbGB5bYVMyQUm6FHc+vouf/S2whInkam29o 9mgcJiTGgnQPKvACyKb4CF82HbsoPVKFcyRSAHjjQUgSDu1OpGs1FRRG1s3zC4teoCQYoUNu OtleMqsHCBo+WjHFn+oX57QQyYgZkegKyoSpI3kVwJ9xAFU9MHCyVKRgOW20dkEug6dsNY8j GgioTjxGfiyvYbp8LI/1U5NXP9Kk4JRstWwXNLF8Ub0nMi4X+1y/qIwFi5BpZGNfkKDqcqg3 IvSrUDrSAdqLa+fVpp7CHSAYVA6tRWBzWIPuTaprv0jb3uSCZnkhKFhbBAOZTLZMZgMeUhki GpZ+qVPeSEIZ8TjUfnUFSzOkw/kF8sp+pB7zD88iXF+5dSJLHGnEICbpCwbBiYAejBywJHEd wSJM/WryYqMYbFAM/k/NH2BUcjoyGE64XWGHkTk5DplKYIVTcSlsIFwfycuTXo+EVduPjZEy XTpK0/oIAm0yINCBUD8qyuCU4xBHqUeqNHKwmGnH9npjqNpzc+3vN5x6sdD/PXC5hBDipqP3 G875x3b4l/mVv+2oXyiveA+v3J6O2N68iqfyEjWQBjo8EGZHze/QCbSfo3wsjIbRMDEGyh6E EG1VTZnSAMeaSV5WKysB8JwdS6vYdTkIO3ph42frYtCn8CsIM1pvRR4taG40apEfySvREZeR ohG8fcBh8DC3ao5U39JJN+t5XJx6EGisV8SnTJIHwGjywt/+e/krs/9e8DA0AON/Bfz5OaKP b+ZP8wVcxh/fAJc//6PGj28fBZX/fJOP5AXQTHMhAFKuIc/WS4yCTPZGAnuDjhVkE3sQ+OYz Prdi3BqXAY/pMNsb4GqnAMjxJVyOlICVAbgcOEgDccLFw/nlBI4XfanDogezQ6yPCL3qEh6d utpRj/plwRasTedVDLm/9itrhzxcGzoLNaMgCsD70LXrlwQ5o+thvOTaL1J6960gT4N6OL+I w2C294MRBg4Pm3pKiqjAAdlTA1fIoYEbemj3DsnT0WVNbXDEz/2KoB3yUGwe04w2bqh30TK8 fRvvybkz/co2jjEVKNYGMM/aTbpH8kptH9TioS211nsNVuq2CDtrA8eH7ShpTAXaGaC2Ya8P sxBCboe4EOrL+lZw9T/BL/cvxS//L7U3HshruP9kw9dxmT+SV+FgfluAY1U/GICuwLb4b5kH BQiIK3mGnQb8qweAo1ktugheRgUFNXlFH2BeDvW24GRpBDzpDZMyavowsFP4NZiqN5IpMN+r axquemMjLx0DKxw8T1wKGAFDBd+NnwJ6HwH88WD0UAGaJpKNPtRUcqyOA/eROVFm/Op8r2nA AFdxwl/oF4qgqqMcuuEL+IWCLJgLw0Fxg5533JPj+aVNa3zH+CgYdWBOmbWfl6Ez4Ff6KgHU dj7E5aKaGxeHzqrxssJlg6vuY3nylwGXSTREfC0OYKiAfq1xmR7OLylrNB7gZVQG7La+Nn0z ECXvY8XlN/rlwBwDn1KAKca74m+43A/QmDkOQAsabDEF58ktLFacCbSZQhKuuDygAewoKwd7 JC8jhmHsUx7ZwJ0XYFmAUbGxo+SARpxZtDLStpNGdWiBxxrNj0PwVsLsIZO8wMItMSfhRj+j nkhxsntHHSZDVpzRZPVEGwa4alahdiv/ML5hhbADwOMQq7tgwDhOfDtQHRiGWbSdQ9s5SP3A nIedIRcJ80WLn4/LhLPf+PXb3vhx8ipnSw4Hn+7jvcNQpqJzDdRXjZHKXZV1Z1ntzA/0V8nt 8FxLTA/lpe8+pf50fD6+js9LWsOYgoC9QYZIwETGGg3XiHu1U0SNpuNpJ5W4SH7sMA1I1nCD JuC/YTnHD0HpS5QlY2zIzQE0l1zjopzT+NGKGc7hl6hxUWiCBb0BBkOmzBzJi1enIwjQ4OBd VNexOsX5Gt8TdScZ34vP53b5JBg6joM/M8fnSVIOIM8Y5Wdc7jQKWKev+ctEQztu8XnJ9GF8 3ighChlo+fTU09oyCpfhBklQUKlxgFt8XqPxbmnA7SJVXLNSMoPDXRwgDQTP8fmMUixD6Icv 2RtwYsZzfL72S6d4JK9Yb5gQ9pEQWBIyFAwiUW4V+XCYfczxeTAUySg9LuG4JWy6YVLGHg6Y +wUjbjAeDLRxtpaHPrUQOyfgxE/x+f5MfJ4JUqhMAx+J+Cw0P4jPW0KqG8zB7sJDrl+gxmV+ edzpFrhXlJah7pQUXH1CDm/sxQtlRUXYbUfKxF5v1Dh2+HFxbJ7OxefLj1DyZdKHw/zK8DPA HDMmz1/qb4Tvdjj3aie3/16E1fdhMEeHtYpW3w4Ky7d/L2qejqj9Sr/wpX9RPT+7XxHcy7D6 bn5Rv/ymX3ldfW6/um0LSHxRPkIntSr5Tl6fPhD8GRhFwSEiE8M4bvYTmjhBMfRYJUZQUrj3 hxWJSxMTQlADEZO8OCMCR0U4j7G1X316SVmShNkP27qv+JneiFpg4MkA4HxY/lBetDbSk9Ql aoSKfUCJb47oQ4p9AlCysY+wU/fRssMqbV//Ehf7bBCf5HWJ4KrrLmmuhWrdIKGzyHbWQZVT e9iZfvWBGCn7KJUWfepXo0FfBwL8Wk6hXzgkii7RX3oJXZTyegSM4gvroK+aE586j5OSXb0M h6/WUBShbmxmefVQtlB9igGjOPXLX6QKgXdxbpThad2i579QPHW1DUJp3i+XZi+vjn6yBGJo IqE21T7wzUiNiYAgVN3ZhKn7ZOm+Ie1aNnn51HMEV2nVL8xA1NPFCBePOI5YdOGzjUODLuo2 +16+pn4h3oO8PqcRkg/mV4zEtXbCkQ7qJ6m2YVVN6OA7pX1wPcwZAl9gsrl9fTlF9o1G6WyP ZYoghNav2H6Sr9cnhRhi8u03+pjrV3gBNp/RgvWnFgH/QowrtWS2Fzqm6e1rxVVv+uuR00D0 884Y1kcdvGoDl+Pgg7/X80FKqTYac9qQv0vt5q/hV368/X4Yfi9+vTW/DhuXfxB+5Xe6/7Rf +ahL33GZd/LKL67d/rD8Zp3XcZjvJsiyYZ4cxp6/qvXX6pjEIeC3TbJOa508rSNd+xSo3Jay 7BQsJEd1CuyBgRCm7RYunAL0aPJy1nmTog6galyt0AHC11q0D4CMsMXiU7jsnYUCHUxmXZXZ qqNbeZlvoBcQrnhnk/vQCXm51U8ItZ1SOAQwgB1yQh7OBnaZAIokS0I/9euiLgLZXmPsGjok 04Xe9PCm/tOv8OvNKZZ62TECWAP4RT/DQ/yKGPCGa1yhVyTMNXhRG3SqOwHcJlwOGFAOWvwY l6tESALLrY9+xi9fcRmwhPgZlz8osgIMkjUuvz3BGn6BpSG0Wtkb8/y6XQGisXXY9s2k4ILJ cElsLVGPXNspAOde2RtTLbAXkL1f2xtd5LFH014cwaqKCsbFfLE1O4Nf0C+rRRdJs6O61SXZ yEuKi3Ccf/o+EjgejIL6Yd0vMe9UwsOAjDAOfR2HB/o3dg0vHY6WxrlfEvpgvSUq6XleU2dg Bisqqh1apwUh6v1hmFQ0wVkowjnLjFqN4Dv80vVqKRg9ddJYP39Yv+adnIZUGxgoP9RRkRIa NfQBmixBujs9Dz4D28jGUUZF+lEvfQojf5Ff+fXKrmeaRwief8hlyy/8yvysS++aHLsO7PT8 vCe/K6b8rAWb435aHGCuKIB+tQ/9ymoSxwX6m30c7noYFtMgLF/iU5tjKXHfr1pgvE112Srw Hf/agNGgvuzR/JL/hTdBOBDyEZL+SM0VQbwLWw8NJ0oWP4X2UZDeHVbEPlrxCCkHJU79AseG 90EQSmNgcxyAU5IYJfZba1XQ/G7gPP2ltUCAEm00/mbJ6pyrHQXqe/IDQTEl8GdBBYOrD6rc r64coCzys19ZnU4BTv1kOexNnIZf2FdHWMk5DpCwq7hshRR6jgN8KrA4pL/i8sk4ADVK9rO/ 3K9aqjc46unkuxNAJ+hQlzDp4uoqxE5+WASVVz+44rK44vJucrWGKsIv7eMmDuADXlDbd9bE qC/hPg5wwl/uEt/EN7bzq29hlhrqAvecftNscfU3UrAorOM2pp+ucH4Q34gOLkON7DR5oUSC 1JSA58/mOIAjFL5RFC+tJbq/yOfaaHPxcGROS0KQ1uJi0Uqh6qVN9ANPg2EKVyEYSEglqDGs m90TljhG0dc4GyUoKtQfOsyRfCNRONMjWw3Ejd5ozkmsjx81j3rWRKGun/8a0ByYXuZFGfHg e3xS3ntx0doNpaReF+CVUj8u+qzf9UXzu2Cfz/vL51zht17mx9SVX+19bG88DSO8G+LI13GU 7+OiT1qXvz/2sJNXXl/WvHyKWH5NZlqYJI/kRTG4iRiTdCE06RqPZaQFKqfiYaIT3hR3EhjH gLBOnOB65HGgTTW/0iFkLcGXqV+CUUmipkLc4vPVRwE0cFN83kl+5vIJDgUGkYxT+MOKo/kF Pj80EPR3jTD39aFmjev162/zmmYAbJwqDHLT9Dwzfax6fm+qgovfcNkBVFRQWuLztsbnFdOy 2k2AX7EDX7VzdhvHfhuXPdHyist3fuXcKAPdANc+foJ19MHBOoIvYHSglYNU7QcELTDX+Dyg 4QaX7/GLC3WpoLGLz+MlPv8phPesf9Sv57NrFZ/f4PLKr9S1eYDFLQQPl1TUfrkL+7hak2Aw mY59uD4rMcfnn9hRcy1KoXZBbv3SSKzj8z2MTCz78NnMTIs7fdregAJrfN71u3EIs+cb45Ww IYUOqgV70nn6KVMILT5f67HkomMAeRGwXR0CPxnzPjKGDiN+kX/jUTlNYB7We0KtXwF+Yowu phrkhldy8A22wdtP4XnvwgkFNRdo66k+hKNx6H2truE+DJFamW/wH66a1PvZN6lVt2Cdm78c jRHvfT3W+8lXqfMrwyvVX/iTA+dCpGULvBxskHn9SvmNV7odllb/zPw/3falh+en23nXr+vC Vn/T9Wf6Pj1v7pant+c/8/v6SPf29cZz4JvT/OqfWFe1qsXt6riWsG3F5t/t73L80nr1/2ud Xrn7VJQ6u2LktiROvrcuxchhTPKg4/URTaOmnZWhYrciUKo0+mmprYZ2OpkfrdO7Xzzo//zz 7IMB7vU6ve2aqj//VCP767K/xn9FeP9lYefnaP/6a3eB+R9/6PHyF50K8eO3v8Qkr850Ckdi wKKZVsH5fsAWw5uDL31b6f7+c22WFyS4FKW3HwWB98zeWEcUeK6Pru3XIsXx6TpzwkWxuj0R R4YRKTficnv+qz7XFgRP1+facJBULxdbnyI6UvXEef08gh5ijl7PL09LZtvnAm79yk/6FbNn S7/K1K/tOj094Ntz9NAEOCBPDbLy1PTyhaT2XFsm5bMgba79cU+e8Svj/XMc8+q4OBqdr89x 7Dquh8BSfbIwF+USXMvlOQ492gIIzOrDntOQZslxNRp4T88FKsbOPBBruSrG5WJHLXsoXvDw Ul6OMdUerdkJn7Hw9LmbyER9aieMLhXBhlHDhdivn6+3yTfP0wn5aC3+OzNNkL/JOnMYDWW3 5bvWAv5+LuDXPxcQTj/XZg8+vZZXOtpWl8T7StoCGs/H7VJ2B9vqetZhorKMoTJQrPpVNkaG mr7BW8wPNHF9cvQN1yLQqugX8lKf+9W75hOsCar5WCGV69tzvqSHt613nbkdkMVN+SiZsZ2f J+pFx/vMOSbDhA7mkqkAnStInPltwDg5uW4ZC0Y0FA16Hjwn+Z68Ct0/zlAAvw6fyy64/jqF x4E6wK9hxmW7et684XKxqvGoTM+1dR5cvwWXAzkjrqqZEQ8DiRMuE355c34dka3d7I279fNo bIvr0TgQQLWrveFX9kbDZV9u/DZkrE9pL/w2gZ2aW3lA7bk2CnDZlWoB5Dfn14G8RrCB1CHv Aaq/2pIh8lgfDKyXoGA4YJYXH9Vgg6FinJGqwKDlbBSMjaS1hJNTtKyCsyJ0KhJG5KWI4cFz HAcn9nt7QyESCq08FbTyVNBlrmba88pIOCYCV6PSXWi4kiSNjNoDfpu6znyjjeo68y/rW2rR 3wS/ErjRm2lbnesvlz/csfj9fq7tt71xQl73/nIg5OwzKvqkvzxZEGivDzNKAFDUj5pyAOjr gzlgt4+NNCtQkkdOVdMlJIx04bcxUhpewA0wY2k6w16y42Z0XMfP1jFXHjxO9uBqgYOjXX3g XwsF3pDk8T15ObLvV2U4W3jMPL7xmBV/1fOJBdE497d6vhvAN3aVXsdPujCzggYa++yXi13o GXmhHArmNtGMx+pX6ozenF+H+HXD5S3vXMPlALhcrrgM/rLc4TKcM7Gfgb1xqXcgOC5Tgwo5 ZffCiUN9rm2xN6hU32NvpLUdtRo4aGwPKVYbBIRUL0l96E/bTRzAYQOiZvOQzsjywhwaujQZ RvpMx7jD0/N6pTM1bkOsek9e9qJ2HXO9SBOvo0FFyOtEH/QnCEnjrARzBRky2uyYUAPWdOpX hp+hlDCMwzC5kaFUp2CAN2AXbMgxnsGwemJ9IDePNqFafCjvySvFPWstbBsmHk74P4ZrO0qM sRLngrZJtcoCfRjrA485jEdxALGNA0TY8PU4gLoLWPzmnfttb3yPvaFPZ1m4naDek5fTMI1e qaei9yWUCk5xUjjWgHJQM0/FMOSSahxhmHl6ACNKAnWRhjK5BcKeIp6DE8dcKqnoAOoD/L70 Wl5O+Mb68py/hPk9EYiwSmUcaxzASMcTibOe71UnUaQKkzyhg+syBqWKwWL4nJtkzugOohng Muey9Br0fCSKvzG/IjuOz28Q/5AHRpu7+Pw2DgA+9RSfb3EAFAS1i71hz5hRo6S2LLiMW3y+ fz2/AnkQn3/ZLyNg/K3j86hseG5NWcfnYwlliQM4fio0Gub4PM1knOLz4aW8ai4bMItYeh4/ HvYDx4oCsoYLOIzSZRLEhifQ9ekyotneIOOl9EMHxst0oYk/hcvdONkbZPyIf5U6aV7bh0pB FdCG5xdQWLTrOPFgPFxSP/LgqVLlEsgBLtf1bJvrKJU0X9a3ROG/CX6BQTXcb/h6fL7k3/7y /8LeiOnrPXBn5JWPN13t3vt5DnZwqUb5HCCqdu8Q3uafV5ydtX3zYRzglbxUv/ddPiNggKp+ yp5/nmiUHNGh+Sk1Pi8Aohc/pfLP93nHP19u/PPDOcKUenLln9cjEpcN/7x/6ZC+jM9vkHQA JWiErVFgsvIrJ/zq4buCU4Y1/7zsAxVX/vlIzugOKhu/DeAym/3lN+PzcDmO4wA3/vlE7zE5 aIsBztmCy37p14LL4cZv0/jnc1r4bSI9NwZTvYdS4/OVfx5JezUQn8srEf4tHMUBrvzz+I5/ XquW/6EHMN7H59koG/88H8U0ixr/PBs5vCf+eXY2Pk8L0zf+eXq9e/ZSH/KDKUfAflnibGQb 2ekpDQmDvceoHlXKhMTGI72Pz1trN5Vn2PAd/PO+/3v4yzVqsNHpw/2GcyPyLonVb5723/GN E/KKd58iIWfH4jH//At55QN3ecB5PLqvN1pW86TVjBfzfb3KL0rj7b4e+NHgw0Ru7Viau2q7 HLgdPTcz/7wdTsXn4cTKPw8K2ghT0KhEek9e9iA+76Cpq/uwt/i8bfF55ep92EM93w3YS4+K Er6x7gMug19JU1/c1QhiZ+K9aHAFwUVhGY8fYOLo8nZ8/jiOffq++RWX1V18/pPz1GjkF9/t zBiMUM6W51a+G58/vu/Q7Kgdr36TV037Nlzj88M+Po8cucXnh96Lwq/884zbM/dTeMBpjs/b j4I99W/ah67Tu455JHM4WJcy2Iut61KyqutSYOfo8nZdSpr4510el1vLHr7BNnhP8fDo/Rm9 UUmoU27887Hxz7s34/PB+12/Yt1WCRpqA1friIr3sNGlaVuGKqDG4KePP5vnVt75N7/95X8W Lv+P5XXPPz9Ugviv4jJ/T17BVIB5MeQP+edNqHeha9FF8NU6WJiZ0Y6DSWG+/w96EL6N2cTc N72hkj+DyxlqiilW/nlbl44FU17KK9cVs7jY5/7rEf+8h4sBQFbtDblZt9z53lT+ecYW/nly zz8PZ5zB5agAtaRlQx8/Kv/8dTnck1IAn0rHWTkfx3Z6DH1j/l/Wma942isuR77in6dr/nlx KmxjCa956AYa7/nnn8yvgYxDP0r/hX7VhdwtvrF7LqDGAbK45oUBO2MwRV/55wHtzswvXfnn Ky4PKDf++WxfyqvUW0Q4vVhPQaPc3XAZKo89jK4SB609iyzN9oYbfV3dBZdqQT2ZIgw8Lxf+ eStPjcMAJ7qJf15v+ecfSyPV5ez5VTXDQUzdS2h5FHEMw/Tcjch7ffiT+ed/49dve+O3vMpt ck76XTN69h7YMf/8E31Y10xj1T/Xu/IgYVfs1TigdqPNU67h48w/TwQSlX+eslt8nk/883N8 Pp+Lz9e4KFWcmBr6b/F5/VJeJZNaD80v8GsfkBjqEm0ArnKNz6Mr/3yu+WGTMbf4vO4DUVf+ +UTO4BdRNT7vrzztTHVv2Rst3P6iXwf887VfpfHPX9fp+Tv++dt987zmn0+nbjuMQ0AtX9vS L5PjW/bGqF6ZNYf887VfE/98jdtYfeOfF4PxcsU/jyr/fMsPO/HP03PxeTh5YDP//KXxz79e fxhpJ8qF0ud+uSP7gWMxMrGjrOiYCQkjofogjg3e9ZZ/HjZ8B/98/N/F54cbwXsekrHG5pyv GyJscHlhdn/v98oWP3gZVlz2QzbnCjou+sGOae/0yR/z2+T0c175V/HPr/qVf153TvIffhfZ XX7GR3Sjc/YzMSf2X+Iu1tLPPLd5RZdzzGyFTWWe2ZNeQNWREAs7WXKY7FikJIadvBFqBoLB YyZ6kpcGF4EnVxkzY+PR1xdvuU6WKTfxishwRCP+kIxIS2FY1Ml5jT+desSbuvk1NlR2Fmrv 61HQVKHjtLOPmt/X1sjrKjdpJWAJSPOFN/Xiek1NFxi1E2+ew7GHy9R5FWceuETOMD/1Fkuh +lj5ba5U7y/5o4RY+Oe3L+oqkVv/kA8WY/BcZSWuq1xAfeWDnXjau5nfBs7B4cpzS5zBeCGP x6cmxo7nNh/zEm8ZDBvFjVC7SQ4irDy3lb9XdQf8vRGmiRKVn7++O3HHc9sbkBJOEzlh6CyP zPX+EiZCQ+3O9kt2FiktrpR8L+SlwLBLlQ35nhbI9iIYokjbCf93RVDNWeVRTsYrJpjvDZr6 ZZOrNDTSpYVhTsE32ApvM/H2KGVPzC84NQTvpfPBa3/PV3l8jlPKpCD36skoFeBPrKGJFFs7 tmdGqSudfGXlSZVU18l4xHPLKL/juWX8vLp9AILma8r79BHrfuXvruC16n+LT+81bf7LZvkT /PNfo9W/w+UnJ+U3mv8u//xNXnm9955/PtIT/NjrFsz88++Mw4pzjhwIVIAqETTAzsojLbZX d9mpJmJtTmOSAAx+4p+vdPFRTfzztYo+BuUTvMPEO8utpicGonPG65l/vmuUVotQHuv5QFBK XQQQvpOj6F2SohK1XXm/V33uwQ7hHnncYJIrhy2J3Yp/3oB6xG7ifa3c87byzzP/ORNwCXVi IPayZ1RN/POXx/zz9/YS4FDaJs244tfEB1u57O5wuYK2T526w2W/5p17wj/P+Wlcrnx6+oh3 7sH8qv3a2xt5sTce8OrPoD3jMjrA5cY/j6/883HNPy95iCdU4mRvVP75yrjb3emNR4MXxWa9 7Q1i04Za5Z/3cx6E6/WI1FzHYXRguFlyy4OgGv98nbN6HifM2cY/L+c8SJJSdUIlQoHBVQp7 B/azUY949bfdopTPeuNuflGQyaQ39Kw3VvNr2alBasY3vXHNW7EZ5QDD9/zzUw6MH+Kc6Z8A y9/NP/8DXE/zDs36dzHcvtGvN9PovG5g3vDB/mBpHdZ6bG/sj4vh7Wu66eYB/3x+zgh5LI6W TynE6X3QuIUQsn6s7zt5LRbyin++bZAYqS8NxAf88weCjghwuebJ2hXxAX5GX3k42cw/v44z fNi2M/a0qnKLsAw9pVO/8Mw/T2ho/PNZfXq24Z/3cUNX+mogUgoFAtAoo678869wOTqcdnzm kyHiUuNNxY031W14Uz2AG7OpAz+/+ZWA34DsMy53gFMccNkJxWf+eX+RlX/eLaDqUTghqT5e +ednXN7Y8/mhvfEwDrDiucWbQb/g8mxvxNTd8sIcxAHcn3/+adwf1yRyWL2vEfON51ZPvPoH en6vbEBYsx11t2Oyo5Y4wIEdNfdrFQeIK/75mkbxjn+eodi1mY4RVic04sQ/Tyv/vOwMekNv VAroT5YYwbsd/BNZP+d5jAr3YV0/v/S2JYHEhCXhXI9MRFjc8883xn3rVszvMSwbvviKGzHr 7wHb+ObOeIBfUSl9zz+/WFffm4hmGxfNPySxzYs4dj6JjF90lx/pjfyy+JOwfSK+kdNZYD74 bh6Xlk9EWFZF5lf3HdYH+eu5c3yeTI7Z6YFjlF/FAe7iNttEVxRbsMrF8ysI+ize74bGhSSa JxCrA8nBsWnyMkoZmTy3JsV2QQ047wJ2cuMvU3zeG3piIBotrYCivNfk4vRh3OZeDsa0+Dxx T8qvjqPaJbcS3jBHYovPG48MW/J/XWynqO49mBeA2y2/XujBV+uciJc5hEPN+3LKvUGCT3EA dovP53SvN+7ui/BjXN6ysR/la6MGTiOP4wCOruLz2GlMFlC9pjV7ayTO8fk5722XHuYpXhcl 2BSffzKXc5rj81s5QuOexOe7Kku8WE2dF5H7PlyazmfM+hMqccpD1ztU76ccx+d3cVCNACZ7 i/xTnUg03RmQFBoXkO2TCZIr6pBFk7xMstEHL2xaEFh6D/PLwlu1LUZIc0IntgI9TFEfnHJm 3fr0KBBqJYiqtuFpHS0Wf3cENA5aLnzlpgd7Cj6GN+IAltFtLsyvgGl+GLfJJ/zy/L7LfDYP 3euR+EYc4AkqPSv/K/bGj3PMH2XNy/d6Ix/mocnv2Af5ZcXuZketr0u8Mycj5zG/PRJX+zfx +Xyzo/KDCxUP/CEZWgAzeaahIXJTiWcVeGEHQFhTqjwmzfwkr5o/wAIk12BEszQcjvANtvg4 pZfhRrETdhwUGACVU4iaoVpqfje+QbudDGTnkuIW10i2x5ZvAA57YhX3ETR4xWXQIMQSv+Cy vHBQjgqt88PqHt7Uf8ytuoX737id04uO4SU/rO9Xe/TR4atwlej3RZLJdWzx+cWvvDUCO4mI q/jl851fefWXDZvMGPCXP2hvORILLguW3taJeRWfV2KTtz3r/GyexH5Ku7ctfooDTOsBpvj8 CiYAk70Li79c4/P7/LBilR8WUC7CAWGJz8d43JRDnbXE5ynYG106ytv+IMTB+YfbyZPYdp9o zg+7NbSIgflkQ1dXZkQfuXZ4nR9WRetMNUfm9FHU6ZpoFd6sbRE1P2x+YyTmOTinvbOGgGNu mJH3fmV+Ivu9eShIW05T88OCjrhbkENhJxjxrt3X0z4yWk37w/ywjG3vmzN2lKM6v4We3xmf f+aAPgeEI1yOR1/zW+j8spHmIBnqMdjl74LRL+JyPltd3sV7j5H2R4Q78r288otwT35nJG6z zLVY+Xv3YZ9bFlN47MnKkG086oG8bouUJvxXqNfvBzQ28Xm18U30HrTWolLfwk5qBlSk77Go iWKjwLf8sBXDGe4DNM5GRCquuR6pUJNeNnlRijgKsuWHbeqm5YeliV/j826Jz+eng2XZyhgU 6GnND4u+WXoUtzk+0eN99jxPbbrmNetvec0iAH7sG6Z5bHjLa7eNz9f8sLzlh1WTu+pJuOje duqaHzagcBSVyMcb+0BMW8+2y6OqXwz5db+udzHm+Dy55oddlh2kWJPH9tC6Wxyg28cBQkAz LhPfGROiusTuetvhySy87+Fsb8Ql7+19HDsfKNI823v78TDdMpmSx/brPHRd7ZqEejyZ5VXt Euji1d4gXmpCUMR0ic/X3LD1htOl5W1HPZYnYn21X1piWvPQXUx/P78ez1P2De9i5vyjt75H suWHlbj31zH7jSYGc06j3sH8olN8Xsfr/NpoFF8zut3yw8bop4RwjxX0Q9yJ+xlk3geIbeFx 9b7Xk2uH6pE+DFJuLTBfE+C8BKk3E5Ov5JV/aoT+/ThAPmEQ5IfOs3nL4T20Q05lxv3OOEB+ kKn+oV2mv4rLX5TXXYtut7qnTxat00a+NRDndcqP8sPuXxx0FsIkPpNlbvlh76IhEasUOsxq WAAjGzGZ85sLzkR9CluKFOWMy9VPgbed8n/qGZffNKclp6rmh7VWT/lh1+uxHwRTg0Yp9c9i Og2y4i3z6XKEFWKJbtfk71qkZZ1e7Ft+WFnzw8643AXsOmeWmiK2j+bMQQ+3+WHfXD9f88Ne yCbN977ow/g89MRf2h3chstmWT9/lx82t/ywnHtHl/ywEdmnvkrajoxH+WGfzq/ar3i0fn7b r2WdXt70C86sEXAwV3owmTbrAZBES35Y3/lekUhUH6b7RIQI84ZjlzdxACQbLrv35OXZp4q9 7PyDUTBt2OaHnRfY4944BBdQeYOpCL3AU79qZtUQo4W/La8rHDp9g7fz8D2CV+8fhz53bagR tlhDdTEGF9YtfbyuMtYqanTvuRJ2fldfcNBOb1t2Ww9iCDYdxQH4drm844zJh6By9katPh0v Pnvn+wvx+Te89VeQY97HrXz2VsQxLuev9e2wGflImTxdp3deMPkdXM4vjo1SvgrfHN5mSM7E PS4/uQmjYn4K9lGo+xto07c46VElYjI1lt/ySoPF7qqyCG0ZX9XzNa90romhUTOgmZJs7bA/ ucp1Qwi1wKqDFMfRv29Hic/wcAS0r9iw4yLoZTIINHXYreLzvcUCIT+ZFfoSegmOnCRuib9K 8Wp2r7rb844h03CZfPi9X/l4Rh3EATYn7NaZ55WTeVtnfo3P+zk+T+f4vK/xeYblAqqSphd3 eNab6v3lzXNt98/dPHRSnj/GWe+b3+yN7XCc+jWv01vbG2BH3eIAYGfYaOIUn89JsXjsfhyr xNneaPH5frt+/vk4jLt+5W01KMgHi8O7il9ce3xbvyGSjMbVtS5yqjcSpyhPAt70Fp9Pb49E UZ91MVBgjfob8XIcXs/liLi9KFdneMIODf5IMI3KR0pC4sQe4jIXm/g8hw2P/b+T4KyfDMN8 Gl4eQ0TtV9m8Bnjdfx/Kj3q58mteSz7648zzbpMt/ppv/pZ4/iAZ/SYr/f0R13z0B+ntX6e6 35zg1hv9fWt/Xj76fPDpf85HdE+Kbwn255kiTvHbTLs/n9CyMI6LoXhHFGMY8SNmlR8mYeIK pnLi3xAC3PbBMinLoGZ+G83FqLgIH417I4z+TIJzzfkojKS+UocXVBh378kr0uEZBwxLaAxi zyHrhZeNn42lUinb8ZXfJimHRq/slG4msaEPOIIHulxsczmVh85D+dznJY+quLJ/fyVf21Ve pC87/vlaZt/HhX9+aPw2aMunNwz0ynPbWzcMup95bmNNqfL+q+h+mPhtpjx0oC3e5e99LK8B j/XiR3HA5UP8lC+g8u/f5DXx3FKBRzKRGA1dIpIUKlG5NFIYSckZokAqcesXGXvxUfMUizf5 e+XHE34bLHHWdJ8XRjJmYHiwUaeAiS0IZsGa32bKBF3hapwTS9UtFcD2Oc9fERKNcy5pXeoI vJ39Sl5P6xkeHNC2DZuPB/lu9JbsKMGGr+e7sTz8v+Cd+4Wv3/36Z73+D6Resu8MIiezAAAA AElFTkSuQmCCUEsDBBQAAAgAAA6sIk0AAAAAAAAAAAAAAAAaAAAAQ29uZmlndXJhdGlvbnMy L3Rvb2xwYW5lbC9QSwMEFAAACAAADqwiTQAAAAAAAAAAAAAAABgAAABDb25maWd1cmF0aW9u czIvbWVudWJhci9QSwMEFAAACAAADqwiTQAAAAAAAAAAAAAAABoAAABDb25maWd1cmF0aW9u czIvc3RhdHVzYmFyL1BLAwQUAAAIAAAOrCJNAAAAAAAAAAAAAAAAHAAAAENvbmZpZ3VyYXRp b25zMi9wcm9ncmVzc2Jhci9QSwMEFAAACAAADqwiTQAAAAAAAAAAAAAAAB8AAABDb25maWd1 cmF0aW9uczIvaW1hZ2VzL0JpdG1hcHMvUEsDBBQAAAgAAA6sIk0AAAAAAAAAAAAAAAAYAAAA Q29uZmlndXJhdGlvbnMyL2Zsb2F0ZXIvUEsDBBQAAAgAAA6sIk0AAAAAAAAAAAAAAAAYAAAA Q29uZmlndXJhdGlvbnMyL3Rvb2xiYXIvUEsDBBQAAAgAAA6sIk0AAAAAAAAAAAAAAAAaAAAA Q29uZmlndXJhdGlvbnMyL3BvcHVwbWVudS9QSwMEFAAACAAADqwiTQAAAAAAAAAAAAAAABwA AABDb25maWd1cmF0aW9uczIvYWNjZWxlcmF0b3IvUEsDBBQACAgIAA6sIk0AAAAAAAAAAAAA AAAIAAAAbWV0YS54bWyNkkFv4yAUhO/9FRbt1YDt1LGRQ6U97KmrrdSstLfIgdeULoEIcJ39 98V2nE1XOfToed8wwzPNw3Gvk3dwXlmzQhmmKAEjrFRmt0K/1t/TCj3wm8a+vCgBTFrR7cGE dA+hTaLVeDaNVqhzhtnWK89MuwfPgmD2AGa2sEuajUGTctTK/Fmh1xAOjJC+73FfYOt2JKvr mozTGZXizB06p0dKCgIahgRPMpyRmR0afrXUwF5WstaegwZ8Kj3G5ZQuyPQ90zsnpb52gcgW JDZsQ5u+K+hvUXK6/sXCc8Tn7Q41eCMFiw7gOc2qlNYpzdd5wYqcZRWu63K5WFaLoiEz1ozl QaoQf1kqO9eGeDJ/WmfVj/v6uSHX559t4q/Q4Hn5H32SJ3YHBqLZOv6otg5+jp3JPS7wEueY 3j0q0x03v6tyUy6SC2RzcPYNRCAFvfvWKS3T/BTz78Qp4fy6fIglfVAiGfXQbjWkwnYmxI2h SRSg9Vkrs/ok2+2QNQ8oIrwhn9ZLrj1l/gFQSwcIYRjW2IYBAAAIAwAAUEsDBBQACAgIAA6s Ik0AAAAAAAAAAAAAAAAKAAAAc3R5bGVzLnhtbN1abW/bNhD+vl8hqMCwAZMly2kaa7GDrV3X AW0wNO32sWAkSuZKiQJF2Ul//Y6kJOs9itM43ewggcjnjnfPvZCScn5xE1Nji3lGWLIy5zPH NHDis4Ak0cr8+OG1dWZerL87Z2FIfOwFzM9jnAgrE7cUZwYIJ5mnJ1dmzhOPoYxkXoJinHnC 91iKk1LIq6M9tZQeUcqmiitwXVrgGzFVWGIbsuh6+soKXJcOONpNFZZY4LQuHrKpwjcZtUJm +SxOkSAtK24oST6vzI0QqWfbu91utlvMGI/s+XK5tNVsZbBf4dKcU4UKfBtTLBfL7PlsbpfY GAs01T6JrZuU5PE15pOpQQJ1oppynAEE3JV5OU1RXaaRX9tocnZtowGa/Q3ik/NMgZupsgim p8oiqMvGSGwG4ntmv4NJ9evd231e8XjqWhLboMrnJJ3spkbX5RljlalSQBe7Mtd1nBNbX9fQ u1H4jhOBeQ3uj8J9RP2KcRb3kQa4uQ0IC29lypdoLp0e1Pzc5jhlXFSGhNObHbDjVqW6ETEd LlU5W0IjHgS9UDBnYUPZQtFYW4J3zxq9bJz/pa1A9f41KjB3bImpSgDo3TdbHlXbQcjyJNBl p8nANynmRE4hqsS8hoZ6xlB2gMpiD6lpaDRWgmlZbZVLvWoYs+LMIgkkGUu9mnSj8rNsIfpC 8eG9LecsuatA3yzWqW2mrrkud86Qwa4ZIh9bAfZptj7XHa8aNvS1NHJlviXQPpXzxhVKIEmh LZXQmNDblfk9Sln2cwunB02joVrirQgn4DUUT7YjWdZApET40F62iBOVG/a4aa/wP+ivfNys GmaKSbeZwPFDbHrLNkQYr/AWJSgCmWG+WsCjWHfFcg5Db5AOkvHy0niPo5wiPmjnsMhXs9ge ystiXB/wSs8CHKKcFse+UnNhsmoplo8pNUt4ijiKOEo3Vgp1hbkgcFbUU4AGLSy1ApIJlMhj 4+I0FXsSZTF1xZSZA8URMo+iJMpRBLM4UQM+tA/BwbqPV2ZbhQVdGyXtVFaYUk8J+bIpZwqF 5cTLy65aeW6g+GZccQXakLbqauqPSxWhHurX5/poVZywGvHQ7Fw6ZgtkFFcxSVSni0AuIBER GbQotVCPzjIa3QVeaXvMu7PgjkiWVI7WRyfNC6EH10mpaKhayvmqZobD3dN+OqtW4AMb0V7B oMEVolHmtTAOx/QNRvKGbySmRlXY1d2f1cyHoairYqQM7gaeOeqj6lPZnZEvoMA9gfLfj+l7 wQTOpYjuh3eYRBvYhK8ZDe7t1yfX+TQvXYC+k1J0azUQxvwg30veDvZ9fnYP34vxQ7x3x713 n8h79xG9/wDWPCih7WHdl0zgg3Qroyq+KokWa9fI/xxxeQC2SgJD+Pj+PoooYgmi1jW1BJfc JLgzJ2C6mpNaGQ/kTbkzewEVZ2SMksB4dubI7/D2WwviQn06QXQeMYivGRPJ1yF72K+Cgil+ EbhrIP7D/boSSOTZY6Xn74wFB+kuzDokQX1fJeiUPHKc09OeZvCYeXSJc8ERPTYpRdVOIWW5 PDopv6KjZ4mkYyIhvt+7ZTwmIX8jnhx6DGqT8k059hvnjB+/Iyg/p8Q6VJ8HUXK/g+Evvg+e Ps1599HdGjnuasCBp92CtENyoWDiSLlw3/LY0zZwTi5oO+yY/ADa7nE4e1LaFqO0LY5NW6A+ XUfs1rOt4hLlgsVIwE12OVEaFWELvGG5aPj9Lo3nZg+o+6xDvr6ADcWKWQBylFvieh/MDdw+ tR+3FGMhHHvhT9NR+fxmU4TKnc/c0yK0MeIRTFEcwoTTHOQa3hq9ZkLItyMvZs5SP3qzh00q bHkCMwVLe2xs2mN3ojAheu7/KXr1Wzt3drLc39rVtp8UBfqNvjN7Pi+FerZsR34rdmoIEgNT nUTpePofyKRvk6+7s9oe7FbFRIyySkXVw4pBqWnsKW69DnqanDZ/fa7+fSIt/mYbjDV6fXFx cW63B4uRtEVCK/Iyis3NY2WGiGaN1yqSm2r1P6UvxYU0Wz+2Xs/L9WpjHRNKVQ3SR02wOzze Re374lXxCLNuh1l9xXFEmM7se5Jt/KBxgghah+jrHztENFZqDKkKaq0eIFH6qN471/fpSzgV VSBri2gun+k68zPLWVoOeOo4tvpxnMIKCVz/ZJQGx3erlqBK9cJbOJ67nL1wYBX35ORELuGp n8rpvvRr+vdkOWnYdaB697NeLutAPfZIuWv3dwu7/x+81v8CUEsHCINp7dSSBgAAICYAAFBL AwQUAAgICAAOrCJNAAAAAAAAAAAAAAAADAAAAG1hbmlmZXN0LnJkZs2TzW6DMBCE7zyFZc7Y QC8FBXIoyrlqn8A1hlgFL/KaEt6+jpNWUaSq6p/U465GM9+OtJvtYRzIi7KowVQ0Yyklykho tekrOrsuuaXbOtrYtisfmh3xaoOlnyq6d24qOV+WhS03DGzPs6IoeJrzPE+8IsHVOHFIDMa0 jggJHo1CafXkfBo5zuIJZldRdOugkHn3ID2L3TqpoLIKYbZSvYe2IJGBQI0JTMqEdIMcuk5L xTOW81E5waHt4sdgvdODojxg8CuOz9jeiAym5V7gvbDuXIPffJVoeu5jenXTxfHfI5RgnDLu T+q7O3n/5/4uz/8Z4q+0dkRsQM6jZ/qQ57TyH1VHr1BLBwi092jSBQEAAIMDAABQSwMEFAAI CAgADqwiTQAAAAAAAAAAAAAAAAwAAABzZXR0aW5ncy54bWztWltz2jgUft9fkfHrToohKY2Z hI5xuaUhIbYxid9kW4AaWfJKMob8+pVN6KYUb6ixO+1MeQEs6TtHR+f2CS4/rkJ8soSMI0qu lPo7VTmBxKcBIvMrZWL3Ti+Uj+2/LulshnzYCqgfh5CIUw6FkFP4iVxOeGszfKXEjLQo4Ii3 CAghbwm/RSNItstar2e3MmGbJyuMyNOVshAiatVqSZK8S87eUTav1TVNq2Wj26k+JTM0P1TU ZvZrUZTSr4LSBRtlMmENVT2vbb4rJy9KvjJNQ2lv7bDdfvvyRcDm7RQJGKa2OXl5nKp2pUiR rSWCyVerKfvWfbvGQRx5GOoMAptGynZQrCM5iIhQ2upl7XuQHwK+gTNRDfIUBWKxD7qhntc/ HA0/gGi+2Kt6s3lxdrD6pyGIThEJ4AoGu7Jgsv+UsjXSv9j6EI1hMgx21OSCSRdQ2qlD1H9I 0xR0R08bSIu8pei3S7gA4hAHNGLGKRtTjoT0/4d9xj4vdpDfIj/udZNiyAPK0DMlAmArwkiM aAB3rb+g7AgPh0wgvyr0He23BiozQl/rXwG+7gu0hBm6Ccg8xzwFD3erb8lZawtr5qWUI3HL zd5b1A4VgoYlAruUhrZEKdWjU1AH4HgXNVO0rha1AZjDNLX+L3qzILi1oEmfod2s7VGKISBK W7AYFgwO4ktjwsCGK3En24oZpskNnAN/nSdrBjDPEbbn4evKlDeclZBD4zgrLjnVK6siR6c5 n1GMPcBy24X6RfPgGvnHn/P8ORXQkW3T05jBtO0o4G+HiHEho5n+vPzgSfFvqagKupqIT1EN iinbgcY0DaF6o3nWaLxvlnCuFVhlALhUPQ6JSZMBBIGkQZUIsRYQCplpKkAf8rtYSOoIrXXo UcwtuFvfSxGSufxAdg847SBkduySNHHm+tMRUTbkFgGRTU3ABdz1qjIEbIDlnjZcqzIJJuTS uXKJRb3xoWC624Xfyy6OhbdiL0BLxHPVLwl8v/JFXWcDr68Qt9ayFWGUoOd8L/29+5wXUr9/ Aofi8HubzYOYgdSbfuQCRxpZgJWV9UsmnO3vQwpehehY2nQskcU19QxAfIgrqItgCZ3N7dcd MTDlVaSDIf8MGdE5AmQcE1/EYA8lLUOQsQAM+DIADBpGDPJ0W6X3h3oU4fWEQ/YJCFD+iWQn LjMEFPEup/QAh83zDiKArZU2uh/8XVMD7IXOGkxH88ngOvKIif25/ku+JmrQs3HHct6eOtX1 kf7Pf/sY6j13It+fvHtdt0INmf2e+mjpK4N05N7fq+7DUDMbTuw+XEeP6869H+I46DtrI9Tk uCM/91Qw1eKx01n6xFw/TrFqhLdLv4+x/6yujLC+8MMg8kJzAYjzHPTr2CP32shIkptPOh/p 0RevsVr6obTvwKRje6hK2c9e32m400ST44nbd5/cBzd6bEy01/ODEH9xbTUxcOfe7N4u0zOC XXMR9LufJ/0ecZ3bCIaTpt131FRnuU9L796ObafTcXpa13E6I9NZzOyJ1jXvL3Rz4CRS7rk9 1Z5cmzbthhse5Wq38ksOA+xDAhnyT15mFgyYWFADYD/GQOSml+IBc4PI0yQKJHb+Zd3ZL9wf 31AQmBKcElykeL4FvzFNj0mmDMOomiOomhj+zhyrOub5k6jPp5cf2yxZXCshPBXS/m7owaBH icjFPkLvaoltpUThDwV8y+t/CgeviGOWRwNzmVXtu9/Ia3n/Hmj/C1BLBwhL/LklFQUAAH8g AABQSwMEFAAICAgADqwiTQAAAAAAAAAAAAAAABUAAABNRVRBLUlORi9tYW5pZmVzdC54bWyt U8tqwzAQvOcrjK7FUptTEXFyKPQL0g9Q7bUtkNbCuwrx31c2TZxSDDbkto/ZmdkVOpyu3mUX 6Ml2WIg3+SoywLKrLDaF+Dp/5u/idNwdvEFbA7G+BVmaQ7qnhYg96s6QJY3GA2kudRcAq66M HpD1X7yelO7Zg4G9OO6yWa+2DvI03w8zuo7O5cFwWwi1RDKXPVTW5DwEKIQJwdnScIKpC1Zy MiwffUoKPZiKWgAWaouVcxv9NxrrSPEtlAGbBSvWmwbU2N+k8tFhbZvYTzvQXq1clCLK9AIy Wlk+MmwTT6M83igxLegyXFmN7U28Htg8nZR4cEDP9/pbk31Vrzh+Qr1stw7M6QOuNX9Q/77n 8QdQSwcIOrq90REBAADZAwAAUEsDBBQACAgIAA6sIk0AAAAAAAAAAAAAAAALAAAAY29udGVu dC54bWztne9v3Dia59/fXxFkgMPOC5fF31SmM4tDD8SVMLMvdvYOCxwWg2q7kvjGdhllp9P9 3x8lKsUnTqXqeWxRUlUxmXF321SZ4vdLfilK/Oinf/3t7vbNr6vN4836/v1btijevlndX62v b+4/vn/7v/+zurBv//XP/+On9YcPN1erd9frq893q/uni6v1/ZP/5xt/9P3ju/DT928/b+7f rZePN4/v7pd3q8d3T1fv1g+r+69HvYOl33W/K3zn8en3W/ThXWF49NPqtyfswW3Zb45d/oL/ zV1hePT1ZvkFe3Bb1jcqPPzDGnvwb4+3Fx/WvtXvHpZPN89q8dvtzf0/37/99PT08O7y8suX L4svYrHefLxkZVledj/dVvhqW+7h8+a2K3V9dbm6XbW/7PGSLdjl17J3q6cltn5tWVil+893 v6w26KZZPi2/U/Vhs3r0RfzptsbEfRA85ht//foR7a5fP/6gma8+LTdon3WFv7WKuMZbRVzD Y++WT59+oK+9/Jv/Yfflb3+NvtrcYX9XW/abprra3DygTzOUhsev1+ttVdsDQmfvqsuLQl6G /walv+wt/mVz87TagOJXe4tfLW+vti2+vtvVaL4cu/QlLla/tpbfdqK2IR5/cAC/DD/eFn68 /uFH/9ff/vr3q0+ru2UsfHO48MXN/ePT8j62zKYV4Ydnqi43q4f15mnbMB/wg69Xi2/r9unp 7vbHQ0f7069FP26ur3cW9dURl34Y8Z344teb1Zc/fDO27vdDedkVguPp3gNYcdmW2XZJL3cc /Dcft/n0Yf35/joMA6ExVr89rDY37Y+Wt91h7775BOjg2/ULPrLPNPAJ3wz0N6vbr71/e0o7 P2a9vrh79Gbwpl8/vANHf5sbm7vfcB/XGn19/eH5Jz7r9FePj+Jpl7b/+R+X7c8u2tj0wdD/ JjBd4G///HVuEMaCx8vtNz74OcLFh+XV6uJ6dXX7+Oefwhi//fab8N9tvd+//euND4yued/8 fXnve5ofiL8Wvbu5/f392/+5fFg//ulZufDNt2+++ei2/MXH1b1vCD9cPH65eXz8psTDzdOV H1B/XW5uOvdd7q/aX1b/b/l/Pu+vFiiDqdLvj0+ru9fU6a/rTzdPb/6y+nV5v/zoj/lxez0r OErt/r7+vPHf+rdlEOnNz//+5j9WHz/fLjc/rOePDxmsxpc/8mX//eXnp7WP2puri+5ztobt vn5zfldrtv1l/Vl045ifOdx+vrt/+/VI+M2LB9//Vpunm9Xjmw/rd79sVst/Xvyy8l3Rf2D7 q79+Yl/8y811m/qMsYWSD0/dCYD67KscH6tywiyYptVNjFU3aReGWDc5Wt3UwhI1VWPVTctF wSh12/yoM2zWX55VzH8H1ir8qP3mp9XNx08+gRhf2PaX76/w58fVxfrh6eZueXsBj37afF7h 6/203F3vr9+889ckq83Fw/Lj6uJrEHxYfr59enZS4ITCBeL1zePD7fL3vj79p7UTWX/5d3G3 vvafdLu5ePrl+6qGy6b+6uni+yr/exu33xR60/+XH8m65vC/+Grlp7F+CAizkLub+4sdP+wP a3/aTg0++l93ffPx5sn/lHX12lGVPfZc/Wg8XN3efv3Jw3LTrhx0/7G3QduDnpt3vbn21fhl /fTUzuqLReG79pvH9e3N9Zs/FN2ft6Dc7eqDt8P9+n4Fv7sJLnn+7af1Q/9NfHdc/WiIHep8 l1f//LhpZ5xtZ177K+k/XHd/KHX80VA7tiY/PJdRFWNHZNFDB7+sAVJ79tsGeH5qE5zwuB0g ntqrDZ+mOX40vTm68SBN8/xohpXSLf0xPqgvlrc3H+8vHrvLoPdvP9z89vWnm9XDarldcfc/ Wt4+puhZ/a/7ta3ylZ8tdBV6//bu5voaXvH5Nlh+3CwfPj07v3gSbWu2yw2k1tevav2+RLcG BX/eTZiObtwy82uLmfZZO5eWOrbRLmUWvGJQm3IIet34v8dLl/t+64Tj3hDXGLMZquZx+XH5 w6XE/ge/rK9/jyvmD5vV8vrx02r19OefwtpBez/i8223vn3xuHpqlwy+LivEz/xw40/mdvnL 6vZx23NCmXZlZBOWTS9Wv7W3Q9uF+l2lvtzcXl8tN9ePcdkk/LD72hcMajw+Ldv7ZOFbUKZ2 BeWbw/qVpx1FuxXTfnkkiB803eWKS/Rn8oOf2V324D9QJKikTPCZh+vZTanyiZ/Viavnn7dZ f9nxYe2C7bPf6z/reVX8t3Yc+6OK7y4drqq+3jVc3n5eXTz9/tANKRs/sr198/X2666f9UNQ WIAMLfN48fiwvL9fXb9/K54V8KcKftqdYPvBD39e+1H85n55+9Nl/42fLp/XfDv4rv3MY3V9 saNRvv/R7tP9frCZb8vcr77MrVHkDNrkzb/cr998XK3vVv6jr97crZb3fxyhnUDP+uZ3+Kq+ vFPv1gPvUfZSPbYN2k0eEM13sM6vrcjqsb1h9LSaQ12WV0+fUSNSen1Wm816M4c2yfpkfUbX ZwR5BhzMX1nXu6tf//HkZfwHo7XcaxPA/9ct0TY7fuWH2/Xy6Vu9fHILafnuaoTy21p0JdNU QoW1BkQlupIvsW37wJ6/qH//dv3h3fu//a//+pf/u/hZ/Pdl+Le/iP/+0xv2xz+96f71svvR H3c6e2f1xcIUVkurtJWaaYU6E3/Q6130A0k1L5GS+pKnJGm1ldRFSV0naUWUVGsvqTFCSykF VlJtkklaWKykhT0lSeutpE2UtOkkramSGimKkvOyVRbXQw5KmsMpYTiVUuFc3xZMUwWJNr0c MprkNppkjCbZRZOkmF4uvN05F6UwsjASZ3p/kEgkqFYaJ6gveEqCVltBXRTUdYJWREGZ5Krg hWFcsUIiBWUykaC2RPZQX/CUBK23gjZR0KYTtCYKWkjGeSH9/NEoaZCCFvsFzbGUMJY4Vxrn uK5kop6Htn05ZDCpbTCpGEyqCyZFm435EUFoZgrOuNG4VPAHJbpc4dyWuDp0JU9J0morqYuS uk7SiiqpKCwzmhWF9q2ElTTRXIMLViAlbUuekqT1VtImStp0ktZESW2p/CUTs8JKi74MLvef SA6nhOGkJHapoCuZaL0CPyljg1436W086RhPuosnTTE+XygtDNcl11Jh46k9KJWoAi2qODVR q62oLorqOlErqqjGWFsWxjCmDMOKmmo9TUmFW3rvSp6WqPVW1CaK2nSi1lRRuTSFLhT3kw9k L2kPGiuivi/YbX6bKLl21abd5vPa6ny+fbp5fU12t8vu62bfk5GX/W3JUes2wkr5MwXN2z2h aLahaGIomi4UDaWrsUUhJW/vifgJvrYCN2ocWqoYvPGl0rhrj67kGRuj2hrDRWO4zhgV1Rgl E8gbVL7syG1uLW7+1pU8bT/oPXaot3Zooh2azg411Q7MGORTBYsiJ/GRJ7EsJK6DdSXHrVv6 mwOUJLbbJLYxiW2XxJbWw1jBGG45pi07apuLkgvkem5b8oz9UG394KIfXOeHiuiHgpXSMJ/C SlqDTmLiReVrjeFrhlxCbEuetjH2JvHWF030RdP5oqb6QjDBtZLCaF2iI3n/knyO5LlHsp9+ SewDY13RUWs3wk0RSiiX21AuYyiXXSiXtM5WGobtYiXxScHXNrmwBW7o7UqesR2qrR1ctIPr 7FAR7VBIzs3XP+hMHne1xJYauRDelTxtY+zN5K0vmuiLpvNFTfVFIbRum1ObAtn47UFTZvIA IThkJr94l+PQmby7XXbfqDBKM9zzeqHoqLUb5T7QMxXtvlRmxTaW/b9uc7n9dvgpLZm19aOX 5Fb5r9iF6yEymiSBLpHP/4WiZ22QKhrEAYO4YJCKapBClppTDTLERIkigbCiQNatK3rqBtk3 j6ujPxrgjyb4oyb7g/m5XMF0ISV6w5w/6Nieg3r9UwZzeQ6KI5/QTfR4LkP3hQGffmIsZiYD mclCZjKK5RX2vtShmenLmxC5iTHRI8FTKFhFBR1Q0AUFK6KCfKEFU8b6qYLQFjf17A5K1COR D1yl2hY7gZ511LMBejZBz5qmpzQLzcpCmv4r8uGXhR7tBmtOoe9Mh3Pc6Tj+Z8ZjBnGQQTxk ECdNuygnkC6Hzk3DKmrogIYuaFhlDY9Awzpq2AANm6BhPaGGOW9y3gyZN5HiwgDGhQWOCyMh IvJYNVHeRGwL0NAFDaus4RFoWEcNG6BhEzSsJ9Qw583prrLhfvtwWROxLAxwWVgAszAS9wHd csnGqLPSroraOaCdC9pVR6fdtGtp42pXR+0aoF0TtKuJ2mFbLufKhLmC5BqdzvzpZxbJKgyg VVhgqzAStoEjb97whHdvzk7BKirogIIuKFhlBWevYB0VbICCTVCwnkzBnDR5xWzIpImQFAYo KSxgUhiJvpBXWybKmshEARq6oGGVNTwCDeuoYQM0bIKG9YQa5rxJmTe45zZYIpLU2CtmkT3C AHyEBfoIo2EFsC2XboxC7rgmPhY8T+2qqJ0D2rmgXUXUDtty6bQ7p35XR+0aoF0TtKsn6nc5 V1LmCvKh9EQcubFzJZI0GEBpsMDSYLRN8tiWSzc2nZN2VdTOAe1c0K46Ou2wG3JOQbs6atcA 7ZqgXU3UDttyE+99HZBHMQBv5Sh5FIluNb+yVsmv+ikUChYxFAxwKFgAUTDSFvN0e3de2eKl Qe7nN+frhCo6wQEnuOCEiuaEluVTlNxYoQrFFJYYNsTWUYotSqQtDoDlj9wWe/ET0RUNcEUT XFGTXcHg39ltZ80xnMjPSDefci/bH8M8Yic4wE7wgJ3gtF3j+PPKQTw/L1TRCw54wQUvVEQv 5CA+HlvsDeLoiga4ogmuqMmuyEGcgzgH8Y4gjiwLDlgWPLAsOGnnfA7io/ZCFb3ggBdc8EJF 9EIO4uOxxf4gjmwN4IomuKImuyIH8RkG8TwXphPe1CFFcER5cIDy4AHlwUkIgXRbpl47oqV4 7vAk9K+i/g7o74L+FU3/gTC3Wf0B1d+frpEfAsRvgvj1JOLnOJ17nCbZMnRcU9YDkRppJRzQ SniglXASJSHlHrF8VTvCVW2kngAvuOCFiuaFfFV7RLbYn7uRowJc0QRX1GRX5KvaM4zhOc5q 5zT0/swjyIUDkAsPIBdOAkqk3T6Xg3iEII5oGOAGF9xQEd2Qg/h4bLE/iCN0BriiCa6oya7I QXyOQZxmw+ArazWP5cWfeSTecEC84YF4w0m0jXT7C09wKjYP/auovwP6u6B/RdQ/Ly/PTv39 6RpRO0D8JohfTyJ+4jgd8BV6Q8SpnEucUl4xNVZ3ItRpvMF07+vyeIT6cAD14QHqw2kwkREH 06z+AOpXUX0H1HdB/Sqrf+zq733RXRS/AeI3Qfx6EvFnSHp4/f7r2ZAepiRlWZzn7WCkBx4J QhwQhHggCHESyQRd+dOknI2tXRW1c0A7F7SrsnYz1q6O2jVAuyZoV0+kXc6VnCtD5UokCHFA EOKBIMRJJJM8No2cK5EgBLRzQbsqazdj7eqoXQO0a4J29UTa5VzJuTJUrkSODwccHx44PpzE 6chj08i5Esk7QDsXtKuydjPWro7aNUC7JmhXT6RdzpVTzZWReaciYmkEwNKIgKURNOhEogX+ rNv3iRJ1c0A3F3Srsm4z1a2OujVAtyboVk+iW86SU82S0a9RRGSrCMBWEYGtIkjkhDzPHfca JWrngHYuaFdl7WasXR21a4B2TdCunki7nCs5V4bKlQgMEQAYIgIwRJCYAXlsGjlXIuwDaOeC dlXWbsba1VG7BmjXBO3qibTLuXKquTL22ldkZgjAzBCBmSFoO+LzGsp4a1+RbwF0c0G3Kus2 U93qqFsDdGuCbvUkuuUsyVkyTJZE8IMA4AcRwA+Ctqk7j0njZUlENADdXNCtyrrNVLc66tYA 3ZqgWz2JbsdDTRiACXKc1IQZ7ppOek1PoSaISE0QgJogAjVBkDZOp1unea0D8BiYs3RBFV3g gAtccEFFcwFbcCymiI8LKmEF8hWhI+OTRrTBXohCdEEDXNAEF9RkF1iD9kGZIzZH7PH2qgMR G1kKArAURGApCNKG6hyxR+qCKrrAARe44IKK5oIcsTO2wf6IjXAF4IImuKAmuyBHbI7YHLE+ RCPVQQCqgwhUBzETMkCO2MQRG/kQwAUuuKCiuSBH7IxtsD9iI2kCuKAJLqjJLsgRmyN2woid B1z1ZxHxFgLgLUTAWwjSNvsMVz0y9auovgPqu6B+ldU/dvX352kkbADxmyB+PYn4OUpzlM56 gnogTCPTQwCmhwhMDzETLkS+Wk0cqpEOAlzgggsqmgvy1eqMbbA/XSNnBLigCS6oyS7IV6s5 YnPELn6WEXAiAeBEBsCJJAEXcsQeqQuq6AIHXOCCCyqaC3LEztgGeyM2uqABLmiCC2qyC3LE 5og9h0WhAwEbmS8SMF9kYL5IEnsiLwkemfpVVN8B9V1Qv8rqH7v6+/M0YmeA+E0Qv55E/MRR OuC71oaIUjmXKM3vW0IqtvddazJibiTA3MiAuZEk3EZ+29aRqV9F9R1Q3wX1q6z+sau/911r UfwGiN8E8etJxJ8PG+HTzePT+uNmeReSU9K0eOUvT0lI4FyitPLl0lRAMWlRNWgLvqQL7Iy5 yN2RgLsjA3dHkjggnC9Ky5nkVhn/tUQ2pz8okaIlx1WhJL5rF60otxqnqC84kKJVVNQBRV1Q tKIpyuyCFdyI0nDt/4nrIN1BiRQVuPHUl0ukqMD2UTFYH62jog1QtAmK1sQ+KhZMxb/IDtIe lPNpYoIPciqRyvmTpFMk+UhA8pGB5CNJZBGpcfX35dJRYXBxwIhhMO88ilQfoKELGlY0DYUo FgJ5p6crm2hmMXVfnCSFIuUH6NgEHWtiX7R6wWz8i+uY3UE5hs4YJDdNCEUYjwQwHhlgPJIE 4CCdwGnCyaYJoYjSARq6oGFF1JByAieq4SQBFEE4QMMmaFgTNaScQF6cO+/UkRb3KJcvN1jm RDqNBHQaGeg0kkSkIFT/VEcr3H0FX26wvIlcGaCfC/pVNP0I1T9N/SRyHVdS13H3ZE0kwgD9 mqBfTex/+OrnpMlJM3bSREiLBJAWGSAtkgRmyEkzQdJEvArQzwX9Kpp+OWkmSJoIRgH6NUG/ mtj/ctIcUdIg772l+eUjv5JBRlKJBKQSGUglkoQrQLZbshEKl3Gvf3XIDHSrom4O6OaCbhVN N2S7JdPtjPpbHXVrgG5N0K2epL/lRDnta5exEyXiOiTAdciA65C0LfqJntDNuu1IlAjYALq5 oFuVdZupbnXUrQG6NUG3ehLdcqLkRBkwUVSkUyhAp1CBTqFoO9LzyDRaokTdHNDNBd2qrNtM daujbg3QrQm61ZPolhMlr3oNmSgRx6AAjkEFHIMi7cnOq14jJkoEKQDdXNCtoumWV71GTJTI QAC6NUG3epL+lhMlX6MMmSiRSqAAlUAFKoGaxb70rNuORIk8AaCbC7pVWbeZ6lZH3RqgWxN0 qyfRLSdK8kTBPd6faJPe2IkSAQAKAABUAAAo0uZiZLslG5lwxIH9/Mcj0a2Kujmgmwu6VTTd kO2WTLcz6m911K0BujVBt3qS/pYTJX2isAK336gtmKYKplS4ft4WHCxd4gZ+BTbwq7CBX5E2 DbOFKEQhFbNay0Ia5HSK+qgtXlKB7H++YJoq2ELglinagoMFT9zPDyR1QdKKLqktS11Yq5XB 7qpfpKK7+M6HYwu0BVNJKnG9tC04WCbFrf1A0iZIWpMlZVZyo8JXnD/9Qfv9OWBEfV9wCCr3 65JrV52Ok81dcIXl3vuSo9YtOTyAQupWESqgAFRABaiAIm1mZouyZFoaLsNXZJcrXzR+vNwa UmvkmzHakuNaIzWTgILxjtZwwBouWKOiWYMvjBFCC12wUhuJO8n2oHGt4Z2LdG1bclxrpEYd EBjf0RkNcEYTnFFTnSF425rKKi1K5KSD55w+jZyW1uCun9qCZ5zSEcOgAIZBBQyDIm0DZ4tC KaV1KbUthcKdpD9ogLwhNL+22KtaS72qPamMjnwHYAwXjFFRjcFLwVvKqmBMWNxIfPA9RoOP GBKLaZMHIG0nndCRGwF80QRf1GRfsEIVXNrSMomd1fP9ayM5oI8joLkpcQHdFjzjgI70CgXo FSrQKxRp9zxbWFkoKaQfiMsSGYPtQaM2v1ASyT/1Bc84oCMWAxjDBWNUVGPIkjEmuVSSaeSN nPagUZsfO2CMPl7MKZ4jbAO4ogmuqGmu0AvDCqb9nI1pbQ32XqzJ+XwK+cwUkrXSFhz50j4p Q4qUzpH6oQD1QwXqh6LRI/wFjy605oora0rcVZIg3/197bQNN02nvkLm1UNwUiwVKZcjUARY wgVLVDRLcL4orIh/cE3fHjRuMONiedxBIi3oihLJkVQCDNEEQ9Q0Q5hi0b97qPuKM0R70JG/ Ejpncie/RV4Z2ZEvjOaUyJGaogA1RQVqiqJRHBZGltJy6WfBSiPfzOQPGrfxsa8hG/dibU6J HIEswBIuWKIiWsIuuFFW80LbUitcd2wPGjeRGXLqPvLEfT6ZHFkvwBJNsERNs4TEjQty6rdJ 5xQeyMZaaFzctQVHrdl47xbem8E6cmY04MzowJnRJO4FqbEHecMwZUArcC0uR67XPHxQRR84 4AMXfFDRfEBo6rFdIDRuadSXO00X7I3aaIIGmKAJJqhpJiC0dA7b0whbrnFzbF/uNDvXgaiN AB4NADw6AHg0DQiCb+qxh1iOu8E28lXWPDxQRQ844AEXPFARPYBt6NEdgGtsfqITrf0RG6lA wABNMEBNNAC2nXPAnkbAznVNeR6D68868og04BHpwCPSJD4KoalHH15nuYg8Dw9U0QMOeMAF D1Q0D6AbemwHzHPJeCYBGyFJwABNMEBNMwC6nXPAnkbA5uXiQxEbAU0aAJp0ADRpGjAmLxcf qw+q6AMHfOCCDyqaD/Jy8XxdsD9qIz0KmKAJJqhpJsjLxWcXtnm5eH/URlqVBrQqHWhVmsTB ycvFR+mBKnrAAQ+44IGK6IG8XDxLB+yP2AjDAgZoggFqogHycvF5BawqcOsXvtxpdq0DARup VxpQr3SgXmkSwIbQ1GMPr7bAMvzOc8E4Aq6AC1xwQUVzAaGpx79pgMRtlWd5JRtZVsAETTBB TTMBoaUnDtrXZ1qCoJVzCdrdrbO734sSt5LZFhy1Zsk5y8/Us3sDNwKsNABY6QCw0mSAlVS8 lFqookADaaik3Nc2vxHIDZe+4LiWTU1rJhijisZwwBguGKMiG0NYxq00QmmrsGDuYoCxi9D8 jJVIy/qCIxsjMfP525rtm5vV0RcN8EUTfFHTfKEXRjJe6EIUTBUGidb2B42V0MNG8euJ4HN5 W8Gkb3jiFvfrfbmX9IWdKRkpUhpQpHSgSGkSFoZQ/dN8QxdHoi19uYH0q6J+Dujngn4VUT98 9U9UP+yq9cvuD+wMnYhlAvo1Qb+aqB+++kf7XpycNEebNJGIpAERSQcikqbhb3LSjJ80EV8E 9HNBv4qoX06a8ZMm0oaAfk3Qrybql5MmJ818kyaSfjQg/ehA+tEkhkdOmgmSJmJ5gH4u6FcR 9ctJM37SRIYO0K8J+tVE/XLS5KTB/XLcEveL1rd3pYyJLBsDWDYmsGwMjWGCq/oLb+Pff777 ZbW5uFrffr67f7zYrB5Wy6fVte9chFsIKZT9cQTExnWgcV1o3IrWuAO9fP44O0WS90v/eOiP ujVAtyboVtN0O/VXPOdh/yiH/chVMYCrYgJXxZCQCnnYfz7sR2AJaFwXGreiNW4e9kcc9iNn BOjWBN1qmm552M/D/rSPcu4c9iPtwwDahwm0D0Pb6H/OI9O4ulVRNwd0c0G3Kus2U93qqFsD dGuCbvUkuuVEyYkyZKJEuIUBcAsT4BaGtJ89j0wjJkqEUQDdXNCtyrrNVLc66tYA3ZqgWz2J bjlRcqIMmSiR4WAAw8EEhoMhbd/OI9OIiRK5C0A3F3Srsm4z1a2OujVAtyboVk+iW06UnChD JkqEFhgALTABWmBIO5XzyDRiokTMANDNBd2qrNtMdaujbg3QrQm61ZPolhMlJ8qQiRJ35Ruw K9+EXfmGtvk6j0zjJUrcNA90c0G3Kus2U93qqFsDdGuCbvUkuuVEyYkyZKLEHewG7GA3YQe7 Ie2gzSPTiIkSd64D3VzQrcq6zVS3OurWAN2aoFs9iW45UZInii1x+3B8uTQVkFziqGdtwcGy Je5ZN2DPugl71g1pz6xcGC5LKQUvDOMahz1qD0okKHJjlaXuC8MLanC4rLbgYKETN7EDQV0Q tKIJqhZMK6lsKQwrbYE7GX8Q8WSw1SkZjjLlyyXrocgH2n3BwdIo7moHgjZB0JraQ7WwXDGh NeMFxxHr/EGjQbu+L5iAXz0As/Ao+dVSYccj9bLx6BUvLUi6hZ5CsTZxa70BW+tN2FpvSFt7 mV5w6/ub5n6AR27Tbo8ZFxxtDJKd2BYc1xZpd+ZTsNbRFg7YwgVbVDRbiGLBi1IUXHClLMM1 fnvQAOYnvbwNCVttC47ri7Q7/gmg62iLBtiiCbaoabbgZmFKYVWpZcnQp9gelNP5BNIZLTkb xNbHmc42IgksQBLYgCSwNCTBQpiCFdpoprSy2K2qI491rES+ya0teK7pHG3hgC1csEVFs4VZ aF4wpQphjR+Lkbh3f9C4tiiwtijGtsVswjm6ogGuaIIraupgoY2S23RGLtQt9P5zzNl8JNks cCtPvtzZJnOkRlhAjbCBGmFJG+T5ovCXRfF/uHP0B42czMibO+W4121zyuVIuwCmcMEUFc0U bCFUYawSshBMIq+a24NGnsUj5/DnmsmRowEc0QRH1ERH4OZm7ODLwXIMH0kMF8gYHuK1RJSF 9XSAJlIER4KHBQQPGwgelkQU4HrBse849iVfcmr76E2vNUo6qBMp/iKaAwjigiAVTZBUDwC9 NllwHhl34TodGoqSdJHvAcRvgvg1TXzc5abIOXcaOadwI686z5SLVBELqCI2UEUsjXIgFsg3 l3Ulc8rtTLmICwGCuCBIRRRknik3mvWPL+UicwSI3wTxa6L4C2RfXKicc6eQcxL7nl85yHt+ 5/ga7QNZF3knFvBObOCdWBJ/gdTYYw+wo0XZ0Xmgih5wwAMueKCieWCu8aqQT6KqcS8kR3PA /oiNEBZggCYYoCYOAviWzheTpxGyTErktMoXPM3udSBkIwLGAgSMDQgYS0NSUBo7h+xcPFBF DzjgARc8UBE9MM+QNbh6mRPVf3/ERioNkL8J8tc0+dHtnAP2NAJ2nqu18xhYf7aRhmMBDccG Go4l0TnQDZ2DdS76V1F/B/R3Qf+Kpv9Mg3WOi8MzidWI5gHiN0H8miY+9vH8HKonEap5afhQ sEYokAVQIBugQJYEKclLw0fpgSp6wAEPuOCBiuaBuYZrXhreF7CRVAQM0AQD1MRBIC8Nn1vI 5qXhQyEb6UgW0JFsoCNZEnslLw0fpQeq6AEHPOCCByqiB+YZsnlpeE/ERvwSkL8J8tc0+fPS 8JkFrEEitny50+xaB+I1spYsYC3ZwFqyJHoKoanHHl61wC1e+XJn6IIqusABF7jggormAkJT jx6yBrfG4sudpgv2x2wEKQETNMEENXEowLf0xEH7+kxLELRyLkG7u3V23xoUyE3vbcFRa5ac 6vtMPbsvcMuITyoBPqkM+KSSRERhC665scpoaWxZIGk0/qBxjaFK3LygLTiyMRLTgQnGqKIx HDCGC8aoqMYQhZHKytKW3hpYY4gBri4JzW8VrmK+3NjjRVrG8Lc12zczq6MrGuCKJriiprlC LkplSilKrrgQGns7t9zf+gPm87BBTDTNK3950jet4CaILNGLJ0qDC3Zf7iW9YWdKRpRRCVBG ZUAZlSRGSWkW2PovbKp3d0yrICsKJBTHFxxIwypq6ICGLmhY0TT09Vqgz2DBUzUjcvE40TtY /KnhpittwYFUrKOKDVCxCSrWxJ7IFkqCv7he6Q/af02VEyh9AqW4OTXn/IkcnxJwfMrA8SlJ 5BBC9U/z3VHTpE8E/wAFXVCwoilIOoGT1XCC7In8HqBhEzSsyRriT+Bo3wWWE+doEycydUrA 1CkDU6ckUTxy4kySOBHCAxR0QcGKpmBOnIkSJ7J0gIZN0LAma5gTJyfO5LeAd6ZNpNqUgGpT BqpNOQuiSdZtR8ZEEg3QzQXdqqzbTHWro24N0K0JutWT6JYTJXmiCIWbAPhyibqYwsLw1ctw +DuzJcJcSgBzKQPMpSSRHOSCC8VLyVj7jAPyCQd/EPG2LF5Q3LghDtARXy6oLLE7GMoXPYi0 M3QimQUI6oKgFVlQwyS3qv+KFTRRBxESZylfLlUP5chbg4p6V2tPGkXWChC0CYLWVEFFKQot lBWWqQL7ViZxWm9MHOABlaN8up4ZjrzGNdQu/MqaJV7CozxjX0YASwkALGUAsJQ0AMfCtC9K tNoURmmNpfWO/TJpqXAvhmkLjmvY1CuDlMfuozEcMIYLxqhoxuALqWRRMsu00Nhe2R40rjEY EnPQFhzbGGmXGwlP4kdfNMAXTfBFTfUFM8YoP9/yhi4Nslsu2Gm9NfFc89lK3Gjny51tOkeK SwkoLmWguJQ0gseC6Xaizkrf2QT2vRb+oJGHOuyDbAX1UbaTSueIdgHGcMEYFdUYRWFLywuh Sm7Rb04sxm1+Y5GvWrajT9rmk82R9wJc0QRX1FRXcKN5WTLGbeE7G3b/Ts7mU8hmXeIy0Jc7 22yO8JcSwF/KAH8paeCPhfRxJjUTzBaKYfdEyZG3KJe45UBf7oyTOfJggC1csEVFtQXXVigp RSFMiZ6yjbyapZAvulYjv+h6VskcMTHAFU1wRU11hSmELZnxszZtLHqZ7bSYMeeazAx7T+Zl t2Tmz4k4kMqRGVMCZkwZmDElCRRBaOrR36vLkYtlfNyroXm4oIoucMAFLrigormA0NSju6DA Xaz7cqfpgv2ZG5kxwARNMEFNNAG+pU+NGTNE0B4jM4YzJDKsLThqzZI/OUVhxvjJ8zZx23/f Rm73g74A8VJYSFEyFb6OeClMMUeJvYtZDnE1RjFH6qewKNwYYA4HzeF6c1Rkc5TKX/GUSjBj OJLpxA5RQobvn9iLMl9w5JolfqKLQI8B3migN5reGzXZG4XkRgvmHVJKg76NsX9qNJ8nke+u fv3Hs2h+/YOHc3kamVmNvMPgCyaqglDYkbMtOdDzjj4eGchPBvOT9flJ4liYhShUOyVRTBok QsyQ6Vp4XRXumZq2YCpdNfaeZltyIF0roKuDurpe14quqzRFIa3SpbUaLSzRqHhhBTJJfMFk wgrknZK25EDC1kDYBgrb9MLWZGGl8hXUjBfc9wLcnMYcejgu59YoucUVcl1AJdr1MVlqcZBa HKYW71OLRMBQYqG40JwX0nApkTeU2oNS6Yq89k9VgclSi4PU4jC1eJ9aNF3louDCFqUKX3G6 tgcl0lXixldf7sRCi4PQ4jC0eB9aVF1LK6wpSm2MZLgV8u6gnFrTp5bA9cNkc8epMkuAzBIw s0SfWSSGhrB2AXYL4jKrOyjRlQ7y5QbU10jMPrIEiCwBI0v0kUWSVRbF4uueXmUt8q133UGJ ZMVdQYt018/TJJYAiSVgYok+sWiyMrHgLP7FydodlCNrBpGF6wOJugDH9gA+WAfwaSRBXEkY V7KPKxKaQyJvGvpyybAqE6uIvK5sCw6XThKkk4TpJPt0IqooFjr+QTIK2oNSUQomlhR36dEW HC6ZJEgmCZNJ9slElFQvBPiDlLQ9KCfT9Mk0LYhwmmRSIJkUTCbVJxMJ00E6idNEfk2UTQpk k4LZpPpsIupIOImT1XGKQFIgkBQMJNUHElVH/EkcD8rttFNo2td/jIwIZYUGCaRhAuk+gWjk GmzrpRu1kI9JJJpIj4wKBfo5qJ/r9auo+mFbL+s3UOJokDgaJo7uE2ca/XLanP41z+hZY0DW GJg1ps8aGlgB+2DqKc6PR88ZA3LGwJwxfc5k7eapXQ20a6B2Ta9dPY12OWFywgyeMBYkjIUJ Y/uEoQEC8ig1YsJYkDAWJoztEyZrN0/taqBdA7Vreu3qabTLCTNOwiD30CZ6EnX0jClBxpQw Y8o+Y4hsDGTrpRupzkm/CujnoH6u16/K+s1Zvxro10D9ml6/eir9ctrk65mhs4YBJASDSAjW IyEYbWd3nhOPlzMMEBsYJDawntiQtZupdjXQroHaNb129TTa5YQZJWEMbkZgTmI+5SMEYBMY xCawHpvAaLuwsa2XbKRCbiAyJzFWVUA/B/VzvX4VVT9s6yXTT+OemdLEx77mqV8N9Gugfk2v X03UD916OW1mkTYl9nV6bcE0VbAlcpN+W3C43AHgAwbBB6wHHzDSRmo/HmmmVCFLpljBsUSz ItErH0uhcANpWzCVrEh6WltwuDgC3AMGuQes5x6QZZWFEVIpo0pVIh+nPQQje0VvVbg2bQsm qkKBRKi2BYdLKYA9YBB7wHrswQt6K/NDGpdSCOR7BA/21gEj6/uCQ6DXX59ku+p1nPh1oSTy rQJtybHrlpqrQEGxMwZ4CwzyFljPW2CkHdzt1N23qeCsLAtWoDvfyC8jEZZjn2b1JUeuW3JC A4XSDgzioEFcb5CKbhCm/LzESqHLQiDvkJBZQ68WwSCnzl3JsQ2SmvVAALgDfzTQH03vj/oF A4ifZwuppR//Cixaelqee07vnN7TpDfATzCIn2A9foKRdrnn9D659AZkCwbJFqwnW5ANktP7 pNIbYDIYxGSwHpNBH0Byeuf0zumNSm+A6GAQ0cF6RAejIQFyep9aegP2B4PsD9azP8gGyel9 UukNmCIMMkVYzxShDyA5vc86vXmBfp3myLGRGqJFS26ANmEQbcJ6tAmjoRUW2BFvMfaAx7Hv Z+ZDvJ+ZVLPENC5aUgNWCoOsFNazUsiG4Jb5kGaysBJ5u38KcyBty0evWWLEFymlAYeFQQ4L 6zks9MHCT4yNEe3DFRbZPUd8h1pO6ZzS80hpAIVhEArDeigMo8Epckofe0oD0gyDpBnWk2bI hsgpfTIpDUg2DJJsWE+yoQ8WOaXPOqVT7Gp7ZZ0SbkCgJTOA6TAI02E9TIfNAsiSHZAuigGS h0EkD+uRPNkBx+6AA3kLuD4Mcn1Yz/WZxAA5YHPAHkXnOhSwgCTEIEmI9SQhRiOZ5OH1uBxQ AQc46ADXO6DKDjh6BxwIWAAzYhBmxHqY0SQGyAGbA/YoOteBgOUAn8QhPon3+CQ+CwRPdkCy gOUAwsQhhIn3EKbsgGN3wP6A5YDkxCHJifckp0kMkAM2B+xRdK5DAQvYURyyo3jPjuIkdk0e Xo/MARVwgIMOcL0DquyAo3fAgYAF+CoO8VW8x1dNYoAcsDlgj6JzHQpYAMniEJLFe0gWp2F3 8vB6XA6ogAMcdIDrHVBlBxy9Aw4ELCBvcUje4j15axID5IDNAXsUnetQwAK0FodoLd6jtTiN jJOH1+NyQAUc4KADXO+AKjvg6B1wIGABHItDOBbv4ViTGCBxwA6QY2kCVs4lYHe30G7JC2GQ j++3JUetW3K48zP97P64BSwsDllYvGdhcSLKhilrWkaJZYWS2M3wbIBtTQQJuCiQW3+6kiPb IzEkmmCPCtjDQXu43h4V0R58oXShC1YUTOsSaY/2oHFHD1Vid4a1JUetW3LY9Lc12zdTq4E7 GuiOpndHTR48jNZalVxzJcsSS4w3+xUYMLeHD2hLk+iVFUj6GrgCOc8qEtH1pWG48aQt+JKe sTs/AY2KQxoV72lUnASTkXqhrWBf/48bhbqDEqnKkKqyVKr6mEA2QjkUXL8CqjqoqutVraiq 8gW3dvs/3My0PSjRG3VYgRxbi0S2kgbJ0WoLDqRqDVRtoKpNr2pNVVUsLJPxf7gTag/KeTV9 XuH8l+iVchNlFeAvcchf4j1/iZOQKlwos0BO4buiiV5PhhtOE72qaaJ8AuAkDsFJvAcn0ZRk SsuF3v7BXvu0B73+vV8z1HSadALAIw6BR7wHHlE1NWYh4h+kpt1BOZ+mz6dJX+87UT4B8hCH 5CHek4c4CSZCOonTfE3zROkEgEEcAoN4Dwwi6kg5iRPVcZpEAnAfDuE+vIf7UPsj4SSO+mWo OYUG+eUWeZ9suFegckDY4ZCww3vCDifhNdAncJqjFvZG1nDvOuWAj8MhH4f3fByiftgTyPoN lDgAb8Mh3ob3eJuJ9Mtpk9MmQdoA3AyHuBne42Y4iTWR02bstAGwGA5hMbyHxRD1y2kzctoA 1guHrBfes14m0i+nzemnTcJHbXdmjQDkFQHJK6Inr4hZcDeydjtyRgBmioDMFNEzU7J2M9Wu Bto1ULum166eRrucMDlhBk8YgB4RED0ievSImAV4Imu3K2EANERAaIjooSFZu5lqVwPtGqhd 02tXT6NdTpicMIMnDGBvCMjeED17Q8yCvJC125UwgJohIDVD9NSMrN1MtauBdg3Urum1q6fR LidMTpjBEwbAJwSET4gePiFmgR7I2u1KGICNEBAbIXpsRNZuptrVQLsGatf02tXTaJcTJifM 4AkDeAsC8hZEz1sQtC3TeZQaMWEADEFAGILoYQhZu5lqVwPtGqhd02tXT6NdTpicMIMnDCAS CEgkED2RQJB2OedRasyEAdwBAbkDoucOZO1mql0NtGugdk2vXT2NdjlhRkkYLoTBvZW9K5mo ErLkSKxBW3K4xAFcAQG5AqLnCgjazuVFoVVZlrJgzLAS+bL7BRXERZBW4jpjVzKVtFhsTFdy uEACoAEBQQOiBw3QpdWFlkVhlCwElvCVTlrJsRxAXzKZtAwtLRtK2hpI20Bpm17ami6tKpnm qpC24Mid160fxoqv7wsmopoTU21XvY6Sai4tx8252oLj1iw1zoBEOBcAcyAg5kD0mANB2lbt +xDnXJdCCcatQM8bB2DLEiQQlhmcBl3Jce2RmpJAwp8LQE8QkJ4genoCzR52wQQ33iBcam4Y LnLbg8a1h+ElbvDoSo48eiSGL1DY6AIwGQRkMoieyUBzh1lY6+dh1kpWciQyuz0mh3YO7fMK bUCGEJAMIXoyhCDtTD+O0OZ+Lo8E/LUlzzm0AXhCQPCE6METNHvIhfAXyZbZQivNFBID7A8a VQKmjUBe57Ulzzi0AdZCQKyF6LEWNHeIhRJaqVKVkiuGWx/zx+zvBTm0c2ifXGgDwIaAgA3R AzYEaYP/cYS2jwski7orec6hDfgdAvI7RM/voNlDLISSJWfMqNKbBBc/7UGjSlBiX7vSFjzj yAZsEAHZIKJng9C8wResYFpaocJX3KTaH5Qz+0QyG0nVpgK1X1mrpNArUlZLACiREFAie0CJ pIESFpwLawtVmoLJEom/XrzsrvjLsxp7C1SMm9NJWUykjJaAfSIh+0T27BOqLTQXipfSp7T2 c1CkLfS4ttAaN1aMe7k/oin2Z7METBUJmSqyZ6qQhwpr4l/s8yYHXs6Tszln88lkM0C7SIh2 kT3aRdIQE8eRzRqZDnrclzTPKJsBNUZCaozsqTFUW9ivuVwoi3R/e9C4YwWu+fW4187zyWZA o5GQRiN7Gg3VE6IwuuT9V+x8TYz2irWczWeZzQk3F9CSGSBxJETiyB6JI0loDnRjk7eMvHZ8 M7hLIl/uDH1QAR846APX+6Ai+oDQ3KM7QeEu0Hy503TCgfwFrB4JWT2yZ/UQjYBv7RF3IuXQ zaE7ZegCSpCElCDZU4IkiVYy29AtkTcKx73UmYcLKuACB13gehdURBegG3tsFyAXRKmbGo/F BQfiFoCLJAQXyR5cRDMBuq1z2OawPZEOdihsATBJQmCS7IFJkgRumW3YznW5eR4+qIAPHPSB 631QEX1AaO6xnTDPFea5BC7gOEnIcZI9x4k4HGDbOgduDtwT6WCHAhfwoyTkR8meHyVJHJvZ Bm5eUj4QuIBFJSGLSvYsKpoP8pLyfJ1wIHIB2EpCsJXswVZEI+Ql5Ry6OXSfhS5AaEmI0JI9 QkuSYDyzDd28pLw3cgFtS0LaluxpWzQX5CXlebrgQNwCLpeEXC7Zc7loJshLyjlsc9h+G7YA wCUhgEv2AC5JYujMNmyZVshnq33BM3RCBZzgoBNc74SK6ARSg4/tBl4USMykL3iabjgQvACt JSFaS/ZoLZoZSO2dw/d0wpdLVSCpsm3JkeuWHBFNi2KA1ZIQqyV7rJakYrWYLoUoheHS/50n oUMWEjln6EqObJDkoGlaQgOwloRgLdmDtagG0VzzkhWSdztKsPt/xw1EbplCIsfbkmMbJDWu mhTaAK0lIVpL9mgtqj+YkMpbRGlRGou7ZcEO0VK/T2/4na/t0J7w9cXqt4fN6vHxZn3/eOnL 9dV79N9cXj9+Wq2e4jd/WV//Hv/ren31+W51/3Rxtb5/8v/88/8HUEsHCKCl0oLsNQAA3yUE AFBLAQIUABQAAAgAAA6sIk2FbDmKLgAAAC4AAAAIAAAAAAAAAAAAAAAAAAAAAABtaW1ldHlw ZVBLAQIUABQAAAgAAA6sIk1sB2Xv8yoAAPMqAAAYAAAAAAAAAAAAAAAAAFQAAABUaHVtYm5h aWxzL3RodW1ibmFpbC5wbmdQSwECFAAUAAAIAAAOrCJNAAAAAAAAAAAAAAAAGgAAAAAAAAAA AAAAAAB9KwAAQ29uZmlndXJhdGlvbnMyL3Rvb2xwYW5lbC9QSwECFAAUAAAIAAAOrCJNAAAA AAAAAAAAAAAAGAAAAAAAAAAAAAAAAAC1KwAAQ29uZmlndXJhdGlvbnMyL21lbnViYXIvUEsB AhQAFAAACAAADqwiTQAAAAAAAAAAAAAAABoAAAAAAAAAAAAAAAAA6ysAAENvbmZpZ3VyYXRp b25zMi9zdGF0dXNiYXIvUEsBAhQAFAAACAAADqwiTQAAAAAAAAAAAAAAABwAAAAAAAAAAAAA AAAAIywAAENvbmZpZ3VyYXRpb25zMi9wcm9ncmVzc2Jhci9QSwECFAAUAAAIAAAOrCJNAAAA AAAAAAAAAAAAHwAAAAAAAAAAAAAAAABdLAAAQ29uZmlndXJhdGlvbnMyL2ltYWdlcy9CaXRt YXBzL1BLAQIUABQAAAgAAA6sIk0AAAAAAAAAAAAAAAAYAAAAAAAAAAAAAAAAAJosAABDb25m aWd1cmF0aW9uczIvZmxvYXRlci9QSwECFAAUAAAIAAAOrCJNAAAAAAAAAAAAAAAAGAAAAAAA AAAAAAAAAADQLAAAQ29uZmlndXJhdGlvbnMyL3Rvb2xiYXIvUEsBAhQAFAAACAAADqwiTQAA AAAAAAAAAAAAABoAAAAAAAAAAAAAAAAABi0AAENvbmZpZ3VyYXRpb25zMi9wb3B1cG1lbnUv UEsBAhQAFAAACAAADqwiTQAAAAAAAAAAAAAAABwAAAAAAAAAAAAAAAAAPi0AAENvbmZpZ3Vy YXRpb25zMi9hY2NlbGVyYXRvci9QSwECFAAUAAgICAAOrCJNYRjW2IYBAAAIAwAACAAAAAAA AAAAAAAAAAB4LQAAbWV0YS54bWxQSwECFAAUAAgICAAOrCJNg2nt1JIGAAAgJgAACgAAAAAA AAAAAAAAAAA0LwAAc3R5bGVzLnhtbFBLAQIUABQACAgIAA6sIk2092jSBQEAAIMDAAAMAAAA AAAAAAAAAAAAAP41AABtYW5pZmVzdC5yZGZQSwECFAAUAAgICAAOrCJNS/y5JRUFAAB/IAAA DAAAAAAAAAAAAAAAAAA9NwAAc2V0dGluZ3MueG1sUEsBAhQAFAAICAgADqwiTTq6vdERAQAA 2QMAABUAAAAAAAAAAAAAAAAAjDwAAE1FVEEtSU5GL21hbmlmZXN0LnhtbFBLAQIUABQACAgI AA6sIk2gpdKC7DUAAN8lBAALAAAAAAAAAAAAAAAAAOA9AABjb250ZW50LnhtbFBLBQYAAAAA EQARAGUEAAAFdAAAAAA= --------------0E1BE69E814B19BC8B911192 Content-Type: application/sql; name="stats-tests.sql" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="stats-tests.sql" LS0gcGFyYWxsZWxpc20ganVzdCBtYWtlcyB0aGUgbnVtYmVycyBtb3JlIGRpZmZpY3VsdCB0 byBpbnRlcnByZXQKc2V0IG1heF9wYXJhbGxlbF93b3JrZXJzX3Blcl9nYXRoZXIgPSAwOwoK LS0gRGVhbidzIGV4YW1wbGUKZHJvcCB0YWJsZSBpZiBleGlzdHMgbWN2X3Rlc3RfMTsKY3Jl YXRlIHRhYmxlIG1jdl90ZXN0XzEgKGEgaW50LCBiIGludCk7Cmluc2VydCBpbnRvIG1jdl90 ZXN0XzEgc2VsZWN0IDEsMSBmcm9tIGdlbmVyYXRlX3NlcmllcygxLDUwMDAwKTsKaW5zZXJ0 IGludG8gbWN2X3Rlc3RfMSBzZWxlY3QgMSwyIGZyb20gZ2VuZXJhdGVfc2VyaWVzKDEsNDAw MDApOwppbnNlcnQgaW50byBtY3ZfdGVzdF8xIHNlbGVjdCAxLHgvMTAgZnJvbSBnZW5lcmF0 ZV9zZXJpZXMoMzAsMjUwMDAwKSBnKHgpOwppbnNlcnQgaW50byBtY3ZfdGVzdF8xIHNlbGVj dCAyLDEgZnJvbSBnZW5lcmF0ZV9zZXJpZXMoMSwzMDAwMCk7Cmluc2VydCBpbnRvIG1jdl90 ZXN0XzEgc2VsZWN0IDIsMiBmcm9tIGdlbmVyYXRlX3NlcmllcygxLDIwMDAwKTsKaW5zZXJ0 IGludG8gbWN2X3Rlc3RfMSBzZWxlY3QgMix4LzEwIGZyb20gZ2VuZXJhdGVfc2VyaWVzKDMw LDUwMDAwMCkgZyh4KTsKaW5zZXJ0IGludG8gbWN2X3Rlc3RfMSBzZWxlY3QgMywxIGZyb20g Z2VuZXJhdGVfc2VyaWVzKDEsMTAwMDApOwppbnNlcnQgaW50byBtY3ZfdGVzdF8xIHNlbGVj dCAzLDIgZnJvbSBnZW5lcmF0ZV9zZXJpZXMoMSw1MDAwKTsKaW5zZXJ0IGludG8gbWN2X3Rl c3RfMSBzZWxlY3QgMyx4IGZyb20gZ2VuZXJhdGVfc2VyaWVzKDMsNjAwMDAwKSBnKHgpOwpp bnNlcnQgaW50byBtY3ZfdGVzdF8xIHNlbGVjdCB4LHgvMTAgZnJvbSBnZW5lcmF0ZV9zZXJp ZXMoNCw3NTAwMDApIGcoeCk7CgphbmFseXNlIG1jdl90ZXN0XzE7CgpleHBsYWluIGFuYWx5 c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVzdF8xIHdoZXJlIGE9MSBhbmQgYj0xOwoKZXhwbGFp biBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X3Rlc3RfMSB3aGVyZSBhPTEgYW5kIGI9MjsK CmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzEgd2hlcmUgYT0xIGFu ZCAoYj0xIG9yIGI9Mik7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVz dF8xIHdoZXJlIChhPTEgb3IgYT0yKSBhbmQgKGI9MSBvciBiPTIpOwoKY3JlYXRlIHN0YXRp c3RpY3MgbWN2X3Rlc3RfbWN2X2FiIChtY3YpIG9uIGEsYiBmcm9tIG1jdl90ZXN0XzE7CmFu YWx5c2UgbWN2X3Rlc3RfMTsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90 ZXN0XzEgd2hlcmUgYT0xIGFuZCBiPTE7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBtY3ZfdGVzdF8xIHdoZXJlIGE9MSBhbmQgYj0yOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVj dCAqIGZyb20gbWN2X3Rlc3RfMSB3aGVyZSBhPTEgYW5kIChiPTEgb3IgYj0yKTsKCmV4cGxh aW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzEgd2hlcmUgKGE9MSBvciBhPTIp IGFuZCAoYj0xIG9yIGI9Mik7CgotLSBwZXJmZWN0bHkgZGVwZW5kZW50IGNvbHVtbnMgd2l0 aCBzbWFsbCBudW1iZXIgb2YgZGlzdGluY3QgdmFsdWVzCmRyb3AgdGFibGUgaWYgZXhpc3Rz IG1jdl90ZXN0XzI7CmNyZWF0ZSB0YWJsZSBtY3ZfdGVzdF8yKGEgaW50LCBiIGludCwgYyBp bnQsIGQgaW50KTsKaW5zZXJ0IGludG8gbWN2X3Rlc3RfMiBzZWxlY3QgbW9kKGksNTApLCBt b2QoaSw1MCksIG1vZChpLDUwKSwgbW9kKGksNTApIGZyb20gZ2VuZXJhdGVfc2VyaWVzKDEs NTAwMDApIHMoaSk7CgphbmFseXNlIG1jdl90ZXN0XzI7CgpleHBsYWluIGFuYWx5c2Ugc2Vs ZWN0ICogZnJvbSBtY3ZfdGVzdF8yIHdoZXJlIGE9MSBhbmQgYj0xOwoKZXhwbGFpbiBhbmFs eXNlIHNlbGVjdCAqIGZyb20gbWN2X3Rlc3RfMiB3aGVyZSBhPTEgYW5kIGI9MSBhbmQgYz0x OwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X3Rlc3RfMiB3aGVyZSBhPTEg YW5kIGI9MSBhbmQgYz0xIGFuZCBkPTE7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBtY3ZfdGVzdF8yIHdoZXJlIGE9MSBhbmQgYj0yOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVj dCAqIGZyb20gbWN2X3Rlc3RfMiB3aGVyZSBhPTEgYW5kIGI9MSBhbmQgYyA8IDEwOwoKZXhw bGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X3Rlc3RfMiB3aGVyZSBhPTEgYW5kIGI9 MSBhbmQgYyA8IDEwIGFuZCBkIDwgMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBtY3ZfdGVzdF8yIHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBjID4gMTA7CgpleHBsYWluIGFu YWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVzdF8yIHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBj ID4gMTAgYW5kIGQgPiAxMDsKCmNyZWF0ZSBzdGF0aXN0aWNzIG1jdl90ZXN0XzJfbWN2X2Fi Y2QgKG1jdikgb24gYSxiLGMsZCBmcm9tIG1jdl90ZXN0XzI7CmFuYWx5c2UgbWN2X3Rlc3Rf MjsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzIgd2hlcmUgYT0x IGFuZCBiPTE7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVzdF8yIHdo ZXJlIGE9MSBhbmQgYj0xIGFuZCBjPTE7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBtY3ZfdGVzdF8yIHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBjPTEgYW5kIGQ9MTsKCmV4cGxh aW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzIgd2hlcmUgYT0xIGFuZCBiPTI7 CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVzdF8yIHdoZXJlIGE9MSBh bmQgYj0xIGFuZCBjIDwgMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3Zf dGVzdF8yIHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBjIDwgMTAgYW5kIGQgPCAxMDsKCmV4cGxh aW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzIgd2hlcmUgYT0xIGFuZCBiPTEg YW5kIGMgPiAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzIg d2hlcmUgYT0xIGFuZCBiPTEgYW5kIGMgPiAxMCBhbmQgZCA+IDEwOwoKLS0gcGVyZmVjdGx5 IGRlcGVuZGVudCBjb2x1bW5zIHdpdGggc21hbGwgbnVtYmVyIG9mIGRpc3RpbmN0IHZhbHVl cywgYnV0Ci0tIHNrZXdlZCB0b3dhcmRzIDAKZHJvcCB0YWJsZSBpZiBleGlzdHMgbWN2X3Rl c3RfMzsKY3JlYXRlIHRhYmxlIG1jdl90ZXN0XzMoYSBpbnQsIGIgaW50LCBjIGludCwgZCBp bnQpOwppbnNlcnQgaW50byBtY3ZfdGVzdF8zIHNlbGVjdCBzcXJ0KG1vZChpLDI1MDApKSwg c3FydChtb2QoaSwyNTAwKSksIHNxcnQobW9kKGksMjUwMCkpLCBzcXJ0KG1vZChpLDI1MDAp KSBmcm9tIGdlbmVyYXRlX3NlcmllcygxLDEwMDAwMCkgcyhpKTsKCmFuYWx5c2UgbWN2X3Rl c3RfMzsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzMgd2hlcmUg YT0xIGFuZCBiPTE7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVzdF8z IHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBjPTE7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICog ZnJvbSBtY3ZfdGVzdF8zIHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBjPTEgYW5kIGQ9MTsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzMgd2hlcmUgYT0xIGFuZCBi PTI7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfdGVzdF8zIHdoZXJlIGE9 MSBhbmQgYj0xIGFuZCBjIDwgMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBt Y3ZfdGVzdF8zIHdoZXJlIGE9MSBhbmQgYj0xIGFuZCBjIDwgMTAgYW5kIGQgPCAxMDsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzMgd2hlcmUgYT0xIGFuZCBi PTEgYW5kIGMgPiAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0 XzMgd2hlcmUgYT0xIGFuZCBiPTEgYW5kIGMgPiAxMCBhbmQgZCA+IDEwOwoKY3JlYXRlIHN0 YXRpc3RpY3MgbWN2X3Rlc3RfM19tY3ZfYWJjZCAobWN2KSBvbiBhLGIsYyxkIGZyb20gbWN2 X3Rlc3RfMzsKYW5hbHlzZSBtY3ZfdGVzdF8zOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAq IGZyb20gbWN2X3Rlc3RfMyB3aGVyZSBhPTEgYW5kIGI9MTsKCmV4cGxhaW4gYW5hbHlzZSBz ZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzMgd2hlcmUgYT0xIGFuZCBiPTEgYW5kIGM9MTsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzMgd2hlcmUgYT0xIGFuZCBi PTEgYW5kIGM9MSBhbmQgZD0xOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2 X3Rlc3RfMyB3aGVyZSBhPTEgYW5kIGI9MjsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBm cm9tIG1jdl90ZXN0XzMgd2hlcmUgYT0xIGFuZCBiPTEgYW5kIGMgPCAxMDsKCmV4cGxhaW4g YW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl90ZXN0XzMgd2hlcmUgYT0xIGFuZCBiPTEgYW5k IGMgPCAxMCBhbmQgZCA8IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2 X3Rlc3RfMyB3aGVyZSBhPTEgYW5kIGI9MSBhbmQgYyA+IDEwOwoKZXhwbGFpbiBhbmFseXNl IHNlbGVjdCAqIGZyb20gbWN2X3Rlc3RfMyB3aGVyZSBhPTEgYW5kIGI9MSBhbmQgYyA+IDEw IGFuZCBkID4gMTA7CgotLSBwZXJmZWN0bHkgZGVwZW5kZW50IHZhbHVlcyB3aXRoIGxhcmdl IG51bWJlciBvZiBkaXN0aW5jdCB2YWx1ZXMsIHNvCi0tIG1vcmUgc3VpdGFibGUgZm9yIGhp c3RvZ3JhbXMKCmRyb3AgdGFibGUgaWYgZXhpc3RzIGhpc3RvZ3JhbV90ZXN0XzQ7CmNyZWF0 ZSB0YWJsZSBoaXN0b2dyYW1fdGVzdF80KGEgaW50LCBiIGludCwgYyBpbnQsIGQgaW50KTsK aW5zZXJ0IGludG8gaGlzdG9ncmFtX3Rlc3RfNCBzZWxlY3QgeCwgeCwgeCwgeCBmcm9tIChz ZWxlY3QgKHJhbmRvbSgpICogMjAwKTo6aW50IGFzIHggZnJvbSBnZW5lcmF0ZV9zZXJpZXMo MSwxMDAwMDApKSBmb287CgphbmFseXNlIGhpc3RvZ3JhbV90ZXN0XzQ7CgpleHBsYWluIGFu YWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPD0gMTAgYW5k IGIgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVz dF80IHdoZXJlIGEgPD0gMTAgYW5kIGIgPD0gMTAgYW5kIGMgPD0gMTA7CgpleHBsYWluIGFu YWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPD0gMTAgYW5k IGIgPD0gMTAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2Vs ZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPSAxICAgYW5kIGIgPD0gMTAg YW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAxICAgYW5kIGMgPD0g MTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dy YW1fdGVzdF80IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAxICAgYW5kIGMgPj0gMTAgYW5kIGQg Pj0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80 IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAxICAgYW5kIGMgPj0gOTAgYW5kIGQgPj0gOTA7Cgpl eHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEg PSAxICAgYW5kIGIgPSAyICAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFu YWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPSAxICAgYW5k IGIgPSAyICAgYW5kIGMgPj0gMTAgYW5kIGQgPj0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2Vs ZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAyICAg YW5kIGMgPj0gOTAgYW5kIGQgPj0gOTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBoaXN0b2dyYW1fdGVzdF80IHdoZXJlIGEgPD0gMTAgYW5kIGIgPj0gOTAgYW5kIGMgPD0g MTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dy YW1fdGVzdF80IHdoZXJlIGEgPD0gNjAgYW5kIGIgPj0gNDAgYW5kIGMgPD0gNjUgYW5kIGQg Pj0gNDU7CgpjcmVhdGUgc3RhdGlzdGljcyBoaXN0b2dyYW1fdGVzdF80X2hpc3RvZ3JhbV9h YmNkIChuZGlzdGluY3QsIGhpc3RvZ3JhbSkgb24gYSxiLGMsZCBmcm9tIGhpc3RvZ3JhbV90 ZXN0XzQ7CmFuYWx5c2UgaGlzdG9ncmFtX3Rlc3RfNDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxl Y3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA8PSAxMCBhbmQgYiA8PSAxMDsK CmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUg YSA8PSAxMCBhbmQgYiA8PSAxMCBhbmQgYyA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxl Y3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA8PSAxMCBhbmQgYiA8PSAxMCBh bmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9t IGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA9IDEgICBhbmQgYiA8PSAxMCBhbmQgYyA8PSAx MCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3Jh bV90ZXN0XzQgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBhbmQgYyA8PSAxMCBhbmQgZCA8 PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQg d2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBhbmQgYyA+PSAxMCBhbmQgZCA+PSAxMDsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA9 IDEgICBhbmQgYiA9IDEgICBhbmQgYyA+PSA5MCBhbmQgZCA+PSA5MDsKCmV4cGxhaW4gYW5h bHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA9IDEgICBhbmQg YiA9IDIgICBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxl Y3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBh bmQgYyA+PSAxMCBhbmQgZCA+PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9t IGhpc3RvZ3JhbV90ZXN0XzQgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBhbmQgYyA+PSA5 MCBhbmQgZCA+PSA5MDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3Jh bV90ZXN0XzQgd2hlcmUgYSA8PSAxMCBhbmQgYiA+PSA5MCBhbmQgYyA8PSAxMCBhbmQgZCA8 PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzQg d2hlcmUgYSA8PSA2MCBhbmQgYiA+PSA0MCBhbmQgYyA8PSA2NSBhbmQgZCA+PSA0NTsKCi0t IHBlcmZlY3RseSBkZXBlbmRlbnQgdmFsdWVzIHdpdGggbGFyZ2UgbnVtYmVyIG9mIGRpc3Rp bmN0IHZhbHVlcywgc28KLS0gbW9yZSBzdWl0YWJsZSBmb3IgaGlzdG9ncmFtcywgc2tld2Vk IHRvd2FyZHMgMAoKZHJvcCB0YWJsZSBpZiBleGlzdHMgaGlzdG9ncmFtX3Rlc3RfNTsKY3Jl YXRlIHRhYmxlIGhpc3RvZ3JhbV90ZXN0XzUoYSBpbnQsIGIgaW50LCBjIGludCwgZCBpbnQp OwppbnNlcnQgaW50byBoaXN0b2dyYW1fdGVzdF81IHNlbGVjdCB4LCB4LCB4LCB4IGZyb20g KHNlbGVjdCAoc3FydChyYW5kb20oKSkgKiAyMDApOjppbnQgYXMgeCBmcm9tIGdlbmVyYXRl X3NlcmllcygxLDEwMDAwMCkpIGZvbzsKCmFuYWx5c2UgaGlzdG9ncmFtX3Rlc3RfNTsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA8 PSAxMCBhbmQgYiA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3Rv Z3JhbV90ZXN0XzUgd2hlcmUgYSA8PSAxMCBhbmQgYiA8PSAxMCBhbmQgYyA8PSAxMDsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA8 PSAxMCBhbmQgYiA8PSAxMCBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5h bHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA9IDEgICBhbmQg YiA8PSAxMCBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxl Y3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBh bmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9t IGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBhbmQgYyA+PSAx MCBhbmQgZCA+PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3Jh bV90ZXN0XzUgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBhbmQgYyA+PSA5MCBhbmQgZCA+ PSA5MDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUg d2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4 cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA9 IDEgICBhbmQgYiA9IDIgICBhbmQgYyA+PSAxMCBhbmQgZCA+PSAxMDsKCmV4cGxhaW4gYW5h bHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA9IDEgICBhbmQg YiA9IDIgICBhbmQgYyA+PSA5MCBhbmQgZCA+PSA5MDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxl Y3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA8PSAxMCBhbmQgYiA+PSA5MCBh bmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9t IGhpc3RvZ3JhbV90ZXN0XzUgd2hlcmUgYSA8PSA2MCBhbmQgYiA+PSA0MCBhbmQgYyA8PSA2 NSBhbmQgZCA+PSA0NTsKCmNyZWF0ZSBzdGF0aXN0aWNzIGhpc3RvZ3JhbV90ZXN0XzVfaGlz dG9ncmFtX2FiY2QgKG5kaXN0aW5jdCwgaGlzdG9ncmFtKSBvbiBhLGIsYyxkIGZyb20gaGlz dG9ncmFtX3Rlc3RfNTsKYW5hbHlzZSBoaXN0b2dyYW1fdGVzdF81OwoKZXhwbGFpbiBhbmFs eXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhIDw9IDEwIGFuZCBi IDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3Rf NSB3aGVyZSBhIDw9IDEwIGFuZCBiIDw9IDEwIGFuZCBjIDw9IDEwOwoKZXhwbGFpbiBhbmFs eXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhIDw9IDEwIGFuZCBi IDw9IDEwIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVj dCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhID0gMSAgIGFuZCBiIDw9IDEwIGFu ZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20g aGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhID0gMSAgIGFuZCBiID0gMSAgIGFuZCBjIDw9IDEw IGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFt X3Rlc3RfNSB3aGVyZSBhID0gMSAgIGFuZCBiID0gMSAgIGFuZCBjID49IDEwIGFuZCBkID49 IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3 aGVyZSBhID0gMSAgIGFuZCBiID0gMSAgIGFuZCBjID49IDkwIGFuZCBkID49IDkwOwoKZXhw bGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhID0g MSAgIGFuZCBiID0gMiAgIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFs eXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhID0gMSAgIGFuZCBi ID0gMiAgIGFuZCBjID49IDEwIGFuZCBkID49IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVj dCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhID0gMSAgIGFuZCBiID0gMiAgIGFu ZCBjID49IDkwIGFuZCBkID49IDkwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20g aGlzdG9ncmFtX3Rlc3RfNSB3aGVyZSBhIDw9IDEwIGFuZCBiID49IDkwIGFuZCBjIDw9IDEw IGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFt X3Rlc3RfNSB3aGVyZSBhIDw9IDYwIGFuZCBiID49IDQwIGFuZCBjIDw9IDY1IGFuZCBkID49 IDQ1OwoKCi0tIHBlcmZlY3RseSBkZXBlbmRlbnQgdmFsdWVzIHdpdGggbGFyZ2UgbnVtYmVy IG9mIGRpc3RpbmN0IHZhbHVlcywgc28KLS0gbW9yZSBzdWl0YWJsZSBmb3IgaGlzdG9ncmFt cywgdmVyeSBmZXcgY29sbGlzaW9ucyAoZG91YmxlKQoKZHJvcCB0YWJsZSBpZiBleGlzdHMg aGlzdG9ncmFtX3Rlc3RfNjsKY3JlYXRlIHRhYmxlIGhpc3RvZ3JhbV90ZXN0XzYoYSBkb3Vi bGUgcHJlY2lzaW9uLCBiIGRvdWJsZSBwcmVjaXNpb24sIGMgZG91YmxlIHByZWNpc2lvbiwg ZCBkb3VibGUgcHJlY2lzaW9uKTsKaW5zZXJ0IGludG8gaGlzdG9ncmFtX3Rlc3RfNiBzZWxl Y3QgeCwgeCwgeCwgeCBmcm9tIChzZWxlY3QgcmFuZG9tKCkgKiAxMDAwMCBhcyB4IGZyb20g Z2VuZXJhdGVfc2VyaWVzKDEsMTAwMDAwKSkgZm9vOwoKYW5hbHlzZSBoaXN0b2dyYW1fdGVz dF82OwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNiB3 aGVyZSBhIDw9IDEwMCBhbmQgYiA8PSAxMDA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICog ZnJvbSBoaXN0b2dyYW1fdGVzdF82IHdoZXJlIGEgPD0gMTAwIGFuZCBiIDw9IDEwMCBhbmQg YyA8PSAxMDA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dyYW1fdGVz dF82IHdoZXJlIGEgPD0gMTAwIGFuZCBiIDw9IDEwMCBhbmQgYyA8PSAxMDAgYW5kIGQgPD0g MTAwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNiB3 aGVyZSBhIDw9IDEwMCBhbmQgYiA+PSA5MDAgYW5kIGMgPD0gMTAwIGFuZCBkIDw9IDEwMDsK CmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhpc3RvZ3JhbV90ZXN0XzYgd2hlcmUg YSA8PSA2MDAgYW5kIGIgPj0gNDAwIGFuZCBjIDw9IDY1MCBhbmQgZCA+PSA0NTA7CgpjcmVh dGUgc3RhdGlzdGljcyBoaXN0b2dyYW1fdGVzdF82X2hpc3RvZ3JhbV9hYmNkIChuZGlzdGlu Y3QsIGhpc3RvZ3JhbSkgb24gYSxiLGMsZCBmcm9tIGhpc3RvZ3JhbV90ZXN0XzY7CmFuYWx5 c2UgaGlzdG9ncmFtX3Rlc3RfNjsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhp c3RvZ3JhbV90ZXN0XzYgd2hlcmUgYSA8PSAxMDAgYW5kIGIgPD0gMTAwOwoKZXhwbGFpbiBh bmFseXNlIHNlbGVjdCAqIGZyb20gaGlzdG9ncmFtX3Rlc3RfNiB3aGVyZSBhIDw9IDEwMCBh bmQgYiA8PSAxMDAgYW5kIGMgPD0gMTAwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZy b20gaGlzdG9ncmFtX3Rlc3RfNiB3aGVyZSBhIDw9IDEwMCBhbmQgYiA8PSAxMDAgYW5kIGMg PD0gMTAwIGFuZCBkIDw9IDEwMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIGhp c3RvZ3JhbV90ZXN0XzYgd2hlcmUgYSA8PSAxMDAgYW5kIGIgPj0gOTAwIGFuZCBjIDw9IDEw MCBhbmQgZCA8PSAxMDA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBoaXN0b2dy YW1fdGVzdF82IHdoZXJlIGEgPD0gNjAwIGFuZCBiID49IDQwMCBhbmQgYyA8PSA2NTAgYW5k IGQgPj0gNDUwOwoKLS0gcGVyZmVjdGx5IGRlcGVuZGVudCB2YWx1ZXMgY29tYmluaW5nIGEg c3Vic2V0IG9mIHZlcnkgY29tbW9uIGNvbWJpbmF0aW9ucywgYW5kIGEgc3Vic2V0Ci0tIG9m IGxhcmdlIG51bWJlciBvZiBkaXN0aW5jdCB2YWx1ZXMsIHNvIHN1aXRhYmxlIGZvciBjb21i aW5hdGlvbiBvZiBtY3YgYW5kIGhpc3RvZ3JhbQoKZHJvcCB0YWJsZSBpZiBleGlzdHMgbWN2 X2hpc3RvZ3JhbV90ZXN0Xzc7CmNyZWF0ZSB0YWJsZSBtY3ZfaGlzdG9ncmFtX3Rlc3RfNyhh IGludCwgYiBpbnQsIGMgaW50LCBkIGludCk7Cmluc2VydCBpbnRvIG1jdl9oaXN0b2dyYW1f dGVzdF83IHNlbGVjdCBtb2QoaSw1MCksIG1vZChpLDUwKSwgbW9kKGksNTApLCBtb2QoaSw1 MCkgZnJvbSBnZW5lcmF0ZV9zZXJpZXMoMSw1MDAwMCkgcyhpKTsKaW5zZXJ0IGludG8gbWN2 X2hpc3RvZ3JhbV90ZXN0Xzcgc2VsZWN0IHgsIHgsIHgsIHggZnJvbSAoc2VsZWN0IChyYW5k b20oKSAqIDIwMCk6OmludCBhcyB4IGZyb20gZ2VuZXJhdGVfc2VyaWVzKDEsNTAwMDApKSBm b287CgphbmFseXNlIG1jdl9oaXN0b2dyYW1fdGVzdF83OwoKZXhwbGFpbiBhbmFseXNlIHNl bGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzcgd2hlcmUgYSA8PSAxMCBhbmQgYiA8 PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVz dF83IHdoZXJlIGEgPD0gMTAgYW5kIGIgPD0gMTAgYW5kIGMgPD0gMTA7CgpleHBsYWluIGFu YWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfNyB3aGVyZSBhIDw9IDEw IGFuZCBiIDw9IDEwIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNl IHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzcgd2hlcmUgYSA9IDEgICBhbmQg YiA8PSAxMCBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxl Y3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF83IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAx ICAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICog ZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfNyB3aGVyZSBhID0gMSAgIGFuZCBiID0gMSAgIGFu ZCBjID49IDEwIGFuZCBkID49IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20g bWN2X2hpc3RvZ3JhbV90ZXN0Xzcgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBhbmQgYyA+ PSA5MCBhbmQgZCA+PSA5MDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9o aXN0b2dyYW1fdGVzdF83IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAyICAgYW5kIGMgPD0gMTAg YW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9n cmFtX3Rlc3RfNyB3aGVyZSBhID0gMSAgIGFuZCBiID0gMiAgIGFuZCBjID49IDEwIGFuZCBk ID49IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90 ZXN0Xzcgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBhbmQgYyA+PSA5MCBhbmQgZCA+PSA5 MDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF83 IHdoZXJlIGEgPD0gMTAgYW5kIGIgPj0gOTAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7Cgpl eHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfNyB3aGVy ZSBhIDw9IDYwIGFuZCBiID49IDQwIGFuZCBjIDw9IDY1IGFuZCBkID49IDQ1OwoKY3JlYXRl IHN0YXRpc3RpY3MgbWN2X2hpc3RvZ3JhbV90ZXN0XzdfaGlzdG9ncmFtX2FiY2QgKG5kaXN0 aW5jdCwgbWN2LCBoaXN0b2dyYW0pIG9uIGEsYixjLGQgZnJvbSBtY3ZfaGlzdG9ncmFtX3Rl c3RfNzsKYW5hbHlzZSBtY3ZfaGlzdG9ncmFtX3Rlc3RfNzsKCmV4cGxhaW4gYW5hbHlzZSBz ZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF83IHdoZXJlIGEgPD0gMTAgYW5kIGIg PD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rl c3RfNyB3aGVyZSBhIDw9IDEwIGFuZCBiIDw9IDEwIGFuZCBjIDw9IDEwOwoKZXhwbGFpbiBh bmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzcgd2hlcmUgYSA8PSAx MCBhbmQgYiA8PSAxMCBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlz ZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF83IHdoZXJlIGEgPSAxICAgYW5k IGIgPD0gMTAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2Vs ZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfNyB3aGVyZSBhID0gMSAgIGFuZCBiID0g MSAgIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAq IGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzcgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBh bmQgYyA+PSAxMCBhbmQgZCA+PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9t IG1jdl9oaXN0b2dyYW1fdGVzdF83IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAxICAgYW5kIGMg Pj0gOTAgYW5kIGQgPj0gOTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3Zf aGlzdG9ncmFtX3Rlc3RfNyB3aGVyZSBhID0gMSAgIGFuZCBiID0gMiAgIGFuZCBjIDw9IDEw IGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3Rv Z3JhbV90ZXN0Xzcgd2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBhbmQgYyA+PSAxMCBhbmQg ZCA+PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1f dGVzdF83IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAyICAgYW5kIGMgPj0gOTAgYW5kIGQgPj0g OTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3Rf NyB3aGVyZSBhIDw9IDEwIGFuZCBiID49IDkwIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoK ZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzcgd2hl cmUgYSA8PSA2MCBhbmQgYiA+PSA0MCBhbmQgYyA8PSA2NSBhbmQgZCA+PSA0NTsKCi0tIHBl cmZlY3RseSBkZXBlbmRlbnQgdmFsdWVzIGNvbWJpbmluZyBhIHN1YnNldCBvZiB2ZXJ5IGNv bW1vbiBjb21iaW5hdGlvbnMsIGFuZCBhIHN1YnNldAotLSBvZiBsYXJnZSBudW1iZXIgb2Yg ZGlzdGluY3QgdmFsdWVzLCBzbyBzdWl0YWJsZSBmb3IgY29tYmluYXRpb24gb2YgbWN2IGFu ZCBoaXN0b2dyYW0KLS0gYm90aCBzdWJzZXRzIGFyZSBza2V3ZWQgdG8gMAoKZHJvcCB0YWJs ZSBpZiBleGlzdHMgbWN2X2hpc3RvZ3JhbV90ZXN0Xzg7CmNyZWF0ZSB0YWJsZSBtY3ZfaGlz dG9ncmFtX3Rlc3RfOChhIGludCwgYiBpbnQsIGMgaW50LCBkIGludCk7Cmluc2VydCBpbnRv IG1jdl9oaXN0b2dyYW1fdGVzdF84IHNlbGVjdCBzcXJ0KG1vZChpLDI1MDApKSwgc3FydCht b2QoaSwyNTAwKSksIHNxcnQobW9kKGksMjUwMCkpLCBzcXJ0KG1vZChpLDI1MDApKSBmcm9t IGdlbmVyYXRlX3NlcmllcygxLDEwMDAwMCkgcyhpKTsKaW5zZXJ0IGludG8gbWN2X2hpc3Rv Z3JhbV90ZXN0Xzggc2VsZWN0IHgsIHgsIHgsIHggZnJvbSAoc2VsZWN0IChzcXJ0KHJhbmRv bSgpKSAqIDIwMCk6OmludCBhcyB4IGZyb20gZ2VuZXJhdGVfc2VyaWVzKDEsMTAwMDAwKSkg Zm9vOwoKYW5hbHlzZSBtY3ZfaGlzdG9ncmFtX3Rlc3RfODsKCmV4cGxhaW4gYW5hbHlzZSBz ZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF84IHdoZXJlIGEgPD0gMTAgYW5kIGIg PD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rl c3RfOCB3aGVyZSBhIDw9IDEwIGFuZCBiIDw9IDEwIGFuZCBjIDw9IDEwOwoKZXhwbGFpbiBh bmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzggd2hlcmUgYSA8PSAx MCBhbmQgYiA8PSAxMCBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlz ZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF84IHdoZXJlIGEgPSAxICAgYW5k IGIgPD0gMTAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2Vs ZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfOCB3aGVyZSBhID0gMSAgIGFuZCBiID0g MSAgIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAq IGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzggd2hlcmUgYSA9IDEgICBhbmQgYiA9IDEgICBh bmQgYyA+PSAxMCBhbmQgZCA+PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9t IG1jdl9oaXN0b2dyYW1fdGVzdF84IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAxICAgYW5kIGMg Pj0gOTAgYW5kIGQgPj0gOTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3Zf aGlzdG9ncmFtX3Rlc3RfOCB3aGVyZSBhID0gMSAgIGFuZCBiID0gMiAgIGFuZCBjIDw9IDEw IGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3Rv Z3JhbV90ZXN0Xzggd2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBhbmQgYyA+PSAxMCBhbmQg ZCA+PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1f dGVzdF84IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAyICAgYW5kIGMgPj0gOTAgYW5kIGQgPj0g OTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3Rf OCB3aGVyZSBhIDw9IDEwIGFuZCBiID49IDkwIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoK ZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzggd2hl cmUgYSA8PSA2MCBhbmQgYiA+PSA0MCBhbmQgYyA8PSA2NSBhbmQgZCA+PSA0NTsKCmNyZWF0 ZSBzdGF0aXN0aWNzIG1jdl9oaXN0b2dyYW1fdGVzdF84X2hpc3RvZ3JhbV9hYmNkIChuZGlz dGluY3QsIG1jdiwgaGlzdG9ncmFtKSBvbiBhLGIsYyxkIGZyb20gbWN2X2hpc3RvZ3JhbV90 ZXN0Xzg7CmFuYWx5c2UgbWN2X2hpc3RvZ3JhbV90ZXN0Xzg7CgpleHBsYWluIGFuYWx5c2Ug c2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfOCB3aGVyZSBhIDw9IDEwIGFuZCBi IDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90 ZXN0Xzggd2hlcmUgYSA8PSAxMCBhbmQgYiA8PSAxMCBhbmQgYyA8PSAxMDsKCmV4cGxhaW4g YW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF84IHdoZXJlIGEgPD0g MTAgYW5kIGIgPD0gMTAgYW5kIGMgPD0gMTAgYW5kIGQgPD0gMTA7CgpleHBsYWluIGFuYWx5 c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFtX3Rlc3RfOCB3aGVyZSBhID0gMSAgIGFu ZCBiIDw9IDEwIGFuZCBjIDw9IDEwIGFuZCBkIDw9IDEwOwoKZXhwbGFpbiBhbmFseXNlIHNl bGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0Xzggd2hlcmUgYSA9IDEgICBhbmQgYiA9 IDEgICBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3Qg KiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF84IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAxICAg YW5kIGMgPj0gMTAgYW5kIGQgPj0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJv bSBtY3ZfaGlzdG9ncmFtX3Rlc3RfOCB3aGVyZSBhID0gMSAgIGFuZCBiID0gMSAgIGFuZCBj ID49IDkwIGFuZCBkID49IDkwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2 X2hpc3RvZ3JhbV90ZXN0Xzggd2hlcmUgYSA9IDEgICBhbmQgYiA9IDIgICBhbmQgYyA8PSAx MCBhbmQgZCA8PSAxMDsKCmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0 b2dyYW1fdGVzdF84IHdoZXJlIGEgPSAxICAgYW5kIGIgPSAyICAgYW5kIGMgPj0gMTAgYW5k IGQgPj0gMTA7CgpleHBsYWluIGFuYWx5c2Ugc2VsZWN0ICogZnJvbSBtY3ZfaGlzdG9ncmFt X3Rlc3RfOCB3aGVyZSBhID0gMSAgIGFuZCBiID0gMiAgIGFuZCBjID49IDkwIGFuZCBkID49 IDkwOwoKZXhwbGFpbiBhbmFseXNlIHNlbGVjdCAqIGZyb20gbWN2X2hpc3RvZ3JhbV90ZXN0 Xzggd2hlcmUgYSA8PSAxMCBhbmQgYiA+PSA5MCBhbmQgYyA8PSAxMCBhbmQgZCA8PSAxMDsK CmV4cGxhaW4gYW5hbHlzZSBzZWxlY3QgKiBmcm9tIG1jdl9oaXN0b2dyYW1fdGVzdF84IHdo ZXJlIGEgPD0gNjAgYW5kIGIgPj0gNDAgYW5kIGMgPD0gNjUgYW5kIGQgPj0gNDU7Cg== --------------0E1BE69E814B19BC8B911192 Content-Type: text/x-patch; name="0002-multivariate-histograms-20180902.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-multivariate-histograms-20180902.patch" From ebdc2d2115b850384f3910f33fbaefc3d57687f3 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Sat, 1 Sep 2018 22:46:45 +0200 Subject: [PATCH 2/2] multivariate histograms --- doc/src/sgml/catalogs.sgml | 14 +- doc/src/sgml/func.sgml | 85 + doc/src/sgml/planstats.sgml | 106 + doc/src/sgml/ref/create_statistics.sgml | 29 +- src/backend/catalog/system_views.sql | 11 + src/backend/commands/statscmds.c | 33 +- src/backend/nodes/outfuncs.c | 2 +- src/backend/optimizer/util/plancat.c | 44 +- src/backend/parser/parse_utilcmd.c | 2 + src/backend/statistics/Makefile | 2 +- src/backend/statistics/README | 4 + src/backend/statistics/README.histogram | 305 +++ src/backend/statistics/dependencies.c | 2 +- src/backend/statistics/extended_stats.c | 252 +- src/backend/statistics/histogram.c | 3019 ++++++++++++++++++++++ src/backend/statistics/mcv.c | 94 +- src/backend/utils/adt/ruleutils.c | 10 + src/backend/utils/adt/selfuncs.c | 7 +- src/bin/psql/describe.c | 9 +- src/include/catalog/pg_cast.dat | 4 + src/include/catalog/pg_proc.dat | 24 + src/include/catalog/pg_statistic_ext.h | 2 + src/include/catalog/pg_type.dat | 7 + src/include/nodes/relation.h | 7 +- src/include/statistics/extended_stats_internal.h | 15 + src/include/statistics/statistics.h | 63 +- src/include/utils/selfuncs.h | 4 + src/test/regress/expected/create_table_like.out | 2 +- src/test/regress/expected/opr_sanity.out | 3 +- src/test/regress/expected/stats_ext.out | 209 +- src/test/regress/expected/type_sanity.out | 3 +- src/test/regress/sql/stats_ext.sql | 133 +- 32 files changed, 4403 insertions(+), 103 deletions(-) create mode 100644 src/backend/statistics/README.histogram create mode 100644 src/backend/statistics/histogram.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index dc7bbe5173..0edd28ad0e 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -6571,8 +6571,9 @@ SCRAM-SHA-256$<iteration count>:&l An array containing codes for the enabled statistic kinds; valid values are: d for n-distinct statistics, - f for functional dependency statistics, and - m for most common values (MCV) list statistics + f for functional dependency statistics, + m for most common values (MCV) list statistics, and + h for histogram statistics @@ -6605,6 +6606,15 @@ SCRAM-SHA-256$<iteration count>:&l + + stxhistogram + pg_histogram + + + Histogram, serialized as pg_histogram type. + + + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 69cfe7bbe9..0a5a62685a 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -20992,6 +20992,91 @@ SELECT m.* FROM pg_statistic_ext, + + Inspecting histograms + + + pg_histogram + pg_histogram_buckets + + + + pg_histogram_buckets returns a list of all buckets + stored in a multi-column histogram, and returns the following columns: + + + + + + Name + Type + Description + + + + + index + int + index of the item in the histogram buckets + + + minvals + text[] + lower boundaries of the histogram bucket + + + maxvals + text[] + upper boundaries of the histogram bucket + + + nullsonly + boolean[] + flags identifying NULL-only dimensions + + + mininclusive + boolean[] + flags identifying which lower boundaries are inclusive + + + maxinclusive + boolean[] + flags identifying which upper boundaries are inclusive + + + frequency + double precision + frequency of this histogram bucket + + + density + double precision + density of this histogram bucket (frequency / volume) + + + bucket_volume + double precision + volume of this histogram bucket (a measure of size) + + + + + + + + The pg_histogram_buckets function can be used like this: + + +SELECT m.* FROM pg_statistic_ext, + pg_histogram_buckets(stxhistogram) m WHERE stxname = 'stts3'; + + + Values of the pg_histogram can be obtained only from the + pg_statistic.stxhistogram column. + + + diff --git a/doc/src/sgml/planstats.sgml b/doc/src/sgml/planstats.sgml index de8ef165c9..67a4f7219c 100644 --- a/doc/src/sgml/planstats.sgml +++ b/doc/src/sgml/planstats.sgml @@ -695,6 +695,112 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a <= 49 AND b > 49; + + Histograms + + + MCV lists, introduced in , + work very well for columns with only a few distinct values, and for + columns with only few common values. In those cases, MCV + lists are a very accurate approximation of the real distribution. + Histograms, briefly described in , + are meant to address the high-cardinality case. + + + + Although the example data we've used in does + not quality as a high-cardinality case, we can try creating a histogram + instead of the MCV list. With the histogram in place, + you may get a plan like this: + + +DROP STATISTICS stts2; +CREATE STATISTICS stts3 (histogram) ON a, b FROM t; +ANALYZE t; +EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1; + QUERY PLAN +------------------------------------------------------------------------------- + Seq Scan on t (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1) + Filter: ((a = 1) AND (b = 1)) + Rows Removed by Filter: 9900 + + + Which seems quite accurate. For other combinations of values the + estimates may be worse, as illustrated by the following query: + + +EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10; + QUERY PLAN +----------------------------------------------------------------------------- + Seq Scan on t (cost=0.00..195.00 rows=100 width=8) (actual rows=0 loops=1) + Filter: ((a = 1) AND (b = 10)) + Rows Removed by Filter: 10000 + + + This happens due to histograms tracking ranges of values, which makes it + impossible to decide how tuples with the exact combination of values there + are in the bucket. + + + + It's also possible to build MCV lists and a histogram, in + which case ANALYZE will build a MCV lists + with the most frequent values, and a histogram on the remaining part of the + sample. + + +DROP STATISTICS stts3; +CREATE STATISTICS stts4 (mcv, histogram) ON a, b FROM t; +ANALYZE t; +EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10; + QUERY PLAN +--------------------------------------------------------------------------- + Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1) + Filter: ((a = 1) AND (b = 10)) + Rows Removed by Filter: 10000 + + + In this case the MCV list and histogram are treated as a + single composed statistics. + + + + Similarly to MCV lists, it is possible to inspect + histograms using a function called pg_histogram_buckets, + which simply lists buckets of a histogram, along with information about + boundaries, frequencies, volume etc. When applied to the histogram from + stts3, you should get something like this: + + +SELECT m.* FROM pg_statistic_ext, + pg_histogram_buckets(stxhistogram) m WHERE stxname = 'stts3'; + index | minvals | maxvals | nullsonly | mininclusive | maxinclusive | frequency | density | bucket_volume +-------+---------+---------+-----------+--------------+--------------+-----------+----------+--------------- + 0 | {0,0} | {3,1} | {f,f} | {t,t} | {f,f} | 0.01 | 2.635 | 0.003795 + 1 | {50,0} | {99,51} | {f,f} | {t,t} | {t,f} | 0.01 | 0.034444 | 0.290323 + 2 | {0,25} | {26,37} | {f,f} | {t,t} | {f,f} | 0.01 | 0.292778 | 0.034156 +... + 61 | {60,56} | {99,62} | {f,f} | {t,t} | {t,f} | 0.02 | 0.752857 | 0.026565 + 62 | {35,25} | {50,37} | {f,f} | {t,t} | {f,f} | 0.02 | 0.878333 | 0.02277 + 63 | {81,85} | {87,99} | {f,f} | {t,t} | {f,t} | 0.02 | 1.756667 | 0.011385 +(64 rows) + + + Which shows that there are 64 buckets, with frequencies ranging between 1% + and 2%. The minvals and maxvals + show the bucket boundaries, nullsonly shows which + columns contain only null values (in the given bucket). + + + + Similarly to MCV lists, the planner applies all conditions + to the buckets, and sums the frequencies of the matching ones. For details, + see histogram_clauselist_selectivity function in + src/backend/statistics/histogram.c. + + + + diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml index fcbfa569d0..ef84341551 100644 --- a/doc/src/sgml/ref/create_statistics.sgml +++ b/doc/src/sgml/ref/create_statistics.sgml @@ -84,7 +84,8 @@ CREATE STATISTICS [ IF NOT EXISTS ] statistics_na ndistinct, which enables n-distinct statistics, and dependencies, which enables functional dependency statistics, and mcv which enables - most-common values lists. + most-common values lists, and histogram which + enables histograms. If this clause is omitted, all supported statistics kinds are included in the statistics object. For more information, see @@ -190,6 +191,32 @@ EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2); + + Create table t3 with two strongly correlated + columns, and a histogram on those two columns: + + +CREATE TABLE t3 ( + a float, + b float +); + +INSERT INTO t3 SELECT mod(i,1000), mod(i,1000) + 50 * (r - 0.5) FROM ( + SELECT i, random() r FROM generate_series(1,1000000) s(i) + ) foo; + +CREATE STATISTICS s3 WITH (histogram) ON (a, b) FROM t3; + +ANALYZE t3; + +-- small overlap +EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 500) AND (b > 500); + +-- no overlap +EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 400) AND (b > 600); + + + diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 7251552419..d823e42125 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1119,6 +1119,17 @@ LANGUAGE INTERNAL STRICT IMMUTABLE PARALLEL SAFE AS 'jsonb_insert'; +CREATE OR REPLACE FUNCTION + pg_histogram_buckets(histogram pg_histogram, otype integer DEFAULT 0, + OUT index integer, OUT minvals text[], OUT maxvals text[], + OUT nullsonly boolean[], OUT mininclusive boolean[], + OUT maxinclusive boolean[], OUT frequency double precision, + OUT density double precision, OUT bucket_volume double precision) +RETURNS SETOF record +LANGUAGE INTERNAL +STRICT IMMUTABLE PARALLEL SAFE +AS 'pg_histogram_buckets'; + -- -- The default permissions for functions mean that anyone can execute them. -- A number of functions shouldn't be executable by just anyone, but rather diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c index 903d8155e0..d7d504360d 100644 --- a/src/backend/commands/statscmds.c +++ b/src/backend/commands/statscmds.c @@ -70,12 +70,13 @@ CreateStatistics(CreateStatsStmt *stmt) Oid relid; ObjectAddress parentobject, myself; - Datum types[3]; /* one for each possible type of statistic */ + Datum types[4]; /* one for each possible type of statistic */ int ntypes; ArrayType *stxkind; bool build_ndistinct; bool build_dependencies; bool build_mcv; + bool build_histogram; bool requested_type = false; int i; ListCell *cell; @@ -271,6 +272,7 @@ CreateStatistics(CreateStatsStmt *stmt) build_ndistinct = false; build_dependencies = false; build_mcv = false; + build_histogram = false; foreach(cell, stmt->stat_types) { char *type = strVal((Value *) lfirst(cell)); @@ -290,6 +292,11 @@ CreateStatistics(CreateStatsStmt *stmt) build_mcv = true; requested_type = true; } + else if (strcmp(type, "histogram") == 0) + { + build_histogram = true; + requested_type = true; + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -302,6 +309,7 @@ CreateStatistics(CreateStatsStmt *stmt) build_ndistinct = true; build_dependencies = true; build_mcv = true; + build_histogram = true; } /* construct the char array of enabled statistic types */ @@ -312,6 +320,8 @@ CreateStatistics(CreateStatsStmt *stmt) types[ntypes++] = CharGetDatum(STATS_EXT_DEPENDENCIES); if (build_mcv) types[ntypes++] = CharGetDatum(STATS_EXT_MCV); + if (build_histogram) + types[ntypes++] = CharGetDatum(STATS_EXT_HISTOGRAM); Assert(ntypes > 0 && ntypes <= lengthof(types)); stxkind = construct_array(types, ntypes, CHAROID, 1, true, 'c'); @@ -331,6 +341,7 @@ CreateStatistics(CreateStatsStmt *stmt) nulls[Anum_pg_statistic_ext_stxndistinct - 1] = true; nulls[Anum_pg_statistic_ext_stxdependencies - 1] = true; nulls[Anum_pg_statistic_ext_stxmcv - 1] = true; + nulls[Anum_pg_statistic_ext_stxhistogram - 1] = true; /* insert it into pg_statistic_ext */ statrel = heap_open(StatisticExtRelationId, RowExclusiveLock); @@ -435,8 +446,9 @@ RemoveStatisticsById(Oid statsOid) * values, this assumption could fail. But that seems like a corner case * that doesn't justify zapping the stats in common cases.) * - * For MCV lists that's not the case, as those statistics store the datums - * internally. In this case we simply reset the statistics value to NULL. + * For MCV lists and histograms that's not the case, as those statistics + * store the datums internally. In those cases we simply reset those + * statistics to NULL. */ void UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, @@ -473,9 +485,10 @@ UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, /* * We can also leave the record as it is if there are no statistics - * including the datum values, like for example MCV lists. + * including the datum values, like for example MCV and histograms. */ - if (statext_is_kind_built(oldtup, STATS_EXT_MCV)) + if (statext_is_kind_built(oldtup, STATS_EXT_MCV) || + statext_is_kind_built(oldtup, STATS_EXT_HISTOGRAM)) reset_stats = true; /* @@ -496,11 +509,11 @@ UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, memset(replaces, 0, Natts_pg_statistic_ext * sizeof(bool)); memset(values, 0, Natts_pg_statistic_ext * sizeof(Datum)); - if (statext_is_kind_built(oldtup, STATS_EXT_MCV)) - { - replaces[Anum_pg_statistic_ext_stxmcv - 1] = true; - nulls[Anum_pg_statistic_ext_stxmcv - 1] = true; - } + replaces[Anum_pg_statistic_ext_stxmcv - 1] = true; + replaces[Anum_pg_statistic_ext_stxhistogram - 1] = true; + + nulls[Anum_pg_statistic_ext_stxmcv - 1] = true; + nulls[Anum_pg_statistic_ext_stxhistogram - 1] = true; rel = heap_open(StatisticExtRelationId, RowExclusiveLock); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index b5af904c18..053cbc498f 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2437,7 +2437,7 @@ _outStatisticExtInfo(StringInfo str, const StatisticExtInfo *node) /* NB: this isn't a complete set of fields */ WRITE_OID_FIELD(statOid); /* don't write rel, leads to infinite recursion in plan tree dump */ - WRITE_CHAR_FIELD(kind); + WRITE_INT_FIELD(kinds); WRITE_BITMAPSET_FIELD(keys); } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 0112450419..1079183ccc 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1324,6 +1324,9 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) HeapTuple htup; Bitmapset *keys = NULL; int i; + int kind = 0; + + StatisticExtInfo *info = makeNode(StatisticExtInfo); htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid)); if (!htup) @@ -1338,42 +1341,25 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) for (i = 0; i < staForm->stxkeys.dim1; i++) keys = bms_add_member(keys, staForm->stxkeys.values[i]); - /* add one StatisticExtInfo for each kind built */ + /* now build the bitmask of statistics kinds */ if (statext_is_kind_built(htup, STATS_EXT_NDISTINCT)) - { - StatisticExtInfo *info = makeNode(StatisticExtInfo); - - info->statOid = statOid; - info->rel = rel; - info->kind = STATS_EXT_NDISTINCT; - info->keys = bms_copy(keys); - - stainfos = lcons(info, stainfos); - } + kind |= STATS_EXT_INFO_NDISTINCT; if (statext_is_kind_built(htup, STATS_EXT_DEPENDENCIES)) - { - StatisticExtInfo *info = makeNode(StatisticExtInfo); - - info->statOid = statOid; - info->rel = rel; - info->kind = STATS_EXT_DEPENDENCIES; - info->keys = bms_copy(keys); - - stainfos = lcons(info, stainfos); - } + kind |= STATS_EXT_INFO_DEPENDENCIES; if (statext_is_kind_built(htup, STATS_EXT_MCV)) - { - StatisticExtInfo *info = makeNode(StatisticExtInfo); + kind |= STATS_EXT_INFO_MCV; - info->statOid = statOid; - info->rel = rel; - info->kind = STATS_EXT_MCV; - info->keys = bms_copy(keys); + if (statext_is_kind_built(htup, STATS_EXT_HISTOGRAM)) + kind |= STATS_EXT_INFO_HISTOGRAM; - stainfos = lcons(info, stainfos); - } + info->statOid = statOid; + info->rel = rel; + info->kinds = kind; + info->keys = bms_copy(keys); + + stainfos = lcons(info, stainfos); ReleaseSysCache(htup); bms_free(keys); diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 29877126d7..58b06fca92 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1683,6 +1683,8 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, stat_types = lappend(stat_types, makeString("dependencies")); else if (enabled[i] == STATS_EXT_MCV) stat_types = lappend(stat_types, makeString("mcv")); + else if (enabled[i] == STATS_EXT_HISTOGRAM) + stat_types = lappend(stat_types, makeString("histogram")); else elog(ERROR, "unrecognized statistics kind %c", enabled[i]); } diff --git a/src/backend/statistics/Makefile b/src/backend/statistics/Makefile index d2815265fb..3e5ad454cd 100644 --- a/src/backend/statistics/Makefile +++ b/src/backend/statistics/Makefile @@ -12,6 +12,6 @@ subdir = src/backend/statistics top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -OBJS = extended_stats.o dependencies.o mcv.o mvdistinct.o +OBJS = extended_stats.o dependencies.o histogram.o mcv.o mvdistinct.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/statistics/README b/src/backend/statistics/README index 8f153a9e85..9de750614f 100644 --- a/src/backend/statistics/README +++ b/src/backend/statistics/README @@ -20,6 +20,8 @@ There are currently two kinds of extended statistics: (c) MCV lists (README.mcv) + (d) histograms (README.histogram) + Compatible clause types ----------------------- @@ -30,6 +32,8 @@ Each type of statistics may be used to estimate some subset of clause types. (b) MCV lists - equality and inequality clauses (AND, OR, NOT), IS NULL + (c) histogram - equality and inequality clauses (AND, OR, NOT), IS NULL + Currently, only OpExprs in the form Var op Const, or Const op Var are supported, however it's feasible to expand the code later to also estimate the selectivities on clauses such as Var op Var. diff --git a/src/backend/statistics/README.histogram b/src/backend/statistics/README.histogram new file mode 100644 index 0000000000..e1a4504502 --- /dev/null +++ b/src/backend/statistics/README.histogram @@ -0,0 +1,305 @@ +Multivariate histograms +======================= + +Histograms on individual attributes consist of buckets represented by ranges, +covering the domain of the attribute. That is, each bucket is a [min,max] +interval, and contains all values in this range. The histogram is built in such +a way that all buckets have about the same frequency. + +Multivariate histograms are an extension into n-dimensional space - the buckets +are n-dimensional intervals (i.e. n-dimensional rectagles), covering the domain +of the combination of attributes. That is, each bucket has a vector of lower +and upper boundaries, denoted min[i] and max[i] (where i = 1..n). + +In addition to the boundaries, each bucket tracks additional info: + + * frequency (fraction of tuples in the bucket) + * whether the boundaries are inclusive or exclusive + * whether the dimension contains only NULL values + * number of distinct values in each dimension (for building only) + +It's possible that in the future we'll multiple histogram types, with different +features. We do however expect all the types to share the same representation +(buckets as ranges) and only differ in how we build them. + +The current implementation builds non-overlapping buckets, that may not be true +for some histogram types and the code should not rely on this assumption. There +are interesting types of histograms (or algorithms) with overlapping buckets. + +When used on low-cardinality data, histograms usually perform considerably worse +than MCV lists (which are a good fit for this kind of data). This is especially +true on label-like values, where ordering of the values is mostly unrelated to +meaning of the data, as proper ordering is crucial for histograms. + +On high-cardinality data the histograms are usually a better choice, because MCV +lists can't represent the distribution accurately enough. + + +Selectivity estimation +---------------------- + +The estimation is implemented in clauselist_mv_selectivity_histogram(), and +works very similarly to clauselist_mv_selectivity_mcvlist. + +The main difference is that while MCV lists support exact matches, histograms +often result in approximate matches - e.g. with equality we can only say if +the constant would be part of the bucket, but not whether it really is there +or what fraction of the bucket it corresponds to. In this case we rely on +some defaults just like in the per-column histograms. + +The current implementation uses histograms to estimates those types of clauses +(think of WHERE conditions): + + (a) equality clauses WHERE (a = 1) AND (b = 2) + (b) inequality clauses WHERE (a < 1) AND (b >= 2) + (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL) + (d) OR-clauses WHERE (a = 1) OR (b = 2) + +Similarly to MCV lists, it's possible to add support for additional types of +clauses, for example: + + (e) multi-var clauses WHERE (a > b) + +and so on. These are tasks for the future, not yet implemented. + + +When evaluating a clause on a bucket, we may get one of three results: + + (a) FULL_MATCH - The bucket definitely matches the clause. + + (b) PARTIAL_MATCH - The bucket matches the clause, but not necessarily all + the tuples it represents. + + (c) NO_MATCH - The bucket definitely does not match the clause. + +This may be illustrated using a range [1, 5], which is essentially a 1-D bucket. +With clause + + WHERE (a < 10) => FULL_MATCH (all range values are below + 10, so the whole bucket matches) + + WHERE (a < 3) => PARTIAL_MATCH (there may be values matching + the clause, but we don't know how many) + + WHERE (a < 0) => NO_MATCH (the whole range is above 1, so + no values from the bucket can match) + +Some clauses may produce only some of those results - for example equality +clauses may never produce FULL_MATCH as we always hit only part of the bucket +(we can't match both boundaries at the same time). This results in less accurate +estimates compared to MCV lists, where we can hit a MCV items exactly (there's +no PARTIAL match in MCV). + +There are also clauses that may not produce any PARTIAL_MATCH results. A nice +example of that is 'IS [NOT] NULL' clause, which either matches the bucket +completely (FULL_MATCH) or not at all (NO_MATCH), thanks to how the NULL-buckets +are constructed. + +Computing the total selectivity estimate is trivial - simply sum selectivities +from all the FULL_MATCH and PARTIAL_MATCH buckets (but for buckets marked with +PARTIAL_MATCH, multiply the frequency by 0.5 to minimize the average error). + + +Building a histogram +--------------------- + +The algorithm of building a histogram in general is quite simple: + + (a) create an initial bucket (containing all sample rows) + + (b) create NULL buckets (by splitting the initial bucket) + + (c) repeat + + (1) choose bucket to split next + + (2) terminate if no bucket that might be split found, or if we've + reached the maximum number of buckets (16384) + + (3) choose dimension to partition the bucket by + + (4) partition the bucket by the selected dimension + +The main complexity is hidden in steps (c.1) and (c.3), i.e. how we choose the +bucket and dimension for the split, as discussed in the next section. + + +Partitioning criteria +--------------------- + +Similarly to one-dimensional histograms, we want to produce buckets with roughly +the same frequency. + +We also need to produce "regular" buckets, because buckets with one dimension +much longer than the others are very likely to match a lot of conditions (which +increases error, even if the bucket frequency is very low). + +This is especially important when handling OR-clauses, because in that case each +clause may add buckets independently. With AND-clauses all the clauses have to +match each bucket, which makes this issue somewhat less concenrning. + +To achieve this, we choose the largest bucket (containing the most sample rows), +but we only choose buckets that can actually be split (have at least 3 different +combinations of values). + +Then we choose the "longest" dimension of the bucket, which is computed by using +the distinct values in the sample as a measure. + +For details see functions select_bucket_to_partition() and partition_bucket(), +which also includes further discussion. + + +The current limit on number of buckets (16384) is mostly arbitrary, but chosen +so that it guarantees we don't exceed the number of distinct values indexable by +uint16 in any of the dimensions. In practice we could handle more buckets as we +index each dimension separately and the splits should use the dimensions evenly. + +Also, histograms this large (with 16k values in multiple dimensions) would be +quite expensive to build and process, so the 16k limit is rather reasonable. + +The actual number of buckets is also related to statistics target, because we +require MIN_BUCKET_ROWS (10) tuples per bucket before a split, so we can't have +more than (2 * 300 * target / 10) buckets. For the default target (100) this +evaluates to ~6k. + + +NULL handling (create_null_buckets) +----------------------------------- + +When building histograms on a single attribute, we first filter out NULL values. +In the multivariate case, we can't really do that because the rows may contain +a mix of NULL and non-NULL values in different columns (so we can't simply +filter all of them out). + +For this reason, the histograms are built in a way so that for each bucket, each +dimension only contains only NULL or non-NULL values. Building the NULL-buckets +happens as the first step in the build, by the create_null_buckets() function. +The number of NULL buckets, as produced by this function, has a clear upper +boundary (2^N) where N is the number of dimensions (attributes the histogram is +built on). Or rather 2^K where K is the number of attributes that are not marked +as not-NULL. + +The buckets with NULL dimensions are then subject to the same build algorithm +(i.e. may be split into smaller buckets) just like any other bucket, but may +only be split by non-NULL dimension. + + +Serialization +------------- + +To store the histogram in pg_statistic_ext table, it is serialized into a more +efficient form. We also use the representation for estimation, i.e. we don't +fully deserialize the histogram. + +For example the boundary values are deduplicated to minimize the required space. +How much redundancy is there, actually? Let's assume there are no NULL values, +so we start with a single bucket - in that case we have 2*N boundaries. Each +time we split a bucket we introduce one new value (in the "middle" of one of +the dimensions), and keep boundries for all the other dimensions. So after K +splits, we have up to + + 2*N + K + +unique boundary values (we may have fewe values, if the same value is used for +several splits). But after K splits we do have (K+1) buckets, so + + (K+1) * 2 * N + +boundary values. Using e.g. N=4 and K=999, we arrive to those numbers: + + 2*N + K = 1007 + (K+1) * 2 * N = 8000 + +wich means a lot of redundancy. It's somewhat counter-intuitive that the number +of distinct values does not really depend on the number of dimensions (except +for the initial bucket, but that's negligible compared to the total). + +By deduplicating the values and replacing them with 16-bit indexes (uint16), we +reduce the required space to + + 1007 * 8 + 8000 * 2 ~= 24kB + +which is significantly less than 64kB required for the 'raw' histogram (assuming +the values are 8B). + +While the bytea compression (pglz) might achieve the same reduction of space, +the deduplicated representation is used to optimize the estimation by caching +results of function calls for already visited values. This significantly +reduces the number of calls to (often quite expensive) operators. + +Note: Of course, this reasoning only holds for histograms built by the algorithm +that simply splits the buckets in half. Other histograms types (e.g. containing +overlapping buckets) may behave differently and require different serialization. + +Serialized histograms are marked with 'magic' constant, to make it easier to +check the bytea value really is a serialized histogram. + + +varlena compression +------------------- + +This serialization may however disable automatic varlena compression, the array +of unique values is placed at the beginning of the serialized form. Which is +exactly the chunk used by pglz to check if the data is compressible, and it +will probably decide it's not very compressible. This is similar to the issue +we had with JSONB initially. + +Maybe storing buckets first would make it work, as the buckets may be better +compressible. + +On the other hand the serialization is actually a context-aware compression, +usually compressing to ~30% (or even less, with large data types). So the lack +of additional pglz compression may be acceptable. + + +Deserialization +--------------- + +The deserialization is not a perfect inverse of the serialization, as we keep +the deduplicated arrays. This reduces the amount of memory and also allows +optimizations during estimation (e.g. we can cache results for the distinct +values, saving expensive function calls). + + +Inspecting the histogram +------------------------ + +Inspecting the regular (per-attribute) histograms is trivial, as it's enough +to select the columns from pg_stats. The data is encoded as anyarrays, and +all the items have the same data type, so anyarray provides a simple way to +get a text representation. + +With multivariate histograms the columns may use different data types, making +it impossible to use anyarrays. It might be possible to produce similar +array-like representation, but that would complicate further processing and +analysis of the histogram. + +So instead the histograms are stored in a custom data type (pg_histogram), +which however makes it more difficult to inspect the contents. To make that +easier, there's a SRF returning detailed information about the histogram. + + SELECT * FROM pg_histogram_buckets(); + +It has two input parameters: + + histogram - OID of the histogram (pg_statistic_ext.stxhistogram) + otype - type of output + +and produces a table with these columns: + + - bucket ID (0...nbuckets-1) + - lower bucket boundaries (string array) + - upper bucket boundaries (string array) + - nulls only dimensions (boolean array) + - lower boundary inclusive (boolean array) + - upper boundary includive (boolean array) + - frequency (double precision) + - density (double precision) + - volume (double precision) + +The 'otype' accepts three values, determining what will be returned in the +lower/upper boundary arrays: + + - 0 - values stored in the histogram, encoded as text + - 1 - indexes into the deduplicated arrays + - 2 - idnexes into the deduplicated arrays, scaled to [0,1] diff --git a/src/backend/statistics/dependencies.c b/src/backend/statistics/dependencies.c index 29e816c4f7..5505feb913 100644 --- a/src/backend/statistics/dependencies.c +++ b/src/backend/statistics/dependencies.c @@ -932,7 +932,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root, int listidx; /* check if there's any stats that might be useful for us. */ - if (!has_stats_of_kind(rel->statlist, STATS_EXT_DEPENDENCIES)) + if (!has_stats_of_kind(rel->statlist, STATS_EXT_INFO_DEPENDENCIES)) return 1.0; list_attnums = (AttrNumber *) palloc(sizeof(AttrNumber) * diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c index 0b66000705..25237cd53c 100644 --- a/src/backend/statistics/extended_stats.c +++ b/src/backend/statistics/extended_stats.c @@ -38,7 +38,6 @@ #include "utils/selfuncs.h" #include "utils/syscache.h" - /* * Used internally to refer to an individual statistics object, i.e., * a pg_statistic_ext entry. @@ -58,7 +57,7 @@ static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs, int nvacatts, VacAttrStats **vacatts); static void statext_store(Relation pg_stext, Oid relid, MVNDistinct *ndistinct, MVDependencies *dependencies, - MCVList * mcvlist, VacAttrStats **stats); + MCVList * mcvlist, MVHistogram * histogram, VacAttrStats **stats); /* @@ -92,10 +91,14 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, StatExtEntry *stat = (StatExtEntry *) lfirst(lc); MVNDistinct *ndistinct = NULL; MVDependencies *dependencies = NULL; + MVHistogram *histogram = NULL; MCVList *mcv = NULL; VacAttrStats **stats; ListCell *lc2; + bool build_mcv = false; + bool build_histogram = false; + /* * Check if we can build these stats based on the column analyzed. If * not, report this fact (except in autovacuum) and move on. @@ -131,12 +134,49 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, dependencies = statext_dependencies_build(numrows, rows, stat->columns, stats); else if (t == STATS_EXT_MCV) - mcv = statext_mcv_build(numrows, rows, stat->columns, stats, - totalrows); + build_mcv = true; + else if (t == STATS_EXT_HISTOGRAM) + build_histogram = true; + } + + /* + * If asked to build both MCV and histogram, first build the MCV part + * and then histogram on the remaining rows. + */ + if (build_mcv && build_histogram) + { + HeapTuple *rows_filtered = NULL; + int numrows_filtered; + + mcv = statext_mcv_build(numrows, rows, stat->columns, stats, + &rows_filtered, &numrows_filtered, + totalrows); + + /* + * Only build the histogram when there are rows not covered by + * MCV. + */ + if (rows_filtered) + { + Assert(numrows_filtered > 0); + + histogram = statext_histogram_build(numrows_filtered, rows_filtered, + stat->columns, stats, numrows); + + /* free this immediately, as we may be building many stats */ + pfree(rows_filtered); + } } + else if (build_mcv) + mcv = statext_mcv_build(numrows, rows, stat->columns, stats, + NULL, NULL, totalrows); + else if (build_histogram) + histogram = statext_histogram_build(numrows, rows, stat->columns, + stats, numrows); /* store the statistics in the catalog */ - statext_store(pg_stext, stat->statOid, ndistinct, dependencies, mcv, stats); + statext_store(pg_stext, stat->statOid, ndistinct, dependencies, mcv, + histogram, stats); } heap_close(pg_stext, RowExclusiveLock); @@ -168,6 +208,10 @@ statext_is_kind_built(HeapTuple htup, char type) attnum = Anum_pg_statistic_ext_stxmcv; break; + case STATS_EXT_HISTOGRAM: + attnum = Anum_pg_statistic_ext_stxhistogram; + break; + default: elog(ERROR, "unexpected statistics type requested: %d", type); } @@ -233,7 +277,8 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) { Assert((enabled[i] == STATS_EXT_NDISTINCT) || (enabled[i] == STATS_EXT_DEPENDENCIES) || - (enabled[i] == STATS_EXT_MCV)); + (enabled[i] == STATS_EXT_MCV) || + (enabled[i] == STATS_EXT_HISTOGRAM)); entry->types = lappend_int(entry->types, (int) enabled[i]); } @@ -308,7 +353,7 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs, static void statext_store(Relation pg_stext, Oid statOid, MVNDistinct *ndistinct, MVDependencies *dependencies, - MCVList * mcv, VacAttrStats **stats) + MCVList * mcv, MVHistogram * histogram, VacAttrStats **stats) { HeapTuple stup, oldtup; @@ -347,10 +392,18 @@ statext_store(Relation pg_stext, Oid statOid, values[Anum_pg_statistic_ext_stxmcv - 1] = PointerGetDatum(data); } + if (histogram != NULL) + { + /* histogram already is a bytea value, not need to serialize */ + nulls[Anum_pg_statistic_ext_stxhistogram - 1] = (histogram == NULL); + values[Anum_pg_statistic_ext_stxhistogram - 1] = PointerGetDatum(histogram); + } + /* always replace the value (either by bytea or NULL) */ replaces[Anum_pg_statistic_ext_stxndistinct - 1] = true; replaces[Anum_pg_statistic_ext_stxdependencies - 1] = true; replaces[Anum_pg_statistic_ext_stxmcv - 1] = true; + replaces[Anum_pg_statistic_ext_stxhistogram - 1] = true; /* there should already be a pg_statistic_ext tuple */ oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid)); @@ -465,6 +518,19 @@ compare_scalars_simple(const void *a, const void *b, void *arg) (SortSupport) arg); } +/* + * qsort_arg comparator for sorting data when partitioning a MV bucket + */ +int +compare_scalars_partition(const void *a, const void *b, void *arg) +{ + Datum da = ((ScalarItem *) a)->value; + Datum db = ((ScalarItem *) b)->value; + SortSupport ssup = (SortSupport) arg; + + return ApplySortComparator(da, false, db, false, ssup); +} + int compare_datums_simple(Datum a, Datum b, SortSupport ssup) { @@ -590,10 +656,11 @@ build_sorted_items(int numrows, HeapTuple *rows, TupleDesc tdesc, /* * has_stats_of_kind - * Check whether the list contains statistic of a given kind + * Check whether the list contains statistic of a given kind (at least + * one of those specified statistics types). */ bool -has_stats_of_kind(List *stats, char requiredkind) +has_stats_of_kind(List *stats, int requiredkinds) { ListCell *l; @@ -601,7 +668,7 @@ has_stats_of_kind(List *stats, char requiredkind) { StatisticExtInfo *stat = (StatisticExtInfo *) lfirst(l); - if (stat->kind == requiredkind) + if (stat->kinds & requiredkinds) return true; } @@ -623,7 +690,7 @@ has_stats_of_kind(List *stats, char requiredkind) * further tiebreakers are needed. */ StatisticExtInfo * -choose_best_statistics(List *stats, Bitmapset *attnums, char requiredkind) +choose_best_statistics(List *stats, Bitmapset *attnums, int requiredkinds) { ListCell *lc; StatisticExtInfo *best_match = NULL; @@ -637,8 +704,8 @@ choose_best_statistics(List *stats, Bitmapset *attnums, char requiredkind) int numkeys; Bitmapset *matched; - /* skip statistics that are not of the correct type */ - if (info->kind != requiredkind) + /* skip statistics that do not match any of the requested types */ + if ((info->kinds & requiredkinds) == 0) continue; /* determine how many attributes of these stats can be matched to */ @@ -843,7 +910,7 @@ statext_is_compatible_clause_internal(Node *clause, Index relid, Bitmapset **att /* * statext_is_compatible_clause - * Determines if the clause is compatible with MCV lists. + * Determines if the clause is compatible with MCV lists and histograms * * Only OpExprs with two arguments using an equality operator are supported. * When returning True attnum is set to the attribute number of the Var within @@ -873,6 +940,89 @@ statext_is_compatible_clause(Node *clause, Index relid, Bitmapset **attnums) } /* + * examine_equality_clause + * Extract variable from a simple top-level equality clause. + * + * For simple equality clause (Var = Const) or (Const = Var) extracts + * the Var. For other clauses returns NULL. + */ +static Var * +examine_equality_clause(PlannerInfo *root, RestrictInfo *rinfo) +{ + OpExpr *expr; + Var *var; + bool ok; + bool varonleft = true; + + if (!IsA(rinfo->clause, OpExpr)) + return NULL; + + expr = (OpExpr *) rinfo->clause; + + if (list_length(expr->args) != 2) + return NULL; + + /* see if it actually has the right */ + ok = (NumRelids((Node *) expr) == 1) && + (is_pseudo_constant_clause(lsecond(expr->args)) || + (varonleft = false, + is_pseudo_constant_clause(linitial(expr->args)))); + + /* unsupported structure (two variables or so) */ + if (!ok) + return NULL; + + if (get_oprrest(expr->opno) != F_EQSEL) + return NULL; + + var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + + return var; +} + +/* + * estimate_equality_groups + * Estimates number of groups for attributes in equality clauses. + * + * Extracts simple top-level equality clauses, and estimates ndistinct + * for that combination (using simplified estimate_num_groups). Then + * returns number of attributes with an equality clause, and a lists + * of equality clauses (to use as conditions for histograms) and also + * remaining non-equality clauses. + */ +static double +estimate_equality_groups(PlannerInfo *root, List *clauses, + List **eqclauses, List **neqclauses) +{ + List *vars = NIL; + ListCell *lc; + + *eqclauses = NIL; + *neqclauses = NIL; + + foreach(lc, clauses) + { + Var *var; + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + Assert(IsA(rinfo, RestrictInfo)); + + var = examine_equality_clause(root, rinfo); + + /* is it a simple equality clause */ + if (var) + { + vars = lappend(vars, var); + *eqclauses = lappend(*eqclauses, rinfo); + } + else + *neqclauses = lappend(*neqclauses, rinfo); + } + + return estimate_num_groups_simple(root, vars); +} + +/* * statext_clauselist_selectivity * Estimate clauses using the best multi-column statistics. * @@ -937,13 +1087,14 @@ statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, mcv_sel, mcv_basesel, mcv_totalsel, + histogram_sel, other_sel, sel; - /* we're interested in MCV lists */ - int types = STATS_EXT_MCV; + /* we're interested in MCV lists and histograms */ + int types = (STATS_EXT_INFO_MCV | STATS_EXT_INFO_HISTOGRAM); - /* check if there's any stats that might be useful for us. */ + /* Check if there's any stats that might be useful for us. */ if (!has_stats_of_kind(rel->statlist, types)) return (Selectivity) 1.0; @@ -994,8 +1145,8 @@ statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, if (!stat) return (Selectivity) 1.0; - /* We only understand MCV lists for now. */ - Assert(stat->kind == STATS_EXT_MCV); + /* We only understand MCV lists and histograms for now. */ + Assert(stat->kinds & (STATS_EXT_INFO_MCV | STATS_EXT_INFO_HISTOGRAM)); /* now filter the clauses to be estimated using the selected MCV */ stat_clauses = NIL; @@ -1018,28 +1169,59 @@ statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, } /* - * First compute "simple" selectivity, i.e. without the extended statistics, - * and essentially assuming independence of the columns/clauses. We'll then - * use the various selectivities computed from MCV list to improve it. + * For statistics with MCV list, we'll estimate the MCV and non-MCV parts. */ - simple_sel = clauselist_selectivity_simple(root, stat_clauses, varRelid, - jointype, sjinfo, NULL); + if (stat->kinds & STATS_EXT_INFO_MCV) + { + /* + * First compute "simple" selectivity, i.e. without the extended statistics, + * and essentially assuming independence of the columns/clauses. We'll then + * use the various selectivities computed from MCV list to improve it. + */ + simple_sel = clauselist_selectivity_simple(root, stat_clauses, varRelid, + jointype, sjinfo, NULL); + + /* + * Now compute the multi-column estimate from the MCV list, along with the + * other selectivities (base & total selectivity). + */ + mcv_sel = mcv_clauselist_selectivity(root, stat, stat_clauses, varRelid, + jointype, sjinfo, rel, + &mcv_basesel, &mcv_totalsel); + + /* Estimated selectivity of values not covered by MCV matches */ + other_sel = simple_sel - mcv_basesel; + CLAMP_PROBABILITY(other_sel); + + /* The non-MCV selectivity can't exceed the 1 - mcv_totalsel. */ + if (other_sel > 1.0 - mcv_totalsel) + other_sel = 1.0 - mcv_totalsel; + } + else + { + /* Otherwise just remember there was no MCV list. */ + mcv_totalsel = 0.0; + } /* - * Now compute the multi-column estimate from the MCV list, along with the - * other selectivities (base & total selectivity). + * If we have a histogram, we'll use it to improve the non-MCV estimate. */ - mcv_sel = mcv_clauselist_selectivity(root, stat, stat_clauses, varRelid, - jointype, sjinfo, rel, - &mcv_basesel, &mcv_totalsel); + if (stat->kinds & STATS_EXT_INFO_HISTOGRAM) + { + List *eqclauses, + *neqclauses; + double ngroups; - /* Estimated selectivity of values not covered by MCV matches */ - other_sel = simple_sel - mcv_basesel; - CLAMP_PROBABILITY(other_sel); + ngroups = estimate_equality_groups(root, stat_clauses, + &eqclauses, &neqclauses); - /* The non-MCV selectivity can't exceed the 1 - mcv_totalsel. */ - if (other_sel > 1.0 - mcv_totalsel) - other_sel = 1.0 - mcv_totalsel; + histogram_sel = histogram_clauselist_selectivity(root, stat, + neqclauses, eqclauses, + varRelid, jointype, + sjinfo, rel); + + other_sel = (1 / ngroups) * histogram_sel; + } /* Overall selectivity is the combination of MCV and non-MCV estimates. */ sel = mcv_sel + other_sel; diff --git a/src/backend/statistics/histogram.c b/src/backend/statistics/histogram.c new file mode 100644 index 0000000000..1ff34a53c0 --- /dev/null +++ b/src/backend/statistics/histogram.c @@ -0,0 +1,3019 @@ +/*------------------------------------------------------------------------- + * + * histogram.c + * POSTGRES multivariate histograms + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/statistics/histogram.c + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include + +#include "access/htup_details.h" +#include "catalog/pg_collation.h" +#include "catalog/pg_statistic_ext.h" +#include "fmgr.h" +#include "funcapi.h" +#include "optimizer/clauses.h" +#include "statistics/extended_stats_internal.h" +#include "statistics/statistics.h" +#include "utils/builtins.h" +#include "utils/bytea.h" +#include "utils/fmgroids.h" +#include "utils/lsyscache.h" +#include "utils/selfuncs.h" +#include "utils/syscache.h" +#include "utils/typcache.h" + + +/* + * Multivariate histograms + */ +typedef struct MVBucketBuild +{ + /* Frequencies of this bucket. */ + float frequency; + + /* + * Information about dimensions being NULL-only. Not yet used. + */ + bool *nullsonly; + + /* lower boundaries - values and information about the inequalities */ + Datum *min; + bool *min_inclusive; + + /* upper boundaries - values and information about the inequalities */ + Datum *max; + bool *max_inclusive; + + /* number of distinct values in each dimension */ + uint32 *ndistincts; + + /* number of distinct combination of values */ + uint32 ndistinct; + + /* aray of sample rows (for this bucket) */ + HeapTuple *rows; + uint32 numrows; + +} MVBucketBuild; + +typedef struct MVHistogramBuild +{ + int32 vl_len_; /* unused: ensure same alignment as + * MVHistogram for serialization */ + uint32 magic; /* magic constant marker */ + uint32 type; /* type of histogram (BASIC) */ + uint32 nbuckets; /* number of buckets (buckets array) */ + uint32 ndimensions; /* number of dimensions */ + Oid types[STATS_MAX_DIMENSIONS]; /* OIDs of data types */ + MVBucketBuild **buckets; /* array of buckets */ +} MVHistogramBuild; + +static MVBucketBuild * create_initial_ext_bucket(int numrows, HeapTuple *rows, + Bitmapset *attrs, VacAttrStats **stats); + +static MVBucketBuild * select_bucket_to_partition(int nbuckets, MVBucketBuild * *buckets); + +static MVBucketBuild * partition_bucket(MVBucketBuild * bucket, Bitmapset *attrs, + VacAttrStats **stats, + int *ndistvalues, Datum **distvalues); + +static MVBucketBuild * copy_ext_bucket(MVBucketBuild * bucket, uint32 ndimensions); + +static void update_bucket_ndistinct(MVBucketBuild * bucket, Bitmapset *attrs, + VacAttrStats **stats); + +static void update_dimension_ndistinct(MVBucketBuild * bucket, int dimension, + Bitmapset *attrs, VacAttrStats **stats, + bool update_boundaries); + +static void create_null_buckets(MVHistogramBuild * histogram, int bucket_idx, + Bitmapset *attrs, VacAttrStats **stats); + +static Datum *build_ndistinct(int numrows, HeapTuple *rows, Bitmapset *attrs, + VacAttrStats **stats, int i, int *nvals); + +static MVHistogram * serialize_histogram(MVHistogramBuild * histogram, + VacAttrStats **stats); + +/* + * Computes size of a serialized histogram bucket, depending on the number + * of dimentions (columns) the statistic is defined on. The datum values + * are stored in a separate array (deduplicated, to minimize the size), and + * so the serialized buckets only store uint16 indexes into that array. + * + * Each serialized bucket needs to store (in this order): + * + * - number of tuples (float) + * - number of distinct (float) + * - min inclusive flags (ndim * sizeof(bool)) + * - max inclusive flags (ndim * sizeof(bool)) + * - null dimension flags (ndim * sizeof(bool)) + * - min boundary indexes (2 * ndim * sizeof(uint16)) + * - max boundary indexes (2 * ndim * sizeof(uint16)) + * + * So in total: + * + * ndim * (4 * sizeof(uint16) + 3 * sizeof(bool)) + (2 * sizeof(float)) + * + * XXX We might save a bit more space by using proper bitmaps instead of + * boolean arrays. + */ +#define BUCKET_SIZE(ndims) \ + (ndims * (4 * sizeof(uint16) + 3 * sizeof(bool)) + sizeof(float)) + +/* + * Macros for convenient access to parts of a serialized bucket. + */ +#define BUCKET_FREQUENCY(b) (*(float*)b) +#define BUCKET_MIN_INCL(b,n) ((bool*)(b + sizeof(float))) +#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n) +#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n) +#define BUCKET_MIN_INDEXES(b,n) ((uint16*)(BUCKET_NULLS_ONLY(b,n) + n)) +#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n)) + +/* + * Minimal number of rows per bucket (can't split smaller buckets). + * + * XXX The single-column statistics (std_typanalyze) pretty much says we + * need 300 rows per bucket. Should we use the same value here? + */ +#define MIN_BUCKET_ROWS 10 + +/* + * Represents match info for a histogram bucket. + */ +typedef struct bucket_match +{ + bool match; /* true/false */ + double fraction; /* fraction of bucket */ +} bucket_match; + +/* + * Builds a multivariate histogram from the set of sampled rows. + * + * The build algorithm is iterative - initially a single bucket containing all + * sample rows is formed, and then repeatedly split into smaller buckets. In + * each round the largest bucket is split into two smaller ones. + * + * The criteria for selecting the largest bucket (and the dimension for the + * split) needs to be elaborate enough to produce buckets of roughly the same + * size, and also regular shape (not very narrow in just one dimension). + * + * The current algorithm works like this: + * + * a) build NULL-buckets (create_null_buckets) + * + * b) while [maximum number of buckets not reached] + * + * c) choose bucket to partition (largest bucket) + * + * c.1) if no bucket eligible to split, terminate the build + * + * c.2) choose bucket dimension to partition (largest dimension) + * + * c.3) split the bucket into two buckets + * + * See the discussion at select_bucket_to_partition and partition_bucket for + * more details about the algorithm. + * + * The function does not update the interan pointers, hence the histogram + * is suitable only for storing. Before using it for estimation, it needs + * to go through statext_histogram_deserialize() first. + */ +MVHistogram * +statext_histogram_build(int numrows, HeapTuple *rows, Bitmapset *attrs, + VacAttrStats **stats, int numrows_total) +{ + int i; + int numattrs = bms_num_members(attrs); + + int *ndistvalues; + Datum **distvalues; + + MVHistogramBuild *histogram; + HeapTuple *rows_copy; + + /* not supposed to build of too few or too many columns */ + Assert((numattrs >= 2) && (numattrs <= STATS_MAX_DIMENSIONS)); + + /* we need to make a copy of the row array, as we'll modify it */ + rows_copy = (HeapTuple *) palloc0(numrows * sizeof(HeapTuple)); + memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows); + + /* build the histogram header */ + + histogram = (MVHistogramBuild *) palloc0(sizeof(MVHistogramBuild)); + + histogram->magic = STATS_HIST_MAGIC; + histogram->type = STATS_HIST_TYPE_BASIC; + histogram->ndimensions = numattrs; + histogram->nbuckets = 1; /* initially just a single bucket */ + + /* + * Allocate space for maximum number of buckets (better than repeatedly + * doing repalloc for short-lived objects). + */ + histogram->buckets + = (MVBucketBuild * *) palloc0(STATS_HIST_MAX_BUCKETS * sizeof(MVBucketBuild)); + + /* Create the initial bucket, covering all sampled rows */ + histogram->buckets[0] + = create_initial_ext_bucket(numrows, rows_copy, attrs, stats); + + /* + * Collect info on distinct values in each dimension (used later to pick + * dimension to partition). + */ + ndistvalues = (int *) palloc0(sizeof(int) * numattrs); + distvalues = (Datum **) palloc0(sizeof(Datum *) * numattrs); + + for (i = 0; i < numattrs; i++) + distvalues[i] = build_ndistinct(numrows, rows, attrs, stats, i, + &ndistvalues[i]); + + /* + * Split the initial bucket into buckets that don't mix NULL and non-NULL + * values in a single dimension. + * + * XXX Maybe this should be happening before the build_ndistinct()? + */ + create_null_buckets(histogram, 0, attrs, stats); + + /* + * Split the buckets into smaller and smaller buckets. The loop will end + * when either all buckets are too small (MIN_BUCKET_ROWS), or there are + * too many buckets in total (STATS_HIST_MAX_BUCKETS). + */ + while (histogram->nbuckets < STATS_HIST_MAX_BUCKETS) + { + MVBucketBuild *bucket = select_bucket_to_partition(histogram->nbuckets, + histogram->buckets); + + /* no bucket eligible for partitioning */ + if (bucket == NULL) + break; + + /* we modify the bucket in-place and add one new bucket */ + histogram->buckets[histogram->nbuckets++] + = partition_bucket(bucket, attrs, stats, ndistvalues, distvalues); + } + + /* Finalize the histogram build - compute bucket frequencies etc. */ + for (i = 0; i < histogram->nbuckets; i++) + { + /* + * The frequency has to be computed from the whole sample, in case + * some of the rows were filtered out in the MCV build. + */ + histogram->buckets[i]->frequency + = (histogram->buckets[i]->numrows * 1.0) / numrows_total; + } + + return serialize_histogram(histogram, stats); +} + +/* + * build_ndistinct + * build array of ndistinct values in a particular column, count them + * + */ +static Datum * +build_ndistinct(int numrows, HeapTuple *rows, Bitmapset *attrs, + VacAttrStats **stats, int i, int *nvals) +{ + int j; + int nvalues, + ndistinct; + Datum *values, + *distvalues; + int *attnums; + + TypeCacheEntry *type; + SortSupportData ssup; + + type = lookup_type_cache(stats[i]->attrtypid, TYPECACHE_LT_OPR); + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(type->lt_opr, &ssup); + + nvalues = 0; + values = (Datum *) palloc0(sizeof(Datum) * numrows); + + attnums = build_attnums(attrs); + + /* collect values from the sample rows, ignore NULLs */ + for (j = 0; j < numrows; j++) + { + Datum value; + bool isnull; + + /* + * remember the index of the sample row, to make the partitioning + * simpler + */ + value = heap_getattr(rows[j], attnums[i], + stats[i]->tupDesc, &isnull); + + if (isnull) + continue; + + values[nvalues++] = value; + } + + /* if no non-NULL values were found, free the memory and terminate */ + if (nvalues == 0) + { + pfree(values); + return NULL; + } + + /* sort the array of values using the SortSupport */ + qsort_arg((void *) values, nvalues, sizeof(Datum), + compare_scalars_simple, (void *) &ssup); + + /* count the distinct values first, and allocate just enough memory */ + ndistinct = 1; + for (j = 1; j < nvalues; j++) + if (compare_scalars_simple(&values[j], &values[j - 1], &ssup) != 0) + ndistinct += 1; + + distvalues = (Datum *) palloc0(sizeof(Datum) * ndistinct); + + /* now collect distinct values into the array */ + distvalues[0] = values[0]; + ndistinct = 1; + + for (j = 1; j < nvalues; j++) + { + if (compare_scalars_simple(&values[j], &values[j - 1], &ssup) != 0) + { + distvalues[ndistinct] = values[j]; + ndistinct += 1; + } + } + + pfree(values); + + *nvals = ndistinct; + return distvalues; +} + +/* + * statext_histogram_load + * Load the histogram list for the indicated pg_statistic_ext tuple +*/ +MVHistogram * +statext_histogram_load(Oid mvoid) +{ + bool isnull = false; + Datum histogram; + HeapTuple htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(mvoid)); + + if (!HeapTupleIsValid(htup)) + elog(ERROR, "cache lookup failed for statistics object %u", mvoid); + + histogram = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxhistogram, &isnull); + + ReleaseSysCache(htup); + + if (isnull) + return NULL; + + return statext_histogram_deserialize(DatumGetByteaP(histogram)); +} + +/* + * Serialize the MV histogram into a bytea value. The basic algorithm is quite + * simple, and mostly mimincs the MCV serialization: + * + * (1) perform deduplication for each attribute (separately) + * + * (a) collect all (non-NULL) attribute values from all buckets + * (b) sort the data (using 'lt' from VacAttrStats) + * (c) remove duplicate values from the array + * + * (2) serialize the arrays into a bytea value + * + * (3) process all buckets + * + * (a) replace min/max values with indexes into the arrays + * + * Each attribute has to be processed separately, as we're mixing different + * datatypes, and we we need to use the right operators to compare/sort them. + * We're also mixing pass-by-value and pass-by-ref types, and so on. + * + * TODO Consider packing boolean flags (NULL) for each item into 'char' or + * a longer type (instead of using an array of bool items). + */ +static MVHistogram * +serialize_histogram(MVHistogramBuild * histogram, VacAttrStats **stats) +{ + int dim, + i; + Size total_length = 0; + + bytea *output = NULL; /* serialized histogram as bytea */ + char *data = NULL; + + DimensionInfo *info; + SortSupport ssup; + + int nbuckets = histogram->nbuckets; + int ndims = histogram->ndimensions; + + /* allocated for serialized bucket data */ + int bucketsize = BUCKET_SIZE(ndims); + char *bucket = palloc0(bucketsize); + + /* values per dimension (and number of non-NULL values) */ + Datum **values = (Datum **) palloc0(sizeof(Datum *) * ndims); + int *counts = (int *) palloc0(sizeof(int) * ndims); + + /* info about dimensions (for deserialize) */ + info = (DimensionInfo *) palloc0(sizeof(DimensionInfo) * ndims); + + /* sort support data */ + ssup = (SortSupport) palloc0(sizeof(SortSupportData) * ndims); + + /* collect and deduplicate values for each dimension separately */ + for (dim = 0; dim < ndims; dim++) + { + int b; + int count; + TypeCacheEntry *type; + + type = lookup_type_cache(stats[dim]->attrtypid, TYPECACHE_LT_OPR); + + /* OID of the data types */ + histogram->types[dim] = stats[dim]->attrtypid; + + /* keep important info about the data type */ + info[dim].typlen = stats[dim]->attrtype->typlen; + info[dim].typbyval = stats[dim]->attrtype->typbyval; + + /* + * Allocate space for all min/max values, including NULLs (we won't + * use them, but we don't know how many are there), and then collect + * all non-NULL values. + */ + values[dim] = (Datum *) palloc0(sizeof(Datum) * nbuckets * 2); + + for (b = 0; b < histogram->nbuckets; b++) + { + /* skip buckets where this dimension is NULL-only */ + if (!histogram->buckets[b]->nullsonly[dim]) + { + values[dim][counts[dim]] = histogram->buckets[b]->min[dim]; + counts[dim] += 1; + + values[dim][counts[dim]] = histogram->buckets[b]->max[dim]; + counts[dim] += 1; + } + } + + /* there are just NULL values in this dimension */ + if (counts[dim] == 0) + continue; + + /* sort and deduplicate */ + ssup[dim].ssup_cxt = CurrentMemoryContext; + ssup[dim].ssup_collation = DEFAULT_COLLATION_OID; + ssup[dim].ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(type->lt_opr, &ssup[dim]); + + qsort_arg(values[dim], counts[dim], sizeof(Datum), + compare_scalars_simple, &ssup[dim]); + + /* + * Walk through the array and eliminate duplicitate values, but keep + * the ordering (so that we can do bsearch later). We know there's at + * least 1 item, so we can skip the first element. + */ + count = 1; /* number of deduplicated items */ + for (i = 1; i < counts[dim]; i++) + { + /* if it's different from the previous value, we need to keep it */ + if (compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim]) != 0) + { + /* XXX: not needed if (count == j) */ + values[dim][count] = values[dim][i]; + count += 1; + } + } + + /* make sure we fit into uint16 */ + Assert(count <= UINT16_MAX); + + /* keep info about the deduplicated count */ + info[dim].nvalues = count; + + /* compute size of the serialized data */ + if (info[dim].typlen > 0) + /* byval or byref, but with fixed length (name, tid, ...) */ + info[dim].nbytes = info[dim].nvalues * info[dim].typlen; + else if (info[dim].typlen == -1) + /* varlena, so just use VARSIZE_ANY */ + for (i = 0; i < info[dim].nvalues; i++) + info[dim].nbytes += VARSIZE_ANY(values[dim][i]); + else if (info[dim].typlen == -2) + /* cstring, so simply strlen */ + for (i = 0; i < info[dim].nvalues; i++) + info[dim].nbytes += strlen(DatumGetPointer(values[dim][i])); + else + elog(ERROR, "unknown data type typbyval=%d typlen=%d", + info[dim].typbyval, info[dim].typlen); + } + + /* + * Now we finally know how much space we'll need for the serialized + * histogram, as it contains these fields: + * + * - length (4B) for varlena + * - magic (4B) + * - type (4B) + * - ndimensions (4B) + * - nbuckets (4B) + * - info (ndim * sizeof(DimensionInfo) + * - arrays of values for each dimension + * - serialized buckets (nbuckets * bucketsize) + * + * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and then + * we'll place the data (and buckets). + */ + total_length = (offsetof(MVHistogram, buckets) + + ndims * sizeof(DimensionInfo) + + nbuckets * bucketsize); + + /* account for the deduplicated data */ + for (dim = 0; dim < ndims; dim++) + total_length += info[dim].nbytes; + + /* + * Enforce arbitrary limit of 1MB on the size of the serialized MCV list. + * This is meant as a protection against someone building MCV list on long + * values (e.g. text documents). + * + * XXX Should we enforce arbitrary limits like this one? Maybe it's not + * even necessary, as long values are usually unique and so won't make it + * into the MCV list in the first place. In the end, we have a 1GB limit + * on bytea values. + */ + if (total_length > (1024 * 1024)) + elog(ERROR, "serialized histogram exceeds 1MB (%ld > %d)", + total_length, (1024 * 1024)); + + /* allocate space for the serialized histogram list, set header */ + output = (bytea *) palloc0(total_length); + + /* + * we'll use 'data' to keep track of the place to write data + * + * XXX No VARDATA() here, as MVHistogramBuild includes the length. + */ + data = (char *) output; + + memcpy(data, histogram, offsetof(MVHistogramBuild, buckets)); + data += offsetof(MVHistogramBuild, buckets); + + memcpy(data, info, sizeof(DimensionInfo) * ndims); + data += sizeof(DimensionInfo) * ndims; + + /* serialize the deduplicated values for all attributes */ + for (dim = 0; dim < ndims; dim++) + { +#ifdef USE_ASSERT_CHECKING + char *tmp = data; +#endif + for (i = 0; i < info[dim].nvalues; i++) + { + Datum v = values[dim][i]; + + if (info[dim].typbyval) /* passed by value */ + { + memcpy(data, &v, info[dim].typlen); + data += info[dim].typlen; + } + else if (info[dim].typlen > 0) /* pased by reference */ + { + memcpy(data, DatumGetPointer(v), info[dim].typlen); + data += info[dim].typlen; + } + else if (info[dim].typlen == -1) /* varlena */ + { + memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v)); + data += VARSIZE_ANY(values[dim][i]); + } + else if (info[dim].typlen == -2) /* cstring */ + { + memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v)) + 1); + data += strlen(DatumGetPointer(v)) + 1; + } + } + + /* make sure we got exactly the amount of data we expected */ + Assert((data - tmp) == info[dim].nbytes); + } + + /* finally serialize the items, with uint16 indexes instead of the values */ + for (i = 0; i < nbuckets; i++) + { + /* don't write beyond the allocated space */ + Assert(data <= (char *) output + total_length - bucketsize); + + /* reset the values for each item */ + memset(bucket, 0, bucketsize); + + BUCKET_FREQUENCY(bucket) = histogram->buckets[i]->frequency; + + for (dim = 0; dim < ndims; dim++) + { + /* do the lookup only for non-NULL values */ + if (!histogram->buckets[i]->nullsonly[dim]) + { + uint16 idx; + Datum *v = NULL; + + /* min boundary */ + v = (Datum *) bsearch_arg(&histogram->buckets[i]->min[dim], + values[dim], info[dim].nvalues, sizeof(Datum), + compare_scalars_simple, &ssup[dim]); + + Assert(v != NULL); /* serialization or deduplication error */ + + /* compute index within the array */ + idx = (v - values[dim]); + + Assert((idx >= 0) && (idx < info[dim].nvalues)); + + BUCKET_MIN_INDEXES(bucket, ndims)[dim] = idx; + + /* max boundary */ + v = (Datum *) bsearch_arg(&histogram->buckets[i]->max[dim], + values[dim], info[dim].nvalues, sizeof(Datum), + compare_scalars_simple, &ssup[dim]); + + Assert(v != NULL); /* serialization or deduplication error */ + + /* compute index within the array */ + idx = (v - values[dim]); + + Assert((idx >= 0) && (idx < info[dim].nvalues)); + + BUCKET_MAX_INDEXES(bucket, ndims)[dim] = idx; + } + } + + /* copy flags (nulls, min/max inclusive) */ + memcpy(BUCKET_NULLS_ONLY(bucket, ndims), + histogram->buckets[i]->nullsonly, sizeof(bool) * ndims); + + memcpy(BUCKET_MIN_INCL(bucket, ndims), + histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims); + + memcpy(BUCKET_MAX_INCL(bucket, ndims), + histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims); + + /* copy the item into the array */ + memcpy(data, bucket, bucketsize); + + data += bucketsize; + } + + /* at this point we expect to match the total_length exactly */ + Assert((data - (char *) output) == total_length); + + /* free the values/counts arrays here */ + pfree(counts); + pfree(info); + pfree(ssup); + + for (dim = 0; dim < ndims; dim++) + pfree(values[dim]); + + pfree(values); + + /* make sure the length is correct */ + SET_VARSIZE(output, total_length); + + return (MVHistogram *)output; +} + +/* +* Reads serialized histogram into MVHistogram structure. + + * Returns histogram in a partially-serialized form (keeps the boundary values + * deduplicated, so that it's possible to optimize the estimation part by + * caching function call results across buckets etc.). + */ +MVHistogram * +statext_histogram_deserialize(bytea *data) +{ + int dim, + i; + + Size expected_size; + char *tmp = NULL; + + MVHistogram *histogram; + DimensionInfo *info; + + int nbuckets; + int ndims; + int bucketsize; + + /* temporary deserialization buffer */ + int bufflen; + char *buff; + char *ptr; + + if (data == NULL) + return NULL; + + /* + * We can't possibly deserialize a histogram if there's not even a + * complete header. + */ + if (VARSIZE_ANY_EXHDR(data) < offsetof(MVHistogram, buckets)) + elog(ERROR, "invalid histogram size %ld (expected at least %ld)", + VARSIZE_ANY_EXHDR(data), offsetof(MVHistogram, buckets)); + + /* read the histogram header */ + histogram + = (MVHistogram *) palloc(sizeof(MVHistogram)); + + /* initialize pointer to data (varlena header is included) */ + tmp = (char *) data; + + /* get the header and perform basic sanity checks */ + memcpy(histogram, tmp, offsetof(MVHistogram, buckets)); + tmp += offsetof(MVHistogram, buckets); + + if (histogram->magic != STATS_HIST_MAGIC) + elog(ERROR, "invalid histogram magic %d (expected %dd)", + histogram->magic, STATS_HIST_MAGIC); + + if (histogram->type != STATS_HIST_TYPE_BASIC) + elog(ERROR, "invalid histogram type %d (expected %dd)", + histogram->type, STATS_HIST_TYPE_BASIC); + + if (histogram->ndimensions == 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid zero-length dimension array in histogram"))); + else if (histogram->ndimensions > STATS_MAX_DIMENSIONS) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid length (%d) dimension array in histogram", + histogram->ndimensions))); + + if (histogram->nbuckets == 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid zero-length bucket array in histogram"))); + else if (histogram->nbuckets > STATS_HIST_MAX_BUCKETS) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid length (%d) bucket array in histogram", + histogram->nbuckets))); + + nbuckets = histogram->nbuckets; + ndims = histogram->ndimensions; + bucketsize = BUCKET_SIZE(ndims); + + /* + * What size do we expect with those parameters (it's incomplete, as we + * yet have to count the array sizes (from DimensionInfo records). + */ + expected_size = offsetof(MVHistogram, buckets) + + ndims * sizeof(DimensionInfo) + + (nbuckets * bucketsize); + + /* check that we have at least the DimensionInfo records */ + if (VARSIZE_ANY(data) < expected_size) + elog(ERROR, "invalid histogram size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* Now it's safe to access the dimention info. */ + info = (DimensionInfo *) (tmp); + tmp += ndims * sizeof(DimensionInfo); + + /* account for the value arrays */ + for (dim = 0; dim < ndims; dim++) + expected_size += info[dim].nbytes; + + if (VARSIZE_ANY(data) != expected_size) + elog(ERROR, "invalid histogram size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* looks OK - not corrupted or something */ + + /* a single buffer for all the values and counts */ + bufflen = (sizeof(int) + sizeof(Datum *)) * ndims; + + for (dim = 0; dim < ndims; dim++) + /* don't allocate space for byval types, matching Datum */ + if (!(info[dim].typbyval && (info[dim].typlen == sizeof(Datum)))) + bufflen += (sizeof(Datum) * info[dim].nvalues); + + /* also, include space for the result, tracking the buckets */ + bufflen += nbuckets * (sizeof(MVBucket *) + /* bucket pointer */ + sizeof(MVBucket)); /* bucket data */ + + buff = palloc0(bufflen); + ptr = buff; + + histogram->nvalues = (int *) ptr; + ptr += (sizeof(int) * ndims); + + histogram->values = (Datum **) ptr; + ptr += (sizeof(Datum *) * ndims); + + /* + * XXX This uses pointers to the original data array (the types not passed + * by value), so when someone frees the memory, e.g. by doing something + * like this: + * + * bytea * data = ... fetch the data from catalog ... + * MVHistogramBuild histogram = deserialize_histogram(data); + * pfree(data); + * + * then 'histogram' references the freed memory. Should copy the pieces. + */ + for (dim = 0; dim < ndims; dim++) + { +#ifdef USE_ASSERT_CHECKING + /* remember where data for this dimension starts */ + char *start = tmp; +#endif + + histogram->nvalues[dim] = info[dim].nvalues; + + if (info[dim].typbyval) + { + /* passed by value / Datum - simply reuse the array */ + if (info[dim].typlen == sizeof(Datum)) + { + histogram->values[dim] = (Datum *) tmp; + tmp += info[dim].nbytes; + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + else + { + histogram->values[dim] = (Datum *) ptr; + ptr += (sizeof(Datum) * info[dim].nvalues); + + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + memcpy(&histogram->values[dim][i], tmp, info[dim].typlen); + tmp += info[dim].typlen; + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + } + else + { + /* all the other types need a chunk of the buffer */ + histogram->values[dim] = (Datum *) ptr; + ptr += (sizeof(Datum) * info[dim].nvalues); + + if (info[dim].typlen > 0) + { + /* pased by reference, but fixed length (name, tid, ...) */ + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + histogram->values[dim][i] = PointerGetDatum(tmp); + tmp += info[dim].typlen; + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + else if (info[dim].typlen == -1) + { + /* varlena */ + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + histogram->values[dim][i] = PointerGetDatum(tmp); + tmp += VARSIZE_ANY(tmp); + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + else if (info[dim].typlen == -2) + { + /* cstring */ + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + histogram->values[dim][i] = PointerGetDatum(tmp); + tmp += (strlen(tmp) + 1); /* don't forget the \0 */ + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + } + + /* check we consumed the serialized data for this dimension exactly */ + Assert((tmp - start) == info[dim].nbytes); + } + + /* now deserialize the buckets and point them into the varlena values */ + histogram->buckets = (MVBucket * *) ptr; + ptr += (sizeof(MVBucket *) * nbuckets); + + for (i = 0; i < nbuckets; i++) + { + MVBucket *bucket = (MVBucket *) ptr; + + ptr += sizeof(MVBucket); + + bucket->frequency = BUCKET_FREQUENCY(tmp); + bucket->nullsonly = BUCKET_NULLS_ONLY(tmp, ndims); + bucket->min_inclusive = BUCKET_MIN_INCL(tmp, ndims); + bucket->max_inclusive = BUCKET_MAX_INCL(tmp, ndims); + + bucket->min = BUCKET_MIN_INDEXES(tmp, ndims); + bucket->max = BUCKET_MAX_INDEXES(tmp, ndims); + + histogram->buckets[i] = bucket; + + Assert(tmp <= (char *) data + VARSIZE_ANY(data)); + + tmp += bucketsize; + } + + /* at this point we expect to match the total_length exactly */ + Assert((tmp - (char *) data) == expected_size); + + /* we should exhaust the output buffer exactly */ + Assert((ptr - buff) == bufflen); + + return histogram; +} + +/* + * create_initial_ext_bucket + * Create an initial bucket, covering all the sampled rows. + */ +static MVBucketBuild * +create_initial_ext_bucket(int numrows, HeapTuple *rows, Bitmapset *attrs, + VacAttrStats **stats) +{ + int i; + int numattrs = bms_num_members(attrs); + + /* TODO allocate bucket as a single piece, including all the fields. */ + MVBucketBuild *bucket = (MVBucketBuild *) palloc0(sizeof(MVBucketBuild)); + + Assert(numrows > 0); + Assert(rows != NULL); + Assert((numattrs >= 2) && (numattrs <= STATS_MAX_DIMENSIONS)); + + /* allocate the per-dimension arrays */ + + /* flags for null-only dimensions */ + bucket->nullsonly = (bool *) palloc0(numattrs * sizeof(bool)); + + /* inclusiveness boundaries - lower/upper bounds */ + bucket->min_inclusive = (bool *) palloc0(numattrs * sizeof(bool)); + bucket->max_inclusive = (bool *) palloc0(numattrs * sizeof(bool)); + + /* lower/upper boundaries */ + bucket->min = (Datum *) palloc0(numattrs * sizeof(Datum)); + bucket->max = (Datum *) palloc0(numattrs * sizeof(Datum)); + + /* number of distinct values (per dimension) */ + bucket->ndistincts = (uint32 *) palloc0(numattrs * sizeof(uint32)); + + /* all the sample rows fall into the initial bucket */ + bucket->numrows = numrows; + bucket->rows = rows; + + /* + * Update the number of ndistinct combinations in the bucket (which we use + * when selecting bucket to partition), and then number of distinct values + * for each partition (which we use when choosing which dimension to + * split). + */ + update_bucket_ndistinct(bucket, attrs, stats); + + /* Update ndistinct (and also set min/max) for all dimensions. */ + for (i = 0; i < numattrs; i++) + update_dimension_ndistinct(bucket, i, attrs, stats, true); + + return bucket; +} + +/* + * Choose the bucket to partition next. + * + * The current criteria is rather simple, chosen so that the algorithm produces + * buckets with about equal frequency and regular size. We select the bucket + * with the highest number of distinct values, and then split it by the longest + * dimension. + * + * The distinct values are uniformly mapped to [0,1] interval, and this is used + * to compute length of the value range. + * + * NOTE: This is not the same array used for deduplication, as this contains + * values for all the tuples from the sample, not just the boundary values. + * + * Returns either pointer to the bucket selected to be partitioned, or NULL if + * there are no buckets that may be split (e.g. if all buckets are too small + * or contain too few distinct values). + * + * + * Tricky example + * -------------- + * + * Consider this table: + * + * CREATE TABLE t AS SELECT i AS a, i AS b + * FROM generate_series(1,1000000) s(i); + * + * CREATE STATISTICS s1 ON t (a,b) WITH (histogram); + * + * ANALYZE t; + * + * It's a very specific (and perhaps artificial) example, because every bucket + * always has exactly the same number of distinct values in all dimensions, + * which makes the partitioning tricky. + * + * Then: + * + * SELECT * FROM t WHERE (a < 100) AND (b < 100); + * + * is estimated to return ~120 rows, while in reality it returns only 99. + * + * QUERY PLAN + * ------------------------------------------------------------- + * Seq Scan on t (cost=0.00..19425.00 rows=117 width=8) + * (actual time=0.129..82.776 rows=99 loops=1) + * Filter: ((a < 100) AND (b < 100)) + * Rows Removed by Filter: 999901 + * Planning time: 1.286 ms + * Execution time: 82.984 ms + * (5 rows) + * + * So this estimate is reasonably close. Let's change the query to OR clause: + * + * SELECT * FROM t WHERE (a < 100) OR (b < 100); + * + * QUERY PLAN + * ------------------------------------------------------------- + * Seq Scan on t (cost=0.00..19425.00 rows=8100 width=8) + * (actual time=0.145..99.910 rows=99 loops=1) + * Filter: ((a < 100) OR (b < 100)) + * Rows Removed by Filter: 999901 + * Planning time: 1.578 ms + * Execution time: 100.132 ms + * (5 rows) + * + * That's clearly a much worse estimate. This happens because the histogram + * contains buckets like this: + * + * bucket 592 [3 30310] [30134 30593] => [0.000233] + * + * i.e. the length of "a" dimension is (30310-3)=30307, while the length of "b" + * is (30593-30134)=459. So the "b" dimension is much narrower than "a". + * Of course, there are also buckets where "b" is the wider dimension. + * + * This is partially mitigated by selecting the "longest" dimension but that + * only happens after we already selected the bucket. So if we never select the + * bucket, this optimization does not apply. + * + * The other reason why this particular example behaves so poorly is due to the + * way we actually split the selected bucket. We do attempt to divide the bucket + * into two parts containing about the same number of tuples, but that does not + * too well when most of the tuples is squashed on one side of the bucket. + * + * For example for columns with data on the diagonal (i.e. when a=b), we end up + * with a narrow bucket on the diagonal and a huge bucket overing the remaining + * part (with much lower density). + * + * So perhaps we need two partitioning strategies - one aiming to split buckets + * with high frequency (number of sampled rows), the other aiming to split + * "large" buckets. And alternating between them, somehow. + * + * TODO Consider using similar lower boundary for row count as for simple + * histograms, i.e. 300 tuples per bucket. + */ +static MVBucketBuild * +select_bucket_to_partition(int nbuckets, MVBucketBuild * *buckets) +{ + int i; + int numrows = 0; + MVBucketBuild *bucket = NULL; + + for (i = 0; i < nbuckets; i++) + { + /* if the number of rows is higher, use this bucket */ + if ((buckets[i]->ndistinct > 2) && + (buckets[i]->numrows > numrows) && + (buckets[i]->numrows >= MIN_BUCKET_ROWS)) + { + bucket = buckets[i]; + numrows = buckets[i]->numrows; + } + } + + /* may be NULL if there are not buckets with (ndistinct>1) */ + return bucket; +} + +/* + * A simple bucket partitioning implementation - we choose the longest bucket + * dimension, measured using the array of distinct values built at the very + * beginning of the build. + * + * We map all the distinct values to a [0,1] interval, uniformly distributed, + * and then use this to measure length. It's essentially a number of distinct + * values within the range, normalized to [0,1]. + * + * Then we choose a 'middle' value splitting the bucket into two parts with + * roughly the same frequency. + * + * This splits the bucket by tweaking the existing one, and returning the new + * bucket (essentially shrinking the existing one in-place and returning the + * other "half" as a new bucket). The caller is responsible for adding the new + * bucket into the list of buckets. + * + * There are multiple histogram options, centered around the partitioning + * criteria, specifying both how to choose a bucket and the dimension most in + * need of a split. For a nice summary and general overview, see "rK-Hist : an + * R-Tree based histogram for multi-dimensional selectivity estimation" thesis + * by J. A. Lopez, Concordia University, p.34-37 (and possibly p. 32-34 for + * explanation of the terms). + * + * It requires care to prevent splitting only one dimension and not splitting + * another one at all (which might happen easily in case of strongly dependent + * columns - e.g. y=x). The current algorithm minimizes this, but may still + * happen for perfectly dependent examples (when all the dimensions have equal + * length, the first one will be selected). + * + * TODO Should probably consider statistics target for the columns (e.g. + * to split dimensions with higher statistics target more frequently). + */ +static MVBucketBuild * +partition_bucket(MVBucketBuild * bucket, Bitmapset *attrs, + VacAttrStats **stats, + int *ndistvalues, Datum **distvalues) +{ + int i; + int dimension; + int numattrs = bms_num_members(attrs); + + Datum split_value; + MVBucketBuild *new_bucket; + + /* needed for sort, when looking for the split value */ + bool isNull; + int nvalues = 0; + TypeCacheEntry *type; + ScalarItem *values; + SortSupportData ssup; + int *attnums; + + int nrows = 1; /* number of rows below current value */ + double delta; + + /* needed when splitting the values */ + HeapTuple *oldrows = bucket->rows; + int oldnrows = bucket->numrows; + + values = (ScalarItem *) palloc0(bucket->numrows * sizeof(ScalarItem)); + + /* + * We can't split buckets with a single distinct value (this also + * disqualifies NULL-only dimensions). Also, there has to be multiple + * sample rows (otherwise, how could there be more distinct values). + */ + Assert(bucket->ndistinct > 1); + Assert(bucket->numrows > 1); + Assert((numattrs >= 2) && (numattrs <= STATS_MAX_DIMENSIONS)); + + /* Look for the next dimension to split. */ + delta = 0.0; + dimension = -1; + + for (i = 0; i < numattrs; i++) + { + Datum *a, + *b; + + type = lookup_type_cache(stats[i]->attrtypid, TYPECACHE_LT_OPR); + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(type->lt_opr, &ssup); + + /* can't split NULL-only dimension */ + if (bucket->nullsonly[i]) + continue; + + /* can't split dimension with a single ndistinct value */ + if (bucket->ndistincts[i] <= 1) + continue; + + /* search for min boundary in the distinct list */ + a = (Datum *) bsearch_arg(&bucket->min[i], + distvalues[i], ndistvalues[i], + sizeof(Datum), compare_scalars_simple, &ssup); + + b = (Datum *) bsearch_arg(&bucket->max[i], + distvalues[i], ndistvalues[i], + sizeof(Datum), compare_scalars_simple, &ssup); + + /* if this dimension is 'larger' then partition by it */ + if (((b - a) * 1.0 / ndistvalues[i]) > delta) + { + delta = ((b - a) * 1.0 / ndistvalues[i]); + dimension = i; + } + } + + /* + * If we haven't found a dimension here, we've done something wrong in + * select_bucket_to_partition. + */ + Assert(dimension != -1); + + /* + * Walk through the selected dimension, collect and sort the values and + * then choose the value to use as the new boundary. + */ + type = lookup_type_cache(stats[dimension]->attrtypid, TYPECACHE_LT_OPR); + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(type->lt_opr, &ssup); + + attnums = build_attnums(attrs); + + for (i = 0; i < bucket->numrows; i++) + { + /* + * remember the index of the sample row, to make the partitioning + * simpler + */ + values[nvalues].value = heap_getattr(bucket->rows[i], attnums[dimension], + stats[dimension]->tupDesc, &isNull); + values[nvalues].tupno = i; + + /* no NULL values allowed here (we never split null-only dimension) */ + Assert(!isNull); + + nvalues++; + } + + /* sort the array of values */ + qsort_arg((void *) values, nvalues, sizeof(ScalarItem), + compare_scalars_partition, (void *) &ssup); + + /* + * We know there are bucket->ndistincts[dimension] distinct values in this + * dimension, and we want to split this into half, so walk through the + * array and stop once we see (ndistinct/2) values. + * + * We always choose the "next" value, i.e. (n/2+1)-th distinct value, and + * use it as an exclusive upper boundary (and inclusive lower boundary). + * + * TODO Maybe we should use "average" of the two middle distinct values + * (at least for even distinct counts), but that would require being able + * to do an average (which does not work for non-numeric types). + * + * TODO Another option is to look for a split that'd give about 50% tuples + * (not distinct values) in each partition. That might work better when + * there are a few very frequent values, and many rare ones. + */ + delta = bucket->numrows; + split_value = values[0].value; + + for (i = 1; i < bucket->numrows; i++) + { + if (values[i].value != values[i - 1].value) + { + /* are we closer to splitting the bucket in half? */ + if (fabs(i - bucket->numrows / 2.0) < delta) + { + /* let's assume we'll use this value for the split */ + split_value = values[i].value; + delta = fabs(i - bucket->numrows / 2.0); + nrows = i; + } + } + } + + Assert(nrows > 0); + Assert(nrows < bucket->numrows); + + /* + * create the new bucket as a (incomplete) copy of the one being + * partitioned. + */ + new_bucket = copy_ext_bucket(bucket, numattrs); + + /* + * Do the actual split of the chosen dimension, using the split value as + * the upper bound for the existing bucket, and lower bound for the new + * one. + */ + bucket->max[dimension] = split_value; + new_bucket->min[dimension] = split_value; + + /* + * We also treat only one side of the new boundary as inclusive, in the + * bucket where it happens to be the upper boundary. We never set the + * min_inclusive[] to false anywhere, but we set it to true anyway. + */ + bucket->max_inclusive[dimension] = false; + new_bucket->min_inclusive[dimension] = true; + + /* + * Redistribute the sample tuples using the 'ScalarItem->tupno' index. We + * know 'nrows' rows should remain in the original bucket and the rest + * goes to the new one. + */ + bucket->numrows = nrows; + new_bucket->numrows = (oldnrows - nrows); + + bucket->rows = (HeapTuple *) palloc0(bucket->numrows * sizeof(HeapTuple)); + new_bucket->rows = (HeapTuple *) palloc0(new_bucket->numrows * sizeof(HeapTuple)); + + /* + * The first nrows should go to the first bucket, the rest should go to + * the new one. Use the tupno field to get the actual HeapTuple row from + * the original array of sample rows. + */ + for (i = 0; i < nrows; i++) + memcpy(&bucket->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple)); + + for (i = nrows; i < oldnrows; i++) + memcpy(&new_bucket->rows[i - nrows], &oldrows[values[i].tupno], sizeof(HeapTuple)); + + /* update ndistinct values for the buckets (total and per dimension) */ + update_bucket_ndistinct(bucket, attrs, stats); + update_bucket_ndistinct(new_bucket, attrs, stats); + + /* + * TODO We don't need to do this for the dimension we used for split, + * because we know how many distinct values went to each partition. + */ + for (i = 0; i < numattrs; i++) + { + update_dimension_ndistinct(bucket, i, attrs, stats, false); + update_dimension_ndistinct(new_bucket, i, attrs, stats, false); + } + + pfree(oldrows); + pfree(values); + + return new_bucket; +} + +/* + * Copy a histogram bucket. The copy does not include the build-time data, i.e. + * sampled rows etc. + */ +static MVBucketBuild * +copy_ext_bucket(MVBucketBuild * bucket, uint32 ndimensions) +{ + /* TODO allocate as a single piece (including all the fields) */ + MVBucketBuild *new_bucket = (MVBucketBuild *) palloc0(sizeof(MVBucketBuild)); + + /* + * Copy only the attributes that will stay the same after the split, and + * we'll recompute the rest after the split. + */ + + /* allocate the per-dimension arrays */ + new_bucket->nullsonly = (bool *) palloc0(ndimensions * sizeof(bool)); + + /* inclusiveness boundaries - lower/upper bounds */ + new_bucket->min_inclusive = (bool *) palloc0(ndimensions * sizeof(bool)); + new_bucket->max_inclusive = (bool *) palloc0(ndimensions * sizeof(bool)); + + /* lower/upper boundaries */ + new_bucket->min = (Datum *) palloc0(ndimensions * sizeof(Datum)); + new_bucket->max = (Datum *) palloc0(ndimensions * sizeof(Datum)); + + /* copy data */ + memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool)); + + memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions * sizeof(bool)); + memcpy(new_bucket->min, bucket->min, ndimensions * sizeof(Datum)); + + memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions * sizeof(bool)); + memcpy(new_bucket->max, bucket->max, ndimensions * sizeof(Datum)); + + /* allocate and copy the interesting part of the build data */ + new_bucket->ndistincts = (uint32 *) palloc0(ndimensions * sizeof(uint32)); + + return new_bucket; +} + +/* + * Counts the number of distinct values in the bucket. This just copies the + * Datum values into a simple array, and sorts them using memcmp-based + * comparator. That means it only works for pass-by-value data types (assuming + * they don't use collations etc.) + */ +static void +update_bucket_ndistinct(MVBucketBuild * bucket, Bitmapset *attrs, VacAttrStats **stats) +{ + int i; + int numattrs = bms_num_members(attrs); + int numrows = bucket->numrows; + + MultiSortSupport mss = multi_sort_init(numattrs); + int *attnums; + SortItem *items; + + attnums = build_attnums(attrs); + + /* prepare the sort function for the first dimension */ + for (i = 0; i < numattrs; i++) + { + VacAttrStats *colstat = stats[i]; + TypeCacheEntry *type; + + type = lookup_type_cache(colstat->attrtypid, TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) /* shouldn't happen */ + elog(ERROR, "cache lookup failed for ordering operator for type %u", + colstat->attrtypid); + + multi_sort_add_dimension(mss, i, type->lt_opr); + } + + /* + * build an array of SortItem(s) sorted using the multi-sort support + * + * XXX This relies on all stats entries pointing to the same tuple + * descriptor. Not sure if that might not be the case. + */ + items = build_sorted_items(numrows, bucket->rows, stats[0]->tupDesc, + mss, numattrs, attnums); + + bucket->ndistinct = 1; + + for (i = 1; i < numrows; i++) + if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0) + bucket->ndistinct += 1; + + pfree(items); +} + +/* + * Count distinct values per bucket dimension. + */ +static void +update_dimension_ndistinct(MVBucketBuild * bucket, int dimension, Bitmapset *attrs, + VacAttrStats **stats, bool update_boundaries) +{ + int j; + int nvalues = 0; + bool isNull; + Datum *values; + SortSupportData ssup; + TypeCacheEntry *type; + int *attnums; + + values = (Datum *) palloc0(bucket->numrows * sizeof(Datum)); + type = lookup_type_cache(stats[dimension]->attrtypid, TYPECACHE_LT_OPR); + + /* we may already know this is a NULL-only dimension */ + if (bucket->nullsonly[dimension]) + bucket->ndistincts[dimension] = 1; + + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(type->lt_opr, &ssup); + + attnums = build_attnums(attrs); + + for (j = 0; j < bucket->numrows; j++) + { + values[nvalues] = heap_getattr(bucket->rows[j], attnums[dimension], + stats[dimension]->tupDesc, &isNull); + + /* ignore NULL values */ + if (!isNull) + nvalues++; + } + + /* there's always at least 1 distinct value (may be NULL) */ + bucket->ndistincts[dimension] = 1; + + /* + * if there are only NULL values in the column, mark it so and continue + * with the next one + */ + if (nvalues == 0) + { + pfree(values); + bucket->nullsonly[dimension] = true; + return; + } + + /* sort the array (pass-by-value datum */ + qsort_arg((void *) values, nvalues, sizeof(Datum), + compare_scalars_simple, (void *) &ssup); + + /* + * Update min/max boundaries to the smallest bounding box. Generally, this + * needs to be done only when constructing the initial bucket. + */ + if (update_boundaries) + { + /* store the min/max values */ + bucket->min[dimension] = values[0]; + bucket->min_inclusive[dimension] = true; + + bucket->max[dimension] = values[nvalues - 1]; + bucket->max_inclusive[dimension] = true; + } + + /* + * Walk through the array and count distinct values by comparing + * succeeding values. + */ + for (j = 1; j < nvalues; j++) + { + if (compare_datums_simple(values[j - 1], values[j], &ssup) != 0) + bucket->ndistincts[dimension] += 1; + } + + pfree(values); +} + +/* + * A properly built histogram must not contain buckets mixing NULL and non-NULL + * values in a single dimension. Each dimension may either be marked as 'nulls + * only', and thus containing only NULL values, or it must not contain any NULL + * values. + * + * Therefore, if the sample contains NULL values in any of the columns, it's + * necessary to build those NULL-buckets. This is done in an iterative way + * using this algorithm, operating on a single bucket: + * + * (1) Check that all dimensions are well-formed (not mixing NULL and + * non-NULL values). + * + * (2) If all dimensions are well-formed, terminate. + * + * (3) If the dimension contains only NULL values, but is not marked as + * NULL-only, mark it as NULL-only and run the algorithm again (on + * this bucket). + * + * (4) If the dimension mixes NULL and non-NULL values, split the bucket + * into two parts - one with NULL values, one with non-NULL values + * (replacing the current one). Then run the algorithm on both buckets. + * + * This is executed in a recursive manner, but the number of executions should + * be quite low - limited by the number of NULL-buckets. Also, in each branch + * the number of nested calls is limited by the number of dimensions + * (attributes) of the histogram. + * + * At the end, there should be buckets with no mixed dimensions. The number of + * buckets produced by this algorithm is rather limited - with N dimensions, + * there may be only 2^N such buckets (each dimension may be either NULL or + * non-NULL). So with 8 dimensions (current value of STATS_MAX_DIMENSIONS) + * there may be only 256 such buckets. + * + * After this, a 'regular' bucket-split algorithm shall run, further optimizing + * the histogram. + */ +static void +create_null_buckets(MVHistogramBuild * histogram, int bucket_idx, + Bitmapset *attrs, VacAttrStats **stats) +{ + int i, + j; + int null_dim = -1; + int null_count = 0; + bool null_found = false; + MVBucketBuild *bucket, + *null_bucket; + int null_idx, + curr_idx; + int *attnums; + + /* remember original values from the bucket */ + int numrows; + HeapTuple *oldrows = NULL; + + Assert(bucket_idx < histogram->nbuckets); + Assert(histogram->ndimensions == bms_num_members(attrs)); + + bucket = histogram->buckets[bucket_idx]; + + numrows = bucket->numrows; + oldrows = bucket->rows; + + attnums = build_attnums(attrs); + + /* + * Walk through all rows / dimensions, and stop once we find NULL in a + * dimension not yet marked as NULL-only. + */ + for (i = 0; i < bucket->numrows; i++) + { + for (j = 0; j < histogram->ndimensions; j++) + { + /* Is this a NULL-only dimension? If yes, skip. */ + if (bucket->nullsonly[j]) + continue; + + /* found a NULL in that dimension? */ + if (heap_attisnull(bucket->rows[i], attnums[j], + stats[j]->tupDesc)) + { + null_found = true; + null_dim = j; + break; + } + } + + /* terminate if we found attribute with NULL values */ + if (null_found) + break; + } + + /* no regular dimension contains NULL values => we're done */ + if (!null_found) + return; + + /* walk through the rows again, count NULL values in 'null_dim' */ + for (i = 0; i < bucket->numrows; i++) + { + if (heap_attisnull(bucket->rows[i], attnums[null_dim], + stats[null_dim]->tupDesc)) + null_count += 1; + } + + Assert(null_count <= bucket->numrows); + + /* + * If (null_count == numrows) the dimension already is NULL-only, but is + * not yet marked like that. It's enough to mark it and repeat the process + * recursively (until we run out of dimensions). + */ + if (null_count == bucket->numrows) + { + bucket->nullsonly[null_dim] = true; + create_null_buckets(histogram, bucket_idx, attrs, stats); + return; + } + + /* + * We have to split the bucket into two - one with NULL values in the + * dimension, one with non-NULL values. We don't need to sort the data or + * anything, but otherwise it's similar to what partition_bucket() does. + */ + + /* create bucket with NULL-only dimension 'dim' */ + null_bucket = copy_ext_bucket(bucket, histogram->ndimensions); + + /* remember the current array info */ + oldrows = bucket->rows; + numrows = bucket->numrows; + + /* we'll keep non-NULL values in the current bucket */ + bucket->numrows = (numrows - null_count); + bucket->rows + = (HeapTuple *) palloc0(bucket->numrows * sizeof(HeapTuple)); + + /* and the NULL values will go to the new one */ + null_bucket->numrows = null_count; + null_bucket->rows + = (HeapTuple *) palloc0(null_bucket->numrows * sizeof(HeapTuple)); + + /* mark the dimension as NULL-only (in the new bucket) */ + null_bucket->nullsonly[null_dim] = true; + + /* walk through the sample rows and distribute them accordingly */ + null_idx = 0; + curr_idx = 0; + for (i = 0; i < numrows; i++) + { + if (heap_attisnull(oldrows[i], attnums[null_dim], + stats[null_dim]->tupDesc)) + /* NULL => copy to the new bucket */ + memcpy(&null_bucket->rows[null_idx++], &oldrows[i], + sizeof(HeapTuple)); + else + memcpy(&bucket->rows[curr_idx++], &oldrows[i], + sizeof(HeapTuple)); + } + + /* update ndistinct values for the buckets (total and per dimension) */ + update_bucket_ndistinct(bucket, attrs, stats); + update_bucket_ndistinct(null_bucket, attrs, stats); + + /* + * TODO We don't need to do this for the dimension we used for split, + * because we know how many distinct values went to each bucket (NULL is + * not a value, so NULL buckets get 0, and the other bucket got all the + * distinct values). + */ + for (i = 0; i < histogram->ndimensions; i++) + { + update_dimension_ndistinct(bucket, i, attrs, stats, false); + update_dimension_ndistinct(null_bucket, i, attrs, stats, false); + } + + pfree(oldrows); + + /* add the NULL bucket to the histogram */ + histogram->buckets[histogram->nbuckets++] = null_bucket; + + /* + * And now run the function recursively on both buckets (the new one + * first, because the call may change number of buckets, and it's used as + * an index). + */ + create_null_buckets(histogram, (histogram->nbuckets - 1), attrs, stats); + create_null_buckets(histogram, bucket_idx, attrs, stats); +} + +/* + * SRF with details about buckets of a histogram: + * + * - bucket ID (0...nbuckets) + * - min values (string array) + * - max values (string array) + * - nulls only (boolean array) + * - min inclusive flags (boolean array) + * - max inclusive flags (boolean array) + * - frequency (double precision) + * + * The input is the OID of the statistics, and there are no rows returned if the + * statistics contains no histogram (or if there's no statistics for the OID). + * + * The second parameter (type) determines what values will be returned + * in the (minvals,maxvals). There are three possible values: + * + * 0 (actual values) + * ----------------- + * - prints actual values + * - using the output function of the data type (as string) + * - handy for investigating the histogram + * + * 1 (distinct index) + * ------------------ + * - prints index of the distinct value (into the serialized array) + * - makes it easier to spot neighbor buckets, etc. + * - handy for plotting the histogram + * + * 2 (normalized distinct index) + * ----------------------------- + * - prints index of the distinct value, but normalized into [0,1] + * - similar to 1, but shows how 'long' the bucket range is + * - handy for plotting the histogram + * + * When plotting the histogram, be careful as the (1) and (2) options skew the + * lengths by distributing the distinct values uniformly. For data types + * without a clear meaning of 'distance' (e.g. strings) that is not a big deal, + * but for numbers it may be confusing. + */ +PG_FUNCTION_INFO_V1(pg_histogram_buckets); + +#define OUTPUT_FORMAT_RAW 0 +#define OUTPUT_FORMAT_INDEXES 1 +#define OUTPUT_FORMAT_DISTINCT 2 + +Datum +pg_histogram_buckets(PG_FUNCTION_ARGS) +{ + FuncCallContext *funcctx; + int call_cntr; + int max_calls; + TupleDesc tupdesc; + AttInMetadata *attinmeta; + + int otype = PG_GETARG_INT32(1); + + if ((otype < 0) || (otype > 2)) + elog(ERROR, "invalid output type specified"); + + /* stuff done only on the first call of the function */ + if (SRF_IS_FIRSTCALL()) + { + MemoryContext oldcontext; + MVHistogram *histogram; + + /* create a function context for cross-call persistence */ + funcctx = SRF_FIRSTCALL_INIT(); + + /* switch to memory context appropriate for multiple function calls */ + oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); + + histogram = statext_histogram_deserialize(PG_GETARG_BYTEA_P(0)); + + funcctx->user_fctx = histogram; + + /* total number of tuples to be returned */ + funcctx->max_calls = 0; + if (funcctx->user_fctx != NULL) + funcctx->max_calls = histogram->nbuckets; + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("function returning record called in context " + "that cannot accept type record"))); + + /* + * generate attribute metadata needed later to produce tuples from raw + * C strings + */ + attinmeta = TupleDescGetAttInMetadata(tupdesc); + funcctx->attinmeta = attinmeta; + + MemoryContextSwitchTo(oldcontext); + } + + /* stuff done on every call of the function */ + funcctx = SRF_PERCALL_SETUP(); + + call_cntr = funcctx->call_cntr; + max_calls = funcctx->max_calls; + attinmeta = funcctx->attinmeta; + + if (call_cntr < max_calls) /* do when there is more left to send */ + { + char **values; + HeapTuple tuple; + Datum result; + double bucket_volume = 1.0; + StringInfo bufs; + + char *format; + int i; + + Oid *outfuncs; + FmgrInfo *fmgrinfo; + + MVHistogram *histogram; + MVBucket *bucket; + + histogram = (MVHistogram *) funcctx->user_fctx; + + Assert(call_cntr < histogram->nbuckets); + + bucket = histogram->buckets[call_cntr]; + + /* + * The scalar values will be formatted directly, using snprintf. + * + * The 'array' values will be formatted through StringInfo. + */ + values = (char **) palloc0(9 * sizeof(char *)); + bufs = (StringInfo) palloc0(9 * sizeof(StringInfoData)); + + values[0] = (char *) palloc(64 * sizeof(char)); + + initStringInfo(&bufs[1]); /* lower boundaries */ + initStringInfo(&bufs[2]); /* upper boundaries */ + initStringInfo(&bufs[3]); /* nulls-only */ + initStringInfo(&bufs[4]); /* lower inclusive */ + initStringInfo(&bufs[5]); /* upper inclusive */ + + values[6] = (char *) palloc(64 * sizeof(char)); + values[7] = (char *) palloc(64 * sizeof(char)); + values[8] = (char *) palloc(64 * sizeof(char)); + + /* we need to do this only when printing the actual values */ + outfuncs = (Oid *) palloc0(sizeof(Oid) * histogram->ndimensions); + fmgrinfo = (FmgrInfo *) palloc0(sizeof(FmgrInfo) * histogram->ndimensions); + + /* + * lookup output functions for all histogram dimensions + * + * XXX This might be one in the first call and stored in user_fctx. + */ + for (i = 0; i < histogram->ndimensions; i++) + { + bool isvarlena; + + getTypeOutputInfo(histogram->types[i], &outfuncs[i], &isvarlena); + + fmgr_info(outfuncs[i], &fmgrinfo[i]); + } + + snprintf(values[0], 64, "%d", call_cntr); /* bucket ID */ + + /* + * for the arrays of lower/upper boundaries, formated according to + * otype + */ + for (i = 0; i < histogram->ndimensions; i++) + { + Datum *vals = histogram->values[i]; + + uint16 minidx = bucket->min[i]; + uint16 maxidx = bucket->max[i]; + + int d = 1; + + /* + * compute bucket volume, using distinct values as a measure + * + * XXX Not really sure what to do for NULL dimensions or + * dimensions with just a single value here, so let's simply count + * them as 1. They will not affect the volume anyway. + */ + if (histogram->nvalues[i] > 1) + d = (histogram->nvalues[i] - 1); + + bucket_volume *= (double) (maxidx - minidx + 1) / d; + + if (i == 0) + format = "{%s"; /* fist dimension */ + else if (i < (histogram->ndimensions - 1)) + format = ", %s"; /* medium dimensions */ + else + format = ", %s}"; /* last dimension */ + + appendStringInfo(&bufs[3], format, bucket->nullsonly[i] ? "t" : "f"); + appendStringInfo(&bufs[4], format, bucket->min_inclusive[i] ? "t" : "f"); + appendStringInfo(&bufs[5], format, bucket->max_inclusive[i] ? "t" : "f"); + + /* + * for NULL-only dimension, simply put there the NULL and + * continue + */ + if (bucket->nullsonly[i]) + { + if (i == 0) + format = "{%s"; + else if (i < (histogram->ndimensions - 1)) + format = ", %s"; + else + format = ", %s}"; + + appendStringInfo(&bufs[1], format, "NULL"); + appendStringInfo(&bufs[2], format, "NULL"); + + continue; + } + + /* otherwise we really need to format the value */ + switch (otype) + { + case OUTPUT_FORMAT_RAW: /* actual boundary values */ + + if (i == 0) + format = "{%s"; + else if (i < (histogram->ndimensions - 1)) + format = ", %s"; + else + format = ", %s}"; + + appendStringInfo(&bufs[1], format, + FunctionCall1(&fmgrinfo[i], vals[minidx])); + + appendStringInfo(&bufs[2], format, + FunctionCall1(&fmgrinfo[i], vals[maxidx])); + + break; + + case OUTPUT_FORMAT_INDEXES: /* indexes into deduplicated + * arrays */ + + if (i == 0) + format = "{%d"; + else if (i < (histogram->ndimensions - 1)) + format = ", %d"; + else + format = ", %d}"; + + appendStringInfo(&bufs[1], format, minidx); + appendStringInfo(&bufs[2], format, maxidx); + + break; + + case OUTPUT_FORMAT_DISTINCT: /* distinct arrays as measure */ + + if (i == 0) + format = "{%f"; + else if (i < (histogram->ndimensions - 1)) + format = ", %f"; + else + format = ", %f}"; + + appendStringInfo(&bufs[1], format, (minidx * 1.0 / d)); + appendStringInfo(&bufs[2], format, (maxidx * 1.0 / d)); + + break; + + default: + elog(ERROR, "unknown output type: %d", otype); + } + } + + values[1] = bufs[1].data; + values[2] = bufs[2].data; + values[3] = bufs[3].data; + values[4] = bufs[4].data; + values[5] = bufs[5].data; + + snprintf(values[6], 64, "%f", bucket->frequency); /* frequency */ + snprintf(values[7], 64, "%f", bucket->frequency / bucket_volume); /* density */ + snprintf(values[8], 64, "%f", bucket_volume); /* volume (as a + * fraction) */ + + /* build a tuple */ + tuple = BuildTupleFromCStrings(attinmeta, values); + + /* make the tuple into a datum */ + result = HeapTupleGetDatum(tuple); + + /* clean up (this is not really necessary) */ + pfree(values[0]); + pfree(values[6]); + pfree(values[7]); + pfree(values[8]); + + resetStringInfo(&bufs[1]); + resetStringInfo(&bufs[2]); + resetStringInfo(&bufs[3]); + resetStringInfo(&bufs[4]); + resetStringInfo(&bufs[5]); + + pfree(bufs); + pfree(values); + + SRF_RETURN_NEXT(funcctx, result); + } + else /* do when there is no more left */ + { + SRF_RETURN_DONE(funcctx); + } +} + +/* + * pg_histogram_in - input routine for type pg_histogram. + * + * pg_histogram is real enough to be a table column, but it has no operations + * of its own, and disallows input too + */ +Datum +pg_histogram_in(PG_FUNCTION_ARGS) +{ + /* + * pg_histogram stores the data in binary form and parsing text input is + * not needed, so disallow this. + */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot accept a value of type %s", "pg_histogram"))); + + PG_RETURN_VOID(); /* keep compiler quiet */ +} + +/* + * pg_histogram_out - output routine for type pg_histogram. + * + * histograms are serialized into a bytea value, so we simply call byteaout() + * to serialize the value into text. But it'd be nice to serialize that into + * a meaningful representation (e.g. for inspection by people). + * + * XXX This should probably return something meaningful, similar to what + * pg_dependencies_out does. Not sure how to deal with the deduplicated + * values, though - do we want to expand that or not? + */ +Datum +pg_histogram_out(PG_FUNCTION_ARGS) +{ + return byteaout(fcinfo); +} + +/* + * pg_histogram_recv - binary input routine for type pg_histogram. + */ +Datum +pg_histogram_recv(PG_FUNCTION_ARGS) +{ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot accept a value of type %s", "pg_histogram"))); + + PG_RETURN_VOID(); /* keep compiler quiet */ +} + +/* + * pg_histogram_send - binary output routine for type pg_histogram. + * + * Histograms are serialized in a bytea value (although the type is named + * differently), so let's just send that. + */ +Datum +pg_histogram_send(PG_FUNCTION_ARGS) +{ + return byteasend(fcinfo); +} + +/* + * selectivity estimation + */ + +/* + * When evaluating conditions on the histogram, we can leverage the fact that + * each bucket boundary value is used by many buckets (each bucket split + * introduces a single new value, duplicating all the other values). That + * allows us to significantly reduce the number of function calls by caching + * the results. + * + * This is one of the reasons why we keep the histogram in partially serialized + * form, with deduplicated values. This allows us to maintain a simple array + * of results indexed by uint16 values. + * + * We only need 2 bits per value, but we allocate a full char as it's more + * convenient and there's not much to gain. 0 means 'unknown' as the function + * was not executed for this value yet. + */ + +#define HIST_CACHE_FALSE 0x01 +#define HIST_CACHE_TRUE 0x03 +#define HIST_CACHE_MASK 0x02 + +/* + * bucket_contains_value + * Decide if the bucket (a range of values in a particular dimension) may + * contain the supplied value. + * + * The function does not simply return true/false, but a "match level" (none, + * partial, full), just like other similar functions. In fact, thise function + * only returns "partial" or "none" levels, as a range can never match exactly + * a value (we never generate histograms with "collapsed" dimensions). + * + * FIXME Should use a better estimate than DEFAULT_EQ_SEL, e.g. derived + * from ndistinct for the variable. But for histograms we shouldn't really + * get here, because equalities are handled as conditions (i.e. we'll get + * here when deciding which buckets match the conditions, but the fraction + * value does not really matter, we only care about the match flag). + */ +static bool +bucket_contains_value(FmgrInfo ltproc, Datum constvalue, + Datum min_value, Datum max_value, + int min_index, int max_index, + bool min_include, bool max_include, + char *callcache, double *fraction) +{ + bool a, + b; + + char min_cached = callcache[min_index]; + char max_cached = callcache[max_index]; + + /* + * First some quick checks on equality - if any of the boundaries equals, + * we have a partial match (so no need to call the comparator). + */ + if (((min_value == constvalue) && (min_include)) || + ((max_value == constvalue) && (max_include))) + { + *fraction = DEFAULT_EQ_SEL; + return true; + } + + /* Keep the values 0/1 because of the XOR at the end. */ + a = ((min_cached & HIST_CACHE_MASK) >> 1); + b = ((max_cached & HIST_CACHE_MASK) >> 1); + + /* + * If result for the bucket lower bound not in cache, evaluate the + * function and store the result in the cache. + */ + if (!min_cached) + { + a = DatumGetBool(FunctionCall2Coll(<proc, + DEFAULT_COLLATION_OID, + constvalue, min_value)); + /* remember the result */ + callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE; + } + + /* And do the same for the upper bound. */ + if (!max_cached) + { + b = DatumGetBool(FunctionCall2Coll(<proc, + DEFAULT_COLLATION_OID, + constvalue, max_value)); + /* remember the result */ + callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE; + } + + *fraction = (a ^ b) ? DEFAULT_EQ_SEL : 0.0; + + return (a ^ b) ? true : false; +} + +/* + * bucket_is_smaller_than_value + * Decide if the bucket (a range of values in a particular dimension) is + * smaller than the supplied value. + * + * The function does not simply return true/false, but a "match level" (none, + * partial, full), just like other similar functions. + * + * Unlike bucket_contains_value this may return all three match levels, i.e. + * "full" (e.g. [10,20] < 30), "partial" (e.g. [10,20] < 15) and "none" + * (e.g. [10,20] < 5). + * + * FIXME Use a better estimate, instead of DEFAULT_INEQ_SEL, i.e. something + * derived in a way similar to convert_to_scalar. + */ +static bool +bucket_is_smaller_than_value(FmgrInfo opproc, Oid typeoid, Datum constvalue, + Datum min_value, Datum max_value, + int min_index, int max_index, + bool min_include, bool max_include, + char *callcache, bool isgt, + double *fraction) +{ + char min_cached = callcache[min_index]; + char max_cached = callcache[max_index]; + + /* Keep the values 0/1 because of the XOR at the end. */ + bool a = ((min_cached & HIST_CACHE_MASK) >> 1); + bool b = ((max_cached & HIST_CACHE_MASK) >> 1); + + if (!min_cached) + { + a = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + min_value, + constvalue)); + /* remember the result */ + callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE; + } + + if (!max_cached) + { + b = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + max_value, + constvalue)); + /* remember the result */ + callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE; + } + + /* + * Now, we need to combine both results into the final answer, and we need + * to be careful about the 'isgt' variable which kinda inverts the + * meaning. + * + * First, we handle the case when each boundary returns different results. + * In that case the outcome can only be 'partial' match, and the fraction + * is computed using convert_to_scalar, just like for 1D histograms. + */ + if (a != b) + { + double val, high, low, binfrac; + + if (convert_to_scalar(constvalue, typeoid, &val, min_value, max_value, + typeoid, &low, &high)) + { + + /* shamelessly copied from ineq_histogram_selectivity */ + if (high <= low) + { + /* cope if bin boundaries appear identical */ + binfrac = 0.5; + } + else if (val <= low) + binfrac = 0.0; + else if (val >= high) + binfrac = 1.0; + else + { + binfrac = (val - low) / (high - low); + + /* + * Watch out for the possibility that we got a NaN or + * Infinity from the division. This can happen + * despite the previous checks, if for example "low" + * is -Infinity. + */ + if (isnan(binfrac) || + binfrac < 0.0 || binfrac > 1.0) + binfrac = 0.5; + } + } + else + binfrac = 0.5; + + *fraction = (isgt) ? binfrac : (1-binfrac); + return true; + } + + /* + * When the results are the same, then it depends on the 'isgt' value. + * There are four options: + * + * isgt=false a=b=true => full match isgt=false a=b=false => empty + * isgt=true a=b=true => empty isgt=true a=b=false => full match + * + * We'll cheat a bit, because we know that (a=b) so we'll use just one of + * them. + */ + if (isgt) + { + *fraction = (!a) ? 1.0 : 0.0; + return (!a); + } + else + { + *fraction = (a) ? 1.0 : 0.0; + return a; + } +} + +/* + * Evaluate clauses using the histogram, and update the match bitmap. + * + * The bitmap may be already partially set, so this is really a way to + * combine results of several clause lists - either when computing + * conditional probability P(A|B) or a combination of AND/OR clauses. + * + * Note: This is not a simple bitmap in the sense that there are three + * possible values for each item - no match, partial match and full match. + * So we need at least 2 bits per item. + * + * TODO: This works with 'bitmap' where each item is represented as a + * char, which is slightly wasteful. Instead, we could use a bitmap + * with 2 bits per item, reducing the size to ~1/4. By using values + * 0, 1 and 3 (instead of 0, 1 and 2), the operations (merging etc.) + * might be performed just like for simple bitmap by using & and |, + * which might be faster than min/max. + */ +static void +histogram_update_match_bitmap(PlannerInfo *root, List *clauses, + Bitmapset *stakeys, + MVHistogram * histogram, + bucket_match *matches, bool is_or) +{ + int i; + ListCell *l; + + /* + * Used for caching function calls, only once per deduplicated value. + * + * We know may have up to (2 * nbuckets) values per dimension. It's + * probably overkill, but let's allocate that once for all clauses, to + * minimize overhead. + * + * Also, we only need two bits per value, but this allocates byte per + * value. Might be worth optimizing. + * + * 0x00 - not yet called 0x01 - called, result is 'false' 0x03 - called, + * result is 'true' + */ + char *callcache = palloc(histogram->nbuckets); + + Assert(histogram != NULL); + Assert(histogram->nbuckets > 0); + + Assert(clauses != NIL); + Assert(list_length(clauses) >= 1); + + /* loop through the clauses and do the estimation */ + foreach(l, clauses) + { + Node *clause = (Node *) lfirst(l); + + /* if it's a RestrictInfo, then extract the clause */ + if (IsA(clause, RestrictInfo)) + clause = (Node *) ((RestrictInfo *) clause)->clause; + + /* it's either OpClause, or NullTest */ + if (is_opclause(clause)) + { + OpExpr *expr = (OpExpr *) clause; + bool varonleft = true; + bool ok; + + FmgrInfo opproc; /* operator */ + + fmgr_info(get_opcode(expr->opno), &opproc); + + /* reset the cache (per clause) */ + memset(callcache, 0, histogram->nbuckets); + + ok = (NumRelids(clause) == 1) && + (is_pseudo_constant_clause(lsecond(expr->args)) || + (varonleft = false, + is_pseudo_constant_clause(linitial(expr->args)))); + + if (ok) + { + FmgrInfo ltproc; + RegProcedure oprrest = get_oprrest(expr->opno); + + Var *var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + Const *cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args); + bool isgt = (!varonleft); + + TypeCacheEntry *typecache + = lookup_type_cache(var->vartype, TYPECACHE_LT_OPR); + + /* lookup dimension for the attribute */ + int idx = bms_member_index(stakeys, var->varattno); + + fmgr_info(get_opcode(typecache->lt_opr), <proc); + + /* + * Check this for all buckets that still have "true" in the + * bitmap + * + * We already know the clauses use suitable operators (because + * that's how we filtered them). + */ + for (i = 0; i < histogram->nbuckets; i++) + { + bool res; + double fraction; + + MVBucket *bucket = histogram->buckets[i]; + + /* histogram boundaries */ + Datum minval, + maxval; + bool mininclude, + maxinclude; + int minidx, + maxidx; + + /* + * For AND-lists, we can also mark NULL buckets as 'no + * match' (and then skip them). For OR-lists this is not + * possible. + */ + if ((!is_or) && bucket->nullsonly[idx]) + matches[i].match = false; + + /* + * XXX There used to be logic to skip buckets that can't + * possibly match, depending on the is_or flag (either + * fully matching or elimated). Once we abandoned the + * concept of NONE/PARTIAL/FULL matches and switched to + * a bool flag + fraction that does not seem possible. + * But maybe we can make it work somehow? + */ + + /* lookup the values and cache of function calls */ + minidx = bucket->min[idx]; + maxidx = bucket->max[idx]; + + minval = histogram->values[idx][bucket->min[idx]]; + maxval = histogram->values[idx][bucket->max[idx]]; + + mininclude = bucket->min_inclusive[idx]; + maxinclude = bucket->max_inclusive[idx]; + + /* + * If it's not a "<" or ">" or "=" operator, just ignore + * the clause. Otherwise note the relid and attnum for the + * variable. + * + * TODO I'm really unsure the handling of 'isgt' flag + * (that is, clauses with reverse order of + * variable/constant) is correct. I wouldn't be surprised + * if there was some mixup. Using the lt/gt operators + * instead of messing with the opproc could make it + * simpler. It would however be using a different operator + * than the query, although it's not any shadier than + * using the selectivity function as is done currently. + */ + switch (oprrest) + { + case F_SCALARLTSEL: /* Var < Const */ + case F_SCALARLESEL: /* Var <= Const */ + case F_SCALARGTSEL: /* Var > Const */ + case F_SCALARGESEL: /* Var >= Const */ + + res = bucket_is_smaller_than_value(opproc, var->vartype, + cst->constvalue, + minval, maxval, + minidx, maxidx, + mininclude, maxinclude, + callcache, isgt, &fraction); + + break; + + case F_EQSEL: + case F_NEQSEL: + + /* + * We only check whether the value is within the + * bucket, using the lt operator, and we also + * check for equality with the boundaries. + */ + + res = bucket_contains_value(ltproc, cst->constvalue, + minval, maxval, + minidx, maxidx, + mininclude, maxinclude, + callcache, &fraction); + + break; + } + + /* + * Merge the result into the bitmap, depending on type + * of the current clause (AND or OR). + */ + if (is_or) + { + Selectivity s1, s2; + + /* OR follows the Max() semantics */ + matches[i].match |= res; + + /* + * Selectivities for an OR clause are combined as s1+s2 - s1*s2 + * to account for the probable overlap of selected tuple sets. + * This is the same formula as in clause_selectivity, because + * the fraction is computed assuming independence (but then we + * also apply geometric mean). + */ + s1 = matches[i].fraction; + s2 = fraction; + + matches[i].fraction = s1 + s2 - s1 * s2; + + CLAMP_PROBABILITY(matches[i].fraction); + } + else + { + /* AND follows Min() semantics */ + matches[i].match &= res; + matches[i].fraction *= fraction; + } + } + } + } + else if (IsA(clause, NullTest)) + { + NullTest *expr = (NullTest *) clause; + Var *var = (Var *) (expr->arg); + + /* lookup index of attribute in the statistics */ + int idx = bms_member_index(stakeys, var->varattno); + + /* + * Walk through the buckets and evaluate the current clause. We + * can skip items that were already ruled out, and terminate if + * there are no remaining buckets that might possibly match. + */ + for (i = 0; i < histogram->nbuckets; i++) + { + char match = false; + MVBucket *bucket = histogram->buckets[i]; + + /* + * Skip buckets that were already eliminated - this is + * impotant considering how we update the info (we only lower + * the match) + */ + if ((!is_or) && (!matches[i].match)) + continue; + else if (is_or && (matches[i].match)) + continue; + + switch (expr->nulltesttype) + { + case IS_NULL: + match = (bucket->nullsonly[idx]) ? true : match; + break; + + case IS_NOT_NULL: + match = (!bucket->nullsonly[idx]) ? true : match; + break; + } + + /* now, update the match bitmap, depending on OR/AND type */ + if (is_or) + { + matches[i].match |= match; + matches[i].fraction = (match) ? 1.0 : matches[i].fraction; + } + else + { + matches[i].match &= match; + matches[i].fraction = (match) ? matches[i].fraction : 0.0; + } + } + } + else if (or_clause(clause) || and_clause(clause)) + { + /* + * AND/OR clause, with all sub-clauses compatible with the stats + */ + + int i; + BoolExpr *bool_clause = ((BoolExpr *) clause); + List *bool_clauses = bool_clause->args; + + /* match/mismatch bitmap for each bucket */ + bucket_match *bool_matches = NULL; + + Assert(bool_clauses != NIL); + Assert(list_length(bool_clauses) >= 2); + + /* by default none of the buckets matches the clauses */ + bool_matches = palloc0(sizeof(bucket_match) * histogram->nbuckets); + + if (or_clause(clause)) + { + /* OR clauses assume nothing matches, initially */ + for (i = 0; i < histogram->nbuckets; i++) + { + bool_matches[i].match = false; + bool_matches[i].fraction = 0.0; + } + } + else + { + /* AND clauses assume nothing matches, initially */ + for (i = 0; i < histogram->nbuckets; i++) + { + bool_matches[i].match = true; + bool_matches[i].fraction = 1.0; + } + } + + /* build the match bitmap for the OR-clauses */ + histogram_update_match_bitmap(root, bool_clauses, + stakeys, histogram, + bool_matches, or_clause(clause)); + + /* + * Merge the bitmap produced by histogram_update_match_bitmap into + * the current one. We need to consider if we're evaluating AND or + * OR condition when merging the results. + */ + for (i = 0; i < histogram->nbuckets; i++) + { + /* Is this OR or AND clause? */ + if (is_or) + { + Selectivity s1, s2; + + matches[i].match |= bool_matches[i].match; + + /* + * Selectivities for an OR clause are combined as s1+s2 - s1*s2 + * to account for the probable overlap of selected tuple sets. + * This is the same formula as in clause_selectivity, because + * the fraction is computed assuming independence (but then we + * also apply geometric mean). + */ + s1 = matches[i].fraction; + s2 = bool_matches[i].fraction; + + matches[i].fraction = s1 + s2 - s1 * s2; + + CLAMP_PROBABILITY(matches[i].fraction); + } + else + { + matches[i].match &= bool_matches[i].match; + matches[i].fraction *= bool_matches[i].fraction; + } + } + + pfree(bool_matches); + + } + else if (not_clause(clause)) + { + /* NOT clause, with all subclauses compatible */ + + int i; + BoolExpr *not_clause = ((BoolExpr *) clause); + List *not_args = not_clause->args; + + /* match/mismatch bitmap for each MCV item */ + bucket_match *not_matches = NULL; + + Assert(not_args != NIL); + Assert(list_length(not_args) == 1); + + /* by default none of the MCV items matches the clauses */ + not_matches = palloc0(sizeof(bucket_match) * histogram->nbuckets); + + /* NOT clauses assume nothing matches, initially + * + * FIXME The comment seems to disagree with the code - not sure + * if nothing should match (code is wrong) or everything should + * match (comment is wrong) by default. + */ + for (i = 0; i < histogram->nbuckets; i++) + { + not_matches[i].match = true; + not_matches[i].fraction = 1.0; + } + + /* build the match bitmap for the OR-clauses */ + histogram_update_match_bitmap(root, not_args, + stakeys, histogram, + not_matches, false); + + /* + * Merge the bitmap produced by histogram_update_match_bitmap into + * the current one. + * + * This is similar to what mcv_update_match_bitmap does, but we + * need to be a tad more careful here, as histograms also track + * what fraction of a bucket matches. + */ + for (i = 0; i < histogram->nbuckets; i++) + { + /* + * When handling a NOT clause, invert the result before + * merging it into the global result. We don't care about + * partial matches here (those invert to partial). + */ + not_matches[i].match = (!not_matches[i].match); + + /* Is this OR or AND clause? */ + if (is_or) + { + Selectivity s1, s2; + + matches[i].match |= not_matches[i].match; + + /* + * Selectivities for an OR clause are combined as s1+s2 - s1*s2 + * to account for the probable overlap of selected tuple sets. + * This is the same formula as in clause_selectivity, because + * the fraction is computed assuming independence (but then we + * also apply geometric mean). + */ + s1 = matches[i].fraction; + s2 = not_matches[i].fraction; + + matches[i].fraction = s1 + s2 - s1 * s2; + + CLAMP_PROBABILITY(matches[i].fraction); + } + else + { + matches[i].match &= not_matches[i].match; + matches[i].fraction *= not_matches[i].fraction; + } + } + + pfree(not_matches); + } + else if (IsA(clause, Var)) + { + /* Var (has to be a boolean Var, possibly from below NOT) */ + + Var *var = (Var *) (clause); + + /* match the attribute to a dimension of the statistic */ + int idx = bms_member_index(stakeys, var->varattno); + + Assert(var->vartype == BOOLOID); + + /* + * Walk through the buckets and evaluate the current clause. + */ + for (i = 0; i < histogram->nbuckets; i++) + { + MVBucket *bucket = histogram->buckets[i]; + bool match = false; + double fraction = 0.0; + + /* + * If the bucket is NULL, it's a mismatch. Otherwise check + * if lower/upper boundaries match and choose partial/full + * match accordingly. + */ + if (!bucket->nullsonly[idx]) + { + int minidx = bucket->min[idx]; + int maxidx = bucket->max[idx]; + + bool a = DatumGetBool(histogram->values[idx][minidx]); + bool b = DatumGetBool(histogram->values[idx][maxidx]); + + /* How many boundary values match? */ + if (a && b) + { + /* both values match - the whole bucket matches */ + match = true; + fraction = 1.0; + } + else if (a || b) + { + /* one value matches - assume half the bucket matches */ + match = true; + fraction = 0.5; + } + } + + /* now, update the match bitmap, depending on OR/AND type */ + if (is_or) + { + Selectivity s1, s2; + + matches[i].match |= match; + + /* + * Selectivities for an OR clause are combined as s1+s2 - s1*s2 + * to account for the probable overlap of selected tuple sets. + * This is the same formula as in clause_selectivity, because + * the fraction is computed assuming independence (but then we + * also apply geometric mean). + */ + s1 = matches[i].fraction; + s2 = fraction; + + matches[i].fraction = s1 + s2 - s1 * s2; + + CLAMP_PROBABILITY(matches[i].fraction); + } + else + { + matches[i].match &= match; + matches[i].fraction *= fraction; + } + } + } + else + elog(ERROR, "unknown clause type: %d", clause->type); + } + + /* free the call cache */ + pfree(callcache); +} + +/* + * Estimate selectivity of clauses using a histogram. + * + * If there's no histogram for the stats, the function returns 0.0. + * + * The general idea of this method is similar to how MCV lists are + * processed, except that this introduces the concept of a partial + * match (MCV only works with full match / mismatch). + * + * The algorithm works like this: + * + * 1) mark all buckets as 'full match' + * 2) walk through all the clauses + * 3) for a particular clause, walk through all the buckets + * 4) skip buckets that are already 'no match' + * 5) check clause for buckets that still match (at least partially) + * 6) sum frequencies for buckets to get selectivity + * + * Unlike MCV lists, histograms have a concept of a partial match. In + * that case we use 1/2 the bucket, to minimize the average error. The + * MV histograms are usually less detailed than the per-column ones, + * meaning the sum is often quite high (thanks to combining a lot of + * "partially hit" buckets). + * + * Maybe we could use per-bucket information with number of distinct + * values it contains (for each dimension), and then use that to correct + * the estimate (so with 10 distinct values, we'd use 1/10 of the bucket + * frequency). We might also scale the value depending on the actual + * ndistinct estimate (not just the values observed in the sample). + * + * Another option would be to multiply the selectivities, i.e. if we get + * 'partial match' for a bucket for multiple conditions, we might use + * 0.5^k (where k is the number of conditions), instead of 0.5. This + * probably does not minimize the average error, though. + * + * TODO: This might use a similar shortcut to MCV lists - count buckets + * marked as partial/full match, and terminate once this drop to 0. + * Not sure if it's really worth it - for MCV lists a situation like + * this is not uncommon, but for histograms it's not that clear. + */ +Selectivity +histogram_clauselist_selectivity(PlannerInfo *root, StatisticExtInfo *stat, + List *clauses, List *conditions, + int varRelid, JoinType jointype, + SpecialJoinInfo *sjinfo, RelOptInfo *rel) +{ + int i; + MVHistogram *histogram; + Selectivity s = 0.0; + Selectivity total_sel = 0.0; + Size len; + int nclauses; + + /* match/mismatch bitmap for each MCV item */ + bucket_match *matches = NULL; + bucket_match *condition_matches = NULL; + + nclauses = list_length(clauses); + + /* load the histogram stored in the statistics object */ + histogram = statext_histogram_load(stat->statOid); + + /* size of the match "bitmap" */ + len = sizeof(bucket_match) * histogram->nbuckets; + + /* by default all the histogram buckets match the clauses fully */ + matches = palloc0(len); + + /* by default all buckets match fully */ + for (i = 0; i < histogram->nbuckets; i++) + { + matches[i].match = true; + matches[i].fraction = 1.0; + } + + histogram_update_match_bitmap(root, clauses, stat->keys, + histogram, matches, false); + + /* if there are condition clauses, build a match bitmap for them */ + if (conditions) + { + /* match bitmap for conditions, by default all buckets match */ + condition_matches = palloc0(len); + + /* by default all buckets match fully */ + for (i = 0; i < histogram->nbuckets; i++) + { + condition_matches[i].match = true; + condition_matches[i].fraction = 1.0; + } + + histogram_update_match_bitmap(root, conditions, stat->keys, + histogram, condition_matches, false); + } + + /* now, walk through the buckets and sum the selectivities */ + for (i = 0; i < histogram->nbuckets; i++) + { + double fraction; + + /* skip buckets that don't satisfy the conditions */ + if (conditions && (!condition_matches[i].match)) + continue; + + /* compute selectivity for buckets matching conditions */ + total_sel += histogram->buckets[i]->frequency; + + /* geometric mean of the bucket fraction */ + fraction = pow(matches[i].fraction, 1.0 / nclauses); + + if (matches[i].match) + s += histogram->buckets[i]->frequency * fraction; + } + + /* conditional selectivity P(clauses|conditions) */ + if (total_sel > 0.0) + return (s / total_sel); + + return 0.0; +} diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c index 533fbdc037..79e6f24deb 100644 --- a/src/backend/statistics/mcv.c +++ b/src/backend/statistics/mcv.c @@ -85,7 +85,8 @@ static int count_distinct_groups(int numrows, SortItem *items, */ MCVList * statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, - VacAttrStats **stats, double totalrows) + VacAttrStats **stats, HeapTuple **rows_filtered, + int *numrows_filtered, double totalrows) { int i, j, @@ -96,6 +97,7 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, double stadistinct; int *mcv_counts; int f1; + int numrows_mcv; int *attnums = build_attnums(attrs); @@ -111,6 +113,9 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, /* transform the sorted rows into groups (sorted by frequency) */ SortItem *groups = build_distinct_groups(numrows, items, mss, &ngroups); + /* Either we have both pointers or none of them. */ + Assert((rows_filtered && numrows_filtered) || (!rows_filtered && !numrows_filtered)); + /* * Maximum number of MCV items to store, based on the attribute with the * largest stats target (and the number of groups we have available). @@ -167,6 +172,9 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, numrows, totalrows); } + /* number of rows represented by MCV items */ + numrows_mcv = 0; + /* * At this point we know the number of items for the MCV list. There might * be none (for uniform distribution with many groups), and in that case @@ -243,9 +251,93 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, item->base_frequency *= (double) count / numrows; } + + /* update the number of sampled rows represented by the MCV list */ + numrows_mcv += groups[i].count; } } + /* Assume we're not returning any filtered rows by default. */ + if (numrows_filtered) + *numrows_filtered = 0; + + if (rows_filtered) + *rows_filtered = NULL; + + /* + * Produce an array with only tuples not covered by the MCV list. This is + * needed when building MCV+histogram pair, where MCV covers the most + * common combinations and histogram covers the remaining part. + * + * We will first sort the groups by the keys (not by count) and then use + * binary search in the group array to check which rows are covered by the + * MCV items. + * + * Do not modify the array in place, as there may be additional stats on + * the table and we need to keep the original array for them. + * + * We only do this when requested by passing non-NULL rows_filtered, and + * when there are rows not covered by the MCV list (that is, when + * numrows_mcv < numrows), or also (nitems < ngroups). + */ + if (rows_filtered && numrows_filtered && (nitems < ngroups)) + { + int i, + j; + + /* used to build the filtered array of tuples */ + HeapTuple *filtered; + int nfiltered; + + /* used for the searches */ + SortItem key; + + /* We do know how many rows we expect (total - MCV rows). */ + nfiltered = (numrows - numrows_mcv); + filtered = (HeapTuple *) palloc(nfiltered * sizeof(HeapTuple)); + + /* wfill this with data from the rows */ + key.values = (Datum *) palloc0(numattrs * sizeof(Datum)); + key.isnull = (bool *) palloc0(numattrs * sizeof(bool)); + + /* + * Sort the groups for bsearch_r (but only the items that actually + * made it to the MCV list). + */ + qsort_arg((void *) groups, nitems, sizeof(SortItem), + multi_sort_compare, mss); + + /* walk through the tuples, compare the values to MCV items */ + nfiltered = 0; + for (i = 0; i < numrows; i++) + { + /* collect the key values from the row */ + for (j = 0; j < numattrs; j++) + key.values[j] + = heap_getattr(rows[i], attnums[j], + stats[j]->tupDesc, &key.isnull[j]); + + /* if not included in the MCV list, keep it in the array */ + if (bsearch_arg(&key, groups, nitems, sizeof(SortItem), + multi_sort_compare, mss) == NULL) + filtered[nfiltered++] = rows[i]; + + /* do not overflow the array */ + Assert(nfiltered <= (numrows - numrows_mcv)); + } + + /* expect to get the right number of remaining rows exactly */ + Assert(nfiltered + numrows_mcv == numrows); + + /* pass the filtered tuples up */ + *numrows_filtered = nfiltered; + *rows_filtered = filtered; + + /* free all the data used here */ + pfree(key.values); + pfree(key.isnull); + } + pfree(items); pfree(groups); pfree(mcv_counts); diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index c941c3310b..837660950e 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -1505,6 +1505,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) bool ndistinct_enabled; bool dependencies_enabled; bool mcv_enabled; + bool histogram_enabled; int i; statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); @@ -1541,6 +1542,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) ndistinct_enabled = false; dependencies_enabled = false; mcv_enabled = false; + histogram_enabled = false; for (i = 0; i < ARR_DIMS(arr)[0]; i++) { @@ -1550,6 +1552,8 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) dependencies_enabled = true; if (enabled[i] == STATS_EXT_MCV) mcv_enabled = true; + if (enabled[i] == STATS_EXT_HISTOGRAM) + histogram_enabled = true; } /* @@ -1578,7 +1582,13 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) } if (mcv_enabled) + { appendStringInfo(&buf, "%smcv", gotone ? ", " : ""); + gotone = true; + } + + if (histogram_enabled) + appendStringInfo(&buf, "%shistogram", gotone ? ", " : ""); appendStringInfoChar(&buf, ')'); } diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index fdfc0d6a1b..bd1cbe8e50 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -172,9 +172,6 @@ static double eqjoinsel_semi(Oid operator, RelOptInfo *inner_rel); static bool estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, List **varinfos, double *ndistinct); -static bool convert_to_scalar(Datum value, Oid valuetypid, double *scaledvalue, - Datum lobound, Datum hibound, Oid boundstypid, - double *scaledlobound, double *scaledhibound); static double convert_numeric_to_scalar(Datum value, Oid typid, bool *failure); static void convert_string_to_scalar(char *value, double *scaledvalue, @@ -4104,7 +4101,7 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, int nshared; /* skip statistics of other kinds */ - if (info->kind != STATS_EXT_NDISTINCT) + if ((info->kinds & STATS_EXT_INFO_NDISTINCT) == 0) continue; /* compute attnums shared by the vars and the statistics object */ @@ -4213,7 +4210,7 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, * The several datatypes representing relative times (intervals) are all * converted to measurements expressed in seconds. */ -static bool +bool convert_to_scalar(Datum value, Oid valuetypid, double *scaledvalue, Datum lobound, Datum hibound, Oid boundstypid, double *scaledlobound, double *scaledhibound) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 394301caa8..929aacf906 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2519,7 +2519,8 @@ describeOneTableDetails(const char *schemaname, " a.attnum = s.attnum AND NOT attisdropped)) AS columns,\n" " 'd' = any(stxkind) AS ndist_enabled,\n" " 'f' = any(stxkind) AS deps_enabled,\n" - " 'm' = any(stxkind) AS mcv_enabled\n" + " 'm' = any(stxkind) AS mcv_enabled,\n" + " 'h' = any(stxkind) AS histogram_enabled\n" "FROM pg_catalog.pg_statistic_ext stat " "WHERE stxrelid = '%s'\n" "ORDER BY 1;", @@ -2562,6 +2563,12 @@ describeOneTableDetails(const char *schemaname, if (strcmp(PQgetvalue(result, i, 7), "t") == 0) { appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : ""); + gotone = true; + } + + if (strcmp(PQgetvalue(result, i, 8), "t") == 0) + { + appendPQExpBuffer(&buf, "%shistogram", gotone ? ", " : ""); } appendPQExpBuffer(&buf, ") ON %s FROM %s", diff --git a/src/include/catalog/pg_cast.dat b/src/include/catalog/pg_cast.dat index dff3a9a08a..99d250bb14 100644 --- a/src/include/catalog/pg_cast.dat +++ b/src/include/catalog/pg_cast.dat @@ -330,6 +330,10 @@ { castsource => 'pg_mcv_list', casttarget => 'text', castfunc => '0', castcontext => 'i', castmethod => 'i' }, +# pg_histogram can be coerced to, but not from, bytea +{ castsource => 'pg_histogram', casttarget => 'bytea', castfunc => '0', + castcontext => 'i', castmethod => 'b' }, + # Datetime category { castsource => 'abstime', casttarget => 'date', castfunc => 'date(abstime)', castcontext => 'a', castmethod => 'f' }, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 3cfcafcb11..cbe2b36923 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -5097,6 +5097,30 @@ proargnames => '{mcv_list,index,values,nulls,frequency,base_frequency}', prosrc => 'pg_stats_ext_mcvlist_items' }, +{ oid => '3426', descr => 'I/O', + proname => 'pg_histogram_in', prorettype => 'pg_histogram', + proargtypes => 'cstring', prosrc => 'pg_histogram_in' }, +{ oid => '3427', descr => 'I/O', + proname => 'pg_histogram_out', prorettype => 'cstring', + proargtypes => 'pg_histogram', prosrc => 'pg_histogram_out' }, +{ oid => '3428', descr => 'I/O', + proname => 'pg_histogram_recv', provolatile => 's', + prorettype => 'pg_histogram', proargtypes => 'internal', + prosrc => 'pg_histogram_recv' }, +{ oid => '3429', descr => 'I/O', + proname => 'pg_histogram_send', provolatile => 's', prorettype => 'bytea', + proargtypes => 'pg_histogram', prosrc => 'pg_histogram_send' }, + +{ oid => '3430', + descr => 'details about histogram buckets', + proname => 'pg_histogram_buckets', prorows => '1000', proisstrict => 'f', + proretset => 't', provolatile => 's', prorettype => 'record', + proargtypes => 'pg_histogram int4', + proallargtypes => '{pg_histogram,int4,int4,_text,_text,_bool,_bool,_bool,float8,float8,float8}', + proargmodes => '{i,i,o,o,o,o,o,o,o,o,o}', + proargnames => '{histogram,otype,index,minvals,maxvals,nullsonly,mininclusive,maxinclusive,frequency,density,bucket_volume}', + prosrc => 'pg_histogram_buckets' }, + { oid => '1928', descr => 'statistics: number of scans done for table/index', proname => 'pg_stat_get_numscans', provolatile => 's', proparallel => 'r', prorettype => 'int8', proargtypes => 'oid', diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h index 7ddbee63c9..514f4230c9 100644 --- a/src/include/catalog/pg_statistic_ext.h +++ b/src/include/catalog/pg_statistic_ext.h @@ -48,6 +48,7 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId) pg_ndistinct stxndistinct; /* ndistinct coefficients (serialized) */ pg_dependencies stxdependencies; /* dependencies (serialized) */ pg_mcv_list stxmcv; /* MCV (serialized) */ + pg_histogram stxhistogram; /* MV histogram (serialized) */ #endif } FormData_pg_statistic_ext; @@ -64,6 +65,7 @@ typedef FormData_pg_statistic_ext *Form_pg_statistic_ext; #define STATS_EXT_NDISTINCT 'd' #define STATS_EXT_DEPENDENCIES 'f' #define STATS_EXT_MCV 'm' +#define STATS_EXT_HISTOGRAM 'h' #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 0ff25e87a7..15e3ab9f93 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -172,6 +172,13 @@ typoutput => 'pg_mcv_list_out', typreceive => 'pg_mcv_list_recv', typsend => 'pg_mcv_list_send', typalign => 'i', typstorage => 'x', typcollation => '100' }, +{ oid => '3425', oid_symbol => 'PGHISTOGRAMOID', + descr => 'multivariate histogram', + typname => 'pg_histogram', typlen => '-1', typbyval => 'f', + typcategory => 'S', typinput => 'pg_histogram_in', + typoutput => 'pg_histogram_out', typreceive => 'pg_histogram_recv', + typsend => 'pg_histogram_send', typalign => 'i', typstorage => 'x', + typcollation => '100' }, { oid => '32', oid_symbol => 'PGDDLCOMMANDOID', descr => 'internal type for passing CollectedCommand', typname => 'pg_ddl_command', typlen => 'SIZEOF_POINTER', typbyval => 't', diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index adb4265047..5849a1d0d4 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -859,10 +859,15 @@ typedef struct StatisticExtInfo Oid statOid; /* OID of the statistics row */ RelOptInfo *rel; /* back-link to statistic's table */ - char kind; /* statistic kind of this entry */ + int kinds; /* statistic kinds of this entry */ Bitmapset *keys; /* attnums of the columns covered */ } StatisticExtInfo; +#define STATS_EXT_INFO_NDISTINCT 1 +#define STATS_EXT_INFO_DEPENDENCIES 2 +#define STATS_EXT_INFO_MCV 4 +#define STATS_EXT_INFO_HISTOGRAM 8 + /* * EquivalenceClasses * diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h index 11159b58ee..ddb0066378 100644 --- a/src/include/statistics/extended_stats_internal.h +++ b/src/include/statistics/extended_stats_internal.h @@ -69,10 +69,16 @@ extern MVDependencies *statext_dependencies_deserialize(bytea *data); extern MCVList * statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, VacAttrStats **stats, + HeapTuple **rows_filtered, int *numrows_filtered, double totalrows); extern bytea *statext_mcv_serialize(MCVList * mcv, VacAttrStats **stats); extern MCVList * statext_mcv_deserialize(bytea *data); +extern MVHistogram * statext_histogram_build(int numrows, HeapTuple *rows, + Bitmapset *attrs, VacAttrStats **stats, + int numrows_total); +extern MVHistogram * statext_histogram_deserialize(bytea *data); + extern MultiSortSupport multi_sort_init(int ndims); extern void multi_sort_add_dimension(MultiSortSupport mss, int sortdim, Oid oper); @@ -83,6 +89,7 @@ extern int multi_sort_compare_dims(int start, int end, const SortItem *a, const SortItem *b, MultiSortSupport mss); extern int compare_scalars_simple(const void *a, const void *b, void *arg); extern int compare_datums_simple(Datum a, Datum b, SortSupport ssup); +extern int compare_scalars_partition(const void *a, const void *b, void *arg); extern void *bsearch_arg(const void *key, const void *base, size_t nmemb, size_t size, @@ -109,4 +116,12 @@ extern Selectivity mcv_clauselist_selectivity(PlannerInfo *root, Selectivity *basesel, Selectivity *totalsel); +extern Selectivity histogram_clauselist_selectivity(PlannerInfo *root, + StatisticExtInfo *stat, + List *clauses, List *conditions, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + RelOptInfo *rel); + #endif /* EXTENDED_STATS_INTERNAL_H */ diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h index e69d6a0232..1d276f0b6d 100644 --- a/src/include/statistics/statistics.h +++ b/src/include/statistics/statistics.h @@ -119,9 +119,68 @@ typedef struct MCVList MCVItem **items; /* array of MCV items */ } MCVList; + +/* used to flag stats serialized to bytea */ +#define STATS_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */ +#define STATS_HIST_TYPE_BASIC 1 /* basic histogram type */ + +/* max buckets in a histogram (mostly arbitrary number) */ +#define STATS_HIST_MAX_BUCKETS 16384 + +/* + * Histogram in a partially serialized form, with deduplicated boundary + * values etc. + */ +typedef struct MVBucket +{ + /* Frequencies of this bucket. */ + float frequency; + + /* + * Information about dimensions being NULL-only. Not yet used. + */ + bool *nullsonly; + + /* lower boundaries - values and information about the inequalities */ + uint16 *min; + bool *min_inclusive; + + /* + * indexes of upper boundaries - values and information about the + * inequalities (exclusive vs. inclusive) + */ + uint16 *max; + bool *max_inclusive; +} MVBucket; + +typedef struct MVHistogram +{ + /* varlena header (do not touch directly!) */ + int32 vl_len_; + uint32 magic; /* magic constant marker */ + uint32 type; /* type of histogram (BASIC) */ + uint32 nbuckets; /* number of buckets (buckets array) */ + uint32 ndimensions; /* number of dimensions */ + Oid types[STATS_MAX_DIMENSIONS]; /* OIDs of data types */ + + /* + * keep this the same with MVHistogram, because of deserialization (same + * offset) + */ + MVBucket **buckets; /* array of buckets */ + + /* + * serialized boundary values, one array per dimension, deduplicated (the + * min/max indexes point into these arrays) + */ + int *nvalues; + Datum **values; +} MVHistogram; + extern MVNDistinct *statext_ndistinct_load(Oid mvoid); extern MVDependencies *statext_dependencies_load(Oid mvoid); extern MCVList * statext_mcv_load(Oid mvoid); +extern MVHistogram * statext_histogram_load(Oid mvoid); extern void BuildRelationExtStatistics(Relation onerel, double totalrows, int numrows, HeapTuple *rows, @@ -141,8 +200,8 @@ extern Selectivity statext_clauselist_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo, RelOptInfo *rel, Bitmapset **estimatedclauses); -extern bool has_stats_of_kind(List *stats, char requiredkind); +extern bool has_stats_of_kind(List *stats, int requiredkinds); extern StatisticExtInfo *choose_best_statistics(List *stats, - Bitmapset *attnums, char requiredkind); + Bitmapset *attnums, int requiredkinds); #endif /* STATISTICS_H */ diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h index 4e9aaca6b5..6c9dd1d85a 100644 --- a/src/include/utils/selfuncs.h +++ b/src/include/utils/selfuncs.h @@ -222,6 +222,10 @@ extern void genericcostestimate(PlannerInfo *root, IndexPath *path, List *qinfos, GenericCosts *costs); +extern bool convert_to_scalar(Datum value, Oid valuetypid, double *scaledvalue, + Datum lobound, Datum hibound, Oid boundstypid, + double *scaledlobound, double *scaledhibound); + /* Functions in array_selfuncs.c */ extern Selectivity scalararraysel_containment(PlannerInfo *root, diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out index 0f97355165..fc672f6cd6 100644 --- a/src/test/regress/expected/create_table_like.out +++ b/src/test/regress/expected/create_table_like.out @@ -243,7 +243,7 @@ Indexes: Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Statistics objects: - "public"."ctlt_all_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM ctlt_all + "public"."ctlt_all_a_b_stat" (ndistinct, dependencies, mcv, histogram) ON a, b FROM ctlt_all SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid; relname | objsubid | description diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out index d13f0928d0..174efe3ce0 100644 --- a/src/test/regress/expected/opr_sanity.out +++ b/src/test/regress/expected/opr_sanity.out @@ -903,11 +903,12 @@ WHERE c.castmethod = 'b' AND pg_ndistinct | bytea | 0 | i pg_dependencies | bytea | 0 | i pg_mcv_list | bytea | 0 | i + pg_histogram | bytea | 0 | i cidr | inet | 0 | i xml | text | 0 | a xml | character varying | 0 | a xml | character | 0 | a -(10 rows) +(11 rows) -- **************** pg_conversion **************** -- Look for illegal values in pg_conversion fields. diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 5d05962c04..67975c91d3 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -58,7 +58,7 @@ ALTER TABLE ab1 DROP COLUMN a; b | integer | | | c | integer | | | Statistics objects: - "public"."ab1_b_c_stats" (ndistinct, dependencies, mcv) ON b, c FROM ab1 + "public"."ab1_b_c_stats" (ndistinct, dependencies, mcv, histogram) ON b, c FROM ab1 -- Ensure statistics are dropped when table is SELECT stxname FROM pg_statistic_ext WHERE stxname LIKE 'ab1%'; @@ -204,9 +204,9 @@ CREATE STATISTICS s10 ON a, b, c FROM ndistinct; ANALYZE ndistinct; SELECT stxkind, stxndistinct FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass; - stxkind | stxndistinct ----------+--------------------------------------------------------- - {d,f,m} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301} + stxkind | stxndistinct +-----------+--------------------------------------------------------- + {d,f,m,h} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301} (1 row) -- Hash Aggregate, thanks to estimates improved by the statistic @@ -270,9 +270,9 @@ INSERT INTO ndistinct (a, b, c, filler1) ANALYZE ndistinct; SELECT stxkind, stxndistinct FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass; - stxkind | stxndistinct ----------+------------------------------------------------------------- - {d,f,m} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000} + stxkind | stxndistinct +-----------+------------------------------------------------------------- + {d,f,m,h} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000} (1 row) -- plans using Group Aggregate, thanks to using correct esimates @@ -758,7 +758,6 @@ EXPLAIN (COSTS OFF) Index Cond: ((a IS NULL) AND (b IS NULL)) (5 rows) -RESET random_page_cost; -- mcv with arrays CREATE TABLE mcv_lists_arrays ( a TEXT[], @@ -822,3 +821,197 @@ EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c; Filter: ((NOT a) AND b AND (NOT c)) (3 rows) +RESET random_page_cost; +-- histograms +CREATE TABLE histograms ( + filler1 TEXT, + filler2 NUMERIC, + a INT, + b TEXT, + filler3 DATE, + c INT, + d TEXT +); +SET random_page_cost = 1.2; +CREATE INDEX histograms_ab_idx ON mcv_lists (a, b); +CREATE INDEX histograms_abc_idx ON histograms (a, b, c); +-- random data (we still get histogram, but as the columns are not +-- correlated, the estimates remain about the same) +INSERT INTO histograms (a, b, c, filler1) + SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i); +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a < 5) AND (b < '5'::text)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a < 5) AND (b < '5'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5' AND c < 5; + QUERY PLAN +--------------------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a < 5) AND (b < '5'::text) AND (c < 5)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a < 5) AND (b < '5'::text) AND (c < 5)) +(4 rows) + +-- create statistics +CREATE STATISTICS histograms_stats (histogram) ON a, b, c FROM histograms; +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a < 5) AND (b < '5'::text)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a < 5) AND (b < '5'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5' AND c < 5; + QUERY PLAN +--------------------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a < 5) AND (b < '5'::text) AND (c < 5)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a < 5) AND (b < '5'::text) AND (c < 5)) +(4 rows) + +-- values correlated along the diagonal +TRUNCATE histograms; +DROP STATISTICS histograms_stats; +INSERT INTO histograms (a, b, c, filler1) + SELECT mod(i,100), mod(i,100) + mod(i,7), mod(i,100) + mod(i,11), i FROM generate_series(1,5000) s(i); +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND c < 3; + QUERY PLAN +--------------------------------------------------- + Index Scan using histograms_abc_idx on histograms + Index Cond: ((a < 3) AND (c < 3)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND b > '2' AND c < 3; + QUERY PLAN +--------------------------------------------------------- + Index Scan using histograms_abc_idx on histograms + Index Cond: ((a < 3) AND (b > '2'::text) AND (c < 3)) +(2 rows) + +-- create statistics +CREATE STATISTICS histograms_stats (histogram) ON a, b, c FROM histograms; +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND c < 3; + QUERY PLAN +----------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a < 3) AND (c < 3)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a < 3) AND (c < 3)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND b > '2' AND c < 3; + QUERY PLAN +--------------------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a < 3) AND (b > '2'::text) AND (c < 3)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a < 3) AND (b > '2'::text) AND (c < 3)) +(4 rows) + +-- almost 5000 unique combinations with NULL values +TRUNCATE histograms; +DROP STATISTICS histograms_stats; +INSERT INTO histograms (a, b, c, filler1) + SELECT + (CASE WHEN mod(i,100) = 0 THEN NULL ELSE mod(i,100) END), + (CASE WHEN mod(i,100) <= 1 THEN NULL ELSE mod(i,100) + mod(i,7) END), + (CASE WHEN mod(i,100) <= 2 THEN NULL ELSE mod(i,100) + mod(i,11) END), + i + FROM generate_series(1,5000) s(i); +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Index Scan using histograms_abc_idx on histograms + Index Cond: ((a IS NULL) AND (b IS NULL)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL AND c IS NULL; + QUERY PLAN +------------------------------------------------------------- + Index Scan using histograms_abc_idx on histograms + Index Cond: ((a IS NULL) AND (b IS NULL) AND (c IS NULL)) +(2 rows) + +-- create statistics +CREATE STATISTICS histograms_stats (histogram) ON a, b, c FROM histograms; +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a IS NULL) AND (b IS NULL)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a IS NULL) AND (b IS NULL)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL AND c IS NULL; + QUERY PLAN +------------------------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a IS NULL) AND (b IS NULL) AND (c IS NULL)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a IS NULL) AND (b IS NULL) AND (c IS NULL)) +(4 rows) + +-- check change of column type resets the histogram statistics +ALTER TABLE histograms ALTER COLUMN c TYPE numeric; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Index Scan using histograms_abc_idx on histograms + Index Cond: ((a IS NULL) AND (b IS NULL)) +(2 rows) + +ANALYZE histograms; +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on histograms + Recheck Cond: ((a IS NULL) AND (b IS NULL)) + -> Bitmap Index Scan on histograms_abc_idx + Index Cond: ((a IS NULL) AND (b IS NULL)) +(4 rows) + +-- histograms with arrays +CREATE TABLE histograms_arrays ( + a TEXT[], + b NUMERIC[], + c INT[] +); +INSERT INTO histograms_arrays (a, b, c) + SELECT + ARRAY[md5(i::text), md5((i-1)::text), md5((i+1)::text)], + ARRAY[(i-1)::numeric/1000, i::numeric/1000, (i+1)::numeric/1000], + ARRAY[(i-1), i, (i+1)] + FROM generate_series(1,5000) s(i); +CREATE STATISTICS histogram_array_stats (histogram) ON a, b, c + FROM histograms_arrays; +ANALYZE histograms_arrays; +RESET random_page_cost; diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out index a56d6c5231..97c292f6f9 100644 --- a/src/test/regress/expected/type_sanity.out +++ b/src/test/regress/expected/type_sanity.out @@ -73,8 +73,9 @@ WHERE p1.typtype not in ('c','d','p') AND p1.typname NOT LIKE E'\\_%' 3361 | pg_ndistinct 3402 | pg_dependencies 4001 | pg_mcv_list + 3425 | pg_histogram 210 | smgr -(5 rows) +(6 rows) -- Make sure typarray points to a varlena array type of our own base SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype, diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index ad1f103217..a949c7e6d1 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -414,8 +414,6 @@ EXPLAIN (COSTS OFF) EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL; -RESET random_page_cost; - -- mcv with arrays CREATE TABLE mcv_lists_arrays ( a TEXT[], @@ -463,3 +461,134 @@ EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c; EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c; EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c; + +RESET random_page_cost; + +-- histograms +CREATE TABLE histograms ( + filler1 TEXT, + filler2 NUMERIC, + a INT, + b TEXT, + filler3 DATE, + c INT, + d TEXT +); + +SET random_page_cost = 1.2; + +CREATE INDEX histograms_ab_idx ON mcv_lists (a, b); +CREATE INDEX histograms_abc_idx ON histograms (a, b, c); + +-- random data (we still get histogram, but as the columns are not +-- correlated, the estimates remain about the same) +INSERT INTO histograms (a, b, c, filler1) + SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i); + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5'; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5' AND c < 5; + +-- create statistics +CREATE STATISTICS histograms_stats (histogram) ON a, b, c FROM histograms; + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5'; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 5 AND b < '5' AND c < 5; + +-- values correlated along the diagonal +TRUNCATE histograms; +DROP STATISTICS histograms_stats; + +INSERT INTO histograms (a, b, c, filler1) + SELECT mod(i,100), mod(i,100) + mod(i,7), mod(i,100) + mod(i,11), i FROM generate_series(1,5000) s(i); + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND c < 3; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND b > '2' AND c < 3; + +-- create statistics +CREATE STATISTICS histograms_stats (histogram) ON a, b, c FROM histograms; + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND c < 3; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a < 3 AND b > '2' AND c < 3; + +-- almost 5000 unique combinations with NULL values +TRUNCATE histograms; +DROP STATISTICS histograms_stats; + +INSERT INTO histograms (a, b, c, filler1) + SELECT + (CASE WHEN mod(i,100) = 0 THEN NULL ELSE mod(i,100) END), + (CASE WHEN mod(i,100) <= 1 THEN NULL ELSE mod(i,100) + mod(i,7) END), + (CASE WHEN mod(i,100) <= 2 THEN NULL ELSE mod(i,100) + mod(i,11) END), + i + FROM generate_series(1,5000) s(i); + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL AND c IS NULL; + +-- create statistics +CREATE STATISTICS histograms_stats (histogram) ON a, b, c FROM histograms; + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL AND c IS NULL; + +-- check change of column type resets the histogram statistics +ALTER TABLE histograms ALTER COLUMN c TYPE numeric; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + +ANALYZE histograms; + +EXPLAIN (COSTS OFF) + SELECT * FROM histograms WHERE a IS NULL AND b IS NULL; + +-- histograms with arrays +CREATE TABLE histograms_arrays ( + a TEXT[], + b NUMERIC[], + c INT[] +); + +INSERT INTO histograms_arrays (a, b, c) + SELECT + ARRAY[md5(i::text), md5((i-1)::text), md5((i+1)::text)], + ARRAY[(i-1)::numeric/1000, i::numeric/1000, (i+1)::numeric/1000], + ARRAY[(i-1), i, (i+1)] + FROM generate_series(1,5000) s(i); + +CREATE STATISTICS histogram_array_stats (histogram) ON a, b, c + FROM histograms_arrays; + +ANALYZE histograms_arrays; + +RESET random_page_cost; -- 2.13.6 --------------0E1BE69E814B19BC8B911192 Content-Type: text/x-patch; name="0001-multivariate-MCV-lists-20180902.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-multivariate-MCV-lists-20180902.patch" From c25a35ea0cbc770b0ff42cef4c86bbfc63667dda Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Sat, 1 Sep 2018 17:48:35 +0200 Subject: [PATCH 1/2] multivariate MCV lists --- doc/src/sgml/catalogs.sgml | 13 +- doc/src/sgml/func.sgml | 77 + doc/src/sgml/planstats.sgml | 114 +- doc/src/sgml/ref/create_statistics.sgml | 28 +- src/backend/commands/analyze.c | 8 +- src/backend/commands/statscmds.c | 107 +- src/backend/optimizer/path/clausesel.c | 89 +- src/backend/optimizer/util/plancat.c | 12 + src/backend/parser/parse_utilcmd.c | 2 + src/backend/statistics/Makefile | 2 +- src/backend/statistics/README | 4 + src/backend/statistics/README.mcv | 140 ++ src/backend/statistics/dependencies.c | 83 +- src/backend/statistics/extended_stats.c | 543 ++++++- src/backend/statistics/mcv.c | 1656 ++++++++++++++++++++++ src/backend/statistics/mvdistinct.c | 28 - src/backend/utils/adt/ruleutils.c | 24 +- src/backend/utils/adt/selfuncs.c | 165 +++ src/bin/psql/describe.c | 9 +- src/include/catalog/pg_cast.dat | 6 + src/include/catalog/pg_proc.dat | 24 + src/include/catalog/pg_statistic_ext.h | 2 + src/include/catalog/pg_type.dat | 7 + src/include/commands/vacuum.h | 6 + src/include/optimizer/cost.h | 6 + src/include/statistics/extended_stats_internal.h | 43 + src/include/statistics/statistics.h | 49 + src/include/utils/selfuncs.h | 2 + src/test/regress/expected/create_table_like.out | 2 +- src/test/regress/expected/opr_sanity.out | 3 +- src/test/regress/expected/stats_ext.out | 319 ++++- src/test/regress/expected/type_sanity.out | 3 +- src/test/regress/sql/stats_ext.sql | 181 +++ 33 files changed, 3630 insertions(+), 127 deletions(-) create mode 100644 src/backend/statistics/README.mcv create mode 100644 src/backend/statistics/mcv.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 07e8b3325f..dc7bbe5173 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -6571,7 +6571,8 @@ SCRAM-SHA-256$<iteration count>:&l An array containing codes for the enabled statistic kinds; valid values are: d for n-distinct statistics, - f for functional dependency statistics + f for functional dependency statistics, and + m for most common values (MCV) list statistics @@ -6594,6 +6595,16 @@ SCRAM-SHA-256$<iteration count>:&l + + stxmcv + pg_mcv_list + + + MCV (most-common values) list statistics, serialized as + pg_mcv_list type. + + + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index bb794e044f..69cfe7bbe9 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -20917,4 +20917,81 @@ CREATE EVENT TRIGGER test_table_rewrite_oid + + Statistics Information Functions + + + function + statistics + + + + To inspect statistics defined using CREATE STATISTICS + command, PostgreSQL provides multiple functions. + + + + Inspecting MCV lists + + + pg_mcv_list_items + pg_mcv_list + + + + pg_mcv_list_items returns a list of all items + stored in a multi-column MCV list, and returns the + following columns: + + + + + + Name + Type + Description + + + + + + index + int + index of the item in the MCV list + + + values + text[] + values stored in the MCV item + + + nulls + boolean[] + flags identifying NULL values + + + frequency + double precision + frequency of this MCV item + + + + + + + + The pg_mcv_list_items function can be used like this: + + +SELECT m.* FROM pg_statistic_ext, + pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts'; + + + Values of the pg_mcv_list can be obtained only from the + pg_statistic.stxmcv column. + + + + + diff --git a/doc/src/sgml/planstats.sgml b/doc/src/sgml/planstats.sgml index ef643ad064..de8ef165c9 100644 --- a/doc/src/sgml/planstats.sgml +++ b/doc/src/sgml/planstats.sgml @@ -455,7 +455,7 @@ rows = (outer_cardinality * inner_cardinality) * selectivity multivariate - + Functional Dependencies @@ -540,7 +540,7 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1; - + Multivariate N-Distinct Counts @@ -585,6 +585,116 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a, b; + + + MCV lists + + + As explained in , functional + dependencies are very cheap and efficient type of statistics, but their + main limitation is their global nature (only tracking dependencies at + the column level, not between individual column values). + + + + This section introduces multivariate variant of MCV + (most-common values) lists, a straight-forward extension of the per-column + statistics described in . This + statistics adresses the limitation by storing individual values, but it + is naturally more expensive, both in terms of storage and planning time. + + + + Let's look at the query from + again, but this time with a MCV list created on the + same set of columns (be sure to drop the functional dependencies, to + make sure the planner uses the newly created statistics). + + +DROP STATISTICS stts; +CREATE STATISTICS stts2 (mcv) ON a, b FROM t; +ANALYZE t; +EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1; + QUERY PLAN +------------------------------------------------------------------------------- + Seq Scan on t (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1) + Filter: ((a = 1) AND (b = 1)) + Rows Removed by Filter: 9900 + + + The estimate is as accurate as with the functional dependencies, mostly + thanks to the table being fairly small and having a simple distribution + with low number of distinct values. Before looking at the second query, + which was not handled by functional dependencies particularly well, + let's inspect the MCV list a bit. + + + + Inspecting the MCV list is possible using + pg_mcv_list_items set-returning function. + + +SELECT m.* FROM pg_statistic_ext, + pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts2'; + index | values | nulls | frequency +-------+---------+-------+----------- + 0 | {0,0} | {f,f} | 0.01 + 1 | {1,1} | {f,f} | 0.01 + 2 | {2,2} | {f,f} | 0.01 +... + 49 | {49,49} | {f,f} | 0.01 + 50 | {50,0} | {f,f} | 0.01 +... + 97 | {97,47} | {f,f} | 0.01 + 98 | {98,48} | {f,f} | 0.01 + 99 | {99,49} | {f,f} | 0.01 +(100 rows) + + + Which confirms there are 100 distinct combinations in the two columns, + and all of them are about equally likely (1% frequency for each one). + Had there been any null values in either of the columns, this would be + identified in the nulls column. + + + + When estimating the selectivity, the planner applies all the conditions + on items in the MCV list, and them sums the frequencies + of the matching ones. See mcv_clauselist_selectivity + in src/backend/statistics/mcv.c for details. + + + + Compared to functional dependencies, MCV lists have two + major advantages. Firstly, the list stores actual values, making it possible + to decide which combinations are compatible. + + +EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10; + QUERY PLAN +--------------------------------------------------------------------------- + Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1) + Filter: ((a = 1) AND (b = 10)) + Rows Removed by Filter: 10000 + + + Secondly, MCV lists handle a wider range of clause types, + not just equality clauses like functional dependencies. See for example the + example range query, presented earlier: + + +EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a <= 49 AND b > 49; + QUERY PLAN +--------------------------------------------------------------------------- + Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1) + Filter: ((a <= 49) AND (b > 49)) + Rows Removed by Filter: 10000 + + + + + + diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml index 539f5bded5..fcbfa569d0 100644 --- a/doc/src/sgml/ref/create_statistics.sgml +++ b/doc/src/sgml/ref/create_statistics.sgml @@ -83,7 +83,8 @@ CREATE STATISTICS [ IF NOT EXISTS ] statistics_na Currently supported kinds are ndistinct, which enables n-distinct statistics, and dependencies, which enables functional - dependency statistics. + dependency statistics, and mcv which enables + most-common values lists. If this clause is omitted, all supported statistics kinds are included in the statistics object. For more information, see @@ -164,6 +165,31 @@ EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 0); conditions are redundant and does not underestimate the row count. + + Create table t2 with two perfectly correlated columns + (containing identical data), and a MCV list on those columns: + + +CREATE TABLE t2 ( + a int, + b int +); + +INSERT INTO t2 SELECT mod(i,100), mod(i,100) + FROM generate_series(1,1000000) s(i); + +CREATE STATISTICS s2 WITH (mcv) ON (a, b) FROM t2; + +ANALYZE t2; + +-- valid combination (found in MCV) +EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1); + +-- invalid combination (not found in MCV) +EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2); + + + diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c index edbdce81f2..bc369df4cd 100644 --- a/src/backend/commands/analyze.c +++ b/src/backend/commands/analyze.c @@ -1759,12 +1759,6 @@ static void compute_scalar_stats(VacAttrStatsP stats, double totalrows); static int compare_scalars(const void *a, const void *b, void *arg); static int compare_mcvs(const void *a, const void *b); -static int analyze_mcv_list(int *mcv_counts, - int num_mcv, - double stadistinct, - double stanullfrac, - int samplerows, - double totalrows); /* @@ -2859,7 +2853,7 @@ compare_mcvs(const void *a, const void *b) * number that are significantly more common than the values not in the list, * and which are therefore deemed worth storing in the table's MCV list. */ -static int +int analyze_mcv_list(int *mcv_counts, int num_mcv, double stadistinct, diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c index 3bb0d24cd2..903d8155e0 100644 --- a/src/backend/commands/statscmds.c +++ b/src/backend/commands/statscmds.c @@ -70,11 +70,12 @@ CreateStatistics(CreateStatsStmt *stmt) Oid relid; ObjectAddress parentobject, myself; - Datum types[2]; /* one for each possible type of statistic */ + Datum types[3]; /* one for each possible type of statistic */ int ntypes; ArrayType *stxkind; bool build_ndistinct; bool build_dependencies; + bool build_mcv; bool requested_type = false; int i; ListCell *cell; @@ -269,6 +270,7 @@ CreateStatistics(CreateStatsStmt *stmt) */ build_ndistinct = false; build_dependencies = false; + build_mcv = false; foreach(cell, stmt->stat_types) { char *type = strVal((Value *) lfirst(cell)); @@ -283,6 +285,11 @@ CreateStatistics(CreateStatsStmt *stmt) build_dependencies = true; requested_type = true; } + else if (strcmp(type, "mcv") == 0) + { + build_mcv = true; + requested_type = true; + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -294,6 +301,7 @@ CreateStatistics(CreateStatsStmt *stmt) { build_ndistinct = true; build_dependencies = true; + build_mcv = true; } /* construct the char array of enabled statistic types */ @@ -302,6 +310,8 @@ CreateStatistics(CreateStatsStmt *stmt) types[ntypes++] = CharGetDatum(STATS_EXT_NDISTINCT); if (build_dependencies) types[ntypes++] = CharGetDatum(STATS_EXT_DEPENDENCIES); + if (build_mcv) + types[ntypes++] = CharGetDatum(STATS_EXT_MCV); Assert(ntypes > 0 && ntypes <= lengthof(types)); stxkind = construct_array(types, ntypes, CHAROID, 1, true, 'c'); @@ -320,6 +330,7 @@ CreateStatistics(CreateStatsStmt *stmt) /* no statistics built yet */ nulls[Anum_pg_statistic_ext_stxndistinct - 1] = true; nulls[Anum_pg_statistic_ext_stxdependencies - 1] = true; + nulls[Anum_pg_statistic_ext_stxmcv - 1] = true; /* insert it into pg_statistic_ext */ statrel = heap_open(StatisticExtRelationId, RowExclusiveLock); @@ -415,23 +426,97 @@ RemoveStatisticsById(Oid statsOid) * null until the next ANALYZE. (Note that the type change hasn't actually * happened yet, so one option that's *not* on the table is to recompute * immediately.) + * + * For both ndistinct and functional-dependencies stats, the on-disk + * representation is independent of the source column data types, and it is + * plausible to assume that the old statistic values will still be good for + * the new column contents. (Obviously, if the ALTER COLUMN TYPE has a USING + * expression that substantially alters the semantic meaning of the column + * values, this assumption could fail. But that seems like a corner case + * that doesn't justify zapping the stats in common cases.) + * + * For MCV lists that's not the case, as those statistics store the datums + * internally. In this case we simply reset the statistics value to NULL. */ void UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, Oid oldColumnType, Oid newColumnType) { + Form_pg_statistic_ext staForm; + HeapTuple stup, + oldtup; + int i; + + /* Do we need to reset anything? */ + bool attribute_referenced; + bool reset_stats = false; + + Relation rel; + + Datum values[Natts_pg_statistic_ext]; + bool nulls[Natts_pg_statistic_ext]; + bool replaces[Natts_pg_statistic_ext]; + + oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statsOid)); + if (!oldtup) + elog(ERROR, "cache lookup failed for statistics object %u", statsOid); + staForm = (Form_pg_statistic_ext) GETSTRUCT(oldtup); + + /* + * If the modified attribute is not referenced by this statistic, we + * can simply leave the statistics alone. + */ + attribute_referenced = false; + for (i = 0; i < staForm->stxkeys.dim1; i++) + if (attnum == staForm->stxkeys.values[i]) + attribute_referenced = true; + /* - * Currently, we don't actually need to do anything here. For both - * ndistinct and functional-dependencies stats, the on-disk representation - * is independent of the source column data types, and it is plausible to - * assume that the old statistic values will still be good for the new - * column contents. (Obviously, if the ALTER COLUMN TYPE has a USING - * expression that substantially alters the semantic meaning of the column - * values, this assumption could fail. But that seems like a corner case - * that doesn't justify zapping the stats in common cases.) - * - * Future types of extended stats will likely require us to work harder. + * We can also leave the record as it is if there are no statistics + * including the datum values, like for example MCV lists. */ + if (statext_is_kind_built(oldtup, STATS_EXT_MCV)) + reset_stats = true; + + /* + * If we can leave the statistics as it is, just do minimal cleanup + * and we're done. + */ + if (!attribute_referenced && reset_stats) + { + ReleaseSysCache(oldtup); + return; + } + + /* + * OK, we need to reset some statistics. So let's build the new tuple, + * replacing the affected statistics types with NULL. + */ + memset(nulls, 0, Natts_pg_statistic_ext * sizeof(bool)); + memset(replaces, 0, Natts_pg_statistic_ext * sizeof(bool)); + memset(values, 0, Natts_pg_statistic_ext * sizeof(Datum)); + + if (statext_is_kind_built(oldtup, STATS_EXT_MCV)) + { + replaces[Anum_pg_statistic_ext_stxmcv - 1] = true; + nulls[Anum_pg_statistic_ext_stxmcv - 1] = true; + } + + rel = heap_open(StatisticExtRelationId, RowExclusiveLock); + + /* replace the old tuple */ + stup = heap_modify_tuple(oldtup, + RelationGetDescr(rel), + values, + nulls, + replaces); + + ReleaseSysCache(oldtup); + CatalogTupleUpdate(rel, &stup->t_self, stup); + + heap_freetuple(stup); + + heap_close(rel, RowExclusiveLock); } /* diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index f4717942c3..fd09ad3c49 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -105,9 +105,6 @@ clauselist_selectivity(PlannerInfo *root, Selectivity s1 = 1.0; RelOptInfo *rel; Bitmapset *estimatedclauses = NULL; - RangeQueryClause *rqlist = NULL; - ListCell *l; - int listidx; /* * If there's exactly one clause, just go directly to @@ -125,6 +122,25 @@ clauselist_selectivity(PlannerInfo *root, if (rel && rel->rtekind == RTE_RELATION && rel->statlist != NIL) { /* + * Estimate selectivity on any clauses applicable by stats tracking + * actual values first, then apply functional dependencies on the + * remaining clauses. The reasoning for this particular order is that + * the more complex stats can track more complex correlations between + * the attributes, and may be considered more reliable. + * + * For example MCV list can give us an exact selectivity for values in + * two columns, while functional dependencies can only provide + * information about overall strength of the dependency. + * + * 'estimatedclauses' is a bitmap of 0-based list positions of clauses + * used that way, so that we can ignore them later (not to estimate + * them twice). + */ + s1 *= statext_clauselist_selectivity(root, clauses, varRelid, + jointype, sjinfo, rel, + &estimatedclauses); + + /* * Perform selectivity estimations on any clauses found applicable by * dependencies_clauselist_selectivity. 'estimatedclauses' will be * filled with the 0-based list positions of clauses used that way, so @@ -133,17 +149,72 @@ clauselist_selectivity(PlannerInfo *root, s1 *= dependencies_clauselist_selectivity(root, clauses, varRelid, jointype, sjinfo, rel, &estimatedclauses); - - /* - * This would be the place to apply any other types of extended - * statistics selectivity estimations for remaining clauses. - */ } /* * Apply normal selectivity estimates for remaining clauses. We'll be * careful to skip any clauses which were already estimated above. - * + */ + return s1 * clauselist_selectivity_simple(root, clauses, varRelid, + jointype, sjinfo, + estimatedclauses); +} + +/* + * clauselist_selectivity_simple - + * Compute the selectivity of an implicitly-ANDed list of boolean + * expression clauses. The list can be empty, in which case 1.0 + * must be returned. List elements may be either RestrictInfos + * or bare expression clauses --- the former is preferred since + * it allows caching of results. + * + * See clause_selectivity() for the meaning of the additional parameters. + * + * Our basic approach is to take the product of the selectivities of the + * subclauses. However, that's only right if the subclauses have independent + * probabilities, and in reality they are often NOT independent. So, + * we want to be smarter where we can. + * + * We also recognize "range queries", such as "x > 34 AND x < 42". Clauses + * are recognized as possible range query components if they are restriction + * opclauses whose operators have scalarltsel or a related function as their + * restriction selectivity estimator. We pair up clauses of this form that + * refer to the same variable. An unpairable clause of this kind is simply + * multiplied into the selectivity product in the normal way. But when we + * find a pair, we know that the selectivities represent the relative + * positions of the low and high bounds within the column's range, so instead + * of figuring the selectivity as hisel * losel, we can figure it as hisel + + * losel - 1. (To visualize this, see that hisel is the fraction of the range + * below the high bound, while losel is the fraction above the low bound; so + * hisel can be interpreted directly as a 0..1 value but we need to convert + * losel to 1-losel before interpreting it as a value. Then the available + * range is 1-losel to hisel. However, this calculation double-excludes + * nulls, so really we need hisel + losel + null_frac - 1.) + * + * If either selectivity is exactly DEFAULT_INEQ_SEL, we forget this equation + * and instead use DEFAULT_RANGE_INEQ_SEL. The same applies if the equation + * yields an impossible (negative) result. + * + * A free side-effect is that we can recognize redundant inequalities such + * as "x < 4 AND x < 5"; only the tighter constraint will be counted. + * + * Of course this is all very dependent on the behavior of the inequality + * selectivity functions; perhaps some day we can generalize the approach. + */ +Selectivity +clauselist_selectivity_simple(PlannerInfo *root, + List *clauses, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + Bitmapset *estimatedclauses) +{ + Selectivity s1 = 1.0; + RangeQueryClause *rqlist = NULL; + ListCell *l; + int listidx; + + /* * Anything that doesn't look like a potential rangequery clause gets * multiplied into s1 and forgotten. Anything that does gets inserted into * an rqlist entry. diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 8369e3ad62..0112450419 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1363,6 +1363,18 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) stainfos = lcons(info, stainfos); } + if (statext_is_kind_built(htup, STATS_EXT_MCV)) + { + StatisticExtInfo *info = makeNode(StatisticExtInfo); + + info->statOid = statOid; + info->rel = rel; + info->kind = STATS_EXT_MCV; + info->keys = bms_copy(keys); + + stainfos = lcons(info, stainfos); + } + ReleaseSysCache(htup); bms_free(keys); } diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 656b1b5f1b..29877126d7 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1681,6 +1681,8 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, stat_types = lappend(stat_types, makeString("ndistinct")); else if (enabled[i] == STATS_EXT_DEPENDENCIES) stat_types = lappend(stat_types, makeString("dependencies")); + else if (enabled[i] == STATS_EXT_MCV) + stat_types = lappend(stat_types, makeString("mcv")); else elog(ERROR, "unrecognized statistics kind %c", enabled[i]); } diff --git a/src/backend/statistics/Makefile b/src/backend/statistics/Makefile index 3404e4554a..d2815265fb 100644 --- a/src/backend/statistics/Makefile +++ b/src/backend/statistics/Makefile @@ -12,6 +12,6 @@ subdir = src/backend/statistics top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -OBJS = extended_stats.o dependencies.o mvdistinct.o +OBJS = extended_stats.o dependencies.o mcv.o mvdistinct.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/statistics/README b/src/backend/statistics/README index a8f00a590e..8f153a9e85 100644 --- a/src/backend/statistics/README +++ b/src/backend/statistics/README @@ -18,6 +18,8 @@ There are currently two kinds of extended statistics: (b) soft functional dependencies (README.dependencies) + (c) MCV lists (README.mcv) + Compatible clause types ----------------------- @@ -26,6 +28,8 @@ Each type of statistics may be used to estimate some subset of clause types. (a) functional dependencies - equality clauses (AND), possibly IS NULL + (b) MCV lists - equality and inequality clauses (AND, OR, NOT), IS NULL + Currently, only OpExprs in the form Var op Const, or Const op Var are supported, however it's feasible to expand the code later to also estimate the selectivities on clauses such as Var op Var. diff --git a/src/backend/statistics/README.mcv b/src/backend/statistics/README.mcv new file mode 100644 index 0000000000..2910eca962 --- /dev/null +++ b/src/backend/statistics/README.mcv @@ -0,0 +1,140 @@ +MCV lists +========= + +Multivariate MCV (most-common values) lists are a straightforward extension of +regular MCV list, tracking most frequent combinations of values for a group of +attributes. + +This works particularly well for columns with a small number of distinct values, +as the list may include all the combinations and approximate the distribution +very accurately. + +For columns with large number of distinct values (e.g. those with continuous +domains), the list will only track the most frequent combinations. If the +distribution is mostly uniform (all combinations about equally frequent), the +MCV list will be empty. + +Estimates of some clauses (e.g. equality) based on MCV lists are more accurate +than when using histograms. + +Also, MCV lists don't necessarily require sorting of the values (the fact that +we use sorting when building them is implementation detail), but even more +importantly the ordering is not built into the approximation (while histograms +are built on ordering). So MCV lists work well even for attributes where the +ordering of the data type is disconnected from the meaning of the data. For +example we know how to sort strings, but it's unlikely to make much sense for +city names (or other label-like attributes). + + +Selectivity estimation +---------------------- + +The estimation, implemented in clauselist_mv_selectivity_mcvlist(), is quite +simple in principle - we need to identify MCV items matching all the clauses +and sum frequencies of all those items. + +Currently MCV lists support estimation of the following clause types: + + (a) equality clauses WHERE (a = 1) AND (b = 2) + (b) inequality clauses WHERE (a < 1) AND (b >= 2) + (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL) + (d) OR clauses WHERE (a < 1) OR (b >= 2) + +It's possible to add support for additional clauses, for example: + + (e) multi-var clauses WHERE (a > b) + +and possibly others. These are tasks for the future, not yet implemented. + + +Estimating equality clauses +--------------------------- + +When computing selectivity estimate for equality clauses + + (a = 1) AND (b = 2) + +we can do this estimate pretty exactly assuming that two conditions are met: + + (1) there's an equality condition on all attributes of the statistic + + (2) we find a matching item in the MCV list + +In this case we know the MCV item represents all tuples matching the clauses, +and the selectivity estimate is complete (i.e. we don't need to perform +estimation using the histogram). This is what we call 'full match'. + +When only (1) holds, but there's no matching MCV item, we don't know whether +there are no such rows or just are not very frequent. We can however use the +frequency of the least frequent MCV item as an upper bound for the selectivity. + +For a combination of equality conditions (not full-match case) we can clamp the +selectivity by the minimum of selectivities for each condition. For example if +we know the number of distinct values for each column, we can use 1/ndistinct +as a per-column estimate. Or rather 1/ndistinct + selectivity derived from the +MCV list. + +We should also probably only use the 'residual ndistinct' by exluding the items +included in the MCV list (and also residual frequency): + + f = (1.0 - sum(MCV frequencies)) / (ndistinct - ndistinct(MCV list)) + +but it's worth pointing out the ndistinct values are multi-variate for the +columns referenced by the equality conditions. + +Note: Only the "full match" limit is currently implemented. + + +Hashed MCV (not yet implemented) +-------------------------------- + +Regular MCV lists have to include actual values for each item, so if those items +are large the list may be quite large. This is especially true for multi-variate +MCV lists, although the current implementation partially mitigates this by +performing de-duplicating the values before storing them on disk. + +It's possible to only store hashes (32-bit values) instead of the actual values, +significantly reducing the space requirements. Obviously, this would only make +the MCV lists useful for estimating equality conditions (assuming the 32-bit +hashes make the collisions rare enough). + +This might also complicate matching the columns to available stats. + + +TODO Consider implementing hashed MCV list, storing just 32-bit hashes instead + of the actual values. This type of MCV list will be useful only for + estimating equality clauses, and will reduce space requirements for large + varlena types (in such cases we usually only want equality anyway). + +TODO Currently there's no logic to consider building only a MCV list (and not + building the histogram at all), except for doing this decision manually in + ADD STATISTICS. + + +Inspecting the MCV list +----------------------- + +Inspecting the regular (per-attribute) MCV lists is trivial, as it's enough +to select the columns from pg_stats. The data is encoded as anyarrays, and +all the items have the same data type, so anyarray provides a simple way to +get a text representation. + +With multivariate MCV lists the columns may use different data types, making +it impossible to use anyarrays. It might be possible to produce similar +array-like representation, but that would complicate further processing and +analysis of the MCV list. + +So instead the MCV lists are stored in a custom data type (pg_mcv_list), +which however makes it more difficult to inspect the contents. To make that +easier, there's a SRF returning detailed information about the MCV lists. + + SELECT * FROM pg_mcv_list_items(stxmcv); + +It accepts one parameter - a pg_mcv_list value (which can only be obtained +from pg_statistic_ext catalog, to defend against malicious input), and +returns these columns: + + - item index (0, ..., (nitems-1)) + - values (string array) + - nulls only (boolean array) + - frequency (double precision) diff --git a/src/backend/statistics/dependencies.c b/src/backend/statistics/dependencies.c index 140783cfb3..29e816c4f7 100644 --- a/src/backend/statistics/dependencies.c +++ b/src/backend/statistics/dependencies.c @@ -201,14 +201,11 @@ static double dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, VacAttrStats **stats, Bitmapset *attrs) { - int i, - j; - int nvalues = numrows * k; + int i; MultiSortSupport mss; SortItem *items; - Datum *values; - bool *isnull; int *attnums; + int *attnums_dep; /* counters valid within a group */ int group_size = 0; @@ -223,26 +220,16 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, /* sort info for all attributes columns */ mss = multi_sort_init(k); - /* data for the sort */ - items = (SortItem *) palloc(numrows * sizeof(SortItem)); - values = (Datum *) palloc(sizeof(Datum) * nvalues); - isnull = (bool *) palloc(sizeof(bool) * nvalues); - - /* fix the pointers to values/isnull */ - for (i = 0; i < numrows; i++) - { - items[i].values = &values[i * k]; - items[i].isnull = &isnull[i * k]; - } - /* - * Transform the bms into an array, to make accessing i-th member easier. + * Transform the bms into an array, to make accessing i-th member easier, + * and then construct a filtered version with only attnums referenced + * by the dependency we validate. */ - attnums = (int *) palloc(sizeof(int) * bms_num_members(attrs)); - i = 0; - j = -1; - while ((j = bms_next_member(attrs, j)) >= 0) - attnums[i++] = j; + attnums = build_attnums(attrs); + + attnums_dep = (int *)palloc(k * sizeof(int)); + for (i = 0; i < k; i++) + attnums_dep[i] = attnums[dependency[i]]; /* * Verify the dependency (a,b,...)->z, using a rather simple algorithm: @@ -254,7 +241,7 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, * (c) for each group count different values in the last column */ - /* prepare the sort function for the first dimension, and SortItem array */ + /* prepare the sort function for the dimensions */ for (i = 0; i < k; i++) { VacAttrStats *colstat = stats[dependency[i]]; @@ -267,19 +254,16 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, /* prepare the sort function for this dimension */ multi_sort_add_dimension(mss, i, type->lt_opr); - - /* accumulate all the data for both columns into an array and sort it */ - for (j = 0; j < numrows; j++) - { - items[j].values[i] = - heap_getattr(rows[j], attnums[dependency[i]], - stats[i]->tupDesc, &items[j].isnull[i]); - } } - /* sort the items so that we can detect the groups */ - qsort_arg((void *) items, numrows, sizeof(SortItem), - multi_sort_compare, mss); + /* + * build an array of SortItem(s) sorted using the multi-sort support + * + * XXX This relies on all stats entries pointing to the same tuple + * descriptor. Not sure if that might not be the case. + */ + items = build_sorted_items(numrows, rows, stats[0]->tupDesc, + mss, k, attnums_dep); /* * Walk through the sorted array, split it into rows according to the @@ -322,9 +306,9 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, } pfree(items); - pfree(values); - pfree(isnull); pfree(mss); + pfree(attnums); + pfree(attnums_dep); /* Compute the 'degree of validity' as (supporting/total). */ return (n_supporting_rows * 1.0 / numrows); @@ -351,7 +335,6 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, VacAttrStats **stats) { int i, - j, k; int numattrs; int *attnums; @@ -364,11 +347,7 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, /* * Transform the bms into an array, to make accessing i-th member easier. */ - attnums = palloc(sizeof(int) * bms_num_members(attrs)); - i = 0; - j = -1; - while ((j = bms_next_member(attrs, j)) >= 0) - attnums[i++] = j; + attnums = build_attnums(attrs); Assert(numattrs >= 2); @@ -915,9 +894,9 @@ find_strongest_dependency(StatisticExtInfo *stats, MVDependencies *dependencies, * using functional dependency statistics, or 1.0 if no useful functional * dependency statistic exists. * - * 'estimatedclauses' is an output argument that gets a bit set corresponding - * to the (zero-based) list index of each clause that is included in the - * estimated selectivity. + * 'estimatedclauses' is an input/output argument that gets a bit set + * corresponding to the (zero-based) list index of each clause that is included + * in the estimated selectivity. * * Given equality clauses on attributes (a,b) we find the strongest dependency * between them, i.e. either (a=>b) or (b=>a). Assuming (a=>b) is the selected @@ -952,9 +931,6 @@ dependencies_clauselist_selectivity(PlannerInfo *root, AttrNumber *list_attnums; int listidx; - /* initialize output argument */ - *estimatedclauses = NULL; - /* check if there's any stats that might be useful for us. */ if (!has_stats_of_kind(rel->statlist, STATS_EXT_DEPENDENCIES)) return 1.0; @@ -969,6 +945,9 @@ dependencies_clauselist_selectivity(PlannerInfo *root, * the attnums for each clause in a list which we'll reference later so we * don't need to repeat the same work again. We'll also keep track of all * attnums seen. + * + * We also skip clauses that we already estimated using different types of + * statistics (we treat them as incompatible). */ listidx = 0; foreach(l, clauses) @@ -976,7 +955,8 @@ dependencies_clauselist_selectivity(PlannerInfo *root, Node *clause = (Node *) lfirst(l); AttrNumber attnum; - if (dependency_is_compatible_clause(clause, rel->relid, &attnum)) + if ((dependency_is_compatible_clause(clause, rel->relid, &attnum)) && + (!bms_is_member(listidx, *estimatedclauses))) { list_attnums[listidx] = attnum; clauses_attnums = bms_add_member(clauses_attnums, attnum); @@ -1046,8 +1026,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root, /* * Skip incompatible clauses, and ones we've already estimated on. */ - if (list_attnums[listidx] == InvalidAttrNumber || - bms_is_member(listidx, *estimatedclauses)) + if (list_attnums[listidx] == InvalidAttrNumber) continue; /* diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c index 2df5f7dc3a..0b66000705 100644 --- a/src/backend/statistics/extended_stats.c +++ b/src/backend/statistics/extended_stats.c @@ -16,6 +16,8 @@ */ #include "postgres.h" +#include + #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -23,6 +25,8 @@ #include "catalog/pg_collation.h" #include "catalog/pg_statistic_ext.h" #include "nodes/relation.h" +#include "optimizer/clauses.h" +#include "optimizer/cost.h" #include "postmaster/autovacuum.h" #include "statistics/extended_stats_internal.h" #include "statistics/statistics.h" @@ -31,6 +35,7 @@ #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "utils/selfuncs.h" #include "utils/syscache.h" @@ -53,7 +58,7 @@ static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs, int nvacatts, VacAttrStats **vacatts); static void statext_store(Relation pg_stext, Oid relid, MVNDistinct *ndistinct, MVDependencies *dependencies, - VacAttrStats **stats); + MCVList * mcvlist, VacAttrStats **stats); /* @@ -87,6 +92,7 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, StatExtEntry *stat = (StatExtEntry *) lfirst(lc); MVNDistinct *ndistinct = NULL; MVDependencies *dependencies = NULL; + MCVList *mcv = NULL; VacAttrStats **stats; ListCell *lc2; @@ -124,10 +130,13 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, else if (t == STATS_EXT_DEPENDENCIES) dependencies = statext_dependencies_build(numrows, rows, stat->columns, stats); + else if (t == STATS_EXT_MCV) + mcv = statext_mcv_build(numrows, rows, stat->columns, stats, + totalrows); } /* store the statistics in the catalog */ - statext_store(pg_stext, stat->statOid, ndistinct, dependencies, stats); + statext_store(pg_stext, stat->statOid, ndistinct, dependencies, mcv, stats); } heap_close(pg_stext, RowExclusiveLock); @@ -155,6 +164,10 @@ statext_is_kind_built(HeapTuple htup, char type) attnum = Anum_pg_statistic_ext_stxdependencies; break; + case STATS_EXT_MCV: + attnum = Anum_pg_statistic_ext_stxmcv; + break; + default: elog(ERROR, "unexpected statistics type requested: %d", type); } @@ -219,7 +232,8 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) for (i = 0; i < ARR_DIMS(arr)[0]; i++) { Assert((enabled[i] == STATS_EXT_NDISTINCT) || - (enabled[i] == STATS_EXT_DEPENDENCIES)); + (enabled[i] == STATS_EXT_DEPENDENCIES) || + (enabled[i] == STATS_EXT_MCV)); entry->types = lappend_int(entry->types, (int) enabled[i]); } @@ -294,7 +308,7 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs, static void statext_store(Relation pg_stext, Oid statOid, MVNDistinct *ndistinct, MVDependencies *dependencies, - VacAttrStats **stats) + MCVList * mcv, VacAttrStats **stats) { HeapTuple stup, oldtup; @@ -325,9 +339,18 @@ statext_store(Relation pg_stext, Oid statOid, values[Anum_pg_statistic_ext_stxdependencies - 1] = PointerGetDatum(data); } + if (mcv != NULL) + { + bytea *data = statext_mcv_serialize(mcv, stats); + + nulls[Anum_pg_statistic_ext_stxmcv - 1] = (data == NULL); + values[Anum_pg_statistic_ext_stxmcv - 1] = PointerGetDatum(data); + } + /* always replace the value (either by bytea or NULL) */ replaces[Anum_pg_statistic_ext_stxndistinct - 1] = true; replaces[Anum_pg_statistic_ext_stxdependencies - 1] = true; + replaces[Anum_pg_statistic_ext_stxmcv - 1] = true; /* there should already be a pg_statistic_ext tuple */ oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid)); @@ -434,6 +457,137 @@ multi_sort_compare_dims(int start, int end, return 0; } +int +compare_scalars_simple(const void *a, const void *b, void *arg) +{ + return compare_datums_simple(*(Datum *) a, + *(Datum *) b, + (SortSupport) arg); +} + +int +compare_datums_simple(Datum a, Datum b, SortSupport ssup) +{ + return ApplySortComparator(a, false, b, false, ssup); +} + +/* simple counterpart to qsort_arg */ +void * +bsearch_arg(const void *key, const void *base, size_t nmemb, size_t size, + int (*compar) (const void *, const void *, void *), + void *arg) +{ + size_t l, + u, + idx; + const void *p; + int comparison; + + l = 0; + u = nmemb; + while (l < u) + { + idx = (l + u) / 2; + p = (void *) (((const char *) base) + (idx * size)); + comparison = (*compar) (key, p, arg); + + if (comparison < 0) + u = idx; + else if (comparison > 0) + l = idx + 1; + else + return (void *) p; + } + + return NULL; +} + +int * +build_attnums(Bitmapset *attrs) +{ + int i, + j; + int numattrs = bms_num_members(attrs); + int *attnums; + + /* build attnums from the bitmapset */ + attnums = (int *) palloc(sizeof(int) * numattrs); + i = 0; + j = -1; + while ((j = bms_next_member(attrs, j)) >= 0) + attnums[i++] = j; + + return attnums; +} + +/* build_sorted_items + * build sorted array of SortItem with values from rows + * + * XXX All the memory is allocated in a single chunk, so that the caller + * can simply pfree the return value to release all of it. + */ +SortItem * +build_sorted_items(int numrows, HeapTuple *rows, TupleDesc tdesc, + MultiSortSupport mss, int numattrs, int *attnums) +{ + int i, + j, + len; + int nvalues = numrows * numattrs; + + /* + * We won't allocate the arrays for each item independenly, but in one + * large chunk and then just set the pointers. This allows the caller to + * simply pfree the return value to release all the memory. + */ + SortItem *items; + Datum *values; + bool *isnull; + char *ptr; + + /* Compute the total amount of memory we need (both items and values). */ + len = numrows * sizeof(SortItem) + nvalues * (sizeof(Datum) + sizeof(bool)); + + /* Allocate the memory and split it into the pieces. */ + ptr = palloc0(len); + + /* items to sort */ + items = (SortItem *) ptr; + ptr += numrows * sizeof(SortItem); + + /* values and null flags */ + values = (Datum *) ptr; + ptr += nvalues * sizeof(Datum); + + isnull = (bool *) ptr; + ptr += nvalues * sizeof(bool); + + /* make sure we consumed the whole buffer exactly */ + Assert((ptr - (char *) items) == len); + + /* fix the pointers to Datum and bool arrays */ + for (i = 0; i < numrows; i++) + { + items[i].values = &values[i * numattrs]; + items[i].isnull = &isnull[i * numattrs]; + + /* load the values/null flags from sample rows */ + for (j = 0; j < numattrs; j++) + { + items[i].values[j] = heap_getattr(rows[i], + attnums[j], /* attnum */ + tdesc, + &items[i].isnull[j]); /* isnull */ + } + } + + /* do the sort, using the multi-sort */ + qsort_arg((void *) items, numrows, sizeof(SortItem), + multi_sort_compare, mss); + + return items; +} + /* * has_stats_of_kind * Check whether the list contains statistic of a given kind @@ -464,7 +618,7 @@ has_stats_of_kind(List *stats, char requiredkind) * object referencing the most of the requested attributes, breaking ties * in favor of objects with fewer keys overall. * - * XXX if multiple statistics objects tie on both criteria, then which object + * XXX If multiple statistics objects tie on both criteria, then which object * is chosen depends on the order that they appear in the stats list. Perhaps * further tiebreakers are needed. */ @@ -514,3 +668,382 @@ choose_best_statistics(List *stats, Bitmapset *attnums, char requiredkind) return best_match; } + +int +bms_member_index(Bitmapset *keys, AttrNumber varattno) +{ + int i, + j; + + i = -1; + j = 0; + while (((i = bms_next_member(keys, i)) >= 0) && (i < varattno)) + j += 1; + + return j; +} + +/* The Duj1 estimator (already used in analyze.c). */ +double +estimate_ndistinct(double totalrows, int numrows, int d, int f1) +{ + double numer, + denom, + ndistinct; + + numer = (double) numrows * (double) d; + + denom = (double) (numrows - f1) + + (double) f1 * (double) numrows / totalrows; + + ndistinct = numer / denom; + + /* Clamp to sane range in case of roundoff error */ + if (ndistinct < (double) d) + ndistinct = (double) d; + + if (ndistinct > totalrows) + ndistinct = totalrows; + + return floor(ndistinct + 0.5); +} + +/* + * statext_is_compatible_clause_internal + * Does the heavy lifting of actually inspecting the clauses for + * statext_is_compatible_clause. + */ +static bool +statext_is_compatible_clause_internal(Node *clause, Index relid, Bitmapset **attnums) +{ + /* We only support plain Vars for now */ + if (IsA(clause, Var)) + { + Var *var = (Var *) clause; + + /* Ensure var is from the correct relation */ + if (var->varno != relid) + return false; + + /* we also better ensure the Var is from the current level */ + if (var->varlevelsup > 0) + return false; + + /* Also skip system attributes (we don't allow stats on those). */ + if (!AttrNumberIsForUserDefinedAttr(var->varattno)) + return false; + + *attnums = bms_add_member(*attnums, var->varattno); + + return true; + } + + /* Var = Const */ + if (is_opclause(clause)) + { + OpExpr *expr = (OpExpr *) clause; + Var *var; + bool varonleft = true; + bool ok; + + /* Only expressions with two arguments are considered compatible. */ + if (list_length(expr->args) != 2) + return false; + + /* see if it actually has the right */ + ok = (NumRelids((Node *) expr) == 1) && + (is_pseudo_constant_clause(lsecond(expr->args)) || + (varonleft = false, + is_pseudo_constant_clause(linitial(expr->args)))); + + /* unsupported structure (two variables or so) */ + if (!ok) + return false; + + /* + * If it's not one of the supported operators ("=", "<", ">", etc.), + * just ignore the clause, as it's not compatible with MCV lists. + * + * This uses the function for estimating selectivity, not the operator + * directly (a bit awkward, but well ...). + */ + if ((get_oprrest(expr->opno) != F_EQSEL) && + (get_oprrest(expr->opno) != F_NEQSEL) && + (get_oprrest(expr->opno) != F_SCALARLTSEL) && + (get_oprrest(expr->opno) != F_SCALARLESEL) && + (get_oprrest(expr->opno) != F_SCALARGTSEL) && + (get_oprrest(expr->opno) != F_SCALARGESEL)) + return false; + + var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + + return statext_is_compatible_clause_internal((Node *) var, relid, attnums); + } + + /* NOT/AND/OR clause */ + if (or_clause(clause) || + and_clause(clause) || + not_clause(clause)) + { + /* + * AND/OR/NOT-clauses are supported if all sub-clauses are supported + * + * Perhaps we could improve this by handling mixed cases, when some of + * the clauses are supported and some are not. Selectivity for the + * supported subclauses would be computed using extended statistics, + * and the remaining clauses would be estimated using the traditional + * algorithm (product of selectivities). + * + * It however seems overly complex, and in a way we already do that + * because if we reject the whole clause as unsupported here, it will + * be eventually passed to clauselist_selectivity() which does exactly + * this (split into supported/unsupported clauses etc). + */ + BoolExpr *expr = (BoolExpr *) clause; + ListCell *lc; + Bitmapset *clause_attnums = NULL; + + foreach(lc, expr->args) + { + /* + * Had we found incompatible clause in the arguments, treat the + * whole clause as incompatible. + */ + if (!statext_is_compatible_clause_internal((Node *) lfirst(lc), + relid, &clause_attnums)) + return false; + } + + /* + * Otherwise the clause is compatible, and we need to merge the + * attnums into the main bitmapset. + */ + *attnums = bms_join(*attnums, clause_attnums); + + return true; + } + + /* Var IS NULL */ + if (IsA(clause, NullTest)) + { + NullTest *nt = (NullTest *) clause; + + /* + * Only simple (Var IS NULL) expressions supported for now. Maybe we + * could use examine_variable to fix this? + */ + if (!IsA(nt->arg, Var)) + return false; + + return statext_is_compatible_clause_internal((Node *) (nt->arg), relid, attnums); + } + + return false; +} + +/* + * statext_is_compatible_clause + * Determines if the clause is compatible with MCV lists. + * + * Only OpExprs with two arguments using an equality operator are supported. + * When returning True attnum is set to the attribute number of the Var within + * the supported clause. + * + * Currently we only support Var = Const, or Const = Var. It may be possible + * to expand on this later. + */ +static bool +statext_is_compatible_clause(Node *clause, Index relid, Bitmapset **attnums) +{ + RestrictInfo *rinfo = (RestrictInfo *) clause; + + if (!IsA(rinfo, RestrictInfo)) + return false; + + /* Pseudoconstants are not really interesting here. */ + if (rinfo->pseudoconstant) + return false; + + /* clauses referencing multiple varnos are incompatible */ + if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON) + return false; + + return statext_is_compatible_clause_internal((Node *) rinfo->clause, + relid, attnums); +} + +/* + * statext_clauselist_selectivity + * Estimate clauses using the best multi-column statistics. + * + * Selects the best extended (multi-column) statistic on a table (measured by + * a number of attributes extracted from the clauses and covered by it), and + * computes the selectivity for supplied clauses. + * + * One of the main challenges with using MCV lists is how to extrapolate the + * estimate to the data not covered by the MCV list. To do that, we compute + * not only the "MCV selectivity" (selectivities for MCV items matching the + * supplied clauses), but also a couple of derived selectivities: + * + * - simple selectivity: Computed without extended statistic, i.e. as if the + * columns/clauses were independent + * + * - base selectivity: Similar to simple selectivity, but is computed using + * the extended statistic by adding up the base frequencies (that we compute + * and store for each MCV item) of matching MCV items. + * + * - total selectivity: Selectivity covered by the whole MCV list. + * + * - other selectivity: A selectivity estimate for data not covered by the MCV + * list (i.e. satisfying the clauses, but not common enough to make it into + * the MCV list) + * + * Note: While simple and base selectivities are defined in a quite similar + * way, the values are computed differently and are not therefore equal. The + * simple selectivity is computed as a product of per-clause estimates, while + * the base selectivity is computed by adding up base frequencies of matching + * items of the multi-column MCV list. So the values may differ for two main + * reasons - (a) the MCV list may not cover 100% of the data and (b) some of + * the MCV items did not match the estimated clauses. + * + * As both (a) and (b) reduce the base selectivity value, it generally holds + * that (simple_selectivity >= base_selectivity). If the MCV list covers all + * the data, the values may be equal. + * + * So (simple_selectivity - base_selectivity) may be seen as a correction for + * the part not covered by the MCV list. + * + * Note: Due to rounding errors and minor differences in how the estimates + * are computed, the inequality may not always hold. Which is why we clamp + * the selectivities to prevent strange estimate (negative etc.). + * + * XXX If we were to use multiple statistics, this is where it would happen. + * We would simply repeat this on a loop on the "remaining" clauses, possibly + * using the already estimated clauses as conditions (and combining the values + * using conditional probability formula). + */ +Selectivity +statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, + JoinType jointype, SpecialJoinInfo *sjinfo, + RelOptInfo *rel, Bitmapset **estimatedclauses) +{ + ListCell *l; + Bitmapset *clauses_attnums = NULL; + Bitmapset **list_attnums; + int listidx; + StatisticExtInfo *stat; + List *stat_clauses; + Selectivity simple_sel, + mcv_sel, + mcv_basesel, + mcv_totalsel, + other_sel, + sel; + + /* we're interested in MCV lists */ + int types = STATS_EXT_MCV; + + /* check if there's any stats that might be useful for us. */ + if (!has_stats_of_kind(rel->statlist, types)) + return (Selectivity) 1.0; + + list_attnums = (Bitmapset **) palloc(sizeof(Bitmapset *) * + list_length(clauses)); + + /* + * Pre-process the clauses list to extract the attnums seen in each item. + * We need to determine if there's any clauses which will be useful for + * dependency selectivity estimations. Along the way we'll record all of + * the attnums for each clause in a list which we'll reference later so we + * don't need to repeat the same work again. We'll also keep track of all + * attnums seen. + * + * We also skip clauses that we already estimated using different types of + * statistics (we treat them as incompatible). + * + * XXX Currently, the estimated clauses are always empty because the extra + * statistics are applied before functional dependencies. Once we decide + * to apply multiple statistics, this may change. + */ + listidx = 0; + foreach(l, clauses) + { + Node *clause = (Node *) lfirst(l); + Bitmapset *attnums = NULL; + + if ((statext_is_compatible_clause(clause, rel->relid, &attnums)) && + (!bms_is_member(listidx, *estimatedclauses))) + { + list_attnums[listidx] = attnums; + clauses_attnums = bms_add_members(clauses_attnums, attnums); + } + else + list_attnums[listidx] = NULL; + + listidx++; + } + + /* We need at least two attributes for MCV lists. */ + if (bms_num_members(clauses_attnums) < 2) + return 1.0; + + /* find the best suited statistics object for these attnums */ + stat = choose_best_statistics(rel->statlist, clauses_attnums, types); + + /* if no matching stats could be found then we've nothing to do */ + if (!stat) + return (Selectivity) 1.0; + + /* We only understand MCV lists for now. */ + Assert(stat->kind == STATS_EXT_MCV); + + /* now filter the clauses to be estimated using the selected MCV */ + stat_clauses = NIL; + + listidx = 0; + foreach(l, clauses) + { + /* + * If the clause is compatible with the selected statistics, mark it + * as estimated and add it to the list to estimate. + */ + if ((list_attnums[listidx] != NULL) && + (bms_is_subset(list_attnums[listidx], stat->keys))) + { + stat_clauses = lappend(stat_clauses, (Node *) lfirst(l)); + *estimatedclauses = bms_add_member(*estimatedclauses, listidx); + } + + listidx++; + } + + /* + * First compute "simple" selectivity, i.e. without the extended statistics, + * and essentially assuming independence of the columns/clauses. We'll then + * use the various selectivities computed from MCV list to improve it. + */ + simple_sel = clauselist_selectivity_simple(root, stat_clauses, varRelid, + jointype, sjinfo, NULL); + + /* + * Now compute the multi-column estimate from the MCV list, along with the + * other selectivities (base & total selectivity). + */ + mcv_sel = mcv_clauselist_selectivity(root, stat, stat_clauses, varRelid, + jointype, sjinfo, rel, + &mcv_basesel, &mcv_totalsel); + + /* Estimated selectivity of values not covered by MCV matches */ + other_sel = simple_sel - mcv_basesel; + CLAMP_PROBABILITY(other_sel); + + /* The non-MCV selectivity can't exceed the 1 - mcv_totalsel. */ + if (other_sel > 1.0 - mcv_totalsel) + other_sel = 1.0 - mcv_totalsel; + + /* Overall selectivity is the combination of MCV and non-MCV estimates. */ + sel = mcv_sel + other_sel; + CLAMP_PROBABILITY(sel); + + return sel; +} diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c new file mode 100644 index 0000000000..533fbdc037 --- /dev/null +++ b/src/backend/statistics/mcv.c @@ -0,0 +1,1656 @@ +/*------------------------------------------------------------------------- + * + * mcv.c + * POSTGRES multivariate MCV lists + * + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/statistics/mcv.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/htup_details.h" +#include "catalog/pg_collation.h" +#include "catalog/pg_statistic_ext.h" +#include "fmgr.h" +#include "funcapi.h" +#include "optimizer/clauses.h" +#include "statistics/extended_stats_internal.h" +#include "statistics/statistics.h" +#include "utils/builtins.h" +#include "utils/bytea.h" +#include "utils/fmgroids.h" +#include "utils/fmgrprotos.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" + +#include + +/* + * Computes size of a serialized MCV item, depending on the number of + * dimensions (columns) the statistic is defined on. The datum values are + * stored in a separate array (deduplicated, to minimize the size), and + * so the serialized items only store uint16 indexes into that array. + * + * Each serialized item store (in this order): + * + * - indexes to values (ndim * sizeof(uint16)) + * - null flags (ndim * sizeof(bool)) + * - frequency (sizeof(double)) + * - base_frequency (sizeof(double)) + * + * So in total each MCV item requires this many bytes: + * + * ndim * (sizeof(uint16) + sizeof(bool)) + 2 * sizeof(double) + */ +#define ITEM_SIZE(ndims) \ + (ndims * (sizeof(uint16) + sizeof(bool)) + 2 * sizeof(double)) + +/* + * Macros for convenient access to parts of a serialized MCV item. + */ +#define ITEM_INDEXES(item) ((uint16*)item) +#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims)) +#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims)) +#define ITEM_BASE_FREQUENCY(item,ndims) ((double*)(ITEM_FREQUENCY(item,ndims) + 1)) + + +static MultiSortSupport build_mss(VacAttrStats **stats, Bitmapset *attrs); + +static SortItem *build_distinct_groups(int numrows, SortItem *items, + MultiSortSupport mss, int *ndistinct); + +static int count_distinct_groups(int numrows, SortItem *items, + MultiSortSupport mss); + +/* + * Builds MCV list from the set of sampled rows. + * + * The algorithm is quite simple: + * + * (1) sort the data (default collation, '<' for the data type) + * + * (2) count distinct groups, decide how many to keep + * + * (3) build the MCV list using the threshold determined in (2) + * + * (4) remove rows represented by the MCV from the sample + * + */ +MCVList * +statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, + VacAttrStats **stats, double totalrows) +{ + int i, + j, + k; + int numattrs = bms_num_members(attrs); + int ngroups; + int nitems; + double stadistinct; + int *mcv_counts; + int f1; + + int *attnums = build_attnums(attrs); + + MCVList *mcvlist = NULL; + + /* comparator for all the columns */ + MultiSortSupport mss = build_mss(stats, attrs); + + /* sort the rows */ + SortItem *items = build_sorted_items(numrows, rows, stats[0]->tupDesc, + mss, numattrs, attnums); + + /* transform the sorted rows into groups (sorted by frequency) */ + SortItem *groups = build_distinct_groups(numrows, items, mss, &ngroups); + + /* + * Maximum number of MCV items to store, based on the attribute with the + * largest stats target (and the number of groups we have available). + */ + nitems = stats[0]->attr->attstattarget; + for (i = 1; i < numattrs; i++) + { + if (stats[i]->attr->attstattarget > nitems) + nitems = stats[i]->attr->attstattarget; + } + if (nitems > ngroups) + nitems = ngroups; + + /* + * Decide how many items to keep in the MCV list. We simply use the same + * algorithm as for per-column MCV lists, to keep it consistent. + * + * One difference is that we do not have a multi-column stanullfrac, and + * we simply treat it as a special item in the MCV list (it it makes it). + * We could compute and store it, of course, but we may have statistics + * on more than two columns, so we'd probably want to store this for + * various combinations of columns - for K columns that'd be 2^K values. + * So we instead store those as items of the multi-column MCV list (if + * common enough). + * + * XXX Conceptually this is similar to the NULL-buckets of histograms. + */ + mcv_counts = (int *) palloc(sizeof(int) * nitems); + f1 = 0; + + for (i = 0; i < nitems; i++) + { + mcv_counts[i] = groups[i].count; + + /* count values that occur exactly once for the ndistinct estimate */ + if (groups[i].count == 1) + f1 += 1; + } + + stadistinct = estimate_ndistinct(totalrows, numrows, ngroups, f1); + + /* + * If we can fit all the items onto the MCV list, do that. Otherwise use + * analyze_mcv_list to decide how many items to keep in the MCV list, just + * like for the single-dimensional MCV list. + * + * XXX Should we also consider stadistinct here, to see if the groups do + * represent all the distinct combinations. + */ + if (ngroups > nitems) + { + nitems = analyze_mcv_list(mcv_counts, nitems, stadistinct, + 0.0, /* stanullfrac */ + numrows, totalrows); + } + + /* + * At this point we know the number of items for the MCV list. There might + * be none (for uniform distribution with many groups), and in that case + * there will be no MCV list. Otherwise construct the MCV list. + */ + if (nitems > 0) + { + /* + * Allocate the MCV list structure, set the global parameters. + */ + mcvlist = (MCVList *) palloc0(sizeof(MCVList)); + + mcvlist->magic = STATS_MCV_MAGIC; + mcvlist->type = STATS_MCV_TYPE_BASIC; + mcvlist->ndimensions = numattrs; + mcvlist->nitems = nitems; + + /* store info about data type OIDs */ + i = 0; + j = -1; + while ((j = bms_next_member(attrs, j)) >= 0) + { + VacAttrStats *colstat = stats[i]; + + mcvlist->types[i] = colstat->attrtypid; + i++; + } + + /* + * Preallocate Datum/isnull arrays (not as a single chunk, as we will + * pass the result outside and thus it needs to be easy to pfree(). + * + * XXX On second thought, we're the only ones dealing with MCV lists, + * so we might allocate everything as a single chunk to reduce palloc + * overhead (chunk headers, etc.) without significant risk. Not sure + * it's worth it, though, as we're not re-building stats very often. + */ + mcvlist->items = (MCVItem * *) palloc0(sizeof(MCVItem *) * nitems); + + for (i = 0; i < nitems; i++) + { + mcvlist->items[i] = (MCVItem *) palloc(sizeof(MCVItem)); + mcvlist->items[i]->values = (Datum *) palloc(sizeof(Datum) * numattrs); + mcvlist->items[i]->isnull = (bool *) palloc(sizeof(bool) * numattrs); + } + + /* Copy the first chunk of groups into the result. */ + for (i = 0; i < nitems; i++) + { + /* just pointer to the proper place in the list */ + MCVItem *item = mcvlist->items[i]; + + /* copy values from the _previous_ group (last item of) */ + memcpy(item->values, groups[i].values, sizeof(Datum) * numattrs); + memcpy(item->isnull, groups[i].isnull, sizeof(bool) * numattrs); + + /* groups should be sorted by frequency in descending order */ + Assert((i == 0) || (groups[i - 1].count >= groups[i].count)); + + /* group frequency */ + item->frequency = (double) groups[i].count / numrows; + + /* base frequency, if the attributes were independent */ + item->base_frequency = 1.0; + for (j = 0; j < numattrs; j++) + { + int count = 0; + + for (k = 0; k < ngroups; k++) + { + if (multi_sort_compare_dim(j, &groups[i], &groups[k], mss) == 0) + count += groups[k].count; + } + + item->base_frequency *= (double) count / numrows; + } + } + } + + pfree(items); + pfree(groups); + pfree(mcv_counts); + + return mcvlist; +} + +/* + * build_mss + * build MultiSortSupport for the attributes passed in attrs + */ +static MultiSortSupport +build_mss(VacAttrStats **stats, Bitmapset *attrs) +{ + int i, + j; + int numattrs = bms_num_members(attrs); + + /* Sort by multiple columns (using array of SortSupport) */ + MultiSortSupport mss = multi_sort_init(numattrs); + + /* prepare the sort functions for all the attributes */ + i = 0; + j = -1; + while ((j = bms_next_member(attrs, j)) >= 0) + { + VacAttrStats *colstat = stats[i]; + TypeCacheEntry *type; + + type = lookup_type_cache(colstat->attrtypid, TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) /* shouldn't happen */ + elog(ERROR, "cache lookup failed for ordering operator for type %u", + colstat->attrtypid); + + multi_sort_add_dimension(mss, i, type->lt_opr); + i++; + } + + return mss; +} + +/* + * count_distinct_groups + * count distinct combinations of SortItems in the array + * + * The array is assumed to be sorted according to the MultiSortSupport. + */ +static int +count_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss) +{ + int i; + int ndistinct; + + ndistinct = 1; + for (i = 1; i < numrows; i++) + { + /* make sure the array really is sorted */ + Assert(multi_sort_compare(&items[i], &items[i - 1], mss) >= 0); + + if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0) + ndistinct += 1; + } + + return ndistinct; +} + +/* + * compare_sort_item_count + * comparator for sorting items by count (frequencies) in descending order + */ +static int +compare_sort_item_count(const void *a, const void *b) +{ + SortItem *ia = (SortItem *) a; + SortItem *ib = (SortItem *) b; + + if (ia->count == ib->count) + return 0; + else if (ia->count > ib->count) + return -1; + + return 1; +} + +/* + * build_distinct_groups + * build array of SortItems for distinct groups and counts matching items + * + * The input array is assumed to be sorted + */ +static SortItem * +build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss, + int *ndistinct) +{ + int i, + j; + int ngroups = count_distinct_groups(numrows, items, mss); + + SortItem *groups = (SortItem *) palloc0(ngroups * sizeof(SortItem)); + + j = 0; + groups[0] = items[0]; + groups[0].count = 1; + + for (i = 1; i < numrows; i++) + { + /* Assume sorted in ascending order. */ + Assert(multi_sort_compare(&items[i], &items[i - 1], mss) >= 0); + + /* New distinct group detected. */ + if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0) + groups[++j] = items[i]; + + groups[j].count++; + } + + /* Sort the distinct groups by frequency (in descending order). */ + pg_qsort((void *) groups, ngroups, sizeof(SortItem), + compare_sort_item_count); + + *ndistinct = ngroups; + return groups; +} + + +/* + * statext_mcv_load + * Load the MCV list for the indicated pg_statistic_ext tuple + */ +MCVList * +statext_mcv_load(Oid mvoid) +{ + bool isnull = false; + Datum mcvlist; + HeapTuple htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(mvoid)); + + if (!HeapTupleIsValid(htup)) + elog(ERROR, "cache lookup failed for statistics object %u", mvoid); + + mcvlist = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxmcv, &isnull); + + ReleaseSysCache(htup); + + if (isnull) + return NULL; + + return statext_mcv_deserialize(DatumGetByteaP(mcvlist)); +} + + +/* + * Serialize MCV list into a bytea value. + * + * The basic algorithm is simple: + * + * (1) perform deduplication (for each attribute separately) + * (a) collect all (non-NULL) attribute values from all MCV items + * (b) sort the data (using 'lt' from VacAttrStats) + * (c) remove duplicate values from the array + * + * (2) serialize the arrays into a bytea value + * + * (3) process all MCV list items + * (a) replace values with indexes into the arrays + * + * Each attribute has to be processed separately, as we may be mixing different + * datatypes, with different sort operators, etc. + * + * We use uint16 values for the indexes in step (3), as we currently don't allow + * more than 8k MCV items anyway, although that's mostly arbitrary limit. We might + * increase this to 65k and still fit into uint16. Furthermore, this limit is on + * the number of distinct values per column, and we usually have few of those + * (and various combinations of them for the those MCV list). So uint16 seems fine. + * + * We don't really expect the serialization to save as much space as for + * histograms, as we are not doing any bucket splits (which is the source + * of high redundancy in histograms). + * + * TODO: Consider packing boolean flags (NULL) for each item into a single char + * (or a longer type) instead of using an array of bool items. + */ +bytea * +statext_mcv_serialize(MCVList * mcvlist, VacAttrStats **stats) +{ + int i; + int dim; + int ndims = mcvlist->ndimensions; + int itemsize = ITEM_SIZE(ndims); + + SortSupport ssup; + DimensionInfo *info; + + Size total_length; + + /* allocate the item just once */ + char *item = palloc0(itemsize); + + /* serialized items (indexes into arrays, etc.) */ + bytea *output; + char *data = NULL; + + /* values per dimension (and number of non-NULL values) */ + Datum **values = (Datum **) palloc0(sizeof(Datum *) * ndims); + int *counts = (int *) palloc0(sizeof(int) * ndims); + + /* + * We'll include some rudimentary information about the attributes (type + * length, etc.), so that we don't have to look them up while + * deserializing the MCV list. + * + * XXX Maybe this is not a great idea? Or maybe we should actually copy + * more fields, e.g. typeid, which would allow us to display the MCV list + * using only the serialized representation (currently we have to fetch + * this info from the relation). + */ + info = (DimensionInfo *) palloc0(sizeof(DimensionInfo) * ndims); + + /* sort support data for all attributes included in the MCV list */ + ssup = (SortSupport) palloc0(sizeof(SortSupportData) * ndims); + + /* collect and deduplicate values for each dimension (attribute) */ + for (dim = 0; dim < ndims; dim++) + { + int ndistinct; + TypeCacheEntry *typentry; + + /* + * Lookup the LT operator (can't get it from stats extra_data, as we + * don't know how to interpret that - scalar vs. array etc.). + */ + typentry = lookup_type_cache(stats[dim]->attrtypid, TYPECACHE_LT_OPR); + + /* copy important info about the data type (length, by-value) */ + info[dim].typlen = stats[dim]->attrtype->typlen; + info[dim].typbyval = stats[dim]->attrtype->typbyval; + + /* allocate space for values in the attribute and collect them */ + values[dim] = (Datum *) palloc0(sizeof(Datum) * mcvlist->nitems); + + for (i = 0; i < mcvlist->nitems; i++) + { + /* skip NULL values - we don't need to deduplicate those */ + if (mcvlist->items[i]->isnull[dim]) + continue; + + values[dim][counts[dim]] = mcvlist->items[i]->values[dim]; + counts[dim] += 1; + } + + /* if there are just NULL values in this dimension, we're done */ + if (counts[dim] == 0) + continue; + + /* sort and deduplicate the data */ + ssup[dim].ssup_cxt = CurrentMemoryContext; + ssup[dim].ssup_collation = DEFAULT_COLLATION_OID; + ssup[dim].ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(typentry->lt_opr, &ssup[dim]); + + qsort_arg(values[dim], counts[dim], sizeof(Datum), + compare_scalars_simple, &ssup[dim]); + + /* + * Walk through the array and eliminate duplicate values, but keep the + * ordering (so that we can do bsearch later). We know there's at + * least one item as (counts[dim] != 0), so we can skip the first + * element. + */ + ndistinct = 1; /* number of distinct values */ + for (i = 1; i < counts[dim]; i++) + { + /* expect sorted array */ + Assert(compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim]) <= 0); + + /* if the value is the same as the previous one, we can skip it */ + if (!compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim])) + continue; + + values[dim][ndistinct] = values[dim][i]; + ndistinct += 1; + } + + /* we must not exceed UINT16_MAX, as we use uint16 indexes */ + Assert(ndistinct <= UINT16_MAX); + + /* + * Store additional info about the attribute - number of deduplicated + * values, and also size of the serialized data. For fixed-length data + * types this is trivial to compute, for varwidth types we need to + * actually walk the array and sum the sizes. + */ + info[dim].nvalues = ndistinct; + + if (info[dim].typlen > 0) /* fixed-length data types */ + info[dim].nbytes = info[dim].nvalues * info[dim].typlen; + else if (info[dim].typlen == -1) /* varlena */ + { + info[dim].nbytes = 0; + for (i = 0; i < info[dim].nvalues; i++) + info[dim].nbytes += VARSIZE_ANY(values[dim][i]); + } + else if (info[dim].typlen == -2) /* cstring */ + { + info[dim].nbytes = 0; + for (i = 0; i < info[dim].nvalues; i++) + info[dim].nbytes += strlen(DatumGetPointer(values[dim][i])); + } + + /* we know (count>0) so there must be some data */ + Assert(info[dim].nbytes > 0); + } + + /* + * Now we can finally compute how much space we'll actually need for the + * whole serialized MCV list, as it contains these fields: + * + * - length (4B) for varlena - magic (4B) - type (4B) - ndimensions (4B) - + * nitems (4B) - info (ndim * sizeof(DimensionInfo) - arrays of values for + * each dimension - serialized items (nitems * itemsize) + * + * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and then we + * will place all the data (values + indexes). We'll however use offsetof + * and sizeof to compute sizes of the structs. + */ + total_length = (sizeof(int32) + offsetof(MCVList, items) + + (ndims * sizeof(DimensionInfo)) + + mcvlist->nitems * itemsize); + + /* add space for the arrays of deduplicated values */ + for (i = 0; i < ndims; i++) + total_length += info[i].nbytes; + + /* + * Enforce arbitrary limit of 1MB on the size of the serialized MCV list. + * This is meant as a protection against someone building MCV list on long + * values (e.g. text documents). + * + * XXX Should we enforce arbitrary limits like this one? Maybe it's not + * even necessary, as long values are usually unique and so won't make it + * into the MCV list in the first place. In the end, we have a 1GB limit + * on bytea values. + */ + if (total_length > (1024 * 1024)) + elog(ERROR, "serialized MCV list exceeds 1MB (%ld)", total_length); + + /* allocate space for the serialized MCV list, set header fields */ + output = (bytea *) palloc0(total_length); + SET_VARSIZE(output, total_length); + + /* 'data' points to the current position in the output buffer */ + data = VARDATA(output); + + /* MCV list header (number of items, ...) */ + memcpy(data, mcvlist, offsetof(MCVList, items)); + data += offsetof(MCVList, items); + + /* information about the attributes */ + memcpy(data, info, sizeof(DimensionInfo) * ndims); + data += sizeof(DimensionInfo) * ndims; + + /* Copy the deduplicated values for all attributes to the output. */ + for (dim = 0; dim < ndims; dim++) + { +#ifdef USE_ASSERT_CHECKING + /* remember the starting point for Asserts later */ + char *tmp = data; +#endif + for (i = 0; i < info[dim].nvalues; i++) + { + Datum v = values[dim][i]; + + if (info[dim].typbyval) /* passed by value */ + { + memcpy(data, &v, info[dim].typlen); + data += info[dim].typlen; + } + else if (info[dim].typlen > 0) /* pased by reference */ + { + memcpy(data, DatumGetPointer(v), info[dim].typlen); + data += info[dim].typlen; + } + else if (info[dim].typlen == -1) /* varlena */ + { + memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v)); + data += VARSIZE_ANY(v); + } + else if (info[dim].typlen == -2) /* cstring */ + { + memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v)) + 1); + data += strlen(DatumGetPointer(v)) + 1; /* terminator */ + } + + /* no underflows or overflows */ + Assert((data > tmp) && ((data - tmp) <= info[dim].nbytes)); + } + + /* + * check we got exactly the amount of data we expected for this + * dimension + */ + Assert((data - tmp) == info[dim].nbytes); + } + + /* Serialize the items, with uint16 indexes instead of the values. */ + for (i = 0; i < mcvlist->nitems; i++) + { + MCVItem *mcvitem = mcvlist->items[i]; + + /* don't write beyond the allocated space */ + Assert(data <= (char *) output + total_length - itemsize); + + /* reset the item (we only allocate it once and reuse it) */ + memset(item, 0, itemsize); + + for (dim = 0; dim < ndims; dim++) + { + Datum *v = NULL; + + /* do the lookup only for non-NULL values */ + if (mcvlist->items[i]->isnull[dim]) + continue; + + v = (Datum *) bsearch_arg(&mcvitem->values[dim], values[dim], + info[dim].nvalues, sizeof(Datum), + compare_scalars_simple, &ssup[dim]); + + Assert(v != NULL); /* serialization or deduplication error */ + + /* compute index within the array */ + ITEM_INDEXES(item)[dim] = (v - values[dim]); + + /* check the index is within expected bounds */ + Assert(ITEM_INDEXES(item)[dim] >= 0); + Assert(ITEM_INDEXES(item)[dim] < info[dim].nvalues); + } + + /* copy NULL and frequency flags into the item */ + memcpy(ITEM_NULLS(item, ndims), mcvitem->isnull, sizeof(bool) * ndims); + memcpy(ITEM_FREQUENCY(item, ndims), &mcvitem->frequency, sizeof(double)); + memcpy(ITEM_BASE_FREQUENCY(item, ndims), &mcvitem->base_frequency, sizeof(double)); + + /* copy the serialized item into the array */ + memcpy(data, item, itemsize); + + data += itemsize; + } + + /* at this point we expect to match the total_length exactly */ + Assert((data - (char *) output) == total_length); + + pfree(item); + pfree(values); + pfree(counts); + + return output; +} + +/* + * Reads serialized MCV list into MCVList structure. + * + * Unlike with histograms, we deserialize the MCV list fully (i.e. we don't + * keep the deduplicated arrays and pointers into them), as we don't expect + * there to be a lot of duplicate values. But perhaps that's not true and we + * should keep the MCV in serialized form too. + * + * XXX See how much memory we could save by keeping the deduplicated version + * (both for typical and corner cases with few distinct values but many items). + */ +MCVList * +statext_mcv_deserialize(bytea *data) +{ + int dim, + i; + Size expected_size; + MCVList *mcvlist; + char *tmp; + + int ndims, + nitems, + itemsize; + DimensionInfo *info = NULL; + Datum **values = NULL; + + /* local allocation buffer (used only for deserialization) */ + int bufflen; + char *buff; + char *ptr; + + /* buffer used for the result */ + int rbufflen; + char *rbuff; + char *rptr; + + if (data == NULL) + return NULL; + + /* + * We can't possibly deserialize a MCV list if there's not even a complete + * header. + */ + if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVList, items)) + elog(ERROR, "invalid MCV Size %ld (expected at least %ld)", + VARSIZE_ANY_EXHDR(data), offsetof(MCVList, items)); + + /* read the MCV list header */ + mcvlist = (MCVList *) palloc0(sizeof(MCVList)); + + /* initialize pointer to the data part (skip the varlena header) */ + tmp = VARDATA_ANY(data); + + /* get the header and perform further sanity checks */ + memcpy(mcvlist, tmp, offsetof(MCVList, items)); + tmp += offsetof(MCVList, items); + + if (mcvlist->magic != STATS_MCV_MAGIC) + elog(ERROR, "invalid MCV magic %d (expected %dd)", + mcvlist->magic, STATS_MCV_MAGIC); + + if (mcvlist->type != STATS_MCV_TYPE_BASIC) + elog(ERROR, "invalid MCV type %d (expected %dd)", + mcvlist->type, STATS_MCV_TYPE_BASIC); + + if (mcvlist->ndimensions == 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid zero-length dimension array in MCVList"))); + else if (mcvlist->ndimensions > STATS_MAX_DIMENSIONS) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid length (%d) dimension array in MCVList", + mcvlist->ndimensions))); + + if (mcvlist->nitems == 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid zero-length item array in MCVList"))); + else if (mcvlist->nitems > STATS_MCVLIST_MAX_ITEMS) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("invalid length (%d) item array in MCVList", + mcvlist->nitems))); + + nitems = mcvlist->nitems; + ndims = mcvlist->ndimensions; + itemsize = ITEM_SIZE(ndims); + + /* + * Check amount of data including DimensionInfo for all dimensions and + * also the serialized items (including uint16 indexes). Also, walk + * through the dimension information and add it to the sum. + */ + expected_size = offsetof(MCVList, items) + + ndims * sizeof(DimensionInfo) + + (nitems * itemsize); + + /* + * Check that we have at least the dimension and info records, along with + * the items. We don't know the size of the serialized values yet. We need + * to do this check first, before accessing the dimension info. + */ + if (VARSIZE_ANY_EXHDR(data) < expected_size) + elog(ERROR, "invalid MCV size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* Now it's safe to access the dimension info. */ + info = (DimensionInfo *) (tmp); + tmp += ndims * sizeof(DimensionInfo); + + /* account for the value arrays */ + for (dim = 0; dim < ndims; dim++) + { + /* + * XXX I wonder if we can/should rely on asserts here. Maybe those + * checks should be done every time? + */ + Assert(info[dim].nvalues >= 0); + Assert(info[dim].nbytes >= 0); + + expected_size += info[dim].nbytes; + } + + /* + * Now we know the total expected MCV size, including all the pieces + * (header, dimension info. items and deduplicated data). So do the final + * check on size. + */ + if (VARSIZE_ANY_EXHDR(data) != expected_size) + elog(ERROR, "invalid MCV size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* + * Allocate one large chunk of memory for the intermediate data, needed + * only for deserializing the MCV list (and allocate densely to minimize + * the palloc overhead). + * + * Let's see how much space we'll actually need, and also include space + * for the array with pointers. + * + * We need an array of Datum pointers values for each dimension, so that + * we can easily translate the uint16 indexes. We also need a top-level + * array of pointers to those per-dimension arrays. + * + * For byval types with size matching sizeof(Datum) we can reuse the + * serialized array directly. + */ + bufflen = sizeof(Datum **) * ndims; /* space for top-level pointers */ + + for (dim = 0; dim < ndims; dim++) + { + /* for full-size byval types, we reuse the serialized value */ + if (!(info[dim].typbyval && info[dim].typlen == sizeof(Datum))) + bufflen += (sizeof(Datum) * info[dim].nvalues); + } + + buff = palloc0(bufflen); + ptr = buff; + + values = (Datum **) buff; + ptr += (sizeof(Datum *) * ndims); + + /* + * XXX This uses pointers to the original data array (the types not passed + * by value), so when someone frees the memory, e.g. by doing something + * like this: + * + * bytea * data = ... fetch the data from catalog ... + * + * MCVList mcvlist = deserialize_mcv_list(data); + * + * pfree(data); + * + * then 'mcvlist' references the freed memory. Should copy the pieces. + */ + for (dim = 0; dim < ndims; dim++) + { +#ifdef USE_ASSERT_CHECKING + /* remember where data for this dimension starts */ + char *start = tmp; +#endif + if (info[dim].typbyval) + { + /* passed by value / size matches Datum - just reuse the array */ + if (info[dim].typlen == sizeof(Datum)) + { + values[dim] = (Datum *) tmp; + tmp += info[dim].nbytes; + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + else + { + values[dim] = (Datum *) ptr; + ptr += (sizeof(Datum) * info[dim].nvalues); + + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + memcpy(&values[dim][i], tmp, info[dim].typlen); + tmp += info[dim].typlen; + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + } + else + { + /* all the other types need a chunk of the buffer */ + values[dim] = (Datum *) ptr; + ptr += (sizeof(Datum) * info[dim].nvalues); + + /* passed by reference, but fixed length (name, tid, ...) */ + if (info[dim].typlen > 0) + { + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + values[dim][i] = PointerGetDatum(tmp); + tmp += info[dim].typlen; + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + else if (info[dim].typlen == -1) + { + /* varlena */ + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + values[dim][i] = PointerGetDatum(tmp); + tmp += VARSIZE_ANY(tmp); + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + else if (info[dim].typlen == -2) + { + /* cstring */ + for (i = 0; i < info[dim].nvalues; i++) + { + /* just point into the array */ + values[dim][i] = PointerGetDatum(tmp); + tmp += (strlen(tmp) + 1); /* don't forget the \0 */ + + /* no overflow of input array */ + Assert(tmp <= start + info[dim].nbytes); + } + } + } + + /* check we consumed the serialized data for this dimension exactly */ + Assert((tmp - start) == info[dim].nbytes); + } + + /* we should have exhausted the buffer exactly */ + Assert((ptr - buff) == bufflen); + + /* allocate space for all the MCV items in a single piece */ + rbufflen = (sizeof(MCVItem *) + sizeof(MCVItem) + + sizeof(Datum) * ndims + sizeof(bool) * ndims) * nitems; + + rbuff = palloc0(rbufflen); + rptr = rbuff; + + mcvlist->items = (MCVItem * *) rbuff; + rptr += (sizeof(MCVItem *) * nitems); + + /* deserialize the MCV items and translate the indexes to Datums */ + for (i = 0; i < nitems; i++) + { + uint16 *indexes = NULL; + MCVItem *item = (MCVItem *) rptr; + + rptr += (sizeof(MCVItem)); + + item->values = (Datum *) rptr; + rptr += (sizeof(Datum) * ndims); + + item->isnull = (bool *) rptr; + rptr += (sizeof(bool) * ndims); + + /* just point to the right place */ + indexes = ITEM_INDEXES(tmp); + + memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims); + memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double)); + memcpy(&item->base_frequency, ITEM_BASE_FREQUENCY(tmp, ndims), sizeof(double)); + + /* translate the values */ + for (dim = 0; dim < ndims; dim++) + if (!item->isnull[dim]) + item->values[dim] = values[dim][indexes[dim]]; + + mcvlist->items[i] = item; + + tmp += ITEM_SIZE(ndims); + + /* check we're not overflowing the input */ + Assert(tmp <= (char *) data + VARSIZE_ANY(data)); + } + + /* check that we processed all the data */ + Assert(tmp == (char *) data + VARSIZE_ANY(data)); + + /* release the temporary buffer */ + pfree(buff); + + return mcvlist; +} + +/* + * SRF with details about buckets of a histogram: + * + * - item ID (0...nitems) + * - values (string array) + * - nulls only (boolean array) + * - frequency (double precision) + * - base_frequency (double precision) + * + * The input is the OID of the statistics, and there are no rows returned if + * the statistics contains no histogram. + */ +PG_FUNCTION_INFO_V1(pg_stats_ext_mcvlist_items); + +Datum +pg_stats_ext_mcvlist_items(PG_FUNCTION_ARGS) +{ + FuncCallContext *funcctx; + int call_cntr; + int max_calls; + TupleDesc tupdesc; + AttInMetadata *attinmeta; + + /* stuff done only on the first call of the function */ + if (SRF_IS_FIRSTCALL()) + { + MemoryContext oldcontext; + MCVList *mcvlist; + + /* create a function context for cross-call persistence */ + funcctx = SRF_FIRSTCALL_INIT(); + + /* switch to memory context appropriate for multiple function calls */ + oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); + + mcvlist = statext_mcv_deserialize(PG_GETARG_BYTEA_P(0)); + + funcctx->user_fctx = mcvlist; + + /* total number of tuples to be returned */ + funcctx->max_calls = 0; + if (funcctx->user_fctx != NULL) + funcctx->max_calls = mcvlist->nitems; + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("function returning record called in context " + "that cannot accept type record"))); + + /* build metadata needed later to produce tuples from raw C-strings */ + attinmeta = TupleDescGetAttInMetadata(tupdesc); + funcctx->attinmeta = attinmeta; + + MemoryContextSwitchTo(oldcontext); + } + + /* stuff done on every call of the function */ + funcctx = SRF_PERCALL_SETUP(); + + call_cntr = funcctx->call_cntr; + max_calls = funcctx->max_calls; + attinmeta = funcctx->attinmeta; + + if (call_cntr < max_calls) /* do when there is more left to send */ + { + char **values; + HeapTuple tuple; + Datum result; + + char *buff = palloc0(1024); + char *format; + + int i; + + Oid *outfuncs; + FmgrInfo *fmgrinfo; + + MCVList *mcvlist; + MCVItem *item; + + mcvlist = (MCVList *) funcctx->user_fctx; + + Assert(call_cntr < mcvlist->nitems); + + item = mcvlist->items[call_cntr]; + + /* + * Prepare a values array for building the returned tuple. This should + * be an array of C strings which will be processed later by the type + * input functions. + */ + values = (char **) palloc(5 * sizeof(char *)); + + values[0] = (char *) palloc(64 * sizeof(char)); + + /* arrays */ + values[1] = (char *) palloc0(1024 * sizeof(char)); + values[2] = (char *) palloc0(1024 * sizeof(char)); + + /* frequency */ + values[3] = (char *) palloc(64 * sizeof(char)); + + /* base frequency */ + values[4] = (char *) palloc(64 * sizeof(char)); + + outfuncs = (Oid *) palloc0(sizeof(Oid) * mcvlist->ndimensions); + fmgrinfo = (FmgrInfo *) palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions); + + for (i = 0; i < mcvlist->ndimensions; i++) + { + bool isvarlena; + + getTypeOutputInfo(mcvlist->types[i], &outfuncs[i], &isvarlena); + + fmgr_info(outfuncs[i], &fmgrinfo[i]); + } + + snprintf(values[0], 64, "%d", call_cntr); /* item ID */ + + for (i = 0; i < mcvlist->ndimensions; i++) + { + Datum val, + valout; + + format = "%s, %s"; + if (i == 0) + format = "{%s%s"; + else if (i == mcvlist->ndimensions - 1) + format = "%s, %s}"; + + if (item->isnull[i]) + valout = CStringGetDatum("NULL"); + else + { + val = item->values[i]; + valout = FunctionCall1(&fmgrinfo[i], val); + } + + snprintf(buff, 1024, format, values[1], DatumGetPointer(valout)); + strncpy(values[1], buff, 1023); + buff[0] = '\0'; + + snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f"); + strncpy(values[2], buff, 1023); + buff[0] = '\0'; + } + + snprintf(values[3], 64, "%f", item->frequency); /* frequency */ + snprintf(values[4], 64, "%f", item->base_frequency); /* base frequency */ + + /* build a tuple */ + tuple = BuildTupleFromCStrings(attinmeta, values); + + /* make the tuple into a datum */ + result = HeapTupleGetDatum(tuple); + + /* clean up (this is not really necessary) */ + pfree(values[0]); + pfree(values[1]); + pfree(values[2]); + pfree(values[3]); + pfree(values[4]); + + pfree(values); + + SRF_RETURN_NEXT(funcctx, result); + } + else /* do when there is no more left */ + { + SRF_RETURN_DONE(funcctx); + } +} + +/* + * pg_mcv_list_in - input routine for type pg_mcv_list. + * + * pg_mcv_list is real enough to be a table column, but it has no operations + * of its own, and disallows input too + */ +Datum +pg_mcv_list_in(PG_FUNCTION_ARGS) +{ + /* + * pg_mcv_list stores the data in binary form and parsing text input is + * not needed, so disallow this. + */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot accept a value of type %s", "pg_mcv_list"))); + + PG_RETURN_VOID(); /* keep compiler quiet */ +} + + +/* + * pg_mcv_list_out - output routine for type PG_MCV_LIST. + * + * MCV lists are serialized into a bytea value, so we simply call byteaout() + * to serialize the value into text. But it'd be nice to serialize that into + * a meaningful representation (e.g. for inspection by people). + * + * XXX This should probably return something meaningful, similar to what + * pg_dependencies_out does. Not sure how to deal with the deduplicated + * values, though - do we want to expand that or not? + */ +Datum +pg_mcv_list_out(PG_FUNCTION_ARGS) +{ + return byteaout(fcinfo); +} + +/* + * pg_mcv_list_recv - binary input routine for type pg_mcv_list. + */ +Datum +pg_mcv_list_recv(PG_FUNCTION_ARGS) +{ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot accept a value of type %s", "pg_mcv_list"))); + + PG_RETURN_VOID(); /* keep compiler quiet */ +} + +/* + * pg_mcv_list_send - binary output routine for type pg_mcv_list. + * + * MCV lists are serialized in a bytea value (although the type is named + * differently), so let's just send that. + */ +Datum +pg_mcv_list_send(PG_FUNCTION_ARGS) +{ + return byteasend(fcinfo); +} + +/* + * mcv_update_match_bitmap + * Evaluate clauses using the MCV list, and update the match bitmap. + * + * A match bitmap keeps match/mismatch status for each MCV item, and we + * update it based on additional clauses. We also use it to skip items + * that can't possibly match (e.g. item marked as "mismatch" can't change + * to "match" when evaluating AND clause list). + * + * The function also returns a flag indicating whether there was an + * equality condition for all attributes, the minimum frequency in the MCV + * list, and a total MCV frequency (sum of frequencies for all items). + * + * XXX Currently the match bitmap uses a char for each MCV item, which is + * somewhat wasteful as we could do with just a single bit, thus reducing + * the size to ~1/8. It would also allow us to combine bitmaps simply using + * & and |, which should be faster than min/max. The bitmaps are fairly + * small, though (as we cap the MCV list size to 8k items). + */ +static void +mcv_update_match_bitmap(PlannerInfo *root, List *clauses, + Bitmapset *keys, MCVList * mcvlist, char *matches, + bool is_or) +{ + int i; + ListCell *l; + + /* The bitmap may be partially built. */ + Assert(clauses != NIL); + Assert(list_length(clauses) >= 1); + Assert(mcvlist != NULL); + Assert(mcvlist->nitems > 0); + Assert(mcvlist->nitems <= STATS_MCVLIST_MAX_ITEMS); + + /* + * Loop through the list of clauses, and for each of them evaluate all the + * MCV items not yet eliminated by the preceding clauses. + */ + foreach(l, clauses) + { + Node *clause = (Node *) lfirst(l); + + /* if it's a RestrictInfo, then extract the clause */ + if (IsA(clause, RestrictInfo)) + clause = (Node *) ((RestrictInfo *) clause)->clause; + + /* + * Handle the various types of clauses - OpClause, NullTest and + * AND/OR/NOT + */ + if (is_opclause(clause)) + { + OpExpr *expr = (OpExpr *) clause; + bool varonleft = true; + bool ok; + FmgrInfo opproc; + + /* get procedure computing operator selectivity */ + RegProcedure oprrest = get_oprrest(expr->opno); + + fmgr_info(get_opcode(expr->opno), &opproc); + + ok = (NumRelids(clause) == 1) && + (is_pseudo_constant_clause(lsecond(expr->args)) || + (varonleft = false, + is_pseudo_constant_clause(linitial(expr->args)))); + + if (ok) + { + + FmgrInfo gtproc; + Var *var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + Const *cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args); + bool isgt = (!varonleft); + + TypeCacheEntry *typecache + = lookup_type_cache(var->vartype, TYPECACHE_GT_OPR); + + /* match the attribute to a dimension of the statistic */ + int idx = bms_member_index(keys, var->varattno); + + fmgr_info(get_opcode(typecache->gt_opr), >proc); + + /* + * Walk through the MCV items and evaluate the current clause. + * We can skip items that were already ruled out, and + * terminate if there are no remaining MCV items that might + * possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + bool mismatch = false; + MCVItem *item = mcvlist->items[i]; + + /* + * For AND-lists, we can also mark NULL items as 'no + * match' (and then skip them). For OR-lists this is not + * possible. + */ + if ((!is_or) && item->isnull[idx]) + matches[i] = STATS_MATCH_NONE; + + /* skip MCV items that were already ruled out */ + if ((!is_or) && (matches[i] == STATS_MATCH_NONE)) + continue; + else if (is_or && (matches[i] == STATS_MATCH_FULL)) + continue; + + switch (oprrest) + { + case F_EQSEL: + case F_NEQSEL: + + /* + * We don't care about isgt in equality, because + * it does not matter whether it's (var op const) + * or (const op var). + */ + mismatch = !DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + item->values[idx])); + + break; + + case F_SCALARLTSEL: /* column < constant */ + case F_SCALARLESEL: /* column <= constant */ + case F_SCALARGTSEL: /* column > constant */ + case F_SCALARGESEL: /* column >= constant */ + + /* + * First check whether the constant is below the + * lower boundary (in that case we can skip the + * bucket, because there's no overlap). + */ + if (isgt) + mismatch = !DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + item->values[idx])); + else + mismatch = !DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + item->values[idx], + cst->constvalue)); + + break; + } + + /* + * XXX The conditions on matches[i] are not needed, as we + * skip MCV items that can't become true/false, depending + * on the current flag. See beginning of the loop over MCV + * items. + */ + + if ((is_or) && (!mismatch)) + { + /* OR - was MATCH_NONE, but will be MATCH_FULL */ + matches[i] = STATS_MATCH_FULL; + continue; + } + else if ((!is_or) && mismatch) + { + /* AND - was MATC_FULL, but will be MATCH_NONE */ + matches[i] = STATS_MATCH_NONE; + continue; + } + + } + } + } + else if (IsA(clause, NullTest)) + { + NullTest *expr = (NullTest *) clause; + Var *var = (Var *) (expr->arg); + + /* match the attribute to a dimension of the statistic */ + int idx = bms_member_index(keys, var->varattno); + + /* + * Walk through the MCV items and evaluate the current clause. We + * can skip items that were already ruled out, and terminate if + * there are no remaining MCV items that might possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + char match = STATS_MATCH_NONE; /* assume mismatch */ + MCVItem *item = mcvlist->items[i]; + + /* if the clause mismatches the MCV item, set it as MATCH_NONE */ + switch (expr->nulltesttype) + { + case IS_NULL: + match = (item->isnull[idx]) ? STATS_MATCH_FULL : match; + break; + + case IS_NOT_NULL: + match = (!item->isnull[idx]) ? STATS_MATCH_FULL : match; + break; + } + + /* now, update the match bitmap, depending on OR/AND type */ + if (is_or) + matches[i] = Max(matches[i], match); + else + matches[i] = Min(matches[i], match); + } + } + else if (or_clause(clause) || and_clause(clause)) + { + /* AND/OR clause, with all subclauses being compatible */ + + int i; + BoolExpr *bool_clause = ((BoolExpr *) clause); + List *bool_clauses = bool_clause->args; + + /* match/mismatch bitmap for each MCV item */ + char *bool_matches = NULL; + + Assert(bool_clauses != NIL); + Assert(list_length(bool_clauses) >= 2); + + /* by default none of the MCV items matches the clauses */ + bool_matches = palloc0(sizeof(char) * mcvlist->nitems); + + if (or_clause(clause)) + { + /* OR clauses assume nothing matches, initially */ + memset(bool_matches, STATS_MATCH_NONE, sizeof(char) * mcvlist->nitems); + } + else + { + /* AND clauses assume everything matches, initially */ + memset(bool_matches, STATS_MATCH_FULL, sizeof(char) * mcvlist->nitems); + } + + /* build the match bitmap for the OR-clauses */ + mcv_update_match_bitmap(root, bool_clauses, keys, + mcvlist, bool_matches, + or_clause(clause)); + + /* + * Merge the bitmap produced by mcv_update_match_bitmap into the + * current one. We need to consider if we're evaluating AND or OR + * condition when merging the results. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + /* Is this OR or AND clause? */ + if (is_or) + matches[i] = Max(matches[i], bool_matches[i]); + else + matches[i] = Min(matches[i], bool_matches[i]); + } + + pfree(bool_matches); + } + else if (not_clause(clause)) + { + /* NOT clause, with all subclauses compatible */ + + int i; + BoolExpr *not_clause = ((BoolExpr *) clause); + List *not_args = not_clause->args; + + /* match/mismatch bitmap for each MCV item */ + char *not_matches = NULL; + + Assert(not_args != NIL); + Assert(list_length(not_args) == 1); + + /* by default none of the MCV items matches the clauses */ + not_matches = palloc0(sizeof(char) * mcvlist->nitems); + + /* NOT clauses assume nothing matches, initially */ + memset(not_matches, STATS_MATCH_FULL, sizeof(char) * mcvlist->nitems); + + /* build the match bitmap for the NOT-clause */ + mcv_update_match_bitmap(root, not_args, keys, + mcvlist, not_matches, false); + + /* + * Merge the bitmap produced by mcv_update_match_bitmap into the + * current one. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + /* + * When handling a NOT clause, we need to invert the result + * before merging it into the global result. + */ + if (not_matches[i] == STATS_MATCH_NONE) + not_matches[i] = STATS_MATCH_FULL; + else + not_matches[i] = STATS_MATCH_NONE; + + /* Is this OR or AND clause? */ + if (is_or) + matches[i] = Max(matches[i], not_matches[i]); + else + matches[i] = Min(matches[i], not_matches[i]); + } + + pfree(not_matches); + } + else if (IsA(clause, Var)) + { + /* Var (has to be a boolean Var, possibly from below NOT) */ + + Var *var = (Var *) (clause); + + /* match the attribute to a dimension of the statistic */ + int idx = bms_member_index(keys, var->varattno); + + Assert(var->vartype == BOOLOID); + + /* + * Walk through the MCV items and evaluate the current clause. We + * can skip items that were already ruled out, and terminate if + * there are no remaining MCV items that might possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + MCVItem *item = mcvlist->items[i]; + bool match = STATS_MATCH_NONE; + + /* if the item is NULL, it's a mismatch */ + if (!item->isnull[idx] && DatumGetBool(item->values[idx])) + match = STATS_MATCH_FULL; + + /* now, update the match bitmap, depending on OR/AND type */ + if (is_or) + matches[i] = Max(matches[i], match); + else + matches[i] = Min(matches[i], match); + } + } + else + { + elog(ERROR, "unknown clause type: %d", clause->type); + } + } +} + + +/* + * mcv_clauselist_selectivity + * Return the selectivity estimate of clauses using MCV list. + * + * It also produces two interesting selectivities - total selectivity of + * all the MCV items combined, and selectivity of the least frequent item + * in the list. + */ +Selectivity +mcv_clauselist_selectivity(PlannerInfo *root, StatisticExtInfo *stat, + List *clauses, int varRelid, + JoinType jointype, SpecialJoinInfo *sjinfo, + RelOptInfo *rel, + Selectivity *basesel, Selectivity *totalsel) +{ + int i; + MCVList *mcv; + Selectivity s = 0.0; + + /* match/mismatch bitmap for each MCV item */ + char *matches = NULL; + + /* load the MCV list stored in the statistics object */ + mcv = statext_mcv_load(stat->statOid); + + /* by default all the MCV items match the clauses fully */ + matches = palloc0(sizeof(char) * mcv->nitems); + memset(matches, STATS_MATCH_FULL, sizeof(char) * mcv->nitems); + + mcv_update_match_bitmap(root, clauses, stat->keys, mcv, + matches, false); + + /* sum frequencies for all the matching MCV items */ + *basesel = 0.0; + *totalsel = 0.0; + for (i = 0; i < mcv->nitems; i++) + { + *totalsel += mcv->items[i]->frequency; + + if (matches[i] != STATS_MATCH_NONE) + { + /* XXX Shouldn't the basesel be outside the if condition? */ + *basesel += mcv->items[i]->base_frequency; + s += mcv->items[i]->frequency; + } + } + + return s; +} diff --git a/src/backend/statistics/mvdistinct.c b/src/backend/statistics/mvdistinct.c index 593c219839..fb74ed3618 100644 --- a/src/backend/statistics/mvdistinct.c +++ b/src/backend/statistics/mvdistinct.c @@ -23,8 +23,6 @@ */ #include "postgres.h" -#include - #include "access/htup_details.h" #include "catalog/pg_statistic_ext.h" #include "utils/fmgrprotos.h" @@ -39,7 +37,6 @@ static double ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, VacAttrStats **stats, int k, int *combination); -static double estimate_ndistinct(double totalrows, int numrows, int d, int f1); static int n_choose_k(int n, int k); static int num_combinations(int n); @@ -508,31 +505,6 @@ ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, return estimate_ndistinct(totalrows, numrows, d, f1); } -/* The Duj1 estimator (already used in analyze.c). */ -static double -estimate_ndistinct(double totalrows, int numrows, int d, int f1) -{ - double numer, - denom, - ndistinct; - - numer = (double) numrows * (double) d; - - denom = (double) (numrows - f1) + - (double) f1 * (double) numrows / totalrows; - - ndistinct = numer / denom; - - /* Clamp to sane range in case of roundoff error */ - if (ndistinct < (double) d) - ndistinct = (double) d; - - if (ndistinct > totalrows) - ndistinct = totalrows; - - return floor(ndistinct + 0.5); -} - /* * n_choose_k * computes binomial coefficients using an algorithm that is both diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index 03e9a28a63..c941c3310b 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -1504,6 +1504,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) bool isnull; bool ndistinct_enabled; bool dependencies_enabled; + bool mcv_enabled; int i; statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); @@ -1539,6 +1540,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) ndistinct_enabled = false; dependencies_enabled = false; + mcv_enabled = false; for (i = 0; i < ARR_DIMS(arr)[0]; i++) { @@ -1546,6 +1548,8 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) ndistinct_enabled = true; if (enabled[i] == STATS_EXT_DEPENDENCIES) dependencies_enabled = true; + if (enabled[i] == STATS_EXT_MCV) + mcv_enabled = true; } /* @@ -1555,13 +1559,27 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) * statistics types on a newer postgres version, if the statistics had all * options enabled on the original version. */ - if (!ndistinct_enabled || !dependencies_enabled) + if (!ndistinct_enabled || !dependencies_enabled || !mcv_enabled) { + bool gotone = false; + appendStringInfoString(&buf, " ("); + if (ndistinct_enabled) + { appendStringInfoString(&buf, "ndistinct"); - else if (dependencies_enabled) - appendStringInfoString(&buf, "dependencies"); + gotone = true; + } + + if (dependencies_enabled) + { + appendStringInfo(&buf, "%sdependencies", gotone ? ", " : ""); + gotone = true; + } + + if (mcv_enabled) + appendStringInfo(&buf, "%smcv", gotone ? ", " : ""); + appendStringInfoChar(&buf, ')'); } diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index f1c78ffb65..fdfc0d6a1b 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -3735,6 +3735,171 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, } /* + * estimate_num_groups_simple + * Estimate number of groups in a relation. + * + * A simplified version of estimate_num_groups, assuming all expressions + * are only plain Vars from a single relation, and that no filtering is + * happenning. + */ +double +estimate_num_groups_simple(PlannerInfo *root, List *vars) +{ + List *varinfos = NIL; + double numdistinct; + ListCell *l; + + RelOptInfo *rel; + double reldistinct = 1; + double relmaxndistinct = reldistinct; + int relvarcount = 0; + + + /* + * If no grouping columns, there's exactly one group. (This can't happen + * for normal cases with GROUP BY or DISTINCT, but it is possible for + * corner cases with set operations.) + */ + if (vars == NIL) + return 1.0; + + /* + * We expect only variables from a single relation. + */ + Assert(NumRelids((Node *) vars) == 1); + + /* + * Find the unique Vars used, treating an expression as a Var if we can + * find stats for it. For each one, record the statistical estimate of + * number of distinct values (total in its table). + */ + numdistinct = 1.0; + + foreach(l, vars) + { + Var *var = (Var *) lfirst(l); + VariableStatData vardata; + + Assert(IsA(var, Var)); + + /* + * If examine_variable is able to deduce anything about the GROUP BY + * expression, treat it as a single variable even if it's really more + * complicated. + */ + examine_variable(root, (Node *) var, 0, &vardata); + if (HeapTupleIsValid(vardata.statsTuple) || vardata.isunique) + { + varinfos = add_unique_group_var(root, varinfos, + (Node *) var, &vardata); + ReleaseVariableStats(vardata); + continue; + } + ReleaseVariableStats(vardata); + } + + Assert(varinfos); + + /* + * Get the numdistinct estimate for the Vars of this rel. + * + * We + * iteratively search for multivariate n-distinct with maximum number + * of vars; assuming that each var group is independent of the others, + * we multiply them together. Any remaining relvarinfos after no more + * multivariate matches are found are assumed independent too, so + * their individual ndistinct estimates are multiplied also. + * + * While iterating, count how many separate numdistinct values we + * apply. We apply a fudge factor below, but only if we multiplied + * more than one such values. + */ + while (varinfos) + { + double mvndistinct; + + rel = ((GroupVarInfo *) linitial(varinfos))->rel; + + if (estimate_multivariate_ndistinct(root, rel, &varinfos, + &mvndistinct)) + { + reldistinct *= mvndistinct; + if (relmaxndistinct < mvndistinct) + relmaxndistinct = mvndistinct; + relvarcount++; + } + else + { + foreach(l, varinfos) + { + GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(l); + + reldistinct *= varinfo->ndistinct; + if (relmaxndistinct < varinfo->ndistinct) + relmaxndistinct = varinfo->ndistinct; + relvarcount++; + } + + /* we're done with this relation */ + varinfos = NIL; + } + } + + /* + * Sanity check --- don't divide by zero if empty relation. + */ + Assert(IS_SIMPLE_REL(rel)); + if (rel->tuples > 0) + { + /* + * Clamp to size of rel, or size of rel / 10 if multiple Vars. The + * fudge factor is because the Vars are probably correlated but we + * don't know by how much. We should never clamp to less than the + * largest ndistinct value for any of the Vars, though, since + * there will surely be at least that many groups. + */ + double clamp = rel->tuples; + + if (relvarcount > 1) + { + clamp *= 0.1; + if (clamp < relmaxndistinct) + { + clamp = relmaxndistinct; + /* for sanity in case some ndistinct is too large: */ + if (clamp > rel->tuples) + clamp = rel->tuples; + } + } + if (reldistinct > clamp) + reldistinct = clamp; + + /* + * We're assuming we are returning all rows. + */ + reldistinct = clamp_row_est(reldistinct); + + /* + * Update estimate of total distinct groups. + */ + numdistinct *= reldistinct; + + /* Guard against out-of-range answers */ + if (numdistinct > rel->tuples) + numdistinct = rel->tuples; + } + + if (numdistinct < 1.0) + numdistinct = 1.0; + + /* Round off */ + numdistinct = ceil(numdistinct); + + return numdistinct; + +} + +/* * Estimate hash bucket statistics when the specified expression is used * as a hash key for the given number of buckets. * diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 4ca0db1d0c..394301caa8 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2518,7 +2518,8 @@ describeOneTableDetails(const char *schemaname, " JOIN pg_catalog.pg_attribute a ON (stxrelid = a.attrelid AND\n" " a.attnum = s.attnum AND NOT attisdropped)) AS columns,\n" " 'd' = any(stxkind) AS ndist_enabled,\n" - " 'f' = any(stxkind) AS deps_enabled\n" + " 'f' = any(stxkind) AS deps_enabled,\n" + " 'm' = any(stxkind) AS mcv_enabled\n" "FROM pg_catalog.pg_statistic_ext stat " "WHERE stxrelid = '%s'\n" "ORDER BY 1;", @@ -2555,6 +2556,12 @@ describeOneTableDetails(const char *schemaname, if (strcmp(PQgetvalue(result, i, 6), "t") == 0) { appendPQExpBuffer(&buf, "%sdependencies", gotone ? ", " : ""); + gotone = true; + } + + if (strcmp(PQgetvalue(result, i, 7), "t") == 0) + { + appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : ""); } appendPQExpBuffer(&buf, ") ON %s FROM %s", diff --git a/src/include/catalog/pg_cast.dat b/src/include/catalog/pg_cast.dat index cf007528fd..dff3a9a08a 100644 --- a/src/include/catalog/pg_cast.dat +++ b/src/include/catalog/pg_cast.dat @@ -324,6 +324,12 @@ { castsource => 'pg_dependencies', casttarget => 'text', castfunc => '0', castcontext => 'i', castmethod => 'i' }, +# pg_mcv_list can be coerced to, but not from, bytea and text +{ castsource => 'pg_mcv_list', casttarget => 'bytea', castfunc => '0', + castcontext => 'i', castmethod => 'b' }, +{ castsource => 'pg_mcv_list', casttarget => 'text', castfunc => '0', + castcontext => 'i', castmethod => 'i' }, + # Datetime category { castsource => 'abstime', casttarget => 'date', castfunc => 'date(abstime)', castcontext => 'a', castmethod => 'f' }, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index a14651010f..3cfcafcb11 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -5073,6 +5073,30 @@ proname => 'pg_dependencies_send', provolatile => 's', prorettype => 'bytea', proargtypes => 'pg_dependencies', prosrc => 'pg_dependencies_send' }, +{ oid => '4002', descr => 'I/O', + proname => 'pg_mcv_list_in', prorettype => 'pg_mcv_list', + proargtypes => 'cstring', prosrc => 'pg_mcv_list_in' }, +{ oid => '4003', descr => 'I/O', + proname => 'pg_mcv_list_out', prorettype => 'cstring', + proargtypes => 'pg_mcv_list', prosrc => 'pg_mcv_list_out' }, +{ oid => '4004', descr => 'I/O', + proname => 'pg_mcv_list_recv', provolatile => 's', + prorettype => 'pg_mcv_list', proargtypes => 'internal', + prosrc => 'pg_mcv_list_recv' }, +{ oid => '4005', descr => 'I/O', + proname => 'pg_mcv_list_send', provolatile => 's', prorettype => 'bytea', + proargtypes => 'pg_mcv_list', prosrc => 'pg_mcv_list_send' }, + +{ oid => '3424', + descr => 'details about MCV list items', + proname => 'pg_mcv_list_items', prorows => '1000', proisstrict => 'f', + proretset => 't', provolatile => 's', prorettype => 'record', + proargtypes => 'pg_mcv_list', + proallargtypes => '{pg_mcv_list,int4,text,_bool,float8,float8}', + proargmodes => '{i,o,o,o,o,o}', + proargnames => '{mcv_list,index,values,nulls,frequency,base_frequency}', + prosrc => 'pg_stats_ext_mcvlist_items' }, + { oid => '1928', descr => 'statistics: number of scans done for table/index', proname => 'pg_stat_get_numscans', provolatile => 's', proparallel => 'r', prorettype => 'int8', proargtypes => 'oid', diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h index 443798ae52..7ddbee63c9 100644 --- a/src/include/catalog/pg_statistic_ext.h +++ b/src/include/catalog/pg_statistic_ext.h @@ -47,6 +47,7 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId) * to build */ pg_ndistinct stxndistinct; /* ndistinct coefficients (serialized) */ pg_dependencies stxdependencies; /* dependencies (serialized) */ + pg_mcv_list stxmcv; /* MCV (serialized) */ #endif } FormData_pg_statistic_ext; @@ -62,6 +63,7 @@ typedef FormData_pg_statistic_ext *Form_pg_statistic_ext; #define STATS_EXT_NDISTINCT 'd' #define STATS_EXT_DEPENDENCIES 'f' +#define STATS_EXT_MCV 'm' #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 48e01cd694..0ff25e87a7 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -165,6 +165,13 @@ typoutput => 'pg_dependencies_out', typreceive => 'pg_dependencies_recv', typsend => 'pg_dependencies_send', typalign => 'i', typstorage => 'x', typcollation => '100' }, +{ oid => '4001', oid_symbol => 'PGMCVLISTOID', + descr => 'multivariate MCV list', + typname => 'pg_mcv_list', typlen => '-1', typbyval => 'f', + typcategory => 'S', typinput => 'pg_mcv_list_in', + typoutput => 'pg_mcv_list_out', typreceive => 'pg_mcv_list_recv', + typsend => 'pg_mcv_list_send', typalign => 'i', typstorage => 'x', + typcollation => '100' }, { oid => '32', oid_symbol => 'PGDDLCOMMANDOID', descr => 'internal type for passing CollectedCommand', typname => 'pg_ddl_command', typlen => 'SIZEOF_POINTER', typbyval => 't', diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h index 5af96fdc8a..775a2bd1c6 100644 --- a/src/include/commands/vacuum.h +++ b/src/include/commands/vacuum.h @@ -198,6 +198,12 @@ extern void analyze_rel(Oid relid, RangeVar *relation, int options, VacuumParams *params, List *va_cols, bool in_outer_xact, BufferAccessStrategy bstrategy); extern bool std_typanalyze(VacAttrStats *stats); +extern int analyze_mcv_list(int *mcv_counts, + int num_mcv, + double stadistinct, + double stanullfrac, + int samplerows, + double totalrows); /* in utils/misc/sampling.c --- duplicate of declarations in utils/sampling.h */ extern double anl_random_fract(void); diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h index 77ca7ff837..e6cded0597 100644 --- a/src/include/optimizer/cost.h +++ b/src/include/optimizer/cost.h @@ -215,6 +215,12 @@ extern Selectivity clause_selectivity(PlannerInfo *root, int varRelid, JoinType jointype, SpecialJoinInfo *sjinfo); +extern Selectivity clauselist_selectivity_simple(PlannerInfo *root, + List *clauses, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + Bitmapset *estimatedclauses); extern void cost_gather_merge(GatherMergePath *path, PlannerInfo *root, RelOptInfo *rel, ParamPathInfo *param_info, Cost input_startup_cost, Cost input_total_cost, diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h index b3ca0c1229..11159b58ee 100644 --- a/src/include/statistics/extended_stats_internal.h +++ b/src/include/statistics/extended_stats_internal.h @@ -31,6 +31,15 @@ typedef struct int tupno; /* position index for tuple it came from */ } ScalarItem; +/* (de)serialization info */ +typedef struct DimensionInfo +{ + int nvalues; /* number of deduplicated values */ + int nbytes; /* number of bytes (serialized) */ + int typlen; /* pg_type.typlen */ + bool typbyval; /* pg_type.typbyval */ +} DimensionInfo; + /* multi-sort */ typedef struct MultiSortSupportData { @@ -44,6 +53,7 @@ typedef struct SortItem { Datum *values; bool *isnull; + int count; } SortItem; extern MVNDistinct *statext_ndistinct_build(double totalrows, @@ -57,6 +67,12 @@ extern MVDependencies *statext_dependencies_build(int numrows, HeapTuple *rows, extern bytea *statext_dependencies_serialize(MVDependencies *dependencies); extern MVDependencies *statext_dependencies_deserialize(bytea *data); +extern MCVList * statext_mcv_build(int numrows, HeapTuple *rows, + Bitmapset *attrs, VacAttrStats **stats, + double totalrows); +extern bytea *statext_mcv_serialize(MCVList * mcv, VacAttrStats **stats); +extern MCVList * statext_mcv_deserialize(bytea *data); + extern MultiSortSupport multi_sort_init(int ndims); extern void multi_sort_add_dimension(MultiSortSupport mss, int sortdim, Oid oper); @@ -65,5 +81,32 @@ extern int multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b, MultiSortSupport mss); extern int multi_sort_compare_dims(int start, int end, const SortItem *a, const SortItem *b, MultiSortSupport mss); +extern int compare_scalars_simple(const void *a, const void *b, void *arg); +extern int compare_datums_simple(Datum a, Datum b, SortSupport ssup); + +extern void *bsearch_arg(const void *key, const void *base, + size_t nmemb, size_t size, + int (*compar) (const void *, const void *, void *), + void *arg); + +extern int *build_attnums(Bitmapset *attrs); + +extern SortItem *build_sorted_items(int numrows, HeapTuple *rows, + TupleDesc tdesc, MultiSortSupport mss, + int numattrs, int *attnums); + +extern int bms_member_index(Bitmapset *keys, AttrNumber varattno); + +extern double estimate_ndistinct(double totalrows, int numrows, int d, int f1); + +extern Selectivity mcv_clauselist_selectivity(PlannerInfo *root, + StatisticExtInfo *stat, + List *clauses, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + RelOptInfo *rel, + Selectivity *basesel, + Selectivity *totalsel); #endif /* EXTENDED_STATS_INTERNAL_H */ diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h index 8009fee322..e69d6a0232 100644 --- a/src/include/statistics/statistics.h +++ b/src/include/statistics/statistics.h @@ -16,6 +16,14 @@ #include "commands/vacuum.h" #include "nodes/relation.h" +/* + * Degree of how much MCV item matches a clause. + * This is then considered when computing the selectivity. + */ +#define STATS_MATCH_NONE 0 /* no match at all */ +#define STATS_MATCH_PARTIAL 1 /* partial match */ +#define STATS_MATCH_FULL 2 /* full match */ + #define STATS_MAX_DIMENSIONS 8 /* max number of attributes */ /* Multivariate distinct coefficients */ @@ -78,8 +86,42 @@ typedef struct MVDependencies /* size of the struct excluding the deps array */ #define SizeOfDependencies (offsetof(MVDependencies, ndeps) + sizeof(uint32)) +/* used to flag stats serialized to bytea */ +#define STATS_MCV_MAGIC 0xE1A651C2 /* marks serialized + * bytea */ +#define STATS_MCV_TYPE_BASIC 1 /* basic MCV list type */ + +/* max items in MCV list (mostly arbitrary number) */ +#define STATS_MCVLIST_MAX_ITEMS 8192 + +/* + * Multivariate MCV (most-common value) lists + * + * A straight-forward extension of MCV items - i.e. a list (array) of + * combinations of attribute values, together with a frequency and null flags. + */ +typedef struct MCVItem +{ + double frequency; /* frequency of this combination */ + double base_frequency; /* frequency if independent */ + bool *isnull; /* lags of NULL values (up to 32 columns) */ + Datum *values; /* variable-length (ndimensions) */ +} MCVItem; + +/* multivariate MCV list - essentally an array of MCV items */ +typedef struct MCVList +{ + uint32 magic; /* magic constant marker */ + uint32 type; /* type of MCV list (BASIC) */ + uint32 nitems; /* number of MCV items in the array */ + AttrNumber ndimensions; /* number of dimensions */ + Oid types[STATS_MAX_DIMENSIONS]; /* OIDs of data types */ + MCVItem **items; /* array of MCV items */ +} MCVList; + extern MVNDistinct *statext_ndistinct_load(Oid mvoid); extern MVDependencies *statext_dependencies_load(Oid mvoid); +extern MCVList * statext_mcv_load(Oid mvoid); extern void BuildRelationExtStatistics(Relation onerel, double totalrows, int numrows, HeapTuple *rows, @@ -92,6 +134,13 @@ extern Selectivity dependencies_clauselist_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo, RelOptInfo *rel, Bitmapset **estimatedclauses); +extern Selectivity statext_clauselist_selectivity(PlannerInfo *root, + List *clauses, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + RelOptInfo *rel, + Bitmapset **estimatedclauses); extern bool has_stats_of_kind(List *stats, char requiredkind); extern StatisticExtInfo *choose_best_statistics(List *stats, Bitmapset *attnums, char requiredkind); diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h index 95e44280c4..4e9aaca6b5 100644 --- a/src/include/utils/selfuncs.h +++ b/src/include/utils/selfuncs.h @@ -209,6 +209,8 @@ extern void mergejoinscansel(PlannerInfo *root, Node *clause, extern double estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, List **pgset); +extern double estimate_num_groups_simple(PlannerInfo *root, List *vars); + extern void estimate_hash_bucket_stats(PlannerInfo *root, Node *hashkey, double nbuckets, Selectivity *mcv_freq, diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out index 8d4543bfe8..0f97355165 100644 --- a/src/test/regress/expected/create_table_like.out +++ b/src/test/regress/expected/create_table_like.out @@ -243,7 +243,7 @@ Indexes: Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Statistics objects: - "public"."ctlt_all_a_b_stat" (ndistinct, dependencies) ON a, b FROM ctlt_all + "public"."ctlt_all_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM ctlt_all SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid; relname | objsubid | description diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out index 3c6d853ffb..d13f0928d0 100644 --- a/src/test/regress/expected/opr_sanity.out +++ b/src/test/regress/expected/opr_sanity.out @@ -902,11 +902,12 @@ WHERE c.castmethod = 'b' AND pg_node_tree | text | 0 | i pg_ndistinct | bytea | 0 | i pg_dependencies | bytea | 0 | i + pg_mcv_list | bytea | 0 | i cidr | inet | 0 | i xml | text | 0 | a xml | character varying | 0 | a xml | character | 0 | a -(9 rows) +(10 rows) -- **************** pg_conversion **************** -- Look for illegal values in pg_conversion fields. diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 054a381dad..5d05962c04 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -58,7 +58,7 @@ ALTER TABLE ab1 DROP COLUMN a; b | integer | | | c | integer | | | Statistics objects: - "public"."ab1_b_c_stats" (ndistinct, dependencies) ON b, c FROM ab1 + "public"."ab1_b_c_stats" (ndistinct, dependencies, mcv) ON b, c FROM ab1 -- Ensure statistics are dropped when table is SELECT stxname FROM pg_statistic_ext WHERE stxname LIKE 'ab1%'; @@ -206,7 +206,7 @@ SELECT stxkind, stxndistinct FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass; stxkind | stxndistinct ---------+--------------------------------------------------------- - {d,f} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301} + {d,f,m} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301} (1 row) -- Hash Aggregate, thanks to estimates improved by the statistic @@ -272,7 +272,7 @@ SELECT stxkind, stxndistinct FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass; stxkind | stxndistinct ---------+------------------------------------------------------------- - {d,f} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000} + {d,f,m} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000} (1 row) -- plans using Group Aggregate, thanks to using correct esimates @@ -509,3 +509,316 @@ EXPLAIN (COSTS OFF) (5 rows) RESET random_page_cost; +-- MCV lists +CREATE TABLE mcv_lists ( + filler1 TEXT, + filler2 NUMERIC, + a INT, + b TEXT, + filler3 DATE, + c INT, + d TEXT +); +SET random_page_cost = 1.2; +CREATE INDEX mcv_lists_ab_idx ON mcv_lists (a, b); +CREATE INDEX mcv_lists_abc_idx ON mcv_lists (a, b, c); +-- random data (no MCV list) +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a = 1) AND (b = '1'::text)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a = 1) AND (b = '1'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + QUERY PLAN +--------------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1)) +(2 rows) + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists; +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a = 1) AND (b = '1'::text)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a = 1) AND (b = '1'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + QUERY PLAN +--------------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1)) +(2 rows) + +-- 100 distinct combinations, all in the MCV list +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + QUERY PLAN +------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a = 1) AND (b = '1'::text)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 1 AND b < '1'; + QUERY PLAN +------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a < 1) AND (b < '1'::text)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0'; + QUERY PLAN +------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a <= 0) AND (b <= '0'::text)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + QUERY PLAN +--------------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5; + QUERY PLAN +--------------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a < 5) AND (b < '1'::text) AND (c < 5)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4; + QUERY PLAN +------------------------------------------------------------ + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a <= 4) AND (b <= '0'::text) AND (c <= 4)) +(2 rows) + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists; +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a = 1) AND (b = '1'::text)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a = 1) AND (b = '1'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 1 AND b < '1'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a < 1) AND (b < '1'::text)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a < 1) AND (b < '1'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0'; + QUERY PLAN +----------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a <= 0) AND (b <= '0'::text)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a <= 0) AND (b <= '0'::text)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a = 1) AND (b = '1'::text)) + Filter: (c = 1) + -> Bitmap Index Scan on mcv_lists_ab_idx + Index Cond: ((a = 1) AND (b = '1'::text)) +(5 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a < 5) AND (b < '1'::text)) + Filter: (c < 5) + -> Bitmap Index Scan on mcv_lists_ab_idx + Index Cond: ((a < 5) AND (b < '1'::text)) +(5 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4; + QUERY PLAN +----------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a <= 4) AND (b <= '0'::text)) + Filter: (c <= 4) + -> Bitmap Index Scan on mcv_lists_ab_idx + Index Cond: ((a <= 4) AND (b <= '0'::text)) +(5 rows) + +-- check change of column type resets the MCV statistics +ALTER TABLE mcv_lists ALTER COLUMN c TYPE numeric; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + QUERY PLAN +------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a = 1) AND (b = '1'::text)) +(2 rows) + +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a = 1) AND (b = '1'::text)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a = 1) AND (b = '1'::text)) +(4 rows) + +-- 100 distinct combinations with NULL values, all in the MCV list +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT + (CASE WHEN mod(i,100) = 1 THEN NULL ELSE mod(i,100) END), + (CASE WHEN mod(i,50) = 1 THEN NULL ELSE mod(i,50) END), + (CASE WHEN mod(i,25) = 1 THEN NULL ELSE mod(i,25) END), + i + FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL; + QUERY PLAN +------------------------------------------------- + Index Scan using mcv_lists_abc_idx on mcv_lists + Index Cond: ((a IS NULL) AND (b IS NULL)) +(2 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL; + QUERY PLAN +------------------------------------------------ + Index Scan using mcv_lists_ab_idx on mcv_lists + Index Cond: ((a IS NULL) AND (b IS NULL)) + Filter: (c IS NULL) +(3 rows) + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists; +ANALYZE mcv_lists; +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a IS NULL) AND (b IS NULL)) + -> Bitmap Index Scan on mcv_lists_abc_idx + Index Cond: ((a IS NULL) AND (b IS NULL)) +(4 rows) + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_lists + Recheck Cond: ((a IS NULL) AND (b IS NULL)) + Filter: (c IS NULL) + -> Bitmap Index Scan on mcv_lists_ab_idx + Index Cond: ((a IS NULL) AND (b IS NULL)) +(5 rows) + +RESET random_page_cost; +-- mcv with arrays +CREATE TABLE mcv_lists_arrays ( + a TEXT[], + b NUMERIC[], + c INT[] +); +INSERT INTO mcv_lists_arrays (a, b, c) + SELECT + ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)], + ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000], + ARRAY[(i/100-1), i/100, (i/100+1)] + FROM generate_series(1,5000) s(i); +CREATE STATISTICS mcv_lists_arrays_stats (mcv) ON a, b, c + FROM mcv_lists_arrays; +ANALYZE mcv_lists_arrays; +-- mcv with bool +CREATE TABLE mcv_lists_bool ( + a BOOL, + b BOOL, + c BOOL +); +INSERT INTO mcv_lists_bool (a, b, c) + SELECT + (mod(i,2) = 0), (mod(i,4) = 0), (mod(i,8) = 0) + FROM generate_series(1,10000) s(i); +CREATE INDEX mcv_lists_bool_ab_idx ON mcv_lists_bool (a, b); +CREATE INDEX mcv_lists_bool_abc_idx ON mcv_lists_bool (a, b, c); +CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c + FROM mcv_lists_bool; +ANALYZE mcv_lists_bool; +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE a AND b AND c; + QUERY PLAN +---------------------------------------------------------------- + Bitmap Heap Scan on mcv_lists_bool + Filter: (a AND b AND c) + -> Bitmap Index Scan on mcv_lists_bool_abc_idx + Index Cond: ((a = true) AND (b = true) AND (c = true)) +(4 rows) + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c; + QUERY PLAN +---------------------------------------------------------- + Index Scan using mcv_lists_bool_ab_idx on mcv_lists_bool + Index Cond: ((a = false) AND (b = true)) + Filter: ((NOT a) AND b AND c) +(3 rows) + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c; + QUERY PLAN +---------------------------------------------------------------- + Index Only Scan using mcv_lists_bool_abc_idx on mcv_lists_bool + Index Cond: ((a = false) AND (b = false) AND (c = true)) + Filter: ((NOT a) AND (NOT b) AND c) +(3 rows) + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c; + QUERY PLAN +---------------------------------------------------------- + Index Scan using mcv_lists_bool_ab_idx on mcv_lists_bool + Index Cond: ((a = false) AND (b = true)) + Filter: ((NOT a) AND b AND (NOT c)) +(3 rows) + diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out index b1419d4bc2..a56d6c5231 100644 --- a/src/test/regress/expected/type_sanity.out +++ b/src/test/regress/expected/type_sanity.out @@ -72,8 +72,9 @@ WHERE p1.typtype not in ('c','d','p') AND p1.typname NOT LIKE E'\\_%' 194 | pg_node_tree 3361 | pg_ndistinct 3402 | pg_dependencies + 4001 | pg_mcv_list 210 | smgr -(4 rows) +(5 rows) -- Make sure typarray points to a varlena array type of our own base SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype, diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index 46acaadb39..ad1f103217 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -282,3 +282,184 @@ EXPLAIN (COSTS OFF) SELECT * FROM functional_dependencies WHERE a = 1 AND b = '1' AND c = 1; RESET random_page_cost; + +-- MCV lists +CREATE TABLE mcv_lists ( + filler1 TEXT, + filler2 NUMERIC, + a INT, + b TEXT, + filler3 DATE, + c INT, + d TEXT +); + +SET random_page_cost = 1.2; + +CREATE INDEX mcv_lists_ab_idx ON mcv_lists (a, b); +CREATE INDEX mcv_lists_abc_idx ON mcv_lists (a, b, c); + +-- random data (no MCV list) +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists; + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + +-- 100 distinct combinations, all in the MCV list +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; + +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 1 AND b < '1'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4; + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists; + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 1 AND b < '1'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0'; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4; + +-- check change of column type resets the MCV statistics +ALTER TABLE mcv_lists ALTER COLUMN c TYPE numeric; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a = 1 AND b = '1'; + +-- 100 distinct combinations with NULL values, all in the MCV list +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; + +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT + (CASE WHEN mod(i,100) = 1 THEN NULL ELSE mod(i,100) END), + (CASE WHEN mod(i,50) = 1 THEN NULL ELSE mod(i,50) END), + (CASE WHEN mod(i,25) = 1 THEN NULL ELSE mod(i,25) END), + i + FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL; + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists; + +ANALYZE mcv_lists; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL; + +EXPLAIN (COSTS OFF) + SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL; + +RESET random_page_cost; + +-- mcv with arrays +CREATE TABLE mcv_lists_arrays ( + a TEXT[], + b NUMERIC[], + c INT[] +); + +INSERT INTO mcv_lists_arrays (a, b, c) + SELECT + ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)], + ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000], + ARRAY[(i/100-1), i/100, (i/100+1)] + FROM generate_series(1,5000) s(i); + +CREATE STATISTICS mcv_lists_arrays_stats (mcv) ON a, b, c + FROM mcv_lists_arrays; + +ANALYZE mcv_lists_arrays; + +-- mcv with bool +CREATE TABLE mcv_lists_bool ( + a BOOL, + b BOOL, + c BOOL +); + +INSERT INTO mcv_lists_bool (a, b, c) + SELECT + (mod(i,2) = 0), (mod(i,4) = 0), (mod(i,8) = 0) + FROM generate_series(1,10000) s(i); + +CREATE INDEX mcv_lists_bool_ab_idx ON mcv_lists_bool (a, b); + +CREATE INDEX mcv_lists_bool_abc_idx ON mcv_lists_bool (a, b, c); + +CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c + FROM mcv_lists_bool; + +ANALYZE mcv_lists_bool; + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE a AND b AND c; + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c; + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c; + +EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c; -- 2.13.6 --------------0E1BE69E814B19BC8B911192--