pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Greg Stark <stark@mit.edu>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: PATCH: Using BRIN indexes for sorted output
Date: Mon, 10 Jul 2023 17:09:44 +0200
Message-ID: <1992ff4f-bf0e-3ad0-a859-cf054f6abc74@enterprisedb.com> (raw)
In-Reply-To: <CAEze2Wj_UjWdpMfDSpYWXFnr2K2yX9zpihsD4iEZ9+JFMLOknw@mail.gmail.com>
References: <1607e037-8c5b-1626-9f98-496f067b9eb5@enterprisedb.com>
	<14568731-ac91-ce2d-1477-0e2504c72775@enterprisedb.com>
	<20221206185143.ivy3afwmkhloksgv@awork3.anarazel.de>
	<4e679562-fd9a-33a3-ce71-cd7a27525f8e@enterprisedb.com>
	<e0b6e395-3ab7-6021-b058-3c5901912b36@enterprisedb.com>
	<20230216161016.GV1653@telsasoft.com>
	<9f73f798-1dac-c75d-d834-aa012ba13b65@enterprisedb.com>
	<8f1374d1-6f4b-ce99-84eb-ffa852c0dadb@enterprisedb.com>
	<CAEze2Whd=MCQ61EEUUWJgohHN5cm3bv027N+vS4o6zOuBTY3tw@mail.gmail.com>
	<30f37aae-57fa-e7ab-9444-7b82969ed68f@enterprisedb.com>
	<CAEze2WgGueCY+ihQDASaKAWNG9C2Ex0vzoNU5inrMLwWyj3KMA@mail.gmail.com>
	<CAEze2Wh5VsGQUyg9FC+xZBLUPLZnm0pOym5uceOM0aG1YnuESA@mail.gmail.com>
	<825f648f-2a31-1870-942f-03fba2b6b3cf@enterprisedb.com>
	<CAEze2WjWaP+vNC9Y_y0vVAWiX9NF7VtXDtY8+cCd27TQiCSsTg@mail.gmail.com>
	<0c331bf8-0cd3-e9fe-ae81-9e3d68f1f95c@enterprisedb.com>
	<CAEze2Wj_UjWdpMfDSpYWXFnr2K2yX9zpihsD4iEZ9+JFMLOknw@mail.gmail.com>

On 7/10/23 14:38, Matthias van de Meent wrote:
> On Mon, 10 Jul 2023 at 13:43, Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>> On 7/10/23 12:22, Matthias van de Meent wrote:
>>> On Fri, 7 Jul 2023 at 20:26, Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>> However, it's not quite clear to me is what you mean by the weight- and
>>>> compare-operators? That is, what are
>>>>
>>>>  - brin_minmax_minorder(PG_FUNCTION_ARGS=brintuple) -> range.min
>>>>  - brin_minmax_maxorder(PG_FUNCTION_ARGS=brintuple) -> range.max
>>>>  - brin_minmax_compare(order, order) -> int
>>>>
>>>> supposed to do? Or what does "PG_FUNCTION_ARGS=brintuple" mean?
>>>
>>> _minorder/_maxorder is for extracting the minimum/maximum relative
>>> order of each range, used for ASC/DESC sorting of operator results
>>> (e.g. to support ORDER BY <->(box_column, '(1,2)'::point) DESC).
>>> PG_FUNCTION_ARGS is mentioned because of PG calling conventions;
>>> though I did forget to describe the second operator argument for the
>>> distance function. We might also want to use only one such "order
>>> extraction function" with DESC/ASC indicated by an argument.
>>>
>>
>> I'm still not sure I understand what "minimum/maximum relative order"
>> is. Isn't it the same as returning min/max value that can appear in the
>> range? Although, that wouldn't work for points/boxes.
> 
> Kind of. For single-dimensional opclasses (minmax, minmax_multi) we
> only need to extract the normal min/max values for ASC/DESC sorts,
> which are readily available in the summary. But for multi-dimensional
> and distance searches (nearest neighbour) we need to calculate the
> distance between the indexed value(s) and the origin value to compare
> the summary against, and the order would thus be asc/desc on distance
> - a distance which may not be precisely represented by float types -
> thus 'relative order' with its own order operation.
> 

Can you give some examples of such data / queries, and how would it
leverage the BRIN sort stuff?

For distance searches, I imagine this as data indexed by BRIN inclusion
opclass, which creates a bounding box. We could return closest/furthest
point on the bounding box (from the point used in the query). Which
seems a bit like a R-tree ...

But I have no idea what would this do for multi-dimensional searches, or
what would those searches do? How would you sort such data other than
lexicographically? Which I think is covered by the current BRIN Sort,
because the data is either stored as multiple columns, in which case we
use the BRIN on the first column. Or it's indexed using BRIN minmax as a
tuple of values, but then it's sorted lexicographically.


>>>> In principle we just need a procedure that tells us min/max for a given
>>>> page range - I guess that's what the minorder/maxorder functions do? But
>>>> why would we need the compare one? We're comparing by the known data
>>>> type, so we can just delegate the comparison to that, no?
>>>
>>> Is there a comparison function for any custom orderable type that we
>>> can just use? GIST distance ordering uses floats, and I don't quite
>>> like that from a user perspective, as it makes ordering operations
>>> imprecise. I'd rather allow (but discourage) any type with its own
>>> compare function.
>>>
>>
>> I haven't really thought about geometric types, just about minmax and
>> minmax-multi. It's not clear to me what the benefit for these types be.
>> I mean, we can probably sort points lexicographically, but is anyone
>> doing that in queries? It seems useless for order by distance.
> 
> Yes, that's why you would sort them by distance, where the distance is
> generated by the opclass as min/max distance between the summary and
> the distance's origin, and then inserted into the tuplesort.
> 

OK, so the query says "order by distance from point X" and we calculate
the min/max distance of values in a given page range.

> (previously)
>>>> I finally had time to look at this patch again. There's a bit of bitrot,
>>>> so here's a rebased version (no other changes).
> 
> It seems like you forgot to attach the rebased patch, so unless you're
> actively working on updating the patchset right now, could you send
> the rebase to make CFBot happy?
> 

Yeah, I forgot about the attachment. So here it is.



regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Attachments: download all (11 files, .tar.gz)

0001-Allow-index-AMs-to-build-and-use-custom-sta-20230710.patch (text/x-patch, 63.4K) download From 0b99eec03885b95d5c748d0709db489f82d02856 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Mon, 17 Oct 2022 18:39:28 +0200 Subject: [PATCH 01/11] Allow index AMs to build and use custom statistics Some indexing AMs work very differently and estimating them using existing statistics is problematic, producing unreliable costing. This applies e.g. to BRIN, which relies on page ranges, not tuple pointers. This adds an optional AM procedure, allowing the opfamily to build custom statistics, store them in pg_statistic and then use them during planning. By default this is disabled, but may be enabled by setting SET enable_indexam_stats = true; Then ANALYZE will call the optional procedure for all indexes. --- src/backend/access/brin/brin.c | 1 + src/backend/access/brin/brin_minmax.c | 902 ++++++++++++++++++ src/backend/commands/analyze.c | 149 ++- src/backend/statistics/extended_stats.c | 2 + src/backend/utils/adt/selfuncs.c | 59 ++ src/backend/utils/cache/lsyscache.c | 41 + src/backend/utils/misc/guc_tables.c | 10 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/access/amapi.h | 2 + src/include/access/brin.h | 63 ++ src/include/access/brin_internal.h | 1 + src/include/catalog/pg_amproc.dat | 64 ++ src/include/catalog/pg_proc.dat | 4 + src/include/catalog/pg_statistic.h | 5 + src/include/commands/vacuum.h | 2 + src/include/utils/lsyscache.h | 1 + src/test/regress/expected/sysviews.out | 3 +- 17 files changed, 1304 insertions(+), 6 deletions(-) diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 3c6a956eaa..5758afdd83 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -95,6 +95,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amstrategies = 0; amroutine->amsupport = BRIN_LAST_OPTIONAL_PROCNUM; amroutine->amoptsprocnum = BRIN_PROCNUM_OPTIONS; + amroutine->amstatsprocnum = BRIN_PROCNUM_STATISTICS; amroutine->amcanorder = false; amroutine->amcanorderbyop = false; amroutine->amcanbackward = false; diff --git a/src/backend/access/brin/brin_minmax.c b/src/backend/access/brin/brin_minmax.c index 8229493c84..64150204fc 100644 --- a/src/backend/access/brin/brin_minmax.c +++ b/src/backend/access/brin/brin_minmax.c @@ -10,17 +10,23 @@ */ #include "postgres.h" +#include "access/brin.h" #include "access/brin_internal.h" +#include "access/brin_revmap.h" #include "access/brin_tuple.h" #include "access/genam.h" #include "access/stratnum.h" #include "catalog/pg_amop.h" #include "catalog/pg_type.h" +#include "executor/executor.h" +#include "miscadmin.h" +#include "storage/bufmgr.h" #include "utils/builtins.h" #include "utils/datum.h" #include "utils/lsyscache.h" #include "utils/rel.h" #include "utils/syscache.h" +#include "utils/timestamp.h" typedef struct MinmaxOpaque { @@ -253,6 +259,902 @@ brin_minmax_union(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +/* FIXME copy of a private struct from brin.c */ +typedef struct BrinOpaque +{ + BlockNumber bo_pagesPerRange; + BrinRevmap *bo_rmAccess; + BrinDesc *bo_bdesc; +} BrinOpaque; + +/* + * Compare ranges by minval (collation and operator are taken from the extra + * argument, which is expected to be TypeCacheEntry). + */ +static int +range_minval_cmp(const void *a, const void *b, void *arg) +{ + BrinRange *ra = *(BrinRange **) a; + BrinRange *rb = *(BrinRange **) b; + TypeCacheEntry *typentry = (TypeCacheEntry *) arg; + FmgrInfo *cmpfunc = &typentry->cmp_proc_finfo; + Datum c; + int r; + + c = FunctionCall2Coll(cmpfunc, typentry->typcollation, + ra->min_value, rb->min_value); + r = DatumGetInt32(c); + + if (r != 0) + return r; + + if (ra->blkno_start < rb->blkno_start) + return -1; + else + return 1; +} + +/* + * Compare ranges by maxval (collation and operator are taken from the extra + * argument, which is expected to be TypeCacheEntry). + */ +static int +range_maxval_cmp(const void *a, const void *b, void *arg) +{ + BrinRange *ra = *(BrinRange **) a; + BrinRange *rb = *(BrinRange **) b; + TypeCacheEntry *typentry = (TypeCacheEntry *) arg; + FmgrInfo *cmpfunc = &typentry->cmp_proc_finfo; + Datum c; + int r; + + c = FunctionCall2Coll(cmpfunc, typentry->typcollation, + ra->max_value, rb->max_value); + r = DatumGetInt32(c); + + if (r != 0) + return r; + + if (ra->blkno_start < rb->blkno_start) + return -1; + else + return 1; +} + +/* compare values using an operator from typcache */ +static int +range_values_cmp(const void *a, const void *b, void *arg) +{ + Datum da = * (Datum *) a; + Datum db = * (Datum *) b; + TypeCacheEntry *typentry = (TypeCacheEntry *) arg; + FmgrInfo *cmpfunc = &typentry->cmp_proc_finfo; + Datum c; + + c = FunctionCall2Coll(cmpfunc, typentry->typcollation, + da, db); + return DatumGetInt32(c); +} + +/* + * minval_end + * Determine first index so that (minval > value). + * + * The array of ranges is expected to be sorted by minvalue, so this is the first + * range that can't possibly intersect with a range having "value" as maxval. + */ +static int +minval_end(BrinRange **ranges, int nranges, Datum value, TypeCacheEntry *typcache) +{ + int start = 0, + end = (nranges - 1); + + // everything matches + if (range_values_cmp(&value, &ranges[end]->min_value, typcache) >= 0) + return nranges; + + // no matches + if (range_values_cmp(&value, &ranges[start]->min_value, typcache) < 0) + return 0; + + while ((end - start) > 0) + { + int midpoint; + int r; + + midpoint = start + (end - start) / 2; + + r = range_values_cmp(&value, &ranges[midpoint]->min_value, typcache); + + if (r >= 0) + start = midpoint + 1; + else + end = midpoint; + } + + Assert(range_values_cmp(&ranges[start]->min_value, &value, typcache) > 0); + Assert(range_values_cmp(&ranges[start-1]->min_value, &value, typcache) <= 0); + + return start; +} + + +/* + * lower_bound + * Determine first index so that (values[index] >= value). + * + * The array of values is sorted, and this returns the first value that + * exceeds (or is equal) to the minvalue. + */ +static int +lower_bound(Datum *values, int nvalues, Datum minvalue, TypeCacheEntry *typcache) +{ + int start = 0, + end = (nvalues - 1); + + /* all values exceed minvalue - return the first element */ + if (range_values_cmp(&minvalue, &values[start], typcache) <= 0) + return 0; + + /* nothing matches - return the element after the last one */ + if (range_values_cmp(&minvalue, &values[end], typcache) > 0) + return nvalues; + + /* + * Now we know the lower boundary is somewhere in the array (and we know + * it's not the first element, because that's covered by the first check + * above). So do a binary search. + */ + while ((end - start) > 0) + { + int midpoint; + int r; + + midpoint = start + (end - start) / 2; + + r = range_values_cmp(&minvalue, &values[midpoint], typcache); + + if (r <= 0) /* minvalue >= midpoint */ + end = midpoint; + else /* midpoint < minvalue */ + start = (midpoint + 1); + } + + Assert(range_values_cmp(&minvalue, &values[start], typcache) <= 0); + Assert(range_values_cmp(&minvalue, &values[start-1], typcache) > 0); + + return start; +} + +/* + * upper_bound + * Determine last index so that (values[index] <= maxvalue). + * + * The array of values is sorted, and this returns the last value that + * does not exceed (or is equal) to the maxvalue. + */ +static int +upper_bound(Datum *values, int nvalues, Datum maxvalue, TypeCacheEntry *typcache) +{ + int start = 0, + end = (nvalues - 1); + + /* everything matches, return the last element */ + if (range_values_cmp(&values[end], &maxvalue, typcache) <= 0) + return (nvalues - 1); + + /* nothing matches, return the element before the first one */ + if (range_values_cmp(&values[start], &maxvalue, typcache) > 0) + return -1; + + /* + * Now we know the lower boundary is somewhere in the array (and we know + * it's not the last element, because that's covered by the first check + * above). So do a binary search. + */ + while ((end - start) > 0) + { + int midpoint; + int r; + + midpoint = start + (end - start) / 2; + + /* Ensure we always move (it might be equal to start due to rounding). */ + midpoint = Max(start+1, midpoint); + + r = range_values_cmp(&values[midpoint], &maxvalue, typcache); + + if (r <= 0) /* value <= maxvalue */ + start = midpoint; + else /* value > maxvalue */ + end = midpoint - 1; + } + + Assert(range_values_cmp(&values[start], &maxvalue, typcache) <= 0); + Assert(range_values_cmp(&values[start+1], &maxvalue, typcache) > 0); + + return start; +} + +/* + * brin_minmax_count_overlaps + * Calculate number of overlaps. + * + * This uses the minranges to quickly eliminate ranges that can't possibly + * intersect. We simply walk minranges until minval > current maxval, and + * we're done. + * + * Unlike brin_minmax_count_overlaps2, this does not have issues with wide + * ranges, so this is what we should use. + */ +static void +brin_minmax_count_overlaps(BrinRange **minranges, int nranges, + TypeCacheEntry *typcache, BrinMinmaxStats *stats) +{ + int64 noverlaps; + + noverlaps = 0; + for (int i = 0; i < nranges; i++) + { + Datum maxval = minranges[i]->max_value; + + /* + * Determine index of the first range with (minval > current maxval) + * by binary search. We know all other ranges can't overlap the + * current one. We simply subtract indexes to count ranges. + */ + int idx = minval_end(minranges, nranges, maxval, typcache); + + /* -1 because we don't count the range as intersecting with itself */ + noverlaps += (idx - i - 1); + } + + /* + * We only count 1/2 the ranges (minval > current minval), so the total + * number of overlaps is twice what we counted. + */ + noverlaps *= 2; + + stats->avg_overlaps = (double) noverlaps / nranges; +} + +/* + * brin_minmax_match_tuples_to_ranges + * Match tuples to ranges, count average number of ranges per tuple. + * + * Alternative to brin_minmax_match_tuples_to_ranges2, leveraging ordering + * of values, not ranges. + * + * XXX This seems like the optimal way to do this. + */ +static void +brin_minmax_match_tuples_to_ranges(BrinRanges *ranges, + int numrows, HeapTuple *rows, + int nvalues, Datum *values, + TypeCacheEntry *typcache, + BrinMinmaxStats *stats) +{ + int64 nmatches = 0; + int64 nmatches_unique = 0; + int64 nvalues_unique = 0; + + int64 *unique = (int64 *) palloc0(sizeof(int64) * nvalues); + + /* + * Build running count of unique values. We know there are unique[i] + * unique values in values array up to index "i". + */ + unique[0] = 1; + for (int i = 1; i < nvalues; i++) + { + if (range_values_cmp(&values[i-1], &values[i], typcache) == 0) + unique[i] = unique[i-1]; + else + unique[i] = unique[i-1] + 1; + } + + nvalues_unique = unique[nvalues-1]; + + /* + * Walk the ranges, for each range determine the first/last mapping + * value. Use the "unique" array to count the unique values. + */ + for (int i = 0; i < ranges->nranges; i++) + { + int start, + end, + nvalues_match, + nunique_match; + + CHECK_FOR_INTERRUPTS(); + + start = lower_bound(values, nvalues, ranges->ranges[i].min_value, typcache); + end = upper_bound(values, nvalues, ranges->ranges[i].max_value, typcache); + + /* if nothing matches (e.g. end=0), skip this range */ + if (end <= start) + continue; + + nvalues_match = (end - start + 1); + nunique_match = (unique[end] - unique[start] + 1); + + Assert((nvalues_match >= 1) && (nvalues_match <= nvalues)); + Assert((nunique_match >= 1) && (nunique_match <= unique[nvalues-1])); + + nmatches += nvalues_match; + nmatches_unique += nunique_match; + } + + Assert(nmatches >= 0); + Assert(nmatches_unique >= 0); + + stats->avg_matches = (double) nmatches / numrows; + stats->avg_matches_unique = (double) nmatches_unique / nvalues_unique; +} + +/* + * brin_minmax_value_stats + * Calculate statistics about minval/maxval values. + * + * We calculate the number of distinct values, and also correlation with respect + * to blkno_start. We don't calculate the regular correlation coefficient, because + * our goal is to estimate how sequential the accesses are. The regular correlation + * would produce 0 for cyclical data sets like mod(i,1000000), but it may be quite + * sequantial access. Maybe it should be called differently, not correlation? + * + * XXX Maybe this should calculate minval vs. maxval correlation too? + * + * XXX I don't know how important the sequentiality is - BRIN generally uses 1MB + * page ranges, which is pretty sequential and the one random seek in between is + * likely going to be negligible. Maybe for small page ranges it'll matter, though. + */ +static void +brin_minmax_value_stats(BrinRange **minranges, BrinRange **maxranges, + int nranges, TypeCacheEntry *typcache, + BrinMinmaxStats *stats) +{ + /* */ + int64 minval_ndist = 1, + maxval_ndist = 1, + minval_corr = 0, + maxval_corr = 0; + + for (int i = 1; i < nranges; i++) + { + if (range_values_cmp(&minranges[i-1]->min_value, &minranges[i]->min_value, typcache) != 0) + minval_ndist++; + + if (range_values_cmp(&maxranges[i-1]->max_value, &maxranges[i]->max_value, typcache) != 0) + maxval_ndist++; + + /* is it immediately sequential? */ + if (minranges[i-1]->blkno_end + 1 == minranges[i]->blkno_start) + minval_corr++; + + /* is it immediately sequential? */ + if (maxranges[i-1]->blkno_end + 1 == maxranges[i]->blkno_start) + maxval_corr++; + } + + stats->minval_ndistinct = minval_ndist; + stats->maxval_ndistinct = maxval_ndist; + + stats->minval_correlation = (double) minval_corr / nranges; + stats->maxval_correlation = (double) maxval_corr / nranges; +} + +/* + * brin_minmax_increment_stats + * Calculate the increment size for minval/maxval steps. + * + * Calculates the minval/maxval increment size, i.e. number of rows that need + * to be added to the sort. This serves as an input to calculation of a good + * watermark step. + */ +static void +brin_minmax_increment_stats(BrinRange **minranges, BrinRange **maxranges, + int nranges, Datum *values, int nvalues, + TypeCacheEntry *typcache, BrinMinmaxStats *stats) +{ + /* */ + int64 minval_ndist = 1, + maxval_ndist = 1; + + double sum_minval = 0, + sum_maxval = 0, + max_minval = 0, + max_maxval = 0; + + for (int i = 1; i < nranges; i++) + { + if (range_values_cmp(&minranges[i-1]->min_value, &minranges[i]->min_value, typcache) != 0) + { + double p; + int start = upper_bound(values, nvalues, minranges[i-1]->min_value, typcache); + int end = upper_bound(values, nvalues, minranges[i]->min_value, typcache); + + /* + * Maybe there are no matching rows, but we still need to count + * this as distinct minval (even though the sample increase is 0). + */ + minval_ndist++; + + Assert(end >= start); + + /* no sample rows match this, so skip */ + if (end == start) + continue; + + p = (double) (end - start) / nvalues; + + max_minval = Max(max_minval, p); + sum_minval += p; + } + + if (range_values_cmp(&maxranges[i-1]->max_value, &maxranges[i]->max_value, typcache) != 0) + { + double p; + int start = upper_bound(values, nvalues, maxranges[i-1]->max_value, typcache); + int end = upper_bound(values, nvalues, maxranges[i]->max_value, typcache); + + /* + * Maybe there are no matching rows, but we still need to count + * this as distinct maxval (even though the sample increase is 0). + */ + maxval_ndist++; + + Assert(end >= start); + + /* no sample rows match this, so skip */ + if (end == start) + continue; + + p = (double) (end - start) / nvalues; + + max_maxval = Max(max_maxval, p); + sum_maxval += p; + } + } + + stats->minval_increment_avg = (sum_minval / minval_ndist); + stats->minval_increment_max = max_minval; + + stats->maxval_increment_avg = (sum_maxval / maxval_ndist); + stats->maxval_increment_max = max_maxval; +} + +/* + * brin_minmax_stats + * Calculate custom statistics for a BRIN minmax index. + * + * At the moment this calculates: + * + * - number of summarized/not-summarized and all/has nulls ranges + * - average number of overlaps for a range + * - average number of rows matching a range + * - number of distinct minval/maxval values + * + * XXX This could also calculate correlation of the range minval, so that + * we can estimate how much random I/O will happen during the BrinSort. + * And perhaps we should also sort the ranges by (minval,block_start) to + * make this as sequential as possible? + * + * XXX Another interesting statistics might be the number of ranges with + * the same minval (or number of distinct minval values), because that's + * essentially what we need to estimate how many ranges will be read in + * one brinsort step. In fact, knowing the number of distinct minval + * values tells us the number of BrinSort loops. + * + * XXX We might also calculate a histogram of minval/maxval values. + * + * XXX I wonder if we could track for each range track probabilities: + * + * - P1 = P(v <= minval) + * - P2 = P(x <= Max(maxval)) for Max(maxval) over preceding ranges + * + * That would allow us to estimate how many ranges we'll have to read to produce + * a particular number of rows, because we need the first probability to exceed + * the requested number of rows (fraction of the table): + * + * (limit rows / reltuples) <= P(v <= minval) + * + * and then the second probability would say how many rows we'll process (either + * sort or spill). And inversely for the DESC ordering. + * + * The difference between P1 for two ranges is how much we'd have to sort + * if we moved the watermark between the ranges (first minval to second one). + * The (P2 - P1) for the new watermark range measures the number of rows in + * the tuplestore. We'll need to aggregate this, though, we can't keep the + * whole data - probably average/median/max for the differences would be nice. + * Might be tricky for different watermark step values, though. + * + * This would also allow estimating how many rows will spill from each range, + * because we have an estimate how many rows match a range on average, and + * we can compare it to the difference between P1. + * + * One issue is we don't have actual tuples from the ranges, so we can't + * measure exactly how many rows would we add. But we can match the sample + * and at least estimate the the probability difference. + * + * Actually - we do know the tuples *are* in those ranges, because if we + * assume the tuple is in some other range, that range would have to have + * a minimal/maximal value so that the value is consistent. Which means + * the range has to be between those ranges. Of course, this only estimates + * the rows we'd going to add to the tuplesort - there might be more rows + * we read and spill to tuplestore, but that's something we can estimate + * using average tuples per range. + */ +Datum +brin_minmax_stats(PG_FUNCTION_ARGS) +{ + Relation heapRel = (Relation) PG_GETARG_POINTER(0); + Relation indexRel = (Relation) PG_GETARG_POINTER(1); + AttrNumber attnum = PG_GETARG_INT16(2); /* index attnum */ + AttrNumber heap_attnum = PG_GETARG_INT16(3); + Expr *expr = (Expr *) PG_GETARG_POINTER(4); + HeapTuple *rows = (HeapTuple *) PG_GETARG_POINTER(5); + int numrows = PG_GETARG_INT32(6); + + BrinOpaque *opaque; + BlockNumber nblocks; + BlockNumber nranges; + BlockNumber heapBlk; + BrinMemTuple *dtup; + BrinTuple *btup = NULL; + Size btupsz = 0; + Buffer buf = InvalidBuffer; + BrinRanges *ranges; + BlockNumber pagesPerRange; + BrinDesc *bdesc; + BrinMinmaxStats *stats; + Form_pg_attribute attr; + + Oid typoid; + TypeCacheEntry *typcache; + BrinRange **minranges, + **maxranges; + int64 prev_min_index; + + /* expression stats */ + EState *estate; + ExprContext *econtext; + ExprState *exprstate; + TupleTableSlot *slot; + + /* attnum or expression has to be supplied */ + Assert(AttributeNumberIsValid(heap_attnum) || (expr != NULL)); + + /* but not both of them at the same time */ + Assert(!(AttributeNumberIsValid(heap_attnum) && (expr != NULL))); + + /* + * Mostly what brinbeginscan does to initialize BrinOpaque, except that + * we use active snapshot instead of the scan snapshot. + */ + opaque = palloc_object(BrinOpaque); + opaque->bo_rmAccess = brinRevmapInitialize(indexRel, + &opaque->bo_pagesPerRange, + GetActiveSnapshot()); + opaque->bo_bdesc = brin_build_desc(indexRel); + + bdesc = opaque->bo_bdesc; + pagesPerRange = opaque->bo_pagesPerRange; + + /* make sure the provided attnum is valid */ + Assert((attnum > 0) && (attnum <= bdesc->bd_tupdesc->natts)); + + /* attribute information */ + attr = TupleDescAttr(bdesc->bd_tupdesc, attnum - 1); + + /* + * We need to know the size of the table so that we know how long to iterate + * on the revmap (and to pre-allocate the arrays). + */ + nblocks = RelationGetNumberOfBlocks(heapRel); + + /* + * How many ranges can there be? We simply look at the number of pages, + * divide it by the pages_per_range. + * + * XXX We need to be careful not to overflow nranges, so we just divide + * and then maybe add 1 for partial ranges. + */ + nranges = (nblocks / pagesPerRange); + if (nblocks % pagesPerRange != 0) + nranges += 1; + + /* allocate for space, and also for the alternative ordering */ + ranges = palloc0(offsetof(BrinRanges, ranges) + nranges * sizeof(BrinRange)); + ranges->nranges = 0; + + /* allocate an initial in-memory tuple, out of the per-range memcxt */ + dtup = brin_new_memtuple(bdesc); + + /* result stats */ + stats = palloc0(sizeof(BrinMinmaxStats)); + SET_VARSIZE(stats, sizeof(BrinMinmaxStats)); + + /* + * Now scan the revmap. We start by querying for heap page 0, + * incrementing by the number of pages per range; this gives us a full + * view of the table. + * + * XXX We count the ranges, and count the special types (not summarized, + * all-null and has-null). The regular ranges are accumulated into an + * array, so that we can calculate additional statistics (overlaps, hits + * for sample tuples, etc). + * + * XXX This needs rethinking to make it work with large indexes with more + * ranges than we can fit into memory (work_mem/maintenance_work_mem). + */ + for (heapBlk = 0; heapBlk < nblocks; heapBlk += pagesPerRange) + { + bool gottuple = false; + BrinTuple *tup; + OffsetNumber off; + Size size; + + stats->n_ranges++; + + CHECK_FOR_INTERRUPTS(); + + tup = brinGetTupleForHeapBlock(opaque->bo_rmAccess, heapBlk, &buf, + &off, &size, BUFFER_LOCK_SHARE, + GetActiveSnapshot()); + if (tup) + { + gottuple = true; + btup = brin_copy_tuple(tup, size, btup, &btupsz); + LockBuffer(buf, BUFFER_LOCK_UNLOCK); + } + + /* Ranges with no indexed tuple are ignored for overlap analysis. */ + if (!gottuple) + { + continue; + } + else + { + dtup = brin_deform_tuple(bdesc, btup, dtup); + if (dtup->bt_placeholder) + { + /* Placeholders can be ignored too, as if not summarized. */ + continue; + } + else + { + BrinValues *bval; + + bval = &dtup->bt_columns[attnum - 1]; + + /* OK this range is summarized */ + stats->n_summarized++; + + if (bval->bv_allnulls) + stats->n_all_nulls++; + + if (bval->bv_hasnulls) + stats->n_has_nulls++; + + if (!bval->bv_allnulls) + { + BrinRange *range; + + range = &ranges->ranges[ranges->nranges++]; + + range->blkno_start = heapBlk; + range->blkno_end = heapBlk + (pagesPerRange - 1); + + range->min_value = datumCopy(bval->bv_values[0], + attr->attbyval, attr->attlen); + range->max_value = datumCopy(bval->bv_values[1], + attr->attbyval, attr->attlen); + } + } + } + } + + if (buf != InvalidBuffer) + ReleaseBuffer(buf); + + /* if we have no regular ranges, we're done */ + if (ranges->nranges == 0) + goto cleanup; + + /* + * Build auxiliary info to optimize the calculation. + * + * We have ranges in the blocknum order, but that is not very useful when + * calculating which ranges interstect - we could cross-check every range + * against every other range, but that's O(N^2) and thus may get extremely + * expensive pretty quick). + * + * To make that cheaper, we'll build two orderings, allowing us to quickly + * eliminate ranges that can't possibly overlap: + * + * - minranges = ranges ordered by min_value + * - maxranges = ranges ordered by max_value + * + * To count intersections, we'll then walk maxranges (i.e. ranges ordered + * by maxval), and for each following range we'll check if it overlaps. + * If yes, we'll proceed to the next one, until we find a range that does + * not overlap. But there might be a later page overlapping - but we can + * use a min_index_lowest tracking the minimum min_index for "future" + * ranges to quickly decide if there are such ranges. If there are none, + * we can terminate (and proceed to the next maxranges element), else we + * have to process additional ranges. + * + * Note: This only counts overlaps with ranges with max_value higher than + * the current one - we want to count all, but the overlaps with preceding + * ranges have already been counted when processing those preceding ranges. + * That is, we'll end up with counting each overlap just for one of those + * ranges, so we get only 1/2 the count. + * + * Note: We don't count the range as overlapping with itself. This needs + * to be considered later, when applying the statistics. + * + * + * XXX This will not work for very many ranges - we can have up to 2^32 of + * them, so allocating a ~32B struct for each would need a lot of memory. + * Not sure what to do about that, perhaps we could sample a couple ranges + * and do some calculations based on that? That is, we could process all + * ranges up to some number (say, statistics_target * 300, as for rows), and + * then sample ranges for larger tables. Then sort the sampled ranges, and + * walk through all ranges once, comparing them to the sample and counting + * overlaps (having them sorted should allow making this quite efficient, + * I think - following algorithm similar to the one implemented here). + */ + + /* info about ordering for the data type */ + typoid = get_atttype(RelationGetRelid(indexRel), attnum); + typcache = lookup_type_cache(typoid, TYPECACHE_CMP_PROC_FINFO); + + /* shouldn't happen, I think - we use this to build the index */ + Assert(OidIsValid(typcache->cmp_proc_finfo.fn_oid)); + + minranges = (BrinRange **) palloc0(ranges->nranges * sizeof(BrinRanges *)); + maxranges = (BrinRange **) palloc0(ranges->nranges * sizeof(BrinRanges *)); + + /* + * Build and sort the ranges min_value / max_value (just pointers + * to the main array). Then go and assign the min_index to each + * range, and finally walk the maxranges array backwards and track + * the min_index_lowest as minimum of "future" indexes. + */ + for (int i = 0; i < ranges->nranges; i++) + { + minranges[i] = &ranges->ranges[i]; + maxranges[i] = &ranges->ranges[i]; + } + + qsort_arg(minranges, ranges->nranges, sizeof(BrinRange *), + range_minval_cmp, typcache); + + qsort_arg(maxranges, ranges->nranges, sizeof(BrinRange *), + range_maxval_cmp, typcache); + + /* + * Update the min_index for each range. If the values are equal, be sure to + * pick the lowest index with that min_value. + */ + minranges[0]->min_index = 0; + for (int i = 1; i < ranges->nranges; i++) + { + if (range_values_cmp(&minranges[i]->min_value, &minranges[i-1]->min_value, typcache) == 0) + minranges[i]->min_index = minranges[i-1]->min_index; + else + minranges[i]->min_index = i; + } + + /* + * Walk the maxranges backward and assign the min_index_lowest as + * a running minimum. + */ + prev_min_index = ranges->nranges; + for (int i = (ranges->nranges - 1); i >= 0; i--) + { + maxranges[i]->min_index_lowest = Min(maxranges[i]->min_index, + prev_min_index); + prev_min_index = maxranges[i]->min_index_lowest; + } + + /* calculate average number of overlapping ranges for any range */ + brin_minmax_count_overlaps(minranges, ranges->nranges, typcache, stats); + + /* calculate minval/maxval stats (distinct values and correlation) */ + brin_minmax_value_stats(minranges, maxranges, + ranges->nranges, typcache, stats); + + /* + * If processing expression, prepare context to evaluate it. + * + * XXX cleanup / refactoring needed + */ + if (expr) + { + estate = CreateExecutorState(); + econtext = GetPerTupleExprContext(estate); + + /* Need a slot to hold the current heap tuple, too */ + slot = MakeSingleTupleTableSlot(RelationGetDescr(heapRel), + &TTSOpsHeapTuple); + + /* Arrange for econtext's scan tuple to be the tuple under test */ + econtext->ecxt_scantuple = slot; + + exprstate = ExecPrepareExpr(expr, estate); + } + + /* match tuples to ranges */ + { + int nvalues = 0; + Datum *values = (Datum *) palloc0(numrows * sizeof(Datum)); + + TupleDesc tdesc = RelationGetDescr(heapRel); + + for (int i = 0; i < numrows; i++) + { + bool isnull; + Datum value; + + if (!expr) + value = heap_getattr(rows[i], heap_attnum, tdesc, &isnull); + else + { + /* + * Reset the per-tuple context each time, to reclaim any cruft + * left behind by evaluating the predicate or index expressions. + */ + ResetExprContext(econtext); + + /* Set up for predicate or expression evaluation */ + ExecStoreHeapTuple(rows[i], slot, false); + + value = ExecEvalExpr(exprstate, + GetPerTupleExprContext(estate), + &isnull); + } + + if (!isnull) + values[nvalues++] = value; + } + + qsort_arg(values, nvalues, sizeof(Datum), range_values_cmp, typcache); + + /* optimized algorithm */ + brin_minmax_match_tuples_to_ranges(ranges, + numrows, rows, nvalues, values, + typcache, stats); + + brin_minmax_increment_stats(minranges, maxranges, ranges->nranges, + values, nvalues, typcache, stats); + } + + /* XXX cleanup / refactoring needed */ + if (expr) + { + ExecDropSingleTupleTableSlot(slot); + FreeExecutorState(estate); + } + + /* + * Possibly quite large, so release explicitly and don't rely + * on the memory context to discard this. + */ + pfree(minranges); + pfree(maxranges); + +cleanup: + /* possibly quite large, so release explicitly */ + pfree(ranges); + + /* free the BrinOpaque, just like brinendscan() would */ + brinRevmapTerminate(opaque->bo_rmAccess); + brin_free_desc(opaque->bo_bdesc); + + PG_RETURN_POINTER(stats); +} + /* * Cache and return the procedure for the given strategy. * diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c index bfd981aa3f..ffc72f2180 100644 --- a/src/backend/commands/analyze.c +++ b/src/backend/commands/analyze.c @@ -16,6 +16,7 @@ #include <math.h> +#include "access/brin_internal.h" #include "access/detoast.h" #include "access/genam.h" #include "access/multixact.h" @@ -30,6 +31,7 @@ #include "catalog/catalog.h" #include "catalog/index.h" #include "catalog/indexing.h" +#include "catalog/pg_am.h" #include "catalog/pg_collation.h" #include "catalog/pg_inherits.h" #include "catalog/pg_namespace.h" @@ -81,6 +83,7 @@ typedef struct AnlIndexData /* Default statistics target (GUC parameter) */ int default_statistics_target = 100; +bool enable_indexam_stats = false; /* A few variables that don't seem worth passing around as parameters */ static MemoryContext anl_context = NULL; @@ -92,7 +95,7 @@ static void do_analyze_rel(Relation onerel, AcquireSampleRowsFunc acquirefunc, BlockNumber relpages, bool inh, bool in_outer_xact, int elevel); static void compute_index_stats(Relation onerel, double totalrows, - AnlIndexData *indexdata, int nindexes, + AnlIndexData *indexdata, Relation *indexRels, int nindexes, HeapTuple *rows, int numrows, MemoryContext col_context); static VacAttrStats *examine_attribute(Relation onerel, int attnum, @@ -454,15 +457,49 @@ do_analyze_rel(Relation onerel, VacuumParams *params, { AnlIndexData *thisdata = &indexdata[ind]; IndexInfo *indexInfo; + bool collectAmStats; + Oid regproc; thisdata->indexInfo = indexInfo = BuildIndexInfo(Irel[ind]); thisdata->tupleFract = 1.0; /* fix later if partial */ - if (indexInfo->ii_Expressions != NIL && va_cols == NIL) + + /* + * Should we collect AM-specific statistics for any of the columns? + * + * If AM-specific statistics are enabled (using a GUC), see if we + * have an optional support procedure to build the statistics. + * + * If there's any such attribute, we just force building stats + * even for regular index keys (not just expressions) and indexes + * without predicates. It'd be good to only build the AM stats, but + * for now this is good enough. + * + * XXX The GUC is there morestly to make it easier to enable/disable + * this during development. + * + * FIXME Only build the AM statistics, not the other stats. And only + * do that for the keys with the optional procedure. not all of them. + */ + collectAmStats = false; + if (enable_indexam_stats && (Irel[ind]->rd_indam->amstatsprocnum != 0)) + { + for (int j = 0; j < indexInfo->ii_NumIndexAttrs; j++) + { + regproc = index_getprocid(Irel[ind], (j+1), Irel[ind]->rd_indam->amstatsprocnum); + if (OidIsValid(regproc)) + { + collectAmStats = true; + break; + } + } + } + + if ((indexInfo->ii_Expressions != NIL || collectAmStats) && va_cols == NIL) { ListCell *indexpr_item = list_head(indexInfo->ii_Expressions); thisdata->vacattrstats = (VacAttrStats **) - palloc(indexInfo->ii_NumIndexAttrs * sizeof(VacAttrStats *)); + palloc0(indexInfo->ii_NumIndexAttrs * sizeof(VacAttrStats *)); tcnt = 0; for (i = 0; i < indexInfo->ii_NumIndexAttrs; i++) { @@ -483,6 +520,12 @@ do_analyze_rel(Relation onerel, VacuumParams *params, if (thisdata->vacattrstats[tcnt] != NULL) tcnt++; } + else + { + thisdata->vacattrstats[tcnt] = + examine_attribute(Irel[ind], i + 1, NULL); + tcnt++; + } } thisdata->attr_cnt = tcnt; } @@ -588,7 +631,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params, if (nindexes > 0) compute_index_stats(onerel, totalrows, - indexdata, nindexes, + indexdata, Irel, nindexes, rows, numrows, col_context); @@ -823,12 +866,93 @@ do_analyze_rel(Relation onerel, VacuumParams *params, anl_context = NULL; } +/* + * compute_indexam_stats + * Call the optional procedure to compute AM-specific statistics. + * + * We simply call the procedure, which is expected to produce a bytea value. + * + * At the moment this only deals with BRIN indexes, and bails out for other + * access methods, but it should be generic - use something like amoptsprocnum + * and just check if the procedure exists. + */ +static void +compute_indexam_stats(Relation onerel, + Relation indexRel, IndexInfo *indexInfo, + double totalrows, AnlIndexData *indexdata, + HeapTuple *rows, int numrows) +{ + int expridx; + + if (!enable_indexam_stats) + return; + + /* ignore index AMs without the optional procedure */ + if (indexRel->rd_indam->amstatsprocnum == 0) + return; + + /* + * Look at attributes, and calculate stats for those that have the + * optional stats proc for the opfamily. + */ + expridx = 0; + for (int i = 0; i < indexInfo->ii_NumIndexAttrs; i++) + { + AttrNumber attno = (i + 1); + AttrNumber attnum = indexInfo->ii_IndexAttrNumbers[i]; /* heap attnum */ + RegProcedure regproc; + FmgrInfo *statsproc; + Datum datum; + VacAttrStats *stats; + MemoryContext oldcxt; + Node *expr = NULL; + + if (!AttributeNumberIsValid(attnum)) + { + expr = (Node *) list_nth(RelationGetIndexExpressions(indexRel), + expridx); + expridx++; + } + + /* do this first, as it doesn't fail when proc not defined */ + regproc = index_getprocid(indexRel, attno, indexRel->rd_indam->amstatsprocnum); + + /* ignore opclasses without the optional procedure */ + if (!RegProcedureIsValid(regproc)) + continue; + + statsproc = index_getprocinfo(indexRel, attno, indexRel->rd_indam->amstatsprocnum); + Assert(statsproc != NULL); + + stats = indexdata->vacattrstats[i]; + + oldcxt = MemoryContextSwitchTo(stats->anl_context); + + /* call the proc, let the AM calculate whatever it wants */ + /* XXX maybe we should just pass the index attno and leave the + * expression handling up to the procedure? */ + datum = FunctionCall7Coll(statsproc, + InvalidOid, /* FIXME correct collation */ + PointerGetDatum(onerel), + PointerGetDatum(indexRel), + Int16GetDatum(attno), + Int16GetDatum(attnum), + PointerGetDatum(expr), + PointerGetDatum(rows), + Int32GetDatum(numrows)); + + stats->staindexam = datum; + + MemoryContextSwitchTo(oldcxt); + } +} + /* * Compute statistics about indexes of a relation */ static void compute_index_stats(Relation onerel, double totalrows, - AnlIndexData *indexdata, int nindexes, + AnlIndexData *indexdata, Relation *indexRels, int nindexes, HeapTuple *rows, int numrows, MemoryContext col_context) { @@ -848,6 +972,7 @@ compute_index_stats(Relation onerel, double totalrows, { AnlIndexData *thisdata = &indexdata[ind]; IndexInfo *indexInfo = thisdata->indexInfo; + Relation indexRel = indexRels[ind]; int attr_cnt = thisdata->attr_cnt; TupleTableSlot *slot; EState *estate; @@ -860,6 +985,13 @@ compute_index_stats(Relation onerel, double totalrows, rowno; double totalindexrows; + /* + * If this is a BRIN index, try calling a procedure to collect + * extra opfamily-specific statistics (if procedure defined). + */ + compute_indexam_stats(onerel, indexRel, indexInfo, totalrows, + thisdata, rows, numrows); + /* Ignore index if no columns to analyze and not partial */ if (attr_cnt == 0 && indexInfo->ii_Predicate == NIL) continue; @@ -1662,6 +1794,13 @@ update_attstats(Oid relid, bool inh, int natts, VacAttrStats **vacattrstats) values[Anum_pg_statistic_stanullfrac - 1] = Float4GetDatum(stats->stanullfrac); values[Anum_pg_statistic_stawidth - 1] = Int32GetDatum(stats->stawidth); values[Anum_pg_statistic_stadistinct - 1] = Float4GetDatum(stats->stadistinct); + + /* optional AM-specific stats */ + if (DatumGetPointer(stats->staindexam) != NULL) + values[Anum_pg_statistic_staindexam - 1] = stats->staindexam; + else + nulls[Anum_pg_statistic_staindexam - 1] = true; + i = Anum_pg_statistic_stakind1 - 1; for (k = 0; k < STATISTIC_NUM_SLOTS; k++) { diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c index 9f67a57724..3e66f6cb2c 100644 --- a/src/backend/statistics/extended_stats.c +++ b/src/backend/statistics/extended_stats.c @@ -2345,6 +2345,8 @@ serialize_expr_stats(AnlExprData *exprdata, int nexprs) values[Anum_pg_statistic_stanullfrac - 1] = Float4GetDatum(stats->stanullfrac); values[Anum_pg_statistic_stawidth - 1] = Int32GetDatum(stats->stawidth); values[Anum_pg_statistic_stadistinct - 1] = Float4GetDatum(stats->stadistinct); + nulls[Anum_pg_statistic_staindexam - 1] = true; + i = Anum_pg_statistic_stakind1 - 1; for (k = 0; k < STATISTIC_NUM_SLOTS; k++) { diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index c4fcd0076e..1058a8d510 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -7834,6 +7834,7 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, Relation indexRel; ListCell *l; VariableStatData vardata; + double averageOverlaps; Assert(rte->rtekind == RTE_RELATION); @@ -7881,6 +7882,7 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, * correlation statistics, we will keep it as 0. */ *indexCorrelation = 0; + averageOverlaps = 0.0; foreach(l, path->indexclauses) { @@ -7890,6 +7892,36 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, /* attempt to lookup stats in relation for this index column */ if (attnum != 0) { + /* + * If AM-specific statistics are enabled, try looking up the stats + * for the index key. We only have this for minmax opclasses, so + * we just cast it like that. But other BRIN opclasses might need + * other stats so either we need to abstract this somehow, or maybe + * just collect a sufficiently generic stats for all BRIN indexes. + * + * XXX Make this non-minmax specific. + */ + if (enable_indexam_stats) + { + BrinMinmaxStats *amstats + = (BrinMinmaxStats *) get_attindexam(index->indexoid, attnum); + + if (amstats) + { + elog(DEBUG1, "found AM stats: attnum %d n_ranges %lld n_summarized %lld n_all_nulls %lld n_has_nulls %lld avg_overlaps %f", + attnum, (long long)amstats->n_ranges, (long long)amstats->n_summarized, + (long long)amstats->n_all_nulls, (long long)amstats->n_has_nulls, + amstats->avg_overlaps); + + /* + * The only thing we use at the moment is the average number + * of overlaps for a single range. Use the other stuff too. + */ + averageOverlaps = Max(averageOverlaps, + 1.0 + amstats->avg_overlaps); + } + } + /* Simple variable -- look to stats for the underlying table */ if (get_relation_stats_hook && (*get_relation_stats_hook) (root, rte, attnum, &vardata)) @@ -7970,6 +8002,14 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, baserel->relid, JOIN_INNER, NULL); + /* + * XXX Can we combine qualSelectivity with the average number of matching + * ranges per value? qualSelectivity estimates how many tuples ar we + * going to match, and average number of matches says how many ranges + * will each of those match on average. We don't know how many will + * be duplicate, but it gives us a worst-case estimate, at least. + */ + /* * Now calculate the minimum possible ranges we could match with if all of * the rows were in the perfect order in the table's heap. @@ -7986,6 +8026,25 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, else estimatedRanges = Min(minimalRanges / *indexCorrelation, indexRanges); + elog(DEBUG1, "before index AM stats: cestimatedRanges = %f", estimatedRanges); + + /* + * If we found some AM stats, look at average number of overlapping ranges, + * and apply that to the currently estimated ranges. + * + * XXX We pretty much combine this with correlation info (because it was + * already applied in the estimatedRanges formula above), which might be + * overly pessimistic. The overlaps stats seems somewhat redundant with + * the correlation, so maybe we should do just one? The AM stats seems + * like a more reliable information, because the correlation is not very + * sensitive to outliers, for example. So maybe let's prefer that, and + * only use the correlation as fallback when AM stats are not available? + */ + if (averageOverlaps > 0.0) + estimatedRanges = Min(estimatedRanges * averageOverlaps, indexRanges); + + elog(DEBUG1, "after index AM stats: cestimatedRanges = %f", estimatedRanges); + /* we expect to visit this portion of the table */ selec = estimatedRanges / indexRanges; diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c index 60978f9415..f8512e3dd3 100644 --- a/src/backend/utils/cache/lsyscache.c +++ b/src/backend/utils/cache/lsyscache.c @@ -3138,6 +3138,47 @@ get_attavgwidth(Oid relid, AttrNumber attnum) return 0; } + +/* + * get_attstaindexam + * + * Given the table and attribute number of a column, get the index AM + * statistics. Return NULL if no data available. + * + * Currently this is only consulted for individual tables, not for inheritance + * trees, so we don't need an "inh" parameter. + */ +bytea * +get_attindexam(Oid relid, AttrNumber attnum) +{ + HeapTuple tp; + + tp = SearchSysCache3(STATRELATTINH, + ObjectIdGetDatum(relid), + Int16GetDatum(attnum), + BoolGetDatum(false)); + if (HeapTupleIsValid(tp)) + { + Datum val; + bytea *retval = NULL; + bool isnull; + + val = SysCacheGetAttr(STATRELATTINH, tp, + Anum_pg_statistic_staindexam, + &isnull); + + if (!isnull) + retval = (bytea *) PG_DETOAST_DATUM(val); + + // staindexam = ((Form_pg_statistic) GETSTRUCT(tp))->staindexam; + ReleaseSysCache(tp); + + return retval; + } + + return NULL; +} + /* * get_attstatsslot * diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index c14456060c..e42f8348ea 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -1025,6 +1025,16 @@ struct config_bool ConfigureNamesBool[] = true, NULL, NULL, NULL }, + { + {"enable_indexam_stats", PGC_USERSET, QUERY_TUNING_METHOD, + gettext_noop("Enables the planner's use of index AM stats."), + NULL, + GUC_EXPLAIN + }, + &enable_indexam_stats, + false, + NULL, NULL, NULL + }, { {"geqo", PGC_USERSET, QUERY_TUNING_GEQO, gettext_noop("Enables genetic query optimization."), diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index e4c0269fa3..9ca523073f 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -379,6 +379,7 @@ #enable_hashagg = on #enable_hashjoin = on #enable_incremental_sort = on +#enable_indexam_stats = off #enable_indexscan = on #enable_indexonlyscan = on #enable_material = on diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 4476ff7fba..e477e2e83c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -216,6 +216,8 @@ typedef struct IndexAmRoutine uint16 amsupport; /* opclass options support function number or 0 */ uint16 amoptsprocnum; + /* opclass statistics support function number or 0 */ + uint16 amstatsprocnum; /* does AM support ORDER BY indexed column's value? */ bool amcanorder; /* does AM support ORDER BY result of an operator on indexed column? */ diff --git a/src/include/access/brin.h b/src/include/access/brin.h index ed66f1b3d5..1d21b816fc 100644 --- a/src/include/access/brin.h +++ b/src/include/access/brin.h @@ -34,6 +34,69 @@ typedef struct BrinStatsData BlockNumber revmapNumPages; } BrinStatsData; +/* + * Info about ranges for BRIN Sort. + */ +typedef struct BrinRange +{ + BlockNumber blkno_start; + BlockNumber blkno_end; + + Datum min_value; + Datum max_value; + bool has_nulls; + bool all_nulls; + bool not_summarized; + + /* + * Index of the range when ordered by min_value (if there are multiple + * ranges with the same min_value, it's the lowest one). + */ + uint32 min_index; + + /* + * Minimum min_index from all ranges with higher max_value (i.e. when + * sorted by max_value). If there are multiple ranges with the same + * max_value, it depends on the ordering (i.e. the ranges may get + * different min_index_lowest, depending on the exact ordering). + */ + uint32 min_index_lowest; +} BrinRange; + +typedef struct BrinRanges +{ + int nranges; + BrinRange ranges[FLEXIBLE_ARRAY_MEMBER]; +} BrinRanges; + +typedef struct BrinMinmaxStats +{ + int32 vl_len_; /* varlena header (do not touch directly!) */ + int64 n_ranges; + int64 n_summarized; + int64 n_all_nulls; + int64 n_has_nulls; + + /* average number of overlapping ranges */ + double avg_overlaps; + + /* average number of matching ranges (per value) */ + double avg_matches; + double avg_matches_unique; + + /* minval/maxval stats (ndistinct, correlation to blkno) */ + int64 minval_ndistinct; + int64 maxval_ndistinct; + double minval_correlation; + double maxval_correlation; + + /* minval/maxval increment stats */ + double minval_increment_avg; + double minval_increment_max; + double maxval_increment_avg; + double maxval_increment_max; + +} BrinMinmaxStats; #define BRIN_DEFAULT_PAGES_PER_RANGE 128 #define BrinGetPagesPerRange(relation) \ diff --git a/src/include/access/brin_internal.h b/src/include/access/brin_internal.h index 97ddc925b2..eac796e6f4 100644 --- a/src/include/access/brin_internal.h +++ b/src/include/access/brin_internal.h @@ -75,6 +75,7 @@ typedef struct BrinDesc #define BRIN_PROCNUM_OPTIONS 5 /* optional */ /* procedure numbers up to 10 are reserved for BRIN future expansion */ #define BRIN_FIRST_OPTIONAL_PROCNUM 11 +#define BRIN_PROCNUM_STATISTICS 11 /* optional */ #define BRIN_LAST_OPTIONAL_PROCNUM 15 #undef BRIN_DEBUG diff --git a/src/include/catalog/pg_amproc.dat b/src/include/catalog/pg_amproc.dat index 5b950129de..9bbd1f14f1 100644 --- a/src/include/catalog/pg_amproc.dat +++ b/src/include/catalog/pg_amproc.dat @@ -804,6 +804,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/bytea_minmax_ops', amproclefttype => 'bytea', amprocrighttype => 'bytea', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/bytea_minmax_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '11', amproc => 'brin_minmax_stats' }, # bloom bytea { amprocfamily => 'brin/bytea_bloom_ops', amproclefttype => 'bytea', @@ -835,6 +837,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/char_minmax_ops', amproclefttype => 'char', amprocrighttype => 'char', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/char_minmax_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '11', amproc => 'brin_minmax_stats' }, # bloom "char" { amprocfamily => 'brin/char_bloom_ops', amproclefttype => 'char', @@ -864,6 +868,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/name_minmax_ops', amproclefttype => 'name', amprocrighttype => 'name', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/name_minmax_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '11', amproc => 'brin_minmax_stats' }, # bloom name { amprocfamily => 'brin/name_bloom_ops', amproclefttype => 'name', @@ -893,6 +899,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int8', amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_stats' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int2', amprocrighttype => 'int2', amprocnum => '1', @@ -905,6 +913,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int2', amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_stats' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', amprocrighttype => 'int4', amprocnum => '1', @@ -917,6 +927,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax multi integer: int2, int4, int8 { amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', @@ -1034,6 +1046,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/text_minmax_ops', amproclefttype => 'text', amprocrighttype => 'text', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/text_minmax_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '11', amproc => 'brin_minmax_stats' }, # bloom text { amprocfamily => 'brin/text_bloom_ops', amproclefttype => 'text', @@ -1062,6 +1076,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/oid_minmax_ops', amproclefttype => 'oid', amprocrighttype => 'oid', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/oid_minmax_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax multi oid { amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', @@ -1110,6 +1126,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/tid_minmax_ops', amproclefttype => 'tid', amprocrighttype => 'tid', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/tid_minmax_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '11', amproc => 'brin_minmax_stats' }, # bloom tid { amprocfamily => 'brin/tid_bloom_ops', amproclefttype => 'tid', @@ -1160,6 +1178,9 @@ { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float4', amprocrighttype => 'float4', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '11', + amproc => 'brin_minmax_stats' }, { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float8', amprocrighttype => 'float8', amprocnum => '1', @@ -1173,6 +1194,9 @@ { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float8', amprocrighttype => 'float8', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi float { amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', @@ -1261,6 +1285,9 @@ { amprocfamily => 'brin/macaddr_minmax_ops', amproclefttype => 'macaddr', amprocrighttype => 'macaddr', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/macaddr_minmax_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi macaddr { amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', @@ -1314,6 +1341,9 @@ { amprocfamily => 'brin/macaddr8_minmax_ops', amproclefttype => 'macaddr8', amprocrighttype => 'macaddr8', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/macaddr8_minmax_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi macaddr8 { amprocfamily => 'brin/macaddr8_minmax_multi_ops', @@ -1366,6 +1396,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/network_minmax_ops', amproclefttype => 'inet', amprocrighttype => 'inet', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/network_minmax_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax multi inet { amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', @@ -1436,6 +1468,9 @@ { amprocfamily => 'brin/bpchar_minmax_ops', amproclefttype => 'bpchar', amprocrighttype => 'bpchar', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/bpchar_minmax_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # bloom character { amprocfamily => 'brin/bpchar_bloom_ops', amproclefttype => 'bpchar', @@ -1467,6 +1502,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/time_minmax_ops', amproclefttype => 'time', amprocrighttype => 'time', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/time_minmax_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax multi time without time zone { amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', @@ -1517,6 +1554,9 @@ { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamp', amprocrighttype => 'timestamp', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '11', + amproc => 'brin_minmax_stats' }, { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamptz', amprocrighttype => 'timestamptz', amprocnum => '1', @@ -1530,6 +1570,9 @@ { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamptz', amprocrighttype => 'timestamptz', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '11', + amproc => 'brin_minmax_stats' }, { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'date', amprocrighttype => 'date', amprocnum => '1', @@ -1542,6 +1585,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'date', amprocrighttype => 'date', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax multi datetime (date, timestamp, timestamptz) { amprocfamily => 'brin/datetime_minmax_multi_ops', @@ -1668,6 +1713,9 @@ { amprocfamily => 'brin/interval_minmax_ops', amproclefttype => 'interval', amprocrighttype => 'interval', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/interval_minmax_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi interval { amprocfamily => 'brin/interval_minmax_multi_ops', @@ -1721,6 +1769,9 @@ { amprocfamily => 'brin/timetz_minmax_ops', amproclefttype => 'timetz', amprocrighttype => 'timetz', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/timetz_minmax_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi time with time zone { amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', @@ -1771,6 +1822,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/bit_minmax_ops', amproclefttype => 'bit', amprocrighttype => 'bit', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/bit_minmax_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax bit varying { amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', @@ -1785,6 +1838,9 @@ { amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', amprocrighttype => 'varbit', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax numeric { amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', @@ -1799,6 +1855,9 @@ { amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', amprocrighttype => 'numeric', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi numeric { amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', @@ -1851,6 +1910,8 @@ amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/uuid_minmax_ops', amproclefttype => 'uuid', amprocrighttype => 'uuid', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/uuid_minmax_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '11', amproc => 'brin_minmax_stats' }, # minmax multi uuid { amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', @@ -1924,6 +1985,9 @@ { amprocfamily => 'brin/pg_lsn_minmax_ops', amproclefttype => 'pg_lsn', amprocrighttype => 'pg_lsn', amprocnum => '4', amproc => 'brin_minmax_union' }, +{ amprocfamily => 'brin/pg_lsn_minmax_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '11', + amproc => 'brin_minmax_stats' }, # minmax multi pg_lsn { amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 6996073989..e50a21fc22 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -8527,6 +8527,10 @@ { oid => '3386', descr => 'BRIN minmax support', proname => 'brin_minmax_union', prorettype => 'bool', proargtypes => 'internal internal internal', prosrc => 'brin_minmax_union' }, +{ oid => '9800', descr => 'BRIN minmax support', + proname => 'brin_minmax_stats', prorettype => 'bool', + proargtypes => 'internal internal int2 int2 internal int4', + prosrc => 'brin_minmax_stats' }, # BRIN minmax multi { oid => '4616', descr => 'BRIN multi minmax support', diff --git a/src/include/catalog/pg_statistic.h b/src/include/catalog/pg_statistic.h index 8770c5b4c6..d3d0bce257 100644 --- a/src/include/catalog/pg_statistic.h +++ b/src/include/catalog/pg_statistic.h @@ -121,6 +121,11 @@ CATALOG(pg_statistic,2619,StatisticRelationId) anyarray stavalues3; anyarray stavalues4; anyarray stavalues5; + + /* + * Statistics calculated by index AM (e.g. BRIN for ranges, etc.). + */ + bytea staindexam; #endif } FormData_pg_statistic; diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h index 4af02940c5..10a8ec83ce 100644 --- a/src/include/commands/vacuum.h +++ b/src/include/commands/vacuum.h @@ -151,6 +151,7 @@ typedef struct VacAttrStats float4 *stanumbers[STATISTIC_NUM_SLOTS]; int numvalues[STATISTIC_NUM_SLOTS]; Datum *stavalues[STATISTIC_NUM_SLOTS]; + Datum staindexam; /* index-specific stats (as bytea) */ /* * These fields describe the stavalues[n] element types. They will be @@ -299,6 +300,7 @@ extern PGDLLIMPORT int vacuum_multixact_freeze_min_age; extern PGDLLIMPORT int vacuum_multixact_freeze_table_age; extern PGDLLIMPORT int vacuum_failsafe_age; extern PGDLLIMPORT int vacuum_multixact_failsafe_age; +extern PGDLLIMPORT bool enable_indexam_stats; /* * Maximum value for default_statistics_target and per-column statistics diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h index 4f5418b972..fcef91d306 100644 --- a/src/include/utils/lsyscache.h +++ b/src/include/utils/lsyscache.h @@ -185,6 +185,7 @@ extern Oid getBaseType(Oid typid); extern Oid getBaseTypeAndTypmod(Oid typid, int32 *typmod); extern int32 get_typavgwidth(Oid typid, int32 typmod); extern int32 get_attavgwidth(Oid relid, AttrNumber attnum); +extern bytea *get_attindexam(Oid relid, AttrNumber attnum); extern bool get_attstatsslot(AttStatsSlot *sslot, HeapTuple statstuple, int reqkind, Oid reqop, int flags); extern void free_attstatsslot(AttStatsSlot *sslot); diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out index 001c6e7eb9..b7fda6fc82 100644 --- a/src/test/regress/expected/sysviews.out +++ b/src/test/regress/expected/sysviews.out @@ -117,6 +117,7 @@ select name, setting from pg_settings where name like 'enable%'; enable_hashagg | on enable_hashjoin | on enable_incremental_sort | on + enable_indexam_stats | off enable_indexonlyscan | on enable_indexscan | on enable_material | on @@ -132,7 +133,7 @@ select name, setting from pg_settings where name like 'enable%'; enable_seqscan | on enable_sort | on enable_tidscan | on -(21 rows) +(22 rows) -- Test that the pg_timezone_names and pg_timezone_abbrevs views are -- more-or-less working. We can't test their contents in any great detail -- 2.41.0
0002-wip-introduce-debug_brin_stats-20230710.patch (text/x-patch, 5.7K) download From a1e5f3886434cda86976b127798aa56027757618 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sat, 29 Oct 2022 17:46:51 +0200 Subject: [PATCH 02/11] wip: introduce debug_brin_stats --- src/backend/access/brin/brin_minmax.c | 80 +++++++++++++++++++++++++++ src/backend/utils/misc/guc_tables.c | 18 ++++++ 2 files changed, 98 insertions(+) diff --git a/src/backend/access/brin/brin_minmax.c b/src/backend/access/brin/brin_minmax.c index 64150204fc..c66b6feafd 100644 --- a/src/backend/access/brin/brin_minmax.c +++ b/src/backend/access/brin/brin_minmax.c @@ -28,6 +28,10 @@ #include "utils/syscache.h" #include "utils/timestamp.h" +#ifdef DEBUG_BRIN_STATS +bool debug_brin_stats = false; +#endif + typedef struct MinmaxOpaque { Oid cached_subtype; @@ -493,6 +497,13 @@ brin_minmax_count_overlaps(BrinRange **minranges, int nranges, { int64 noverlaps; +#ifdef DEBUG_BRIN_STATS + TimestampTz start_ts; + + if (debug_brin_stats) + start_ts = GetCurrentTimestamp(); +#endif + noverlaps = 0; for (int i = 0; i < nranges; i++) { @@ -515,6 +526,16 @@ brin_minmax_count_overlaps(BrinRange **minranges, int nranges, */ noverlaps *= 2; +#ifdef DEBUG_BRIN_STATS + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_count_overlaps -----"); + elog(WARNING, "noverlaps = %ld", noverlaps); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } +#endif + stats->avg_overlaps = (double) noverlaps / nranges; } @@ -540,6 +561,13 @@ brin_minmax_match_tuples_to_ranges(BrinRanges *ranges, int64 *unique = (int64 *) palloc0(sizeof(int64) * nvalues); +#ifdef DEBUG_BRIN_STATS + TimestampTz start_ts; + + if (debug_brin_stats) + start_ts = GetCurrentTimestamp(); +#endif + /* * Build running count of unique values. We know there are unique[i] * unique values in values array up to index "i". @@ -588,6 +616,18 @@ brin_minmax_match_tuples_to_ranges(BrinRanges *ranges, Assert(nmatches >= 0); Assert(nmatches_unique >= 0); +#ifdef DEBUG_BRIN_STATS + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_match_tuples_to_ranges -----"); + elog(WARNING, "nmatches = %ld %f", nmatches, (double) nmatches / numrows); + elog(WARNING, "nmatches unique = %ld %ld %f", nmatches_unique, nvalues_unique, + (double) nmatches_unique / nvalues_unique); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } +#endif + stats->avg_matches = (double) nmatches / numrows; stats->avg_matches_unique = (double) nmatches_unique / nvalues_unique; } @@ -619,6 +659,13 @@ brin_minmax_value_stats(BrinRange **minranges, BrinRange **maxranges, minval_corr = 0, maxval_corr = 0; +#ifdef DEBUG_BRIN_STATS + TimestampTz start_ts; + + if (debug_brin_stats) + start_ts = GetCurrentTimestamp(); +#endif + for (int i = 1; i < nranges; i++) { if (range_values_cmp(&minranges[i-1]->min_value, &minranges[i]->min_value, typcache) != 0) @@ -641,6 +688,19 @@ brin_minmax_value_stats(BrinRange **minranges, BrinRange **maxranges, stats->minval_correlation = (double) minval_corr / nranges; stats->maxval_correlation = (double) maxval_corr / nranges; + +#ifdef DEBUG_BRIN_STATS + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_value_stats -----"); + elog(WARNING, "minval ndistinct " INT64_FORMAT " correlation %f", + stats->minval_ndistinct, stats->minval_correlation); + elog(WARNING, "maxval ndistinct " INT64_FORMAT " correlation %f", + stats->maxval_ndistinct, stats->maxval_correlation); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } +#endif } /* @@ -665,6 +725,13 @@ brin_minmax_increment_stats(BrinRange **minranges, BrinRange **maxranges, max_minval = 0, max_maxval = 0; +#ifdef DEBUG_BRIN_STATS + TimestampTz start_ts; + + if (debug_brin_stats) + start_ts = GetCurrentTimestamp(); +#endif + for (int i = 1; i < nranges; i++) { if (range_values_cmp(&minranges[i-1]->min_value, &minranges[i]->min_value, typcache) != 0) @@ -716,6 +783,19 @@ brin_minmax_increment_stats(BrinRange **minranges, BrinRange **maxranges, } } +#ifdef DEBUG_BRIN_STATS + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_increment_stats -----"); + elog(WARNING, "minval ndistinct %ld sum %f max %f avg %f", + minval_ndist, sum_minval, max_minval, sum_minval / minval_ndist); + elog(WARNING, "maxval ndistinct %ld sum %f max %f avg %f", + maxval_ndist, sum_maxval, max_maxval, sum_maxval / maxval_ndist); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } +#endif + stats->minval_increment_avg = (sum_minval / minval_ndist); stats->minval_increment_max = max_minval; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index e42f8348ea..9711a23c34 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -98,6 +98,10 @@ extern char *temp_tablespaces; extern bool ignore_checksum_failure; extern bool ignore_invalid_pages; +#ifdef DEBUG_BRIN_STATS +extern bool debug_brin_stats; +#endif + #ifdef TRACE_SYNCSCAN extern bool trace_syncscan; #endif @@ -1253,6 +1257,20 @@ struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, +#ifdef DEBUG_BRIN_STATS + /* this is undocumented because not exposed in a standard build */ + { + {"debug_brin_stats", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Print info about calculated BRIN statistics."), + NULL, + GUC_NOT_IN_SAMPLE + }, + &debug_brin_stats, + false, + NULL, NULL, NULL + }, +#endif + { {"exit_on_error", PGC_USERSET, ERROR_HANDLING_OPTIONS, gettext_noop("Terminate session on any error."), -- 2.41.0
0003-wip-introduce-debug_brin_cross_check-20230710.patch (text/x-patch, 19.7K) download From 385c5cd92007f7336efb9594752f21a1e27604e9 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sat, 29 Oct 2022 20:47:31 +0200 Subject: [PATCH 03/11] wip: introduce debug_brin_cross_check --- src/backend/access/brin/brin_minmax.c | 574 ++++++++++++++++++++++++++ src/backend/utils/misc/guc_tables.c | 10 + 2 files changed, 584 insertions(+) diff --git a/src/backend/access/brin/brin_minmax.c b/src/backend/access/brin/brin_minmax.c index c66b6feafd..a4c00caafd 100644 --- a/src/backend/access/brin/brin_minmax.c +++ b/src/backend/access/brin/brin_minmax.c @@ -30,6 +30,7 @@ #ifdef DEBUG_BRIN_STATS bool debug_brin_stats = false; +bool debug_brin_cross_check = false; #endif typedef struct MinmaxOpaque @@ -340,6 +341,50 @@ range_values_cmp(const void *a, const void *b, void *arg) return DatumGetInt32(c); } +#ifdef DEBUG_BRIN_STATS +/* + * maxval_start + * Determine first index so that (maxvalue >= value). + * + * The array of ranges is expected to be sorted by maxvalue, so this is the first + * range that can possibly intersect with range having "value" as minval. + */ +static int +maxval_start(BrinRange **ranges, int nranges, Datum value, TypeCacheEntry *typcache) +{ + int start = 0, + end = (nranges - 1); + + // everything matches + if (range_values_cmp(&value, &ranges[start]->max_value, typcache) <= 0) + return 0; + + // no matches + if (range_values_cmp(&value, &ranges[end]->max_value, typcache) > 0) + return nranges; + + while ((end - start) > 0) + { + int midpoint; + int r; + + midpoint = start + (end - start) / 2; + + r = range_values_cmp(&value, &ranges[midpoint]->max_value, typcache); + + if (r <= 0) + end = midpoint; + else + start = (midpoint + 1); + } + + Assert(ranges[start]->max_value >= value); + Assert(ranges[start-1]->max_value < value); + + return start; +} +#endif + /* * minval_end * Determine first index so that (minval > value). @@ -632,6 +677,316 @@ brin_minmax_match_tuples_to_ranges(BrinRanges *ranges, stats->avg_matches_unique = (double) nmatches_unique / nvalues_unique; } +#ifdef DEBUG_BRIN_STATS +/* + * Simple histogram, with bins tracking value and two overlap counts. + * + * XXX Maybe we should have two separate histograms, one for all counts and + * another one for "unique" values. + * + * XXX Serialize the histogram. There might be a data set where we have very + * many distinct buckets (values having very different number of matching + * ranges) - not sure if there's some sort of upper limit (but hard to say for + * other opclasses, like bloom). And we don't want arbitrarily large histogram, + * to keep the statistics fairly small, I guess. So we'd need to pick a subset, + * merge buckets with "similar" counts, or approximate it somehow. For now we + * don't serialize it, because we don't use the histogram. + */ +typedef struct histogram_bin_t +{ + int64 value; + int64 count; +} histogram_bin_t; + +typedef struct histogram_t +{ + int nbins; + int nbins_max; + histogram_bin_t bins[FLEXIBLE_ARRAY_MEMBER]; +} histogram_t; + +#define HISTOGRAM_BINS_START 32 + +/* allocate histogram with default number of bins */ +static histogram_t * +histogram_init(void) +{ + histogram_t *hist; + + hist = (histogram_t *) palloc0(offsetof(histogram_t, bins) + + sizeof(histogram_bin_t) * HISTOGRAM_BINS_START); + hist->nbins_max = HISTOGRAM_BINS_START; + + return hist; +} + +/* + * histogram_add + * Add a hit for a particular value to the histogram. + * + * XXX We don't sort the bins, so just do binary sort. For large number of values + * this might be an issue, for small number of values a linear search is fine. + */ +static histogram_t * +histogram_add(histogram_t *hist, int value) +{ + bool found = false; + histogram_bin_t *bin; + + for (int i = 0; i < hist->nbins; i++) + { + if (hist->bins[i].value == value) + { + bin = &hist->bins[i]; + found = true; + } + } + + if (!found) + { + if (hist->nbins == hist->nbins_max) + { + int nbins = (2 * hist->nbins_max); + + hist = repalloc(hist, offsetof(histogram_t, bins) + + sizeof(histogram_bin_t) * nbins); + hist->nbins_max = nbins; + } + + Assert(hist->nbins < hist->nbins_max); + + bin = &hist->bins[hist->nbins++]; + bin->value = value; + bin->count = 0; + } + + bin->count += 1; + + Assert(bin->value == value); + Assert(bin->count >= 0); + + return hist; +} + +/* used to sort histogram bins by value */ +static int +histogram_bin_cmp(const void *a, const void *b) +{ + histogram_bin_t *ba = (histogram_bin_t *) a; + histogram_bin_t *bb = (histogram_bin_t *) b; + + if (ba->value < bb->value) + return -1; + + if (bb->value < ba->value) + return 1; + + return 0; +} + +static void +histogram_print(histogram_t *hist) +{ + return; + + elog(WARNING, "----- histogram -----"); + for (int i = 0; i < hist->nbins; i++) + { + elog(WARNING, "bin %d value %ld count %ld", + i, hist->bins[i].value, hist->bins[i].count); + } +} + +/* + * brin_minmax_match_tuples_to_ranges2 + * Match tuples to ranges, count average number of ranges per tuple. + * + * Match sample tuples to the ranges, so that we can count how many ranges + * a value matches on average. This might seem redundant to the number of + * overlaps, because the value is ~avg_overlaps/2. + * + * Imagine ranges arranged in "shifted" uniformly by 1/overlaps, e.g. with 3 + * overlaps [0,100], [33,133], [66, 166] and so on. A random value will hit + * only half of there ranges, thus 1/2. This can be extended to randomly + * overlapping ranges. + * + * However, we may not be able to count overlaps for some opclasses (e.g. for + * bloom ranges), in which case we have at least this. + * + * This simply walks the values, and determines matching ranges by looking + * for lower/upper bound in ranges ordered by minval/maxval. + * + * XXX The other question is what to do about duplicate values. If we have a + * very frequent value in the sample, it's likely in many places/ranges. Which + * will skew the average, because it'll be added repeatedly. So we also count + * avg_ranges for unique values. + * + * XXX The relationship that (average_matches ~ average_overlaps/2) only + * works for minmax opclass, and can't be extended to minmax-multi. The + * overlaps can only consider the two extreme values (essentially treating + * the summary as a single minmax range), because that's what brinsort + * needs. But the minmax-multi range may have "gaps" (kinda the whole point + * of these opclasses), which affects matching tuples to ranges. + * + * XXX This also builds histograms of the number of matches, both for the + * raw and unique values. At the moment we don't do anything with the + * results, though (except for printing those). + */ +static void +brin_minmax_match_tuples_to_ranges2(BrinRanges *ranges, + BrinRange **minranges, BrinRange **maxranges, + int numrows, HeapTuple *rows, + int nvalues, Datum *values, + TypeCacheEntry *typcache, + BrinMinmaxStats *stats) +{ + int64 nmatches = 0; + int64 nmatches_unique = 0; + int64 nmatches_value = 0; + int64 nvalues_unique = 0; + + histogram_t *hist = histogram_init(); + histogram_t *hist_unique = histogram_init(); + TimestampTz start_ts = GetCurrentTimestamp(); + + for (int i = 0; i < nvalues; i++) + { + int start; + int end; + + CHECK_FOR_INTERRUPTS(); + + /* + * Same value as preceding, so just use the preceding count. + * We don't increment the unique counters, because this is + * a duplicate. + */ + if ((i > 0) && (range_values_cmp(&values[i-1], &values[i], typcache) == 0)) + { + nmatches += nmatches_value; + hist = histogram_add(hist, nmatches_value); + continue; + } + + nmatches_value = 0; + + start = maxval_start(maxranges, ranges->nranges, values[i], typcache); + end = minval_end(minranges, ranges->nranges, values[i], typcache); + + for (int j = start; j < ranges->nranges; j++) + { + if (maxranges[j]->min_index >= end) + continue; + + if (maxranges[j]->min_index_lowest >= end) + break; + + nmatches_value++; + } + + hist = histogram_add(hist, nmatches_value); + hist_unique = histogram_add(hist_unique, nmatches_value); + + nmatches += nmatches_value; + nmatches_unique += nmatches_value; + nvalues_unique++; + } + + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_match_tuples_to_ranges2 -----"); + elog(WARNING, "nmatches = %ld %f", nmatches, (double) nmatches / numrows); + elog(WARNING, "nmatches unique = %ld %ld %f", + nmatches_unique, nvalues_unique, (double) nmatches_unique / nvalues_unique); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } + + if (stats->avg_matches != (double) nmatches / numrows) + elog(ERROR, "brin_minmax_match_tuples_to_ranges2: avg_matches mismatch %f != %f", + stats->avg_matches, (double) nmatches / numrows); + + if (stats->avg_matches_unique != (double) nmatches_unique / nvalues_unique) + elog(ERROR, "brin_minmax_match_tuples_to_ranges2: avg_matches_unique mismatch %f != %f", + stats->avg_matches_unique, (double) nmatches_unique / nvalues_unique); + + pg_qsort(hist->bins, hist->nbins, sizeof(histogram_bin_t), histogram_bin_cmp); + pg_qsort(hist_unique->bins, hist_unique->nbins, sizeof(histogram_bin_t), histogram_bin_cmp); + + histogram_print(hist); + histogram_print(hist_unique); + + pfree(hist); + pfree(hist_unique); +} + +/* + * brin_minmax_match_tuples_to_ranges_bruteforce + * Match tuples to ranges, count average number of ranges per tuple. + * + * Bruteforce approach, used mostly for cross-checking. + */ +static void +brin_minmax_match_tuples_to_ranges_bruteforce(BrinRanges *ranges, + int numrows, HeapTuple *rows, + int nvalues, Datum *values, + TypeCacheEntry *typcache, + BrinMinmaxStats *stats) +{ + int64 nmatches = 0; + int64 nmatches_unique = 0; + int64 nvalues_unique = 0; + + TimestampTz start_ts = GetCurrentTimestamp(); + + for (int i = 0; i < nvalues; i++) + { + bool is_unique; + int64 nmatches_value = 0; + + CHECK_FOR_INTERRUPTS(); + + /* is this a new value? */ + is_unique = ((i == 0) || (range_values_cmp(&values[i-1], &values[i], typcache) != 0)); + + /* count unique values */ + nvalues_unique += (is_unique) ? 1 : 0; + + for (int j = 0; j < ranges->nranges; j++) + { + if (range_values_cmp(&values[i], &ranges->ranges[j].min_value, typcache) < 0) + continue; + + if (range_values_cmp(&values[i], &ranges->ranges[j].max_value, typcache) > 0) + continue; + + nmatches_value++; + } + + nmatches += nmatches_value; + nmatches_unique += (is_unique) ? nmatches_value : 0; + } + + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_match_tuples_to_ranges_bruteforce -----"); + elog(WARNING, "nmatches = %ld %f", nmatches, (double) nmatches / numrows); + elog(WARNING, "nmatches unique = %ld %ld %f", nmatches_unique, nvalues_unique, + (double) nmatches_unique / nvalues_unique); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } + + if (stats->avg_matches != (double) nmatches / numrows) + elog(ERROR, "brin_minmax_match_tuples_to_ranges_bruteforce: avg_matches mismatch %f != %f", + stats->avg_matches, (double) nmatches / numrows); + + if (stats->avg_matches_unique != (double) nmatches_unique / nvalues_unique) + elog(ERROR, "brin_minmax_match_tuples_to_ranges_bruteforce: avg_matches_unique mismatch %f != %f", + stats->avg_matches_unique, (double) nmatches_unique / nvalues_unique); +} +#endif + /* * brin_minmax_value_stats * Calculate statistics about minval/maxval values. @@ -803,6 +1158,198 @@ brin_minmax_increment_stats(BrinRange **minranges, BrinRange **maxranges, stats->maxval_increment_max = max_maxval; } +#ifdef DEBUG_BRIN_STATS +/* + * brin_minmax_count_overlaps2 + * Calculate number of overlaps. + * + * This uses the minranges/maxranges to quickly eliminate ranges that can't + * possibly intersect. + * + * XXX Seems rather complicated and works poorly for wide ranges (with outlier + * values), brin_minmax_count_overlaps is likely better. + */ +static void +brin_minmax_count_overlaps2(BrinRanges *ranges, + BrinRange **minranges, BrinRange **maxranges, + TypeCacheEntry *typcache, BrinMinmaxStats *stats) +{ + int64 noverlaps; + TimestampTz start_ts = GetCurrentTimestamp(); + + /* + * Walk the ranges ordered by max_values, see how many ranges overlap. + * + * Once we get to a state where (min_value > current.max_value) for + * all future ranges, we know none of them can overlap and we can + * terminate. This is what min_index_lowest is for. + * + * XXX If there are very wide ranges (with outlier min/max values), + * the min_index_lowest is going to be pretty useless, because the + * range will be sorted at the very end by max_value, but will have + * very low min_index, so this won't work. + * + * XXX We could collect a more elaborate stuff, like for example a + * histogram of number of overlaps, or maximum number of overlaps. + * So we'd have average, but then also an info if there are some + * ranges with very many overlaps. + */ + noverlaps = 0; + for (int i = 0; i < ranges->nranges; i++) + { + int idx = (i + 1); + BrinRange *ra = maxranges[i]; + uint64 min_index = ra->min_index; + + CHECK_FOR_INTERRUPTS(); + +#ifdef NOT_USED + /* + * XXX Not needed, we can just count "future" ranges and then + * we just multiply by 2. + */ + + /* + * What's the first range that might overlap with this one? + * needs to have maxval > current.minval. + */ + while (idx > 0) + { + BrinRange *rb = maxranges[idx - 1]; + + /* the range is before the current one, so can't intersect */ + if (range_values_cmp(&rb->max_value, &ra->min_value, typcache) < 0) + break; + + idx--; + } +#endif + + /* + * Find the first min_index that is higher than the max_value, + * so that we can compare that instead of the values in the + * next loop. There should be fewer value comparisons than in + * the next loop, so we'll save on function calls. + */ + while (min_index < ranges->nranges) + { + if (range_values_cmp(&minranges[min_index]->min_value, + &ra->max_value, typcache) > 0) + break; + + min_index++; + } + + /* + * Walk the following ranges (ordered by max_value), and check + * if it overlaps. If it matches, we look at the next one. If + * not, we check if there can be more ranges. + */ + for (int j = idx; j < ranges->nranges; j++) + { + BrinRange *rb = maxranges[j]; + + /* the range overlaps - just continue with the next one */ + // if (range_values_cmp(&rb->min_value, &ra->max_value, typcache) <= 0) + if (rb->min_index < min_index) + { + noverlaps++; + continue; + } + + /* + * Are there any future ranges that might overlap? We can + * check the min_index_lowest to decide quickly. + */ + if (rb->min_index_lowest >= min_index) + break; + } + } + + /* + * We only count intersect for "following" ranges when ordered by maxval, + * so we only see 1/2 the overlaps. So double the result. + */ + noverlaps *= 2; + + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_count_overlaps2 -----"); + elog(WARNING, "noverlaps = %ld", noverlaps); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } + + if (stats->avg_overlaps != (double) noverlaps / ranges->nranges) + elog(ERROR, "brin_minmax_count_overlaps2: mismatch %f != %f", + stats->avg_overlaps, (double) noverlaps / ranges->nranges); +} + +/* + * brin_minmax_count_overlaps_bruteforce + * Calculate number of overlaps by brute force. + * + * Actually compares every range to every other range. Quite expensive, used + * primarily to cross-check the other algorithms. + */ +static void +brin_minmax_count_overlaps_bruteforce(BrinRanges *ranges, + TypeCacheEntry *typcache, + BrinMinmaxStats *stats) +{ + int64 noverlaps; + TimestampTz start_ts = GetCurrentTimestamp(); + + /* + * Brute force calculation of overlapping ranges, comparing each + * range to every other range - bound to be pretty expensive, as + * it's pretty much O(N^2). Kept mostly for easy cross-check with + * the preceding "optimized" code. + */ + noverlaps = 0; + for (int i = 0; i < ranges->nranges; i++) + { + BrinRange *ra = &ranges->ranges[i]; + + for (int j = 0; j < ranges->nranges; j++) + { + BrinRange *rb = &ranges->ranges[j]; + + CHECK_FOR_INTERRUPTS(); + + if (i == j) + continue; + + if (range_values_cmp(&ra->max_value, &rb->min_value, typcache) < 0) + continue; + + if (range_values_cmp(&rb->max_value, &ra->min_value, typcache) < 0) + continue; + +#if 0 + elog(DEBUG1, "[%ld,%ld] overlaps [%ld,%ld]", + ra->min_value, ra->max_value, + rb->min_value, rb->max_value); +#endif + + noverlaps++; + } + } + + if (debug_brin_stats) + { + elog(WARNING, "----- brin_minmax_count_overlaps_bruteforce -----"); + elog(WARNING, "noverlaps = %ld", noverlaps); + elog(WARNING, "duration = %ld", TimestampDifferenceMilliseconds(start_ts, + GetCurrentTimestamp())); + } + + if (stats->avg_overlaps != (double) noverlaps / ranges->nranges) + elog(ERROR, "brin_minmax_count_overlaps2: mismatch %f != %f", + stats->avg_overlaps, (double) noverlaps / ranges->nranges); +} +#endif + /* * brin_minmax_stats * Calculate custom statistics for a BRIN minmax index. @@ -814,6 +1361,11 @@ brin_minmax_increment_stats(BrinRange **minranges, BrinRange **maxranges, * - average number of rows matching a range * - number of distinct minval/maxval values * + * There are multiple ways to calculate some of the metrics, so to allow + * cross-checking during development it's possible to run and compare all. + * To do that, define STATS_CROSS_CHECK. There's also STATS_DEBUG define + * that simply prints the calculated results. + * * XXX This could also calculate correlation of the range minval, so that * we can estimate how much random I/O will happen during the BrinSort. * And perhaps we should also sort the ranges by (minval,block_start) to @@ -1141,6 +1693,14 @@ brin_minmax_stats(PG_FUNCTION_ARGS) /* calculate average number of overlapping ranges for any range */ brin_minmax_count_overlaps(minranges, ranges->nranges, typcache, stats); +#ifdef DEBUG_BRIN_STATS + if (debug_brin_cross_check) + { + brin_minmax_count_overlaps2(ranges, minranges, maxranges, typcache, stats); + brin_minmax_count_overlaps_bruteforce(ranges, typcache, stats); + } +#endif + /* calculate minval/maxval stats (distinct values and correlation) */ brin_minmax_value_stats(minranges, maxranges, ranges->nranges, typcache, stats); @@ -1206,6 +1766,20 @@ brin_minmax_stats(PG_FUNCTION_ARGS) numrows, rows, nvalues, values, typcache, stats); +#ifdef DEBUG_BRIN_STATS + if (debug_brin_cross_check) + { + brin_minmax_match_tuples_to_ranges2(ranges, minranges, maxranges, + numrows, rows, nvalues, values, + typcache, stats); + + brin_minmax_match_tuples_to_ranges_bruteforce(ranges, + numrows, rows, + nvalues, values, + typcache, stats); + } +#endif + brin_minmax_increment_stats(minranges, maxranges, ranges->nranges, values, nvalues, typcache, stats); } diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 9711a23c34..2fa7ac0052 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -100,6 +100,7 @@ extern bool ignore_invalid_pages; #ifdef DEBUG_BRIN_STATS extern bool debug_brin_stats; +extern bool debug_brin_cross_check; #endif #ifdef TRACE_SYNCSCAN @@ -1269,6 +1270,15 @@ struct config_bool ConfigureNamesBool[] = false, NULL, NULL, NULL }, + { + {"debug_brin_cross_check", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Cross-check calculation of BRIN statistics."), + NULL + }, + &debug_brin_cross_check, + false, + NULL, NULL, NULL + }, #endif { -- 2.41.0
0004-Allow-BRIN-indexes-to-produce-sorted-output-20230710.patch (text/x-patch, 132.2K) download From 1cb497ba5ace624194e2c36f251aad9e1259621a Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sun, 9 Oct 2022 11:33:37 +0200 Subject: [PATCH 04/11] Allow BRIN indexes to produce sorted output Some BRIN indexes can be used to produce sorted output, by using the range information to sort tuples incrementally. This is particularly interesting for LIMIT queries, which only need to scan the first few rows, and alternative plans (e.g. Seq Scan + Sort) have a very high startup cost. Of course, if there are e.g. BTREE indexes this is going to be slower, but people are unlikely to have both index types on the same column. This is disabled by default, use enable_brinsort GUC to enable it. --- src/backend/access/brin/brin_minmax.c | 402 ++++ src/backend/commands/explain.c | 44 + src/backend/executor/Makefile | 1 + src/backend/executor/execProcnode.c | 10 + src/backend/executor/meson.build | 1 + src/backend/executor/nodeBrinSort.c | 1612 +++++++++++++++++ src/backend/optimizer/path/costsize.c | 254 +++ src/backend/optimizer/path/indxpath.c | 183 ++ src/backend/optimizer/path/pathkeys.c | 49 + src/backend/optimizer/plan/createplan.c | 260 +++ src/backend/optimizer/plan/setrefs.c | 19 + src/backend/optimizer/util/pathnode.c | 57 + src/backend/utils/misc/guc_tables.c | 28 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/backend/utils/sort/tuplesort.c | 12 + src/include/access/brin.h | 35 - src/include/access/brin_internal.h | 1 + src/include/catalog/pg_amproc.dat | 64 + src/include/catalog/pg_proc.dat | 3 + src/include/executor/nodeBrinSort.h | 47 + src/include/nodes/execnodes.h | 108 ++ src/include/nodes/pathnodes.h | 11 + src/include/nodes/plannodes.h | 32 + src/include/optimizer/cost.h | 3 + src/include/optimizer/pathnode.h | 9 + src/include/optimizer/paths.h | 3 + src/include/utils/tuplesort.h | 1 + src/test/regress/expected/sysviews.out | 3 +- 28 files changed, 3217 insertions(+), 36 deletions(-) create mode 100644 src/backend/executor/nodeBrinSort.c create mode 100644 src/include/executor/nodeBrinSort.h diff --git a/src/backend/access/brin/brin_minmax.c b/src/backend/access/brin/brin_minmax.c index a4c00caafd..162650cf73 100644 --- a/src/backend/access/brin/brin_minmax.c +++ b/src/backend/access/brin/brin_minmax.c @@ -16,6 +16,10 @@ #include "access/brin_tuple.h" #include "access/genam.h" #include "access/stratnum.h" +#include "access/table.h" +#include "access/tableam.h" +#include "catalog/index.h" +#include "catalog/pg_am.h" #include "catalog/pg_amop.h" #include "catalog/pg_type.h" #include "executor/executor.h" @@ -43,6 +47,9 @@ static FmgrInfo *minmax_get_strategy_procinfo(BrinDesc *bdesc, uint16 attno, Oid subtype, uint16 strategynum); +/* print info about ranges */ +#define BRINSORT_DEBUG + Datum brin_minmax_opcinfo(PG_FUNCTION_ARGS) { @@ -1809,6 +1816,401 @@ cleanup: PG_RETURN_POINTER(stats); } +/* + * brin_minmax_range_tupdesc + * Create a tuple descriptor to store BrinRange data. + */ +static TupleDesc +brin_minmax_range_tupdesc(BrinDesc *brdesc, AttrNumber attnum) +{ + TupleDesc tupdesc; + AttrNumber attno = 1; + + /* expect minimum and maximum */ + Assert(brdesc->bd_info[attnum - 1]->oi_nstored == 2); + + tupdesc = CreateTemplateTupleDesc(7); + + /* blkno_start */ + TupleDescInitEntry(tupdesc, attno++, NULL, INT4OID, -1, 0); + + /* blkno_end (could be calculated as blkno_start + pages_per_range) */ + TupleDescInitEntry(tupdesc, attno++, NULL, INT4OID, -1, 0); + + /* has_nulls */ + TupleDescInitEntry(tupdesc, attno++, NULL, BOOLOID, -1, 0); + + /* all_nulls */ + TupleDescInitEntry(tupdesc, attno++, NULL, BOOLOID, -1, 0); + + /* not_summarized */ + TupleDescInitEntry(tupdesc, attno++, NULL, BOOLOID, -1, 0); + + /* min_value */ + TupleDescInitEntry(tupdesc, attno++, NULL, + brdesc->bd_info[attnum - 1]->oi_typcache[0]->type_id, + -1, 0); + + /* max_value */ + TupleDescInitEntry(tupdesc, attno++, NULL, + brdesc->bd_info[attnum - 1]->oi_typcache[0]->type_id, + -1, 0); + + return tupdesc; +} + +/* + * brin_minmax_scan_init + * Prepare the BrinRangeScanDesc including the sorting info etc. + * + * We want to have the ranges in roughly this order + * + * - not-summarized + * - summarized, non-null values + * - summarized, all-nulls + * + * We do it this way, because the not-summarized ranges need to be + * scanned always (both to produce NULL and non-NULL values), and + * we need to read all of them into the tuplesort before producing + * anything. So placing them at the beginning is reasonable. + * + * The all-nulls ranges are placed last, because when processing + * NULLs we need to scan everything anyway (some of the ranges might + * have has_nulls=true). But for non-NULL values we can abort once + * we hit the first all-nulls range. + * + * The regular ranges are sorted by blkno_start, to make it maybe + * a bit more sequential (but this only helps if there are ranges + * with the same minval). + */ +static BrinRangeScanDesc * +brin_minmax_scan_init(BrinDesc *bdesc, Oid collation, AttrNumber attnum, bool asc) +{ + BrinRangeScanDesc *scan; + + /* sort by (not_summarized, minval, blkno_start, all_nulls) */ + AttrNumber keys[4]; + Oid collations[4]; + bool nullsFirst[4]; + Oid operators[4]; + Oid typid; + TypeCacheEntry *typcache; + + /* we expect to have min/max value for each range, same type for both */ + Assert(bdesc->bd_info[attnum - 1]->oi_nstored == 2); + Assert(bdesc->bd_info[attnum - 1]->oi_typcache[0]->type_id == + bdesc->bd_info[attnum - 1]->oi_typcache[1]->type_id); + + scan = (BrinRangeScanDesc *) palloc0(sizeof(BrinRangeScanDesc)); + + /* build tuple descriptor for range data */ + scan->tdesc = brin_minmax_range_tupdesc(bdesc, attnum); + + /* initialize ordering info */ + keys[0] = 5; /* not_summarized */ + keys[1] = 4; /* all_nulls */ + keys[2] = (asc) ? 6 : 7; /* min_value (asc) or max_value (desc) */ + keys[3] = 1; /* blkno_start */ + + collations[0] = InvalidOid; /* FIXME */ + collations[1] = InvalidOid; /* FIXME */ + collations[2] = collation; /* FIXME */ + collations[3] = InvalidOid; /* FIXME */ + + /* unrelated to the ordering desired by the user */ + nullsFirst[0] = false; + nullsFirst[1] = false; + nullsFirst[2] = false; + nullsFirst[3] = false; + + /* lookup sort operator for the boolean type (used for not_summarized) */ + typcache = lookup_type_cache(BOOLOID, TYPECACHE_GT_OPR); + operators[0] = typcache->gt_opr; + + /* lookup sort operator for the boolean type (used for all_nulls) */ + typcache = lookup_type_cache(BOOLOID, TYPECACHE_LT_OPR); + operators[1] = typcache->lt_opr; + + /* lookup sort operator for the min/max type */ + typid = bdesc->bd_info[attnum - 1]->oi_typcache[0]->type_id; + typcache = lookup_type_cache(typid, TYPECACHE_LT_OPR | TYPECACHE_GT_OPR); + operators[2] = (asc) ? typcache->lt_opr : typcache->gt_opr; + + /* lookup sort operator for the bigint type (used for blkno_start) */ + typcache = lookup_type_cache(INT4OID, TYPECACHE_LT_OPR); + operators[3] = typcache->lt_opr; + + /* + * XXX better to keep this small enough to fit into L2/L3, large values + * of work_mem may easily make this slower. + */ + scan->ranges = tuplesort_begin_heap(scan->tdesc, + 4, /* nkeys */ + keys, + operators, + collations, + nullsFirst, + work_mem, + NULL, + TUPLESORT_RANDOMACCESS); + + scan->slot = MakeSingleTupleTableSlot(scan->tdesc, + &TTSOpsMinimalTuple); + + return scan; +} + +/* + * brin_minmax_scan_add_tuple + * Form and store a tuple representing the BRIN range to the tuplestore. + */ +static void +brin_minmax_scan_add_tuple(BrinRangeScanDesc *scan, TupleTableSlot *slot, + BlockNumber block_start, BlockNumber block_end, + bool has_nulls, bool all_nulls, bool not_summarized, + Datum min_value, Datum max_value) +{ + ExecClearTuple(slot); + + memset(slot->tts_isnull, false, 7 * sizeof(bool)); + + slot->tts_values[0] = UInt32GetDatum(block_start); + slot->tts_values[1] = UInt32GetDatum(block_end); + slot->tts_values[2] = BoolGetDatum(has_nulls); + slot->tts_values[3] = BoolGetDatum(all_nulls); + slot->tts_values[4] = BoolGetDatum(not_summarized); + slot->tts_values[5] = min_value; + slot->tts_values[6] = max_value; + + if (all_nulls || not_summarized) + { + slot->tts_isnull[5] = true; + slot->tts_isnull[6] = true; + } + + ExecStoreVirtualTuple(slot); + + tuplesort_puttupleslot(scan->ranges, slot); + + scan->nranges++; +} + +#ifdef BRINSORT_DEBUG +/* + * brin_minmax_scan_next + * Return the next BRIN range information from the tuplestore. + * + * Returns NULL when there are no more ranges. + */ +static BrinRange * +brin_minmax_scan_next(BrinRangeScanDesc *scan) +{ + if (tuplesort_gettupleslot(scan->ranges, true, false, scan->slot, NULL)) + { + bool isnull; + BrinRange *range = (BrinRange *) palloc(sizeof(BrinRange)); + + range->blkno_start = slot_getattr(scan->slot, 1, &isnull); + range->blkno_end = slot_getattr(scan->slot, 2, &isnull); + range->has_nulls = slot_getattr(scan->slot, 3, &isnull); + range->all_nulls = slot_getattr(scan->slot, 4, &isnull); + range->not_summarized = slot_getattr(scan->slot, 5, &isnull); + range->min_value = slot_getattr(scan->slot, 6, &isnull); + range->max_value = slot_getattr(scan->slot, 7, &isnull); + + return range; + } + + return NULL; +} + +/* + * brin_minmax_scan_dump + * Print info about all page ranges stored in the tuplestore. + */ +static void +brin_minmax_scan_dump(BrinRangeScanDesc *scan) +{ + BrinRange *range; + + if (!message_level_is_interesting(WARNING)) + return; + + elog(WARNING, "===== dumping ====="); + while ((range = brin_minmax_scan_next(scan)) != NULL) + { + elog(WARNING, "[%u %u] has_nulls %d all_nulls %d not_summarized %d values [%ld %ld]", + range->blkno_start, range->blkno_end, + range->has_nulls, range->all_nulls, range->not_summarized, + range->min_value, range->max_value); + + pfree(range); + } + + /* reset the tuplestore, so that we can start scanning again */ + tuplesort_rescan(scan->ranges); +} +#endif + +static void +brin_minmax_scan_finalize(BrinRangeScanDesc *scan) +{ + tuplesort_performsort(scan->ranges); +} + +/* + * brin_minmax_ranges + * Load the BRIN ranges and sort them. + */ +Datum +brin_minmax_ranges(PG_FUNCTION_ARGS) +{ + IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0); + AttrNumber attnum = PG_GETARG_INT16(1); + bool asc = PG_GETARG_BOOL(2); + Oid colloid = PG_GET_COLLATION(); + BrinOpaque *opaque; + Relation indexRel; + Relation heapRel; + BlockNumber nblocks; + BlockNumber heapBlk; + Oid heapOid; + BrinMemTuple *dtup; + BrinTuple *btup = NULL; + Size btupsz = 0; + Buffer buf = InvalidBuffer; + BlockNumber pagesPerRange; + BrinDesc *bdesc; + BrinRangeScanDesc *brscan; + TupleTableSlot *slot; + + /* + * Determine how many BRIN ranges could there be, allocate space and read + * all the min/max values. + */ + opaque = (BrinOpaque *) scan->opaque; + bdesc = opaque->bo_bdesc; + pagesPerRange = opaque->bo_pagesPerRange; + + indexRel = bdesc->bd_index; + + /* make sure the provided attnum is valid */ + Assert((attnum > 0) && (attnum <= bdesc->bd_tupdesc->natts)); + + /* + * We need to know the size of the table so that we know how long to iterate + * on the revmap (and to pre-allocate the arrays). + */ + heapOid = IndexGetRelation(RelationGetRelid(indexRel), false); + heapRel = table_open(heapOid, AccessShareLock); + nblocks = RelationGetNumberOfBlocks(heapRel); + table_close(heapRel, AccessShareLock); + + /* allocate an initial in-memory tuple, out of the per-range memcxt */ + dtup = brin_new_memtuple(bdesc); + + /* initialize the scan describing scan of ranges sorted by minval */ + brscan = brin_minmax_scan_init(bdesc, colloid, attnum, asc); + + slot = MakeSingleTupleTableSlot(brscan->tdesc, &TTSOpsVirtual); + + /* + * Now scan the revmap. We start by querying for heap page 0, + * incrementing by the number of pages per range; this gives us a full + * view of the table. + * + * XXX The sort may be quite expensive, e.g. for small BRIN ranges. Maybe + * we could optimize this somehow? For example, we know the not-summarized + * ranges are always going to be first, and all-null ranges last, so maybe + * we could stash those somewhere, and not sort them? But there are likely + * only very few such ranges, in most cases. Moreover, how would we then + * prepend/append those ranges to the sorted ones? Probably would have to + * store them in a tuplestore, or something. + * + * XXX Seems that having large work_mem can be quite detrimental, because + * then it overflows L2/L3 caches, making the sort much slower. + * + * XXX If there are other indexes, would be great to filter the ranges, so + * that we only sort the interesting ones - reduces the number of ranges, + * makes the sort faster. + * + * XXX Another option is making this incremental - e.g. only ask for the + * first 1000 ranges, using a top-N sort. And then if it's not enough we + * could request another chunk. But the second request would have to be + * rather unlikely (because quite expensive), and the top-N sort does not + * seem all that faster (as long as we don't overflow L2/L3). + */ + for (heapBlk = 0; heapBlk < nblocks; heapBlk += pagesPerRange) + { + bool gottuple = false; + BrinTuple *tup; + OffsetNumber off; + Size size; + + CHECK_FOR_INTERRUPTS(); + + tup = brinGetTupleForHeapBlock(opaque->bo_rmAccess, heapBlk, &buf, + &off, &size, BUFFER_LOCK_SHARE, + scan->xs_snapshot); + if (tup) + { + gottuple = true; + btup = brin_copy_tuple(tup, size, btup, &btupsz); + LockBuffer(buf, BUFFER_LOCK_UNLOCK); + } + + /* + * Ranges with no indexed tuple may contain anything. + */ + if (!gottuple) + { + brin_minmax_scan_add_tuple(brscan, slot, + heapBlk, heapBlk + (pagesPerRange - 1), + false, false, true, 0, 0); + } + else + { + dtup = brin_deform_tuple(bdesc, btup, dtup); + if (dtup->bt_placeholder) + { + /* + * Placeholder tuples are treated as if not summarized. + * + * XXX Is this correct? + */ + brin_minmax_scan_add_tuple(brscan, slot, + heapBlk, heapBlk + (pagesPerRange - 1), + false, false, true, 0, 0); + } + else + { + BrinValues *bval; + + bval = &dtup->bt_columns[attnum - 1]; + + brin_minmax_scan_add_tuple(brscan, slot, + heapBlk, heapBlk + (pagesPerRange - 1), + bval->bv_hasnulls, bval->bv_allnulls, false, + bval->bv_values[0], bval->bv_values[1]); + } + } + } + + ExecDropSingleTupleTableSlot(slot); + + if (buf != InvalidBuffer) + ReleaseBuffer(buf); + + /* do the sort and any necessary post-processing */ + brin_minmax_scan_finalize(brscan); + +#ifdef BRINSORT_DEBUG + brin_minmax_scan_dump(brscan); +#endif + + PG_RETURN_POINTER(brscan); +} + /* * Cache and return the procedure for the given strategy. * diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 8570b14f62..d2eae9ea1f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -85,6 +85,8 @@ static void show_sort_keys(SortState *sortstate, List *ancestors, ExplainState *es); static void show_incremental_sort_keys(IncrementalSortState *incrsortstate, List *ancestors, ExplainState *es); +static void show_brinsort_keys(BrinSortState *sortstate, List *ancestors, + ExplainState *es); static void show_merge_append_keys(MergeAppendState *mstate, List *ancestors, ExplainState *es); static void show_agg_keys(AggState *astate, List *ancestors, @@ -1111,6 +1113,7 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used) case T_IndexScan: case T_IndexOnlyScan: case T_BitmapHeapScan: + case T_BrinSort: case T_TidScan: case T_TidRangeScan: case T_SubqueryScan: @@ -1273,6 +1276,9 @@ ExplainNode(PlanState *planstate, List *ancestors, case T_IndexOnlyScan: pname = sname = "Index Only Scan"; break; + case T_BrinSort: + pname = sname = "BRIN Sort"; + break; case T_BitmapIndexScan: pname = sname = "Bitmap Index Scan"; break; @@ -1519,6 +1525,16 @@ ExplainNode(PlanState *planstate, List *ancestors, ExplainScanTarget((Scan *) indexonlyscan, es); } break; + case T_BrinSort: + { + BrinSort *brinsort = (BrinSort *) plan; + + ExplainIndexScanDetails(brinsort->indexid, + brinsort->indexorderdir, + es); + ExplainScanTarget((Scan *) brinsort, es); + } + break; case T_BitmapIndexScan: { BitmapIndexScan *bitmapindexscan = (BitmapIndexScan *) plan; @@ -1804,6 +1820,18 @@ ExplainNode(PlanState *planstate, List *ancestors, ExplainPropertyFloat("Heap Fetches", NULL, planstate->instrument->ntuples2, 0, es); break; + case T_BrinSort: + show_scan_qual(((BrinSort *) plan)->indexqualorig, + "Index Cond", planstate, ancestors, es); + if (((BrinSort *) plan)->indexqualorig) + show_instrumentation_count("Rows Removed by Index Recheck", 2, + planstate, es); + show_scan_qual(plan->qual, "Filter", planstate, ancestors, es); + show_brinsort_keys(castNode(BrinSortState, planstate), ancestors, es); + if (plan->qual) + show_instrumentation_count("Rows Removed by Filter", 1, + planstate, es); + break; case T_BitmapIndexScan: show_scan_qual(((BitmapIndexScan *) plan)->indexqualorig, "Index Cond", planstate, ancestors, es); @@ -2403,6 +2431,21 @@ show_incremental_sort_keys(IncrementalSortState *incrsortstate, ancestors, es); } +/* + * Show the sort keys for a BRIN Sort node. + */ +static void +show_brinsort_keys(BrinSortState *sortstate, List *ancestors, ExplainState *es) +{ + BrinSort *plan = (BrinSort *) sortstate->ss.ps.plan; + + show_sort_group_keys((PlanState *) sortstate, "Sort Key", + plan->numCols, 0, plan->sortColIdx, + plan->sortOperators, plan->collations, + plan->nullsFirst, + ancestors, es); +} + /* * Likewise, for a MergeAppend node. */ @@ -3823,6 +3866,7 @@ ExplainTargetRel(Plan *plan, Index rti, ExplainState *es) case T_ForeignScan: case T_CustomScan: case T_ModifyTable: + case T_BrinSort: /* Assert it's on a real relation */ Assert(rte->rtekind == RTE_RELATION); objectname = get_rel_name(rte->relid); diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile index 11118d0ce0..bcaa2ce8e2 100644 --- a/src/backend/executor/Makefile +++ b/src/backend/executor/Makefile @@ -38,6 +38,7 @@ OBJS = \ nodeBitmapHeapscan.o \ nodeBitmapIndexscan.o \ nodeBitmapOr.o \ + nodeBrinSort.o \ nodeCtescan.o \ nodeCustom.o \ nodeForeignscan.o \ diff --git a/src/backend/executor/execProcnode.c b/src/backend/executor/execProcnode.c index 4d288bc8d4..93d1007809 100644 --- a/src/backend/executor/execProcnode.c +++ b/src/backend/executor/execProcnode.c @@ -79,6 +79,7 @@ #include "executor/nodeBitmapHeapscan.h" #include "executor/nodeBitmapIndexscan.h" #include "executor/nodeBitmapOr.h" +#include "executor/nodeBrinSort.h" #include "executor/nodeCtescan.h" #include "executor/nodeCustom.h" #include "executor/nodeForeignscan.h" @@ -226,6 +227,11 @@ ExecInitNode(Plan *node, EState *estate, int eflags) estate, eflags); break; + case T_BrinSort: + result = (PlanState *) ExecInitBrinSort((BrinSort *) node, + estate, eflags); + break; + case T_BitmapIndexScan: result = (PlanState *) ExecInitBitmapIndexScan((BitmapIndexScan *) node, estate, eflags); @@ -639,6 +645,10 @@ ExecEndNode(PlanState *node) ExecEndIndexOnlyScan((IndexOnlyScanState *) node); break; + case T_BrinSortState: + ExecEndBrinSort((BrinSortState *) node); + break; + case T_BitmapIndexScanState: ExecEndBitmapIndexScan((BitmapIndexScanState *) node); break; diff --git a/src/backend/executor/meson.build b/src/backend/executor/meson.build index 65f9457c9b..ed7f38a139 100644 --- a/src/backend/executor/meson.build +++ b/src/backend/executor/meson.build @@ -26,6 +26,7 @@ backend_sources += files( 'nodeBitmapHeapscan.c', 'nodeBitmapIndexscan.c', 'nodeBitmapOr.c', + 'nodeBrinSort.c', 'nodeCtescan.c', 'nodeCustom.c', 'nodeForeignscan.c', diff --git a/src/backend/executor/nodeBrinSort.c b/src/backend/executor/nodeBrinSort.c new file mode 100644 index 0000000000..9505eafc54 --- /dev/null +++ b/src/backend/executor/nodeBrinSort.c @@ -0,0 +1,1612 @@ +/*------------------------------------------------------------------------- + * + * nodeBrinSort.c + * Routines to support sorted scan of relations using a BRIN index + * + * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * The overall algorithm is roughly this: + * + * 0) initialize a tuplestore and a tuplesort + * + * 1) fetch list of page ranges from the BRIN index, sorted by minval + * (with the not-summarized ranges first, and all-null ranges last) + * + * 2) for NULLS FIRST ordering, walk all ranges that may contain NULL + * values and output them (and return to the beginning of the list) + * + * 3) while there are ranges in the list, do this: + * + * a) get next (distinct) minval from the list, call it watermark + * + * b) if there are any tuples in the tuplestore, move them to tuplesort + * + * c) process all ranges with (minval < watermark) - read tuples and feed + * them either into tuplestore (when value < watermark) or tuplestore + * + * d) sort the tuplestore, output all the tuples + * + * 4) if some tuples remain in the tuplestore, sort and output them + * + * 5) for NULLS LAST ordering, walk all ranges that may contain NULL + * values and output them (and return to the beginning of the list) + * + * + * For DESC orderings the process is almost the same, except that we look + * at maxval and use '>' operator (but that's transparent). + * + * There's a couple possible things that might be done in different ways: + * + * 1) Not using tuplestore, and feeding tuples only to a tuplesort. Then + * while producing the tuples, we'd only output tuples up to the current + * watermark, and then we'd keep the remaining tuples for the next round. + * Either we'd need to transfer them into a second tuplesort, or allow + * "reopening" the tuplesort and adding more tuples. And then only the + * part since the watermark would get sorted (possibly using a merge-sort + * with the already sorted part). + * + * + * 2) The other question is what to do with NULL values - at the moment we + * just read the ranges, output the NULL tuples and that's it - we're not + * retaining any non-NULL tuples, so that we'll read the ranges again in + * the second range. The logic here is that either there are very few + * such ranges, so it's won't cost much to just re-read them. Or maybe + * there are very many such ranges, and we'd do a lot of spilling to the + * tuplestore, and it's not much more expensive to just re-read the source + * data. There are counter-examples, though - e.g., there might be many + * has_nulls ranges, but with very few non-NULL tuples. In this case it + * might be better to actually spill the tuples instead of re-reading all + * the ranges. Maybe this is something we can do at run-time, or maybe we + * could estimate this at planning time. We do know the null_frac for the + * column, so we know the number of NULL rows. And we also know the number + * of all_nulls and has_nulls ranges. We can estimate the number of rows + * per range, and we can estimate how many non-NULL rows are in the + * has_nulls ranges (we don't need to re-read all-nulls ranges). There's + * also the filter, which may reduce the amount of rows to store. + * + * So we'd need to compare two metrics calculated roughly like this: + * + * cost(re-reading has-nulls ranges) + * = cost(random_page_cost * n_has_nulls + seq_page_cost * pages_per_range) + * + * cost(spilling non-NULL rows from has-nulls ranges) + * = cost(numrows * width / BLCKSZ * seq_page_cost * 2) + * + * where numrows is the number of non-NULL rows in has_null ranges, which + * can be calculated like this: + * + * // estimated number of rows in has-null ranges + * rows_in_has_nulls = (reltuples / relpages) * pages_per_range * n_has_nulls + * + * // number of NULL rows in the has-nulls ranges + * nulls_in_ranges = reltuples * null_frac - n_all_nulls * (reltuples / relpages) + * + * // numrows is the difference, multiplied by selectivity of the index + * // filter condition (value between 0.0 and 1.0) + * numrows = (rows_in_has_nulls - nulls_in_ranges) * selectivity + * + * This ignores non-summarized ranges, but there should be only very few of + * those, so it should not make a huge difference. Otherwise we can divide + * them between regular, has-nulls and all-nulls pages to keep the ratio. + * + * + * 3) How large step to make when updating the watermark? + * + * When updating the watermark, one option is to simply proceed to the next + * distinct minval value, which is the smallest possible step we can make. + * This may be both fine and very inefficient, depending on how many rows + * end up in the tuplesort and how many rows we end up spilling (possibly + * repeatedly to the tuplestore). + * + * When having to sort large number of rows, it's inefficient to run many + * tiny sorts, even if it produces correct result. For example when sorting + * 1M rows, we may split this as either (a) 100000x sorts of 10 rows, or + * (b) 1000 sorts of 1000 rows. The (b) option is almost certainly more + * efficient. Maybe sorts of 10k rows would be even better, if it fits + * into work_mem. + * + * This gets back to how large the page ranges are, and if/how much they + * overlap. With tiny ranges (e.g. a single-page ranges), a single range + * can only add as many rows as we can fit on a single page. So we need + * more ranges by default - how many watermark steps that is depends on + * how many distinct minval values are there ... + * + * Then there's overlaps - if ranges do not overlap, we're done and we'll + * add the whole range because the next watermark is above maxval. But + * when the ranges overlap, we'll only add the first part (assuming the + * minval of the next range is the watermark). Assume 10 overlapping + * ranges - imagine for example ranges shifted by 10%, so something like + * + * [0,100] [10,110], [20,120], [30, 130], ..., [90, 190] + * + * In the first step we use watermark=10 and load the first range, with + * maybe 1000 rows in total. But assuming uniform distribution, only about + * 100 rows will go into the tuplesort, the remaining 900 rows will go into + * the tuplestore (assuming uniform distribution). Then in the second step + * we sort another 100 rows and the remaining 800 rows will be moved into + * a new tuplestore. And so on and so on. + * + * This means that incrementing the watermarks by single steps may be + * quite inefficient, and we need to reflect both the range size and + * how much the ranges overlap. + * + * In fact, maybe we should not determine the step as number of minval + * values to skip, but how many ranges would that mean reading. Because + * if we have a minval with many duplicates, that may load many rows. + * Or even better, we could look at how many rows would that mean loading + * into the tuplestore - if we track P(x<minval) for each range (e.g. by + * calculating average value during ANALYZE, or perhaps by estimating + * it from per-column stats), then we know the increment is going to be + * about + * + * P(x < minval[i]) - P(x < minval[i-1]) + * + * and we can stop once we'd exceed work_mem (with some slack). See comment + * for brin_minmax_stats() for more thoughts. + * + * + * 4) LIMIT/OFFSET vs. full sort + * + * There's one case where very small sorts may be actually optimal, and + * that's queries that need to process only very few rows - say, LIMIT + * queries with very small bound. + * + * + * FIXME handling of other brin opclasses (minmax-multi) + * + * FIXME improve costing + * + * + * Improvement ideas: + * + * 1) multiple tuplestores for overlapping ranges + * + * When there are many overlapping ranges (so that maxval > current.maxval), + * we're loading all the "future" tuples into a new tuplestore. However, if + * there are multiple such ranges (imagine ranges "shifting" by 10%, which + * gives us 9 more ranges), we know in the next round we'll only need rows + * until the next maxval. We'll not sort these rows, but we'll still shuffle + * them around until we get to the proper range (so about 10x each row). + * Maybe we should pre-allocate the tuplestores (or maybe even tuplesorts) + * for future ranges, and route the tuples to the correct one? Maybe we + * could be a bit smarter and discard tuples once we have enough rows for + * the preceding ranges (say, with LIMIT queries). We'd also need to worry + * about work_mem, though - we can't just use many tuplestores, each with + * whole work_mem. So we'd probably use e.g. work_mem/2 for the next one, + * and then /4, /8 etc. for the following ones. That's work_mem in total. + * And there'd need to be some limit on number of tuplestores, I guess. + * + * 2) handling NULL values + * + * We need to handle NULLS FIRST / NULLS LAST cases. The question is how + * to do that - the easiest way is to simply do a separate scan of ranges + * that might contain NULL values, processing just rows with NULLs, and + * discarding other rows. And then process non-NULL values as currently. + * The NULL scan would happen before/after this regular phase. + * + * Byt maybe we could be smarter, and not do separate scans. When reading + * a page, we might stash the tuple in a tuplestore, so that we can read + * it the next round. Obviously, this might be expensive if we need to + * keep too many rows, so the tuplestore would grow too large - in that + * case it might be better to just do the two scans. + * + * 3) parallelism + * + * Presumably we could do a parallel version of this. The leader or first + * worker would prepare the range information, and the workers would then + * grab ranges (in a kinda round robin manner), sort them independently, + * and then the results would be merged by Gather Merge. + * + * IDENTIFICATION + * src/backend/executor/nodeBrinSort.c + * + *------------------------------------------------------------------------- + */ +/* + * INTERFACE ROUTINES + * ExecBrinSort scans a relation using an index + * IndexNext retrieve next tuple using index + * ExecInitBrinSort creates and initializes state info. + * ExecReScanBrinSort rescans the indexed relation. + * ExecEndBrinSort releases all storage. + * ExecBrinSortMarkPos marks scan position. + * ExecBrinSortRestrPos restores scan position. + * ExecBrinSortEstimate estimates DSM space needed for parallel index scan + * ExecBrinSortInitializeDSM initialize DSM for parallel BrinSort + * ExecBrinSortReInitializeDSM reinitialize DSM for fresh scan + * ExecBrinSortInitializeWorker attach to DSM info in parallel worker + */ +#include "postgres.h" + +#include "access/brin.h" +#include "access/brin_internal.h" +#include "access/nbtree.h" +#include "access/relscan.h" +#include "access/table.h" +#include "access/tableam.h" +#include "catalog/index.h" +#include "catalog/pg_am.h" +#include "executor/execdebug.h" +#include "executor/nodeBrinSort.h" +#include "lib/pairingheap.h" +#include "miscadmin.h" +#include "nodes/nodeFuncs.h" +#include "utils/array.h" +#include "utils/datum.h" +#include "utils/lsyscache.h" +#include "utils/memutils.h" +#include "utils/rel.h" + + +static TupleTableSlot *IndexNext(BrinSortState *node); +static bool IndexRecheck(BrinSortState *node, TupleTableSlot *slot); +static void ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate); + +#ifdef DEBUG_BRIN_SORT +bool debug_brin_sort = false; +#endif + +/* do various consistency checks */ +static void +AssertCheckRanges(BrinSortState *node) +{ +#ifdef USE_ASSERT_CHECKING + +#endif +} + +/* + * brinsort_start_tidscan + * Start scanning tuples from a given page range. + * + * We open a TID range scan for the given range, and initialize the tuplesort. + * Optionally, we update the watermark (with either high/low value). We only + * need to do this for the main page range, not for the intersecting ranges. + * + * XXX Maybe we should initialize the tidscan only once, and then do rescan + * for the following ranges? And similarly for the tuplesort? + */ +static void +brinsort_start_tidscan(BrinSortState *node) +{ + BrinSort *plan = (BrinSort *) node->ss.ps.plan; + EState *estate = node->ss.ps.state; + BrinRange *range = node->bs_range; + + /* There must not be any TID scan in progress yet. */ + Assert(node->ss.ss_currentScanDesc == NULL); + + /* Initialize the TID range scan, for the provided block range. */ + if (node->ss.ss_currentScanDesc == NULL) + { + TableScanDesc tscandesc; + ItemPointerData mintid, + maxtid; + + ItemPointerSetBlockNumber(&mintid, range->blkno_start); + ItemPointerSetOffsetNumber(&mintid, 0); + + ItemPointerSetBlockNumber(&maxtid, range->blkno_end); + ItemPointerSetOffsetNumber(&maxtid, MaxHeapTuplesPerPage); + + elog(DEBUG1, "loading range blocks [%u, %u]", + range->blkno_start, range->blkno_end); + + tscandesc = table_beginscan_tidrange(node->ss.ss_currentRelation, + estate->es_snapshot, + &mintid, &maxtid); + node->ss.ss_currentScanDesc = tscandesc; + } + + if (node->bs_tuplesortstate == NULL) + { + TupleDesc tupDesc = (node->ss.ps.ps_ResultTupleDesc); + + node->bs_tuplesortstate = tuplesort_begin_heap(tupDesc, + plan->numCols, + plan->sortColIdx, + plan->sortOperators, + plan->collations, + plan->nullsFirst, + work_mem, + NULL, + TUPLESORT_NONE); + } + + if (node->bs_tuplestore == NULL) + { + node->bs_tuplestore = tuplestore_begin_heap(false, false, work_mem); + } +} + +/* + * brinsort_end_tidscan + * Finish the TID range scan. + */ +static void +brinsort_end_tidscan(BrinSortState *node) +{ + /* get the first range, read all tuples using a tid range scan */ + if (node->ss.ss_currentScanDesc != NULL) + { + table_endscan(node->ss.ss_currentScanDesc); + node->ss.ss_currentScanDesc = NULL; + } +} + +/* + * brinsort_update_watermark + * Advance the watermark to the next minval (or maxval for DESC). + * + * We could could actually advance the watermark by multiple steps (not to + * the immediately following minval, but a couple more), to accumulate more + * rows in the tuplesort. The number of steps we make correlates with the + * amount of data we sort in a given step, but we don't know in advance + * how many rows (or bytes) will that actually be. We could do some simple + * heuristics (measure past sorts and extrapolate). + * + * XXX With a separate _set and _empty flags, we don't really need to pass + * a separate "first" parameter - "set=false" has the same meaning. + */ +static void +brinsort_update_watermark(BrinSortState *node, bool asc) +{ + int cmp; + bool found = false; + + tuplesort_markpos(node->bs_scan->ranges); + + while (tuplesort_gettupleslot(node->bs_scan->ranges, true, false, node->bs_scan->slot, NULL)) + { + bool isnull; + Datum value; + bool all_nulls; + bool not_summarized; + + all_nulls = DatumGetBool(slot_getattr(node->bs_scan->slot, 4, &isnull)); + Assert(!isnull); + + not_summarized = DatumGetBool(slot_getattr(node->bs_scan->slot, 5, &isnull)); + Assert(!isnull); + + /* we ignore ranges that are either all_nulls or not summarized */ + if (all_nulls || not_summarized) + continue; + + /* use either minval or maxval, depending on the ASC / DESC */ + if (asc) + value = slot_getattr(node->bs_scan->slot, 6, &isnull); + else + value = slot_getattr(node->bs_scan->slot, 7, &isnull); + + if (!node->bs_watermark_set) + { + node->bs_watermark_set = true; + node->bs_watermark = value; + continue; + } + + cmp = ApplySortComparator(node->bs_watermark, false, value, false, + &node->bs_sortsupport); + + if (cmp < 0) + { + node->bs_watermark_set = true; + node->bs_watermark = value; + found = true; + break; + } + } + + tuplesort_restorepos(node->bs_scan->ranges); + + node->bs_watermark_empty = (!found); +} + +/* + * brinsort_load_tuples + * Load tuples from the TID range scan, add them to tuplesort/store. + * + * When called for the "current" range, we don't need to check the watermark, + * we know the tuple goes into the tuplesort. So with check_watermark we + * skip the comparator call to save CPU cost. + */ +static void +brinsort_load_tuples(BrinSortState *node, bool check_watermark, bool null_processing) +{ + BrinSort *plan = (BrinSort *) node->ss.ps.plan; + TableScanDesc scan; + EState *estate; + ScanDirection direction; + TupleTableSlot *slot; + BrinRange *range = node->bs_range; + ProjectionInfo *projInfo; + + estate = node->ss.ps.state; + direction = estate->es_direction; + projInfo = node->bs_ProjInfo; + + slot = node->ss.ss_ScanTupleSlot; + + Assert(node->bs_range != NULL); + + /* + * If we're not processign NULLS, and this is all-nulls range, we can + * just skip it - we won't find any non-NULL tuples in it. + * + * XXX Shouldn't happen, thanks to logic in brinsort_next_range(). + */ + if (!null_processing && range->all_nulls) + return; + + /* + * Similarly, if we're processing NULLs and this range does not have + * has_nulls flag, we can skip it. + * + * XXX Shouldn't happen, thanks to logic in brinsort_next_range(). + */ + if (null_processing && !(range->has_nulls || range->not_summarized || range->all_nulls)) + return; + + brinsort_start_tidscan(node); + + scan = node->ss.ss_currentScanDesc; + + /* + * Read tuples, evaluate the filter (so that we don't keep tuples only to + * discard them later), and decide if it goes into the current range + * (tuplesort) or overflow (tuplestore). + */ + while (table_scan_getnextslot_tidrange(scan, direction, slot)) + { + ExprContext *econtext; + ExprState *qual; + + /* + * Fetch data from node + */ + qual = node->bs_qual; + econtext = node->ss.ps.ps_ExprContext; + + /* + * place the current tuple into the expr context + */ + econtext->ecxt_scantuple = slot; + + /* + * check that the current tuple satisfies the qual-clause + * + * check for non-null qual here to avoid a function call to ExecQual() + * when the qual is null ... saves only a few cycles, but they add up + * ... + * + * XXX Done here, because in ExecScan we'll get different slot type + * (minimal tuple vs. buffered tuple). Scan expects slot while reading + * from the table (like here), but we're stashing it into a tuplesort. + * + * XXX Maybe we could eliminate many tuples by leveraging the BRIN + * range, by executing the consistent function. But we don't have + * the qual in appropriate format at the moment, so we'd preprocess + * the keys similarly to bringetbitmap(). In which case we should + * probably evaluate the stuff while building the ranges? Although, + * if the "consistent" function is expensive, it might be cheaper + * to do that incrementally, as we need the ranges. Would be a win + * for LIMIT queries, for example. + * + * XXX However, maybe we could also leverage other bitmap indexes, + * particularly for BRIN indexes because that makes it simpler to + * eliminate the ranges incrementally - we know which ranges to + * load from the index, while for other indexes (e.g. btree) we + * have to read the whole index and build a bitmap in order to have + * a bitmap for any range. Although, if the condition is very + * selective, we may need to read only a small fraction of the + * index, so maybe that's OK. + */ + if (qual == NULL || ExecQual(qual, econtext)) + { + int cmp = 0; /* matters for check_watermark=false */ + Datum value; + bool isnull; + TupleTableSlot *tmpslot; + + if (projInfo) + tmpslot = ExecProject(projInfo); + else + tmpslot = slot; + + value = slot_getattr(tmpslot, plan->sortColIdx[0], &isnull); + + /* + * Handle NULL values - stash them into the tuplestore, and then + * we'll output them in "process" stage. + * + * XXX Can we be a bit smarter for LIMIT queries and stop reading + * rows once we get the number we need to produce? Probably not, + * because the ordering may reference other columns (which we may + * satisfy through IncrementalSort). But all NULL columns are + * considered equal, so we need all the rows to properly compare + * the other keys. + */ + if (null_processing) + { + /* Stash it to the tuplestore (when NULL, or ignore + * it (when not-NULL). */ + if (isnull) + tuplestore_puttupleslot(node->bs_tuplestore, tmpslot); + + /* NULL or not, we're done */ + continue; + } + + /* we're not processing NULL values, so ignore NULLs */ + if (isnull) + continue; + + /* + * Otherwise compare to watermark, and stash it either to the + * tuplesort or tuplestore. + */ + if (check_watermark && node->bs_watermark_set && !node->bs_watermark_empty) + cmp = ApplySortComparator(value, false, + node->bs_watermark, false, + &node->bs_sortsupport); + + if (cmp <= 0) + tuplesort_puttupleslot(node->bs_tuplesortstate, tmpslot); + else + { + /* + * XXX We can be a bit smarter for LIMIT queries - once we + * know we have more rows in the tuplesort than we need to + * output, we can stop spilling - those rows are not going + * to be needed. We can discard the tuplesort (no need to + * respill) and stop spilling. + */ + tuplestore_puttupleslot(node->bs_tuplestore, tmpslot); + } + } + + ExecClearTuple(slot); + } + + ExecClearTuple(slot); + + brinsort_end_tidscan(node); +} + +/* + * brinsort_load_spill_tuples + * Load tuples from the spill tuplestore, and either stash them into + * a tuplesort or a new tuplestore. + * + * After processing the last range, we want to process all remaining ranges, + * so with check_watermark=false we skip the check. + */ +static void +brinsort_load_spill_tuples(BrinSortState *node, bool check_watermark) +{ + BrinSort *plan = (BrinSort *) node->ss.ps.plan; + Tuplestorestate *tupstore; + TupleTableSlot *slot; + ProjectionInfo *projInfo; + + projInfo = node->bs_ProjInfo; + + if (node->bs_tuplestore == NULL) + return; + + /* start scanning the existing tuplestore (XXX needed?) */ + tuplestore_rescan(node->bs_tuplestore); + + /* + * Create a new tuplestore, for tuples that exceed the watermark and so + * should not be included in the current sort. + */ + tupstore = tuplestore_begin_heap(false, false, work_mem); + + /* + * We need a slot for minimal tuples. The scan slot uses buffered tuples, + * so it'd trigger an error in the loop. + */ + if (projInfo) + slot = node->ss.ps.ps_ResultTupleSlot; + else + slot = MakeSingleTupleTableSlot(RelationGetDescr(node->ss.ss_currentRelation), + &TTSOpsMinimalTuple); + + while (tuplestore_gettupleslot(node->bs_tuplestore, true, true, slot)) + { + int cmp = 0; /* matters for check_watermark=false */ + bool isnull; + Datum value; + + value = slot_getattr(slot, plan->sortColIdx[0], &isnull); + + /* We shouldn't have NULL values in the spill, at least not now. */ + Assert(!isnull); + + if (check_watermark && node->bs_watermark_set && !node->bs_watermark_empty) + cmp = ApplySortComparator(value, false, + node->bs_watermark, false, + &node->bs_sortsupport); + + if (cmp <= 0) + tuplesort_puttupleslot(node->bs_tuplesortstate, slot); + else + { + /* + * XXX We can be a bit smarter for LIMIT queries - once we + * know we have more rows in the tuplesort than we need to + * output, we can stop spilling - those rows are not going + * to be needed. We can discard the tuplesort (no need to + * respill) and stop spilling. + */ + tuplestore_puttupleslot(tupstore, slot); + } + } + + /* + * Discard the existing tuplestore (that we just processed), use the new + * one instead. + */ + tuplestore_end(node->bs_tuplestore); + node->bs_tuplestore = tupstore; + + if (!projInfo) + ExecDropSingleTupleTableSlot(slot); +} + +static bool +brinsort_next_range(BrinSortState *node, bool asc) +{ + /* FIXME free the current bs_range, if any */ + node->bs_range = NULL; + + /* + * Mark the position, so that we can restore it in case we reach the + * current watermark. + */ + tuplesort_markpos(node->bs_scan->ranges); + + /* + * Get the next range and return it, unless we can prove it's the last + * range that can possibly match the current conditon (thanks to how we + * order the ranges). + * + * Also skip ranges that can't possibly match (e.g. because we are in + * NULL processing, and the range has no NULLs). + */ + while (tuplesort_gettupleslot(node->bs_scan->ranges, true, false, node->bs_scan->slot, NULL)) + { + bool isnull; + Datum value; + + BrinRange *range = (BrinRange *) palloc(sizeof(BrinRange)); + + range->blkno_start = DatumGetUInt32(slot_getattr(node->bs_scan->slot, 1, &isnull)); + range->blkno_end = DatumGetUInt32(slot_getattr(node->bs_scan->slot, 2, &isnull)); + range->has_nulls = DatumGetBool(slot_getattr(node->bs_scan->slot, 3, &isnull)); + range->all_nulls = DatumGetBool(slot_getattr(node->bs_scan->slot, 4, &isnull)); + range->not_summarized = DatumGetBool(slot_getattr(node->bs_scan->slot, 5, &isnull)); + range->min_value = slot_getattr(node->bs_scan->slot, 6, &isnull); + range->max_value = slot_getattr(node->bs_scan->slot, 7, &isnull); + + /* + * Not-summarized ranges match irrespectedly of the watermark (if + * it's set at all). + */ + if (range->not_summarized) + { + node->bs_range = range; + return true; + } + + /* + * The range is summarized, but maybe the watermark is not? That + * would mean we're processing NULL values, so we skip ranges that + * can't possibly match (i.e. with all_nulls=has_nulls=false). + */ + if (!node->bs_watermark_set) + { + if (range->all_nulls || range->has_nulls) + { + node->bs_range = range; + return true; + } + + /* update the position and try the next range */ + tuplesort_markpos(node->bs_scan->ranges); + pfree(range); + + continue; + } + + /* + * Watermark is set, but it's empty - everything matches (except + * for NULL-only ranges, because we're definitely not processing + * NULLS, because then we wouldn't have watermark set). + */ + if (node->bs_watermark_empty) + { + node->bs_range = range; + return true; + } + + /* + * So now we have a summarized range, and we know the watermark + * is set too (so we're not processing NULLs). We place the ranges + * with only nulls last, so once we hit one we're done. + */ + if (range->all_nulls) + { + pfree(range); + return false; /* no more matching ranges */ + } + + /* + * Compare the range to the watermark, using either the minval or + * maxval, depending on ASC/DESC ordering. If the range precedes the + * watermark, return it. Otherwise abort, all the future ranges are + * either not matching the watermark (thanks to ordering) or contain + * only NULL values. + */ + + /* use minval or maxval, depending on ASC / DESC */ + value = (asc) ? range->min_value : range->max_value; + + /* + * compare it to the current watermark (if set) + * + * XXX We don't use (... <= 0) here, because then we'd load ranges + * with that minval (and there might be multiple), but most of the + * rows would go into the tuplestore, because only rows matching the + * minval exactly would be loaded into tuplesort. + */ + if (ApplySortComparator(value, false, + node->bs_watermark, false, + &node->bs_sortsupport) < 0) + { + node->bs_range = range; + return true; + } + + pfree(range); + break; + } + + /* not a matching range, we're done */ + tuplesort_restorepos(node->bs_scan->ranges); + + return false; +} + +static bool +brinsort_range_with_nulls(BrinSortState *node) +{ + BrinRange *range = node->bs_range; + + if (range->all_nulls || range->has_nulls || range->not_summarized) + return true; + + return false; +} + +static void +brinsort_rescan(BrinSortState *node) +{ + tuplesort_rescan(node->bs_scan->ranges); +} + +/* ---------------------------------------------------------------- + * IndexNext + * + * Retrieve a tuple from the BrinSort node's currentRelation + * using the index specified in the BrinSortState information. + * ---------------------------------------------------------------- + */ +static TupleTableSlot * +IndexNext(BrinSortState *node) +{ + BrinSort *plan = (BrinSort *) node->ss.ps.plan; + EState *estate; + ScanDirection direction; + IndexScanDesc scandesc; + TupleTableSlot *slot; + bool nullsFirst; + bool asc; + + /* + * extract necessary information from index scan node + */ + estate = node->ss.ps.state; + direction = estate->es_direction; + + /* flip direction if this is an overall backward scan */ + /* XXX For BRIN indexes this is always forward direction */ + // if (ScanDirectionIsBackward(((BrinSort *) node->ss.ps.plan)->indexorderdir)) + if (false) + { + if (ScanDirectionIsForward(direction)) + direction = BackwardScanDirection; + else if (ScanDirectionIsBackward(direction)) + direction = ForwardScanDirection; + } + scandesc = node->iss_ScanDesc; + slot = node->ss.ss_ScanTupleSlot; + + nullsFirst = plan->nullsFirst[0]; + asc = ScanDirectionIsForward(plan->indexorderdir); + + if (scandesc == NULL) + { + /* + * We reach here if the index scan is not parallel, or if we're + * serially executing an index scan that was planned to be parallel. + */ + scandesc = index_beginscan(node->ss.ss_currentRelation, + node->iss_RelationDesc, + estate->es_snapshot, + node->iss_NumScanKeys, + node->iss_NumOrderByKeys); + + node->iss_ScanDesc = scandesc; + + /* + * If no run-time keys to calculate or they are ready, go ahead and + * pass the scankeys to the index AM. + */ + if (node->iss_NumRuntimeKeys == 0 || node->iss_RuntimeKeysReady) + index_rescan(scandesc, + node->iss_ScanKeys, node->iss_NumScanKeys, + node->iss_OrderByKeys, node->iss_NumOrderByKeys); + + /* + * Load info about BRIN ranges, sort them to match the desired ordering. + */ + ExecInitBrinSortRanges(plan, node); + node->bs_phase = BRINSORT_START; + } + + /* + * ok, now that we have what we need, fetch the next tuple. + */ + while (node->bs_phase != BRINSORT_FINISHED) + { + CHECK_FOR_INTERRUPTS(); + + elog(DEBUG1, "phase = %d", node->bs_phase); + + AssertCheckRanges(node); + + switch (node->bs_phase) + { + case BRINSORT_START: + + elog(DEBUG1, "phase = START"); + + /* + * If we have NULLS FIRST, move to that stage. Otherwise + * start scanning regular ranges. + */ + if (nullsFirst) + node->bs_phase = BRINSORT_LOAD_NULLS; + else + { + node->bs_phase = BRINSORT_LOAD_RANGE; + + /* set the first watermark */ + brinsort_update_watermark(node, asc); + } + + break; + + case BRINSORT_LOAD_RANGE: + { + elog(DEBUG1, "phase = LOAD_RANGE"); + + /* + * Load tuples matching the new watermark from the existing + * spill tuplestore. We do this before loading tuples from + * the next chunk of ranges, because those will add tuples + * to the spill, and we'd end up processing those twice. + */ + brinsort_load_spill_tuples(node, true); + + /* + * Load tuples from ranges, until we find a range that has + * min_value >= watermark. + * + * XXX In fact, we are guaranteed to find an exact match + * for the watermark, because of how we pick the watermark. + */ + while (brinsort_next_range(node, asc)) + brinsort_load_tuples(node, true, false); + + /* + * If we have loaded any tuples into the tuplesort, try + * sorting it and move to producing the tuples. + * + * XXX The range might have no rows matching the current + * watermark, in which case the tuplesort is empty. + */ + if (node->bs_tuplesortstate) + { +#ifdef DEBUG_BRIN_SORT + tuplesort_reset_stats(node->bs_tuplesortstate); +#endif + + tuplesort_performsort(node->bs_tuplesortstate); + +#ifdef DEBUG_BRIN_SORT + if (debug_brin_sort) + { + TuplesortInstrumentation stats; + + memset(&stats, 0, sizeof(TuplesortInstrumentation)); + tuplesort_get_stats(node->bs_tuplesortstate, &stats); + + tuplesort_get_stats(node->bs_tuplesortstate, &stats); + + elog(WARNING, "method: %s space: %ld kB (%s)", + tuplesort_method_name(stats.sortMethod), + stats.spaceUsed, + tuplesort_space_type_name(stats.spaceType)); + } +#endif + } + + node->bs_phase = BRINSORT_PROCESS_RANGE; + break; + } + + case BRINSORT_PROCESS_RANGE: + + elog(DEBUG1, "phase BRINSORT_PROCESS_RANGE"); + + slot = node->ss.ps.ps_ResultTupleSlot; + + /* read tuples from the tuplesort range, and output them */ + if (node->bs_tuplesortstate != NULL) + { + if (tuplesort_gettupleslot(node->bs_tuplesortstate, + ScanDirectionIsForward(direction), + false, slot, NULL)) + return slot; + + /* once we're done with the tuplesort, reset it */ + tuplesort_reset(node->bs_tuplesortstate); + } + + /* + * Now that we processed tuples from the last range batch, + * see if we reached the end of if we should try updating + * the watermark once again. If the watermark is not set, + * we've already processed the last range. + */ + if (node->bs_watermark_empty) + { + if (nullsFirst) + node->bs_phase = BRINSORT_FINISHED; + else + { + brinsort_rescan(node); + node->bs_phase = BRINSORT_LOAD_NULLS; + node->bs_watermark_set = false; + node->bs_watermark_empty = false; + } + } + else + { + /* updte the watermark and try reading more ranges */ + node->bs_phase = BRINSORT_LOAD_RANGE; + brinsort_update_watermark(node, asc); + } + + break; + + case BRINSORT_LOAD_NULLS: + { + elog(DEBUG1, "phase = LOAD_NULLS"); + + /* + * Try loading another range. If there are no more ranges, + * we're done and we move either to loading regular ranges. + * Otherwise check if this range can contain NULL values. + * If yes, process the range. If not, try loading another + * one from the list. + */ + while (true) + { + /* no more ranges - terminate or load regular ranges */ + if (!brinsort_next_range(node, asc)) + { + if (nullsFirst) + { + brinsort_rescan(node); + node->bs_phase = BRINSORT_LOAD_RANGE; + brinsort_update_watermark(node, asc); + } + else + node->bs_phase = BRINSORT_FINISHED; + + break; + } + + /* If this range (may) have nulls, proces them */ + if (brinsort_range_with_nulls(node)) + break; + } + + if (node->bs_range == NULL) + break; + + /* + * There should be nothing left in the tuplestore, because + * we flush that at the end of processing regular tuples, + * and we don't retain tuples between NULL ranges. + */ + // Assert(node->bs_tuplestore == NULL); + + /* + * Load the next unprocessed / NULL range. We don't need to + * check watermark while processing NULLS. + */ + brinsort_load_tuples(node, false, true); + + node->bs_phase = BRINSORT_PROCESS_NULLS; + break; + } + + break; + + case BRINSORT_PROCESS_NULLS: + + elog(DEBUG1, "phase = LOAD_NULLS"); + + slot = node->ss.ps.ps_ResultTupleSlot; + + Assert(node->bs_tuplestore != NULL); + + /* read tuples from the tuplesort range, and output them */ + if (node->bs_tuplestore != NULL) + { + + while (tuplestore_gettupleslot(node->bs_tuplestore, true, true, slot)) + return slot; + + tuplestore_end(node->bs_tuplestore); + node->bs_tuplestore = NULL; + + node->bs_phase = BRINSORT_LOAD_NULLS; /* load next range */ + } + + break; + + case BRINSORT_FINISHED: + elog(ERROR, "unexpected BrinSort phase: FINISHED"); + break; + } + } + + /* + * if we get here it means the index scan failed so we are at the end of + * the scan.. + */ + node->iss_ReachedEnd = true; + return ExecClearTuple(slot); +} + +/* + * IndexRecheck -- access method routine to recheck a tuple in EvalPlanQual + */ +static bool +IndexRecheck(BrinSortState *node, TupleTableSlot *slot) +{ + ExprContext *econtext; + + /* + * extract necessary information from index scan node + */ + econtext = node->ss.ps.ps_ExprContext; + + /* Does the tuple meet the indexqual condition? */ + econtext->ecxt_scantuple = slot; + return ExecQualAndReset(node->indexqualorig, econtext); +} + + +/* ---------------------------------------------------------------- + * ExecBrinSort(node) + * ---------------------------------------------------------------- + */ +static TupleTableSlot * +ExecBrinSort(PlanState *pstate) +{ + BrinSortState *node = castNode(BrinSortState, pstate); + + /* + * If we have runtime keys and they've not already been set up, do it now. + */ + if (node->iss_NumRuntimeKeys != 0 && !node->iss_RuntimeKeysReady) + ExecReScan((PlanState *) node); + + return ExecScan(&node->ss, + (ExecScanAccessMtd) IndexNext, + (ExecScanRecheckMtd) IndexRecheck); +} + +/* ---------------------------------------------------------------- + * ExecReScanBrinSort(node) + * + * Recalculates the values of any scan keys whose value depends on + * information known at runtime, then rescans the indexed relation. + * + * ---------------------------------------------------------------- + */ +void +ExecReScanBrinSort(BrinSortState *node) +{ + /* + * If we are doing runtime key calculations (ie, any of the index key + * values weren't simple Consts), compute the new key values. But first, + * reset the context so we don't leak memory as each outer tuple is + * scanned. Note this assumes that we will recalculate *all* runtime keys + * on each call. + */ + if (node->iss_NumRuntimeKeys != 0) + { + ExprContext *econtext = node->iss_RuntimeContext; + + ResetExprContext(econtext); + ExecIndexEvalRuntimeKeys(econtext, + node->iss_RuntimeKeys, + node->iss_NumRuntimeKeys); + } + node->iss_RuntimeKeysReady = true; + + /* reset index scan */ + if (node->iss_ScanDesc) + index_rescan(node->iss_ScanDesc, + node->iss_ScanKeys, node->iss_NumScanKeys, + node->iss_OrderByKeys, node->iss_NumOrderByKeys); + node->iss_ReachedEnd = false; + + ExecScanReScan(&node->ss); +} + + +/* ---------------------------------------------------------------- + * ExecEndBrinSort + * ---------------------------------------------------------------- + */ +void +ExecEndBrinSort(BrinSortState *node) +{ + Relation indexRelationDesc; + IndexScanDesc IndexScanDesc; + + /* + * extract information from the node + */ + indexRelationDesc = node->iss_RelationDesc; + IndexScanDesc = node->iss_ScanDesc; + + /* + * clear out tuple table slots + */ + if (node->ss.ps.ps_ResultTupleSlot) + ExecClearTuple(node->ss.ps.ps_ResultTupleSlot); + ExecClearTuple(node->ss.ss_ScanTupleSlot); + + /* + * close the index relation (no-op if we didn't open it) + */ + if (IndexScanDesc) + index_endscan(IndexScanDesc); + if (indexRelationDesc) + index_close(indexRelationDesc, NoLock); + + if (node->ss.ss_currentScanDesc != NULL) + table_endscan(node->ss.ss_currentScanDesc); + + if (node->bs_tuplestore != NULL) + tuplestore_end(node->bs_tuplestore); + node->bs_tuplestore = NULL; + + if (node->bs_tuplesortstate != NULL) + tuplesort_end(node->bs_tuplesortstate); + node->bs_tuplesortstate = NULL; + + if (node->bs_scan->ranges != NULL) + tuplesort_end(node->bs_scan->ranges); + node->bs_scan->ranges = NULL; +} + +/* ---------------------------------------------------------------- + * ExecBrinSortMarkPos + * + * Note: we assume that no caller attempts to set a mark before having read + * at least one tuple. Otherwise, iss_ScanDesc might still be NULL. + * ---------------------------------------------------------------- + */ +void +ExecBrinSortMarkPos(BrinSortState *node) +{ + EState *estate = node->ss.ps.state; + EPQState *epqstate = estate->es_epq_active; + + if (epqstate != NULL) + { + /* + * We are inside an EvalPlanQual recheck. If a test tuple exists for + * this relation, then we shouldn't access the index at all. We would + * instead need to save, and later restore, the state of the + * relsubs_done flag, so that re-fetching the test tuple is possible. + * However, given the assumption that no caller sets a mark at the + * start of the scan, we can only get here with relsubs_done[i] + * already set, and so no state need be saved. + */ + Index scanrelid = ((Scan *) node->ss.ps.plan)->scanrelid; + + Assert(scanrelid > 0); + if (epqstate->relsubs_slot[scanrelid - 1] != NULL || + epqstate->relsubs_rowmark[scanrelid - 1] != NULL) + { + /* Verify the claim above */ + if (!epqstate->relsubs_done[scanrelid - 1]) + elog(ERROR, "unexpected ExecBrinSortMarkPos call in EPQ recheck"); + return; + } + } + + index_markpos(node->iss_ScanDesc); +} + +/* ---------------------------------------------------------------- + * ExecIndexRestrPos + * ---------------------------------------------------------------- + */ +void +ExecBrinSortRestrPos(BrinSortState *node) +{ + EState *estate = node->ss.ps.state; + EPQState *epqstate = estate->es_epq_active; + + if (estate->es_epq_active != NULL) + { + /* See comments in ExecIndexMarkPos */ + Index scanrelid = ((Scan *) node->ss.ps.plan)->scanrelid; + + Assert(scanrelid > 0); + if (epqstate->relsubs_slot[scanrelid - 1] != NULL || + epqstate->relsubs_rowmark[scanrelid - 1] != NULL) + { + /* Verify the claim above */ + if (!epqstate->relsubs_done[scanrelid - 1]) + elog(ERROR, "unexpected ExecBrinSortRestrPos call in EPQ recheck"); + return; + } + } + + index_restrpos(node->iss_ScanDesc); +} + +/* + * somewhat crippled verson of bringetbitmap + * + * XXX We don't call consistent function (or any other function), so unlike + * bringetbitmap we don't set a separate memory context. If we end up filtering + * the ranges somehow (e.g. by WHERE conditions), this might be necessary. + * + * XXX Should be part of opclass, to somewhere in brin_minmax.c etc. + */ +static void +ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate) +{ + IndexScanDesc scan = planstate->iss_ScanDesc; + Relation indexRel = planstate->iss_RelationDesc; + int attno; + FmgrInfo *rangeproc; + BrinRangeScanDesc *brscan; + bool asc; + + /* BRIN Sort only allows ORDER BY using a single column */ + Assert(node->numCols == 1); + + attno = node->attnums[0]; + + /* + * Make sure we matched the sort key - if not, we should not have got + * to this place at all (try sorting using this index). + */ + Assert(AttrNumberIsForUserDefinedAttr(attno)); + + /* + * get procedure to generate sort ranges + * + * FIXME we can't rely on a particular procnum to identify which opclass + * allows building sort ranges, because the optinal procnums are not + * unique (e.g. inclusion_ops have 12 too). So we probably need a flag + * for the opclass. + */ + rangeproc = index_getprocinfo(indexRel, attno, BRIN_PROCNUM_RANGES); + + /* + * Should not get here without a proc, thanks to the check before + * building the BrinSort path. + */ + Assert(OidIsValid(rangeproc->fn_oid)); + + memset(&planstate->bs_sortsupport, 0, sizeof(SortSupportData)); + + planstate->bs_sortsupport.ssup_collation = node->collations[0]; + planstate->bs_sortsupport.ssup_cxt = CurrentMemoryContext; // FIXME + + PrepareSortSupportFromOrderingOp(node->sortOperators[0], &planstate->bs_sortsupport); + + /* + * Determine if this ASC or DESC sort, so that we can request the + * ranges in the appropriate order (ordered either by minval for + * ASC, or by maxval for DESC). + */ + asc = ScanDirectionIsForward(node->indexorderdir); + + /* + * Ask the opclass to produce ranges in appropriate ordering. + * + * XXX Pass info about ASC/DESC, NULLS FIRST/LAST. + */ + brscan = (BrinRangeScanDesc *) DatumGetPointer(FunctionCall3Coll(rangeproc, + node->collations[0], + PointerGetDatum(scan), + Int16GetDatum(attno), + BoolGetDatum(asc))); + + /* allocate for space, and also for the alternative ordering */ + planstate->bs_scan = brscan; +} + +/* ---------------------------------------------------------------- + * ExecInitBrinSort + * + * Initializes the index scan's state information, creates + * scan keys, and opens the base and index relations. + * + * Note: index scans have 2 sets of state information because + * we have to keep track of the base relation and the + * index relation. + * ---------------------------------------------------------------- + */ +BrinSortState * +ExecInitBrinSort(BrinSort *node, EState *estate, int eflags) +{ + BrinSortState *indexstate; + Relation currentRelation; + LOCKMODE lockmode; + + /* + * create state structure + */ + indexstate = makeNode(BrinSortState); + indexstate->ss.ps.plan = (Plan *) node; + indexstate->ss.ps.state = estate; + indexstate->ss.ps.ExecProcNode = ExecBrinSort; + + /* + * Miscellaneous initialization + * + * create expression context for node + */ + ExecAssignExprContext(estate, &indexstate->ss.ps); + + /* + * open the scan relation + */ + currentRelation = ExecOpenScanRelation(estate, node->scan.scanrelid, eflags); + + indexstate->ss.ss_currentRelation = currentRelation; + indexstate->ss.ss_currentScanDesc = NULL; /* no heap scan here */ + + /* + * get the scan type from the relation descriptor. + */ + ExecInitScanTupleSlot(estate, &indexstate->ss, + RelationGetDescr(currentRelation), + table_slot_callbacks(currentRelation)); + + /* + * Initialize result type and projection. + */ + ExecInitResultTupleSlotTL(&indexstate->ss.ps, &TTSOpsMinimalTuple); + // ExecInitResultTypeTL(&indexstate->ss.ps); + // ExecAssignScanProjectionInfo(&indexstate->ss); + // ExecInitResultSlot(&indexstate->ss.ps, &TTSOpsVirtual); + + indexstate->bs_ProjInfo = ExecBuildProjectionInfo(((Plan *) node)->targetlist, + indexstate->ss.ps.ps_ExprContext, + indexstate->ss.ps.ps_ResultTupleSlot, + &indexstate->ss.ps, + indexstate->ss.ss_ScanTupleSlot->tts_tupleDescriptor); + + /* + * initialize child expressions + * + * Note: we don't initialize all of the indexqual expression, only the + * sub-parts corresponding to runtime keys (see below). Likewise for + * indexorderby, if any. But the indexqualorig expression is always + * initialized even though it will only be used in some uncommon cases --- + * would be nice to improve that. (Problem is that any SubPlans present + * in the expression must be found now...) + */ + indexstate->ss.ps.qual = + ExecInitQual(node->scan.plan.qual, (PlanState *) indexstate); + indexstate->indexqualorig = + ExecInitQual(node->indexqualorig, (PlanState *) indexstate); + + /* + * If we are just doing EXPLAIN (ie, aren't going to run the plan), stop + * here. This allows an index-advisor plugin to EXPLAIN a plan containing + * references to nonexistent indexes. + */ + if (eflags & EXEC_FLAG_EXPLAIN_ONLY) + return indexstate; + + /* Open the index relation. */ + lockmode = exec_rt_fetch(node->scan.scanrelid, estate)->rellockmode; + indexstate->iss_RelationDesc = index_open(node->indexid, lockmode); + + /* + * Initialize index-specific scan state + */ + indexstate->iss_RuntimeKeysReady = false; + indexstate->iss_RuntimeKeys = NULL; + indexstate->iss_NumRuntimeKeys = 0; + + /* + * build the index scan keys from the index qualification + */ + ExecIndexBuildScanKeys((PlanState *) indexstate, + indexstate->iss_RelationDesc, + node->indexqual, + false, + &indexstate->iss_ScanKeys, + &indexstate->iss_NumScanKeys, + &indexstate->iss_RuntimeKeys, + &indexstate->iss_NumRuntimeKeys, + NULL, /* no ArrayKeys */ + NULL); + + /* + * If we have runtime keys, we need an ExprContext to evaluate them. The + * node's standard context won't do because we want to reset that context + * for every tuple. So, build another context just like the other one... + * -tgl 7/11/00 + */ + if (indexstate->iss_NumRuntimeKeys != 0) + { + ExprContext *stdecontext = indexstate->ss.ps.ps_ExprContext; + + ExecAssignExprContext(estate, &indexstate->ss.ps); + indexstate->iss_RuntimeContext = indexstate->ss.ps.ps_ExprContext; + indexstate->ss.ps.ps_ExprContext = stdecontext; + } + else + { + indexstate->iss_RuntimeContext = NULL; + } + + indexstate->bs_tuplesortstate = NULL; + indexstate->bs_qual = indexstate->ss.ps.qual; + indexstate->ss.ps.qual = NULL; + // ExecInitResultTupleSlotTL(&indexstate->ss.ps, &TTSOpsMinimalTuple); + + /* + * all done. + */ + return indexstate; +} + +/* ---------------------------------------------------------------- + * Parallel Scan Support + * ---------------------------------------------------------------- + */ + +/* ---------------------------------------------------------------- + * ExecBrinSortEstimate + * + * Compute the amount of space we'll need in the parallel + * query DSM, and inform pcxt->estimator about our needs. + * ---------------------------------------------------------------- + */ +void +ExecBrinSortEstimate(BrinSortState *node, + ParallelContext *pcxt) +{ + EState *estate = node->ss.ps.state; + + node->iss_PscanLen = index_parallelscan_estimate(node->iss_RelationDesc, + estate->es_snapshot); + shm_toc_estimate_chunk(&pcxt->estimator, node->iss_PscanLen); + shm_toc_estimate_keys(&pcxt->estimator, 1); +} + +/* ---------------------------------------------------------------- + * ExecBrinSortInitializeDSM + * + * Set up a parallel index scan descriptor. + * ---------------------------------------------------------------- + */ +void +ExecBrinSortInitializeDSM(BrinSortState *node, + ParallelContext *pcxt) +{ + EState *estate = node->ss.ps.state; + ParallelIndexScanDesc piscan; + + piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); + index_parallelscan_initialize(node->ss.ss_currentRelation, + node->iss_RelationDesc, + estate->es_snapshot, + piscan); + shm_toc_insert(pcxt->toc, node->ss.ps.plan->plan_node_id, piscan); + node->iss_ScanDesc = + index_beginscan_parallel(node->ss.ss_currentRelation, + node->iss_RelationDesc, + node->iss_NumScanKeys, + node->iss_NumOrderByKeys, + piscan); + + /* + * If no run-time keys to calculate or they are ready, go ahead and pass + * the scankeys to the index AM. + */ + if (node->iss_NumRuntimeKeys == 0 || node->iss_RuntimeKeysReady) + index_rescan(node->iss_ScanDesc, + node->iss_ScanKeys, node->iss_NumScanKeys, + node->iss_OrderByKeys, node->iss_NumOrderByKeys); +} + +/* ---------------------------------------------------------------- + * ExecBrinSortReInitializeDSM + * + * Reset shared state before beginning a fresh scan. + * ---------------------------------------------------------------- + */ +void +ExecBrinSortReInitializeDSM(BrinSortState *node, + ParallelContext *pcxt) +{ + index_parallelrescan(node->iss_ScanDesc); +} + +/* ---------------------------------------------------------------- + * ExecBrinSortInitializeWorker + * + * Copy relevant information from TOC into planstate. + * ---------------------------------------------------------------- + */ +void +ExecBrinSortInitializeWorker(BrinSortState *node, + ParallelWorkerContext *pwcxt) +{ + ParallelIndexScanDesc piscan; + + piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); + node->iss_ScanDesc = + index_beginscan_parallel(node->ss.ss_currentRelation, + node->iss_RelationDesc, + node->iss_NumScanKeys, + node->iss_NumOrderByKeys, + piscan); + + /* + * If no run-time keys to calculate or they are ready, go ahead and pass + * the scankeys to the index AM. + */ + if (node->iss_NumRuntimeKeys == 0 || node->iss_RuntimeKeysReady) + index_rescan(node->iss_ScanDesc, + node->iss_ScanKeys, node->iss_NumScanKeys, + node->iss_OrderByKeys, node->iss_NumOrderByKeys); +} diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index ef475d95a1..94b5c8df81 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -792,6 +792,260 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count, path->path.total_cost = startup_cost + run_cost; } +void +cost_brinsort(BrinSortPath *path, PlannerInfo *root, double loop_count, + bool partial_path) +{ + IndexOptInfo *index = path->ipath.indexinfo; + RelOptInfo *baserel = index->rel; + amcostestimate_function amcostestimate; + List *qpquals; + Cost startup_cost = 0; + Cost run_cost = 0; + Cost cpu_run_cost = 0; + Cost indexStartupCost; + Cost indexTotalCost; + Selectivity indexSelectivity; + double indexCorrelation, + csquared; + double spc_seq_page_cost, + spc_random_page_cost; + Cost min_IO_cost, + max_IO_cost; + QualCost qpqual_cost; + Cost cpu_per_tuple; + double tuples_fetched; + double pages_fetched; + double rand_heap_pages; + double index_pages; + + /* Should only be applied to base relations */ + Assert(IsA(baserel, RelOptInfo) && + IsA(index, IndexOptInfo)); + Assert(baserel->relid > 0); + Assert(baserel->rtekind == RTE_RELATION); + + /* + * Mark the path with the correct row estimate, and identify which quals + * will need to be enforced as qpquals. We need not check any quals that + * are implied by the index's predicate, so we can use indrestrictinfo not + * baserestrictinfo as the list of relevant restriction clauses for the + * rel. + */ + if (path->ipath.path.param_info) + { + path->ipath.path.rows = path->ipath.path.param_info->ppi_rows; + /* qpquals come from the rel's restriction clauses and ppi_clauses */ + qpquals = list_concat(extract_nonindex_conditions(path->ipath.indexinfo->indrestrictinfo, + path->ipath.indexclauses), + extract_nonindex_conditions(path->ipath.path.param_info->ppi_clauses, + path->ipath.indexclauses)); + } + else + { + path->ipath.path.rows = baserel->rows; + /* qpquals come from just the rel's restriction clauses */ + qpquals = extract_nonindex_conditions(path->ipath.indexinfo->indrestrictinfo, + path->ipath.indexclauses); + } + + if (!enable_indexscan) + startup_cost += disable_cost; + /* we don't need to check enable_indexonlyscan; indxpath.c does that */ + + /* + * Call index-access-method-specific code to estimate the processing cost + * for scanning the index, as well as the selectivity of the index (ie, + * the fraction of main-table tuples we will have to retrieve) and its + * correlation to the main-table tuple order. We need a cast here because + * pathnodes.h uses a weak function type to avoid including amapi.h. + */ + amcostestimate = (amcostestimate_function) index->amcostestimate; + amcostestimate(root, &path->ipath, loop_count, + &indexStartupCost, &indexTotalCost, + &indexSelectivity, &indexCorrelation, + &index_pages); + + /* + * Save amcostestimate's results for possible use in bitmap scan planning. + * We don't bother to save indexStartupCost or indexCorrelation, because a + * bitmap scan doesn't care about either. + */ + path->ipath.indextotalcost = indexTotalCost; + path->ipath.indexselectivity = indexSelectivity; + + /* all costs for touching index itself included here */ + startup_cost += indexStartupCost; + run_cost += indexTotalCost - indexStartupCost; + + /* estimate number of main-table tuples fetched */ + tuples_fetched = clamp_row_est(indexSelectivity * baserel->tuples); + + /* fetch estimated page costs for tablespace containing table */ + get_tablespace_page_costs(baserel->reltablespace, + &spc_random_page_cost, + &spc_seq_page_cost); + + /*---------- + * Estimate number of main-table pages fetched, and compute I/O cost. + * + * When the index ordering is uncorrelated with the table ordering, + * we use an approximation proposed by Mackert and Lohman (see + * index_pages_fetched() for details) to compute the number of pages + * fetched, and then charge spc_random_page_cost per page fetched. + * + * When the index ordering is exactly correlated with the table ordering + * (just after a CLUSTER, for example), the number of pages fetched should + * be exactly selectivity * table_size. What's more, all but the first + * will be sequential fetches, not the random fetches that occur in the + * uncorrelated case. So if the number of pages is more than 1, we + * ought to charge + * spc_random_page_cost + (pages_fetched - 1) * spc_seq_page_cost + * For partially-correlated indexes, we ought to charge somewhere between + * these two estimates. We currently interpolate linearly between the + * estimates based on the correlation squared (XXX is that appropriate?). + * + * If it's an index-only scan, then we will not need to fetch any heap + * pages for which the visibility map shows all tuples are visible. + * Hence, reduce the estimated number of heap fetches accordingly. + * We use the measured fraction of the entire heap that is all-visible, + * which might not be particularly relevant to the subset of the heap + * that this query will fetch; but it's not clear how to do better. + *---------- + */ + if (loop_count > 1) + { + /* + * For repeated indexscans, the appropriate estimate for the + * uncorrelated case is to scale up the number of tuples fetched in + * the Mackert and Lohman formula by the number of scans, so that we + * estimate the number of pages fetched by all the scans; then + * pro-rate the costs for one scan. In this case we assume all the + * fetches are random accesses. + */ + pages_fetched = index_pages_fetched(tuples_fetched * loop_count, + baserel->pages, + (double) index->pages, + root); + + rand_heap_pages = pages_fetched; + + max_IO_cost = (pages_fetched * spc_random_page_cost) / loop_count; + + /* + * In the perfectly correlated case, the number of pages touched by + * each scan is selectivity * table_size, and we can use the Mackert + * and Lohman formula at the page level to estimate how much work is + * saved by caching across scans. We still assume all the fetches are + * random, though, which is an overestimate that's hard to correct for + * without double-counting the cache effects. (But in most cases + * where such a plan is actually interesting, only one page would get + * fetched per scan anyway, so it shouldn't matter much.) + */ + pages_fetched = ceil(indexSelectivity * (double) baserel->pages); + + pages_fetched = index_pages_fetched(pages_fetched * loop_count, + baserel->pages, + (double) index->pages, + root); + + min_IO_cost = (pages_fetched * spc_random_page_cost) / loop_count; + } + else + { + /* + * Normal case: apply the Mackert and Lohman formula, and then + * interpolate between that and the correlation-derived result. + */ + pages_fetched = index_pages_fetched(tuples_fetched, + baserel->pages, + (double) index->pages, + root); + + rand_heap_pages = pages_fetched; + + /* max_IO_cost is for the perfectly uncorrelated case (csquared=0) */ + max_IO_cost = pages_fetched * spc_random_page_cost; + + /* min_IO_cost is for the perfectly correlated case (csquared=1) */ + pages_fetched = ceil(indexSelectivity * (double) baserel->pages); + + if (pages_fetched > 0) + { + min_IO_cost = spc_random_page_cost; + if (pages_fetched > 1) + min_IO_cost += (pages_fetched - 1) * spc_seq_page_cost; + } + else + min_IO_cost = 0; + } + + if (partial_path) + { + /* + * Estimate the number of parallel workers required to scan index. Use + * the number of heap pages computed considering heap fetches won't be + * sequential as for parallel scans the pages are accessed in random + * order. + */ + path->ipath.path.parallel_workers = compute_parallel_worker(baserel, + rand_heap_pages, + index_pages, + max_parallel_workers_per_gather); + + /* + * Fall out if workers can't be assigned for parallel scan, because in + * such a case this path will be rejected. So there is no benefit in + * doing extra computation. + */ + if (path->ipath.path.parallel_workers <= 0) + return; + + path->ipath.path.parallel_aware = true; + } + + /* + * Now interpolate based on estimated index order correlation to get total + * disk I/O cost for main table accesses. + */ + csquared = indexCorrelation * indexCorrelation; + + run_cost += max_IO_cost + csquared * (min_IO_cost - max_IO_cost); + + /* + * Estimate CPU costs per tuple. + * + * What we want here is cpu_tuple_cost plus the evaluation costs of any + * qual clauses that we have to evaluate as qpquals. + */ + cost_qual_eval(&qpqual_cost, qpquals, root); + + startup_cost += qpqual_cost.startup; + cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple; + + cpu_run_cost += cpu_per_tuple * tuples_fetched; + + /* tlist eval costs are paid per output row, not per tuple scanned */ + startup_cost += path->ipath.path.pathtarget->cost.startup; + cpu_run_cost += path->ipath.path.pathtarget->cost.per_tuple * path->ipath.path.rows; + + /* Adjust costing for parallelism, if used. */ + if (path->ipath.path.parallel_workers > 0) + { + double parallel_divisor = get_parallel_divisor(&path->ipath.path); + + path->ipath.path.rows = clamp_row_est(path->ipath.path.rows / parallel_divisor); + + /* The CPU cost is divided among all the workers. */ + cpu_run_cost /= parallel_divisor; + } + + run_cost += cpu_run_cost; + + path->ipath.path.startup_cost = startup_cost; + path->ipath.path.total_cost = startup_cost + run_cost; +} + /* * extract_nonindex_conditions * diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index 6a93d767a5..35679a25ed 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -17,12 +17,16 @@ #include <math.h> +#include "access/brin_internal.h" +#include "access/relation.h" #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_am.h" #include "catalog/pg_operator.h" +#include "catalog/pg_opclass.h" #include "catalog/pg_opfamily.h" #include "catalog/pg_type.h" +#include "miscadmin.h" #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" #include "nodes/supportnodes.h" @@ -32,10 +36,13 @@ #include "optimizer/paths.h" #include "optimizer/prep.h" #include "optimizer/restrictinfo.h" +#include "utils/rel.h" #include "utils/lsyscache.h" #include "utils/selfuncs.h" +bool enable_brinsort = true; + /* XXX see PartCollMatchesExprColl */ #define IndexCollMatchesExprColl(idxcollation, exprcollation) \ ((idxcollation) == InvalidOid || (idxcollation) == (exprcollation)) @@ -1106,6 +1113,182 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, } } + /* + * If this is a BRIN index with suitable opclass (minmax or such), we may + * try doing BRIN sort. BRIN indexes are not ordered and amcanorderbyop + * is set to false, so we probably will need some new opclass flag to + * mark indexes that support this. + */ + if (enable_brinsort && pathkeys_possibly_useful) + { + ListCell *lc; + Relation rel2 = relation_open(index->indexoid, NoLock); + int idx; + + /* + * Try generating sorted paths for each key with the right opclass. + */ + idx = -1; + foreach(lc, index->indextlist) + { + TargetEntry *indextle = (TargetEntry *) lfirst(lc); + BrinSortPath *bpath; + Oid rangeproc; + AttrNumber attnum; + + idx++; + attnum = (idx + 1); + + + /* XXX ignore non-BRIN indexes */ + if (rel2->rd_rel->relam != BRIN_AM_OID) + continue; + + /* + * XXX Ignore keys not using an opclass with the "ranges" proc. + * For now we only do this for some minmax opclasses, but adding + * it to all minmax is simple, and adding it to minmax-multi + * should not be very hard. + */ + rangeproc = index_getprocid(rel2, attnum, BRIN_PROCNUM_RANGES); + if (!OidIsValid(rangeproc)) + continue; + + /* + * XXX stuff extracted from build_index_pathkeys, except that we + * only deal with a single index key (producing a single pathkey), + * so we only sort on a single column. I guess we could use more + * index keys and sort on more expressions? Would that mean these + * keys need to be rather well correlated? In any case, it seems + * rather complex to implement, so I leave it as a possible + * future improvement. + * + * XXX This could also use the other BRIN keys (even from other + * indexes) in a different way - we might use the other ranges + * to quickly eliminate some of the chunks, essentially like a + * bitmap, but maybe without using the bitmap. Or we might use + * other indexes through bitmaps. + * + * XXX This fakes a number of parameters, because we don't store + * the btree opclass in the index, instead we use the default + * one for the key data type. And BRIN does not allow specifying + * + * XXX We don't add the path to result, because this function is + * supposed to generate IndexPaths. Instead, we just add the path + * using add_path(). We should be building this in a different + * place, perhaps in create_index_paths() or so. + * + * XXX By building it elsewhere, we could also leverage the index + * paths we've built here, particularly the bitmap index paths, + * which we could use to eliminate many of the ranges. + * + * XXX We don't have any explicit ordering associated with the + * BRIN index, e.g. we don't have ASC/DESC and NULLS FIRST/LAST. + * So this is not encoded in the index, and we can satisfy all + * these cases - but we need to add paths for each combination. + * I wonder if there's a better way to do this. + */ + + /* ASC NULLS LAST */ + index_pathkeys = build_index_pathkeys_brin(root, index, indextle, + idx, + false, /* reverse_sort */ + false); /* nulls_first */ + + useful_pathkeys = truncate_useless_pathkeys(root, rel, + index_pathkeys); + + if (useful_pathkeys != NIL) + { + bpath = create_brinsort_path(root, index, + index_clauses, + useful_pathkeys, + ForwardScanDirection, + index_only_scan, + outer_relids, + loop_count, + false); + + /* cheat and add it anyway */ + add_path(rel, (Path *) bpath); + } + + /* DESC NULLS LAST */ + index_pathkeys = build_index_pathkeys_brin(root, index, indextle, + idx, + true, /* reverse_sort */ + false); /* nulls_first */ + + useful_pathkeys = truncate_useless_pathkeys(root, rel, + index_pathkeys); + + if (useful_pathkeys != NIL) + { + bpath = create_brinsort_path(root, index, + index_clauses, + useful_pathkeys, + BackwardScanDirection, + index_only_scan, + outer_relids, + loop_count, + false); + + /* cheat and add it anyway */ + add_path(rel, (Path *) bpath); + } + + /* ASC NULLS FIRST */ + index_pathkeys = build_index_pathkeys_brin(root, index, indextle, + idx, + false, /* reverse_sort */ + true); /* nulls_first */ + + useful_pathkeys = truncate_useless_pathkeys(root, rel, + index_pathkeys); + + if (useful_pathkeys != NIL) + { + bpath = create_brinsort_path(root, index, + index_clauses, + useful_pathkeys, + ForwardScanDirection, + index_only_scan, + outer_relids, + loop_count, + false); + + /* cheat and add it anyway */ + add_path(rel, (Path *) bpath); + } + + /* DESC NULLS FIRST */ + index_pathkeys = build_index_pathkeys_brin(root, index, indextle, + idx, + true, /* reverse_sort */ + true); /* nulls_first */ + + useful_pathkeys = truncate_useless_pathkeys(root, rel, + index_pathkeys); + + if (useful_pathkeys != NIL) + { + bpath = create_brinsort_path(root, index, + index_clauses, + useful_pathkeys, + BackwardScanDirection, + index_only_scan, + outer_relids, + loop_count, + false); + + /* cheat and add it anyway */ + add_path(rel, (Path *) bpath); + } + } + + relation_close(rel2, NoLock); + } + return result; } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index e53ea84224..3d66c0d871 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -27,6 +27,7 @@ #include "optimizer/paths.h" #include "partitioning/partbounds.h" #include "utils/lsyscache.h" +#include "utils/typcache.h" static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); @@ -622,6 +623,54 @@ build_index_pathkeys(PlannerInfo *root, return retval; } + +List * +build_index_pathkeys_brin(PlannerInfo *root, + IndexOptInfo *index, + TargetEntry *tle, + int idx, + bool reverse_sort, + bool nulls_first) +{ + TypeCacheEntry *typcache; + PathKey *cpathkey; + Oid sortopfamily; + + /* + * Get default btree opfamily for the type, extracted from the + * entry in index targetlist. + * + * XXX Is there a better / more correct way to do this? + */ + typcache = lookup_type_cache(exprType((Node *) tle->expr), + TYPECACHE_BTREE_OPFAMILY); + sortopfamily = typcache->btree_opf; + + /* + * OK, try to make a canonical pathkey for this sort key. Note we're + * underneath any outer joins, so nullable_relids should be NULL. + */ + cpathkey = make_pathkey_from_sortinfo(root, + tle->expr, + sortopfamily, + index->opcintype[idx], + index->indexcollations[idx], + reverse_sort, + nulls_first, + 0, + index->rel->relids, + false); + + /* + * There may be no pathkey if we haven't matched any sortkey, in which + * case ignore it. + */ + if (!cpathkey) + return NIL; + + return list_make1(cpathkey); +} + /* * partkey_is_bool_constant_for_query * diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ec73789bc2..4eea942dbf 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -18,6 +18,7 @@ #include <math.h> +#include "access/genam.h" #include "access/sysattr.h" #include "catalog/pg_class.h" #include "foreign/fdwapi.h" @@ -41,6 +42,7 @@ #include "parser/parsetree.h" #include "partitioning/partprune.h" #include "utils/lsyscache.h" +#include "utils/rel.h" /* @@ -124,6 +126,8 @@ static SampleScan *create_samplescan_plan(PlannerInfo *root, Path *best_path, List *tlist, List *scan_clauses); static Scan *create_indexscan_plan(PlannerInfo *root, IndexPath *best_path, List *tlist, List *scan_clauses, bool indexonly); +static BrinSort *create_brinsort_plan(PlannerInfo *root, BrinSortPath *best_path, + List *tlist, List *scan_clauses); static BitmapHeapScan *create_bitmap_scan_plan(PlannerInfo *root, BitmapHeapPath *best_path, List *tlist, List *scan_clauses); @@ -191,6 +195,9 @@ static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual, List *indexorderby, List *indextlist, ScanDirection indexscandir); +static BrinSort *make_brinsort(List *qptlist, List *qpqual, Index scanrelid, + Oid indexid, List *indexqual, List *indexqualorig, + ScanDirection indexscandir); static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid, List *indexqual, List *indexqualorig); @@ -410,6 +417,9 @@ create_plan_recurse(PlannerInfo *root, Path *best_path, int flags) case T_CustomScan: plan = create_scan_plan(root, best_path, flags); break; + case T_BrinSort: + plan = create_scan_plan(root, best_path, flags); + break; case T_HashJoin: case T_MergeJoin: case T_NestLoop: @@ -776,6 +786,13 @@ create_scan_plan(PlannerInfo *root, Path *best_path, int flags) scan_clauses); break; + case T_BrinSort: + plan = (Plan *) create_brinsort_plan(root, + (BrinSortPath *) best_path, + tlist, + scan_clauses); + break; + default: elog(ERROR, "unrecognized node type: %d", (int) best_path->pathtype); @@ -3170,6 +3187,223 @@ create_indexscan_plan(PlannerInfo *root, return scan_plan; } +/* + * create_brinsort_plan + * Returns a brinsort plan for the base relation scanned by 'best_path' + * with restriction clauses 'scan_clauses' and targetlist 'tlist'. + * + * This is mostly a slighly simplified version of create_indexscan_plan, with + * the unecessary parts removed (we don't support indexonly scans, or reordering + * and similar stuff). + */ +static BrinSort * +create_brinsort_plan(PlannerInfo *root, + BrinSortPath *best_path, + List *tlist, + List *scan_clauses) +{ + BrinSort *brinsort_plan; + List *indexclauses = best_path->ipath.indexclauses; + Index baserelid = best_path->ipath.path.parent->relid; + IndexOptInfo *indexinfo = best_path->ipath.indexinfo; + Oid indexoid = indexinfo->indexoid; + Relation indexRel; + List *indexprs; + List *qpqual; + List *stripped_indexquals; + List *fixed_indexquals; + ListCell *l; + + List *pathkeys = best_path->ipath.path.pathkeys; + + /* it should be a base rel... */ + Assert(baserelid > 0); + Assert(best_path->ipath.path.parent->rtekind == RTE_RELATION); + + /* + * Extract the index qual expressions (stripped of RestrictInfos) from the + * IndexClauses list, and prepare a copy with index Vars substituted for + * table Vars. (This step also does replace_nestloop_params on the + * fixed_indexquals.) + */ + fix_indexqual_references(root, &best_path->ipath, + &stripped_indexquals, + &fixed_indexquals); + + /* + * The qpqual list must contain all restrictions not automatically handled + * by the index, other than pseudoconstant clauses which will be handled + * by a separate gating plan node. All the predicates in the indexquals + * will be checked (either by the index itself, or by nodeIndexscan.c), + * but if there are any "special" operators involved then they must be + * included in qpqual. The upshot is that qpqual must contain + * scan_clauses minus whatever appears in indexquals. + * + * is_redundant_with_indexclauses() detects cases where a scan clause is + * present in the indexclauses list or is generated from the same + * EquivalenceClass as some indexclause, and is therefore redundant with + * it, though not equal. (The latter happens when indxpath.c prefers a + * different derived equality than what generate_join_implied_equalities + * picked for a parameterized scan's ppi_clauses.) Note that it will not + * match to lossy index clauses, which is critical because we have to + * include the original clause in qpqual in that case. + * + * In some situations (particularly with OR'd index conditions) we may + * have scan_clauses that are not equal to, but are logically implied by, + * the index quals; so we also try a predicate_implied_by() check to see + * if we can discard quals that way. (predicate_implied_by assumes its + * first input contains only immutable functions, so we have to check + * that.) + * + * Note: if you change this bit of code you should also look at + * extract_nonindex_conditions() in costsize.c. + */ + qpqual = NIL; + foreach(l, scan_clauses) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, l); + + if (rinfo->pseudoconstant) + continue; /* we may drop pseudoconstants here */ + if (is_redundant_with_indexclauses(rinfo, indexclauses)) + continue; /* dup or derived from same EquivalenceClass */ + if (!contain_mutable_functions((Node *) rinfo->clause) && + predicate_implied_by(list_make1(rinfo->clause), stripped_indexquals, + false)) + continue; /* provably implied by indexquals */ + qpqual = lappend(qpqual, rinfo); + } + + /* Sort clauses into best execution order */ + qpqual = order_qual_clauses(root, qpqual); + + /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */ + qpqual = extract_actual_clauses(qpqual, false); + + /* + * We have to replace any outer-relation variables with nestloop params in + * the indexqualorig, qpqual, and indexorderbyorig expressions. A bit + * annoying to have to do this separately from the processing in + * fix_indexqual_references --- rethink this when generalizing the inner + * indexscan support. But note we can't really do this earlier because + * it'd break the comparisons to predicates above ... (or would it? Those + * wouldn't have outer refs) + */ + if (best_path->ipath.path.param_info) + { + stripped_indexquals = (List *) + replace_nestloop_params(root, (Node *) stripped_indexquals); + qpqual = (List *) + replace_nestloop_params(root, (Node *) qpqual); + } + + /* Finally ready to build the plan node */ + brinsort_plan = make_brinsort(tlist, + qpqual, + baserelid, + indexoid, + fixed_indexquals, + stripped_indexquals, + best_path->ipath.indexscandir); + + Assert(list_length(pathkeys) == 1); + + if (pathkeys != NIL) + { + /* + * Compute sort column info, and adjust the Append's tlist as needed. + * Because we pass adjust_tlist_in_place = true, we may ignore the + * function result; it must be the same plan node. However, we then + * need to detect whether any tlist entries were added. + */ + (void) prepare_sort_from_pathkeys((Plan *) brinsort_plan, pathkeys, + best_path->ipath.path.parent->relids, + NULL, + true, + &brinsort_plan->numCols, + &brinsort_plan->sortColIdx, + &brinsort_plan->sortOperators, + &brinsort_plan->collations, + &brinsort_plan->nullsFirst); + //tlist_was_changed = (orig_tlist_length != list_length(plan->plan.targetlist)); + for (int i = 0; i < brinsort_plan->numCols; i++) + elog(DEBUG1, "%d => %d %d %d %d", i, + brinsort_plan->sortColIdx[i], + brinsort_plan->sortOperators[i], + brinsort_plan->collations[i], + brinsort_plan->nullsFirst[i]); + } + + copy_generic_path_info(&brinsort_plan->scan.plan, &best_path->ipath.path); + + /* + * Now lookup the index attnums for sort expressions. + * + * Determine index attnum we're interested in. sortColIdx is an index into + * the target list, so we need to grab the expression and try to match it + * to the index. The expression may be either plain Var (in which case we + * match it to indkeys value), or an expression (in which case we match it + * to indexprs). + * + * XXX We've already matched the sort key to the index, otherwise we would + * not get here. So maybe we could just remember it, somehow? Also, we must + * keep the decisions made in these two places consistent - if we fail to + * match a sort key here (which we matched before), we have a problem. + * + * FIXME lock mode for index_open + */ + indexRel = index_open(indexoid, NoLock); + indexprs = RelationGetIndexExpressions(indexRel); + + brinsort_plan->attnums + = (AttrNumber *) palloc0(sizeof(AttrNumber) * brinsort_plan->numCols); + + for (int i = 0; i < brinsort_plan->numCols; i++) + { + TargetEntry *tle; + int expridx = 0; /* expression index */ + + tle = list_nth(brinsort_plan->scan.plan.targetlist, + brinsort_plan->sortColIdx[i] - 1); /* FIXME proper colidx */ + + /* find the index key matching the expression from the target entry */ + for (int j = 0; j < indexRel->rd_index->indnatts; j++) + { + AttrNumber indkey = indexRel->rd_index->indkey.values[j]; + + if (AttributeNumberIsValid(indkey)) + { + Var *var = (Var *) tle->expr; + + if (!IsA(tle->expr, Var)) + continue; + + if (var->varattno == indkey) + { + brinsort_plan->attnums[i] = (j + 1); + break; + } + } + else + { + Node *expr = (Node *) list_nth(indexprs, expridx); + + if (equal(expr, tle->expr)) + { + brinsort_plan->attnums[i] = (j + 1); + break; + } + + expridx++; + } + } + } + + index_close(indexRel, NoLock); + + return brinsort_plan; +} + /* * create_bitmap_scan_plan * Returns a bitmap scan plan for the base relation scanned by 'best_path' @@ -5524,6 +5758,31 @@ make_indexscan(List *qptlist, return node; } +static BrinSort * +make_brinsort(List *qptlist, + List *qpqual, + Index scanrelid, + Oid indexid, + List *indexqual, + List *indexqualorig, + ScanDirection indexscandir) +{ + BrinSort *node = makeNode(BrinSort); + Plan *plan = &node->scan.plan; + + plan->targetlist = qptlist; + plan->qual = qpqual; + plan->lefttree = NULL; + plan->righttree = NULL; + node->scan.scanrelid = scanrelid; + node->indexid = indexid; + node->indexqual = indexqual; + node->indexqualorig = indexqualorig; + node->indexorderdir = indexscandir; + + return node; +} + static IndexOnlyScan * make_indexonlyscan(List *qptlist, List *qpqual, @@ -7158,6 +7417,7 @@ is_projection_capable_path(Path *path) case T_Memoize: case T_Sort: case T_IncrementalSort: + case T_BrinSort: case T_Unique: case T_SetOp: case T_LockRows: diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index c63758cb2b..b198119c9b 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -681,6 +681,25 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) return set_indexonlyscan_references(root, splan, rtoffset); } break; + case T_BrinSort: + { + BrinSort *splan = (BrinSort *) plan; + + splan->scan.scanrelid += rtoffset; + splan->scan.plan.targetlist = + fix_scan_list(root, splan->scan.plan.targetlist, + rtoffset, NUM_EXEC_TLIST(plan)); + splan->scan.plan.qual = + fix_scan_list(root, splan->scan.plan.qual, + rtoffset, NUM_EXEC_QUAL(plan)); + splan->indexqual = + fix_scan_list(root, splan->indexqual, + rtoffset, 1); + splan->indexqualorig = + fix_scan_list(root, splan->indexqualorig, + rtoffset, NUM_EXEC_QUAL(plan)); + } + break; case T_BitmapIndexScan: { BitmapIndexScan *splan = (BitmapIndexScan *) plan; diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 5f5596841c..1ad4235cd2 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -1028,6 +1028,63 @@ create_index_path(PlannerInfo *root, return pathnode; } + +/* + * create_brinsort_path + * Creates a path node for sorted brin sort scan. + * + * 'index' is a usable index. + * 'indexclauses' is a list of IndexClause nodes representing clauses + * to be enforced as qual conditions in the scan. + * 'indexorderbys' is a list of bare expressions (no RestrictInfos) + * to be used as index ordering operators in the scan. + * 'indexorderbycols' is an integer list of index column numbers (zero based) + * the ordering operators can be used with. + * 'pathkeys' describes the ordering of the path. + * 'indexscandir' is ForwardScanDirection or BackwardScanDirection + * for an ordered index, or NoMovementScanDirection for + * an unordered index. + * 'indexonly' is true if an index-only scan is wanted. + * 'required_outer' is the set of outer relids for a parameterized path. + * 'loop_count' is the number of repetitions of the indexscan to factor into + * estimates of caching behavior. + * 'partial_path' is true if constructing a parallel index scan path. + * + * Returns the new path node. + */ +BrinSortPath * +create_brinsort_path(PlannerInfo *root, + IndexOptInfo *index, + List *indexclauses, + List *pathkeys, + ScanDirection indexscandir, + bool indexonly, + Relids required_outer, + double loop_count, + bool partial_path) +{ + BrinSortPath *pathnode = makeNode(BrinSortPath); + RelOptInfo *rel = index->rel; + + pathnode->ipath.path.pathtype = T_BrinSort; + pathnode->ipath.path.parent = rel; + pathnode->ipath.path.pathtarget = rel->reltarget; + pathnode->ipath.path.param_info = get_baserel_parampathinfo(root, rel, + required_outer); + pathnode->ipath.path.parallel_aware = false; + pathnode->ipath.path.parallel_safe = rel->consider_parallel; + pathnode->ipath.path.parallel_workers = 0; + pathnode->ipath.path.pathkeys = pathkeys; + + pathnode->ipath.indexinfo = index; + pathnode->ipath.indexclauses = indexclauses; + pathnode->ipath.indexscandir = indexscandir; + + cost_brinsort(pathnode, root, loop_count, partial_path); + + return pathnode; +} + /* * create_bitmap_heap_path * Creates a path node for a bitmap scan. diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 2fa7ac0052..e696029cf3 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -103,6 +103,10 @@ extern bool debug_brin_stats; extern bool debug_brin_cross_check; #endif +#ifdef DEBUG_BRIN_SORT +extern bool debug_brin_sort; +#endif + #ifdef TRACE_SYNCSCAN extern bool trace_syncscan; #endif @@ -1040,6 +1044,16 @@ struct config_bool ConfigureNamesBool[] = false, NULL, NULL, NULL }, + { + {"enable_brinsort", PGC_USERSET, QUERY_TUNING_METHOD, + gettext_noop("Enables the planner's use of BRIN sort plans."), + NULL, + GUC_EXPLAIN + }, + &enable_brinsort, + true, + NULL, NULL, NULL + }, { {"geqo", PGC_USERSET, QUERY_TUNING_GEQO, gettext_noop("Enables genetic query optimization."), @@ -1281,6 +1295,20 @@ struct config_bool ConfigureNamesBool[] = }, #endif +#ifdef DEBUG_BRIN_SORT + /* this is undocumented because not exposed in a standard build */ + { + {"debug_brin_sort", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Print info about BRIN sorting."), + NULL, + GUC_NOT_IN_SAMPLE + }, + &debug_brin_sort, + false, + NULL, NULL, NULL + }, +#endif + { {"exit_on_error", PGC_USERSET, ERROR_HANDLING_OPTIONS, gettext_noop("Terminate session on any error."), diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 9ca523073f..af59d9048d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -375,6 +375,7 @@ #enable_async_append = on #enable_bitmapscan = on +#enable_brinsort = off #enable_gathermerge = on #enable_hashagg = on #enable_hashjoin = on diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c index e5a4e5b371..0c2c2e1121 100644 --- a/src/backend/utils/sort/tuplesort.c +++ b/src/backend/utils/sort/tuplesort.c @@ -2574,6 +2574,18 @@ tuplesort_get_stats(Tuplesortstate *state, } } +/* + * tuplesort_reset_stats - reset summary statistics + * + * This can be called before tuplesort_performsort() starts. + */ +void +tuplesort_reset_stats(Tuplesortstate *state) +{ + state->isMaxSpaceDisk = false; + state->maxSpace = 0; +} + /* * Convert TuplesortMethod to a string. */ diff --git a/src/include/access/brin.h b/src/include/access/brin.h index 1d21b816fc..cdfa7421ae 100644 --- a/src/include/access/brin.h +++ b/src/include/access/brin.h @@ -34,41 +34,6 @@ typedef struct BrinStatsData BlockNumber revmapNumPages; } BrinStatsData; -/* - * Info about ranges for BRIN Sort. - */ -typedef struct BrinRange -{ - BlockNumber blkno_start; - BlockNumber blkno_end; - - Datum min_value; - Datum max_value; - bool has_nulls; - bool all_nulls; - bool not_summarized; - - /* - * Index of the range when ordered by min_value (if there are multiple - * ranges with the same min_value, it's the lowest one). - */ - uint32 min_index; - - /* - * Minimum min_index from all ranges with higher max_value (i.e. when - * sorted by max_value). If there are multiple ranges with the same - * max_value, it depends on the ordering (i.e. the ranges may get - * different min_index_lowest, depending on the exact ordering). - */ - uint32 min_index_lowest; -} BrinRange; - -typedef struct BrinRanges -{ - int nranges; - BrinRange ranges[FLEXIBLE_ARRAY_MEMBER]; -} BrinRanges; - typedef struct BrinMinmaxStats { int32 vl_len_; /* varlena header (do not touch directly!) */ diff --git a/src/include/access/brin_internal.h b/src/include/access/brin_internal.h index eac796e6f4..6dd3f3e3c1 100644 --- a/src/include/access/brin_internal.h +++ b/src/include/access/brin_internal.h @@ -76,6 +76,7 @@ typedef struct BrinDesc /* procedure numbers up to 10 are reserved for BRIN future expansion */ #define BRIN_FIRST_OPTIONAL_PROCNUM 11 #define BRIN_PROCNUM_STATISTICS 11 /* optional */ +#define BRIN_PROCNUM_RANGES 12 /* optional */ #define BRIN_LAST_OPTIONAL_PROCNUM 15 #undef BRIN_DEBUG diff --git a/src/include/catalog/pg_amproc.dat b/src/include/catalog/pg_amproc.dat index 9bbd1f14f1..b499699b35 100644 --- a/src/include/catalog/pg_amproc.dat +++ b/src/include/catalog/pg_amproc.dat @@ -806,6 +806,8 @@ amprocrighttype => 'bytea', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/bytea_minmax_ops', amproclefttype => 'bytea', amprocrighttype => 'bytea', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/bytea_minmax_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # bloom bytea { amprocfamily => 'brin/bytea_bloom_ops', amproclefttype => 'bytea', @@ -839,6 +841,8 @@ amprocrighttype => 'char', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/char_minmax_ops', amproclefttype => 'char', amprocrighttype => 'char', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/char_minmax_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # bloom "char" { amprocfamily => 'brin/char_bloom_ops', amproclefttype => 'char', @@ -870,6 +874,8 @@ amprocrighttype => 'name', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/name_minmax_ops', amproclefttype => 'name', amprocrighttype => 'name', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/name_minmax_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # bloom name { amprocfamily => 'brin/name_bloom_ops', amproclefttype => 'name', @@ -901,6 +907,8 @@ amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int8', amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '12', amproc => 'brin_minmax_ranges' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int2', amprocrighttype => 'int2', amprocnum => '1', @@ -915,6 +923,8 @@ amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int2', amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '12', amproc => 'brin_minmax_ranges' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', amprocrighttype => 'int4', amprocnum => '1', @@ -929,6 +939,8 @@ amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax multi integer: int2, int4, int8 { amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', @@ -1048,6 +1060,8 @@ amprocrighttype => 'text', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/text_minmax_ops', amproclefttype => 'text', amprocrighttype => 'text', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/text_minmax_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # bloom text { amprocfamily => 'brin/text_bloom_ops', amproclefttype => 'text', @@ -1078,6 +1092,8 @@ amprocrighttype => 'oid', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/oid_minmax_ops', amproclefttype => 'oid', amprocrighttype => 'oid', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/oid_minmax_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax multi oid { amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', @@ -1128,6 +1144,8 @@ amprocrighttype => 'tid', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/tid_minmax_ops', amproclefttype => 'tid', amprocrighttype => 'tid', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/tid_minmax_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # bloom tid { amprocfamily => 'brin/tid_bloom_ops', amproclefttype => 'tid', @@ -1181,6 +1199,9 @@ { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float4', amprocrighttype => 'float4', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float8', amprocrighttype => 'float8', amprocnum => '1', @@ -1197,6 +1218,9 @@ { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float8', amprocrighttype => 'float8', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi float { amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', @@ -1288,6 +1312,9 @@ { amprocfamily => 'brin/macaddr_minmax_ops', amproclefttype => 'macaddr', amprocrighttype => 'macaddr', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/macaddr_minmax_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi macaddr { amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', @@ -1344,6 +1371,9 @@ { amprocfamily => 'brin/macaddr8_minmax_ops', amproclefttype => 'macaddr8', amprocrighttype => 'macaddr8', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/macaddr8_minmax_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi macaddr8 { amprocfamily => 'brin/macaddr8_minmax_multi_ops', @@ -1398,6 +1428,8 @@ amprocrighttype => 'inet', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/network_minmax_ops', amproclefttype => 'inet', amprocrighttype => 'inet', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/network_minmax_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax multi inet { amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', @@ -1471,6 +1503,9 @@ { amprocfamily => 'brin/bpchar_minmax_ops', amproclefttype => 'bpchar', amprocrighttype => 'bpchar', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/bpchar_minmax_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # bloom character { amprocfamily => 'brin/bpchar_bloom_ops', amproclefttype => 'bpchar', @@ -1504,6 +1539,8 @@ amprocrighttype => 'time', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/time_minmax_ops', amproclefttype => 'time', amprocrighttype => 'time', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/time_minmax_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax multi time without time zone { amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', @@ -1557,6 +1594,9 @@ { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamp', amprocrighttype => 'timestamp', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamptz', amprocrighttype => 'timestamptz', amprocnum => '1', @@ -1573,6 +1613,9 @@ { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamptz', amprocrighttype => 'timestamptz', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'date', amprocrighttype => 'date', amprocnum => '1', @@ -1587,6 +1630,8 @@ amprocrighttype => 'date', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'date', amprocrighttype => 'date', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/datetime_minmax_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax multi datetime (date, timestamp, timestamptz) { amprocfamily => 'brin/datetime_minmax_multi_ops', @@ -1716,6 +1761,9 @@ { amprocfamily => 'brin/interval_minmax_ops', amproclefttype => 'interval', amprocrighttype => 'interval', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/interval_minmax_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi interval { amprocfamily => 'brin/interval_minmax_multi_ops', @@ -1772,6 +1820,9 @@ { amprocfamily => 'brin/timetz_minmax_ops', amproclefttype => 'timetz', amprocrighttype => 'timetz', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/timetz_minmax_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi time with time zone { amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', @@ -1824,6 +1875,8 @@ amprocrighttype => 'bit', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/bit_minmax_ops', amproclefttype => 'bit', amprocrighttype => 'bit', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/bit_minmax_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax bit varying { amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', @@ -1841,6 +1894,9 @@ { amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', amprocrighttype => 'varbit', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax numeric { amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', @@ -1858,6 +1914,9 @@ { amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', amprocrighttype => 'numeric', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi numeric { amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', @@ -1912,6 +1971,8 @@ amprocrighttype => 'uuid', amprocnum => '4', amproc => 'brin_minmax_union' }, { amprocfamily => 'brin/uuid_minmax_ops', amproclefttype => 'uuid', amprocrighttype => 'uuid', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/uuid_minmax_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '12', amproc => 'brin_minmax_ranges' }, # minmax multi uuid { amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', @@ -1988,6 +2049,9 @@ { amprocfamily => 'brin/pg_lsn_minmax_ops', amproclefttype => 'pg_lsn', amprocrighttype => 'pg_lsn', amprocnum => '11', amproc => 'brin_minmax_stats' }, +{ amprocfamily => 'brin/pg_lsn_minmax_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '12', + amproc => 'brin_minmax_ranges' }, # minmax multi pg_lsn { amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index e50a21fc22..3280751380 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -8531,6 +8531,9 @@ proname => 'brin_minmax_stats', prorettype => 'bool', proargtypes => 'internal internal int2 int2 internal int4', prosrc => 'brin_minmax_stats' }, +{ oid => '9801', descr => 'BRIN minmax support', + proname => 'brin_minmax_ranges', prorettype => 'bool', + proargtypes => 'internal int2 bool', prosrc => 'brin_minmax_ranges' }, # BRIN minmax multi { oid => '4616', descr => 'BRIN multi minmax support', diff --git a/src/include/executor/nodeBrinSort.h b/src/include/executor/nodeBrinSort.h new file mode 100644 index 0000000000..3cac599d81 --- /dev/null +++ b/src/include/executor/nodeBrinSort.h @@ -0,0 +1,47 @@ +/*------------------------------------------------------------------------- + * + * nodeBrinSort.h + * + * + * + * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/executor/nodeBrinSort.h + * + *------------------------------------------------------------------------- + */ +#ifndef NODEBRIN_SORT_H +#define NODEBRIN_SORT_H + +#include "access/genam.h" +#include "access/parallel.h" +#include "nodes/execnodes.h" + +extern BrinSortState *ExecInitBrinSort(BrinSort *node, EState *estate, int eflags); +extern void ExecEndBrinSort(BrinSortState *node); +extern void ExecBrinSortMarkPos(BrinSortState *node); +extern void ExecBrinSortRestrPos(BrinSortState *node); +extern void ExecReScanBrinSort(BrinSortState *node); +extern void ExecBrinSortEstimate(BrinSortState *node, ParallelContext *pcxt); +extern void ExecBrinSortInitializeDSM(BrinSortState *node, ParallelContext *pcxt); +extern void ExecBrinSortReInitializeDSM(BrinSortState *node, ParallelContext *pcxt); +extern void ExecBrinSortInitializeWorker(BrinSortState *node, + ParallelWorkerContext *pwcxt); + +/* + * These routines are exported to share code with nodeIndexonlyscan.c and + * nodeBitmapBrinSort.c + */ +extern void ExecIndexBuildScanKeys(PlanState *planstate, Relation index, + List *quals, bool isorderby, + ScanKey *scanKeys, int *numScanKeys, + IndexRuntimeKeyInfo **runtimeKeys, int *numRuntimeKeys, + IndexArrayKeyInfo **arrayKeys, int *numArrayKeys); +extern void ExecIndexEvalRuntimeKeys(ExprContext *econtext, + IndexRuntimeKeyInfo *runtimeKeys, int numRuntimeKeys); +extern bool ExecIndexEvalArrayKeys(ExprContext *econtext, + IndexArrayKeyInfo *arrayKeys, int numArrayKeys); +extern bool ExecIndexAdvanceArrayKeys(IndexArrayKeyInfo *arrayKeys, int numArrayKeys); + +#endif /* NODEBRIN_SORT_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index cb714f4a19..cbaa540768 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1583,6 +1583,114 @@ typedef struct IndexScanState Size iss_PscanLen; } IndexScanState; +typedef enum { + BRINSORT_START, + BRINSORT_LOAD_RANGE, + BRINSORT_PROCESS_RANGE, + BRINSORT_LOAD_NULLS, + BRINSORT_PROCESS_NULLS, + BRINSORT_FINISHED +} BrinSortPhase; + +typedef struct BrinRangeScanDesc +{ + /* range info tuple descriptor */ + TupleDesc tdesc; + + /* ranges, sorted by minval, blkno_start */ + Tuplesortstate *ranges; + + /* number of ranges in the tuplesort */ + int64 nranges; + + /* distinct minval (sorted) */ + Tuplestorestate *minvals; + + /* slot for accessing the tuplesort/tuplestore */ + TupleTableSlot *slot; + +} BrinRangeScanDesc; + +/* + * Info about ranges for BRIN Sort. + */ +typedef struct BrinRange +{ + BlockNumber blkno_start; + BlockNumber blkno_end; + + Datum min_value; + Datum max_value; + bool has_nulls; + bool all_nulls; + bool not_summarized; + + /* + * Index of the range when ordered by min_value (if there are multiple + * ranges with the same min_value, it's the lowest one). + */ + uint32 min_index; + + /* + * Minimum min_index from all ranges with higher max_value (i.e. when + * sorted by max_value). If there are multiple ranges with the same + * max_value, it depends on the ordering (i.e. the ranges may get + * different min_index_lowest, depending on the exact ordering). + */ + uint32 min_index_lowest; +} BrinRange; + +typedef struct BrinRanges +{ + int nranges; + BrinRange ranges[FLEXIBLE_ARRAY_MEMBER]; +} BrinRanges; + +typedef struct BrinSortState +{ + ScanState ss; /* its first field is NodeTag */ + ExprState *indexqualorig; + List *indexorderbyorig; + struct ScanKeyData *iss_ScanKeys; + int iss_NumScanKeys; + struct ScanKeyData *iss_OrderByKeys; + int iss_NumOrderByKeys; + IndexRuntimeKeyInfo *iss_RuntimeKeys; + int iss_NumRuntimeKeys; + bool iss_RuntimeKeysReady; + ExprContext *iss_RuntimeContext; + Relation iss_RelationDesc; + struct IndexScanDescData *iss_ScanDesc; + + /* These are needed for re-checking ORDER BY expr ordering */ + pairingheap *iss_ReorderQueue; + bool iss_ReachedEnd; + Datum *iss_OrderByValues; + bool *iss_OrderByNulls; + SortSupport iss_SortSupport; + bool *iss_OrderByTypByVals; + int16 *iss_OrderByTypLens; + Size iss_PscanLen; + + /* */ + BrinRangeScanDesc *bs_scan; + BrinRange *bs_range; + ExprState *bs_qual; + Datum bs_watermark; + bool bs_watermark_set; + bool bs_watermark_empty; + BrinSortPhase bs_phase; + SortSupportData bs_sortsupport; + ProjectionInfo *bs_ProjInfo; + + /* + * We need two tuplesort instances - one for current range, one for + * spill-over tuples from the overlapping ranges + */ + void *bs_tuplesortstate; + Tuplestorestate *bs_tuplestore; +} BrinSortState; + /* ---------------- * IndexOnlyScanState information * diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index c17b53f7ad..0dbe00ef7d 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -1684,6 +1684,17 @@ typedef struct IndexPath Selectivity indexselectivity; } IndexPath; +/* + * read sorted data from brin index + * + * We use IndexPath, because that's what amcostestimate is expecting, but + * we typedef it as a separate struct. + */ +typedef struct BrinSortPath +{ + IndexPath ipath; +} BrinSortPath; + /* * Each IndexClause references a RestrictInfo node from the query's WHERE * or JOIN conditions, and shows how that restriction can be applied to diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index 1b787fe031..b4fa5faff5 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -498,6 +498,38 @@ typedef struct IndexOnlyScan ScanDirection indexorderdir; /* forward or backward or don't care */ } IndexOnlyScan; +/* + * XXX Does it make sense (is it possible) to have a sort by more than one + * column, using a BRIN index? + */ +typedef struct BrinSort +{ + Scan scan; + Oid indexid; /* OID of index to scan */ + List *indexqual; /* list of index quals (usually OpExprs) */ + List *indexqualorig; /* the same in original form */ + ScanDirection indexorderdir; /* forward or backward or don't care */ + + /* number of sort-key columns */ + int numCols; + + /* attnums in the index */ + AttrNumber *attnums pg_node_attr(array_size(numCols)); + + /* their indexes in the target list */ + AttrNumber *sortColIdx pg_node_attr(array_size(numCols)); + + /* OIDs of operators to sort them by */ + Oid *sortOperators pg_node_attr(array_size(numCols)); + + /* OIDs of collations */ + Oid *collations pg_node_attr(array_size(numCols)); + + /* NULLS FIRST/LAST directions */ + bool *nullsFirst pg_node_attr(array_size(numCols)); + +} BrinSort; + /* ---------------- * bitmap index scan node * diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h index 6cf49705d3..1fed43645a 100644 --- a/src/include/optimizer/cost.h +++ b/src/include/optimizer/cost.h @@ -70,6 +70,7 @@ extern PGDLLIMPORT bool enable_parallel_hash; extern PGDLLIMPORT bool enable_partition_pruning; extern PGDLLIMPORT bool enable_presorted_aggregate; extern PGDLLIMPORT bool enable_async_append; +extern PGDLLIMPORT bool enable_brinsort; extern PGDLLIMPORT int constraint_exclusion; extern double index_pages_fetched(double tuples_fetched, BlockNumber pages, @@ -80,6 +81,8 @@ extern void cost_samplescan(Path *path, PlannerInfo *root, RelOptInfo *baserel, ParamPathInfo *param_info); extern void cost_index(IndexPath *path, PlannerInfo *root, double loop_count, bool partial_path); +extern void cost_brinsort(BrinSortPath *path, PlannerInfo *root, + double loop_count, bool partial_path); extern void cost_bitmap_heap_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, ParamPathInfo *param_info, Path *bitmapqual, double loop_count); diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index 001e75b5b7..ebb64d525b 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -49,6 +49,15 @@ extern IndexPath *create_index_path(PlannerInfo *root, Relids required_outer, double loop_count, bool partial_path); +extern BrinSortPath *create_brinsort_path(PlannerInfo *root, + IndexOptInfo *index, + List *indexclauses, + List *pathkeys, + ScanDirection indexscandir, + bool indexonly, + Relids required_outer, + double loop_count, + bool partial_path); extern BitmapHeapPath *create_bitmap_heap_path(PlannerInfo *root, RelOptInfo *rel, Path *bitmapqual, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 50bc3b503a..9fae9bae00 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -216,6 +216,9 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths, extern Path *get_cheapest_parallel_safe_total_inner(List *paths); extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index, ScanDirection scandir); +extern List *build_index_pathkeys_brin(PlannerInfo *root, IndexOptInfo *index, + TargetEntry *tle, int idx, + bool reverse_sort, bool nulls_first); extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel, ScanDirection scandir, bool *partialkeys); extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr, diff --git a/src/include/utils/tuplesort.h b/src/include/utils/tuplesort.h index af057b6358..fb4e65cf0f 100644 --- a/src/include/utils/tuplesort.h +++ b/src/include/utils/tuplesort.h @@ -367,6 +367,7 @@ extern void tuplesort_reset(Tuplesortstate *state); extern void tuplesort_get_stats(Tuplesortstate *state, TuplesortInstrumentation *stats); +extern void tuplesort_reset_stats(Tuplesortstate *state); extern const char *tuplesort_method_name(TuplesortMethod m); extern const char *tuplesort_space_type_name(TuplesortSpaceType t); diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out index b7fda6fc82..308e912c21 100644 --- a/src/test/regress/expected/sysviews.out +++ b/src/test/regress/expected/sysviews.out @@ -113,6 +113,7 @@ select name, setting from pg_settings where name like 'enable%'; --------------------------------+--------- enable_async_append | on enable_bitmapscan | on + enable_brinsort | on enable_gathermerge | on enable_hashagg | on enable_hashjoin | on @@ -133,7 +134,7 @@ select name, setting from pg_settings where name like 'enable%'; enable_seqscan | on enable_sort | on enable_tidscan | on -(22 rows) +(23 rows) -- Test that the pg_timezone_names and pg_timezone_abbrevs views are -- more-or-less working. We can't test their contents in any great detail -- 2.41.0
0005-wip-brinsort-explain-stats-20230710.patch (text/x-patch, 12.8K) download From 938158dc3fa88f6bed1e16bcf45aab33f9f15dcf Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sat, 29 Oct 2022 22:01:01 +0200 Subject: [PATCH 05/11] wip: brinsort explain stats Show some internal stats about BRIN Sort in EXPLAIN output. --- src/backend/commands/explain.c | 132 ++++++++++++++++++++++++++++ src/backend/executor/nodeBrinSort.c | 66 +++++++++++--- src/include/nodes/execnodes.h | 39 ++++++++ 3 files changed, 223 insertions(+), 14 deletions(-) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index d2eae9ea1f..6d1609011e 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -87,6 +87,8 @@ static void show_incremental_sort_keys(IncrementalSortState *incrsortstate, List *ancestors, ExplainState *es); static void show_brinsort_keys(BrinSortState *sortstate, List *ancestors, ExplainState *es); +static void show_brinsort_stats(BrinSortState *sortstate, List *ancestors, + ExplainState *es); static void show_merge_append_keys(MergeAppendState *mstate, List *ancestors, ExplainState *es); static void show_agg_keys(AggState *astate, List *ancestors, @@ -1828,6 +1830,7 @@ ExplainNode(PlanState *planstate, List *ancestors, planstate, es); show_scan_qual(plan->qual, "Filter", planstate, ancestors, es); show_brinsort_keys(castNode(BrinSortState, planstate), ancestors, es); + show_brinsort_stats(castNode(BrinSortState, planstate), ancestors, es); if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, planstate, es); @@ -2446,6 +2449,135 @@ show_brinsort_keys(BrinSortState *sortstate, List *ancestors, ExplainState *es) ancestors, es); } +static void +show_brinsort_stats(BrinSortState *sortstate, List *ancestors, ExplainState *es) +{ + BrinSortStats *stats = &sortstate->bs_stats; + + if (sortstate->bs_scan != NULL && + sortstate->bs_scan->ranges != NULL) + { + TuplesortInstrumentation stats; + + memset(&stats, 0, sizeof(TuplesortInstrumentation)); + tuplesort_get_stats(sortstate->bs_scan->ranges, &stats); + + ExplainIndentText(es); + appendStringInfo(es->str, "Ranges: " INT64_FORMAT " Build time: " INT64_FORMAT " Method: %s Space: " INT64_FORMAT " kB (%s)\n", + sortstate->bs_scan->nranges, + sortstate->bs_stats.ranges_build_ms, + tuplesort_method_name(stats.sortMethod), + stats.spaceUsed, + tuplesort_space_type_name(stats.spaceType)); + } + + if (stats->sort_count > 0) + { + ExplainPropertyInteger("Ranges Processed", NULL, (int64) + stats->range_count, es); + + if (es->format == EXPLAIN_FORMAT_TEXT) + { + ExplainPropertyInteger("Sorts", NULL, (int64) + stats->sort_count, es); + + ExplainIndentText(es); + appendStringInfo(es->str, "Tuples Sorted: " INT64_FORMAT " Per-sort: " INT64_FORMAT " Direct: " INT64_FORMAT " Spilled: " INT64_FORMAT " Respilled: " INT64_FORMAT "\n", + stats->ntuples_tuplesort_all, + stats->ntuples_tuplesort_all / stats->sort_count, + stats->ntuples_tuplesort_direct, + stats->ntuples_spilled, + stats->ntuples_respilled); + } + else + { + ExplainOpenGroup("Sorts", "Sorts", true, es); + + ExplainPropertyInteger("Count", NULL, (int64) + stats->sort_count, es); + + ExplainPropertyInteger("Tuples per sort", NULL, (int64) + stats->ntuples_tuplesort_all / stats->sort_count, es); + + ExplainPropertyInteger("Sorted tuples (all)", NULL, (int64) + stats->ntuples_tuplesort_all, es); + + ExplainPropertyInteger("Sorted tuples (direct)", NULL, (int64) + stats->ntuples_tuplesort_direct, es); + + ExplainPropertyInteger("Spilled tuples", NULL, (int64) + stats->ntuples_spilled, es); + + ExplainPropertyInteger("Respilled tuples", NULL, (int64) + stats->ntuples_respilled, es); + + ExplainCloseGroup("Sorts", "Sorts", true, es); + } + } + + if (stats->sort_count_in_memory > 0) + { + if (es->format == EXPLAIN_FORMAT_TEXT) + { + ExplainIndentText(es); + appendStringInfo(es->str, "Sorts (in-memory) Count: " INT64_FORMAT " Space Total: " INT64_FORMAT " kB Maximum: " INT64_FORMAT " kB Average: " INT64_FORMAT " kB\n", + stats->sort_count_in_memory, + stats->total_space_used_in_memory, + stats->max_space_used_in_memory, + stats->total_space_used_in_memory / stats->sort_count_in_memory); + } + else + { + ExplainOpenGroup("In-Memory Sorts", "In-Memory Sorts", true, es); + + ExplainPropertyInteger("Count", NULL, (int64) + stats->sort_count_in_memory, es); + + ExplainPropertyInteger("Average space", "kB", (int64) + stats->total_space_used_in_memory / stats->sort_count_in_memory, es); + + ExplainPropertyInteger("Maximum space", "kB", (int64) + stats->max_space_used_in_memory, es); + + ExplainPropertyInteger("Total space", "kB", (int64) + stats->total_space_used_in_memory, es); + + ExplainCloseGroup("In-Memory Sorts", "In-Memory Sorts", true, es); + } + } + + if (stats->sort_count_on_disk > 0) + { + if (es->format == EXPLAIN_FORMAT_TEXT) + { + ExplainIndentText(es); + appendStringInfo(es->str, "Sorts (on-disk) Count: " INT64_FORMAT " Space Total: " INT64_FORMAT " kB Maximum: " INT64_FORMAT " kB Average: " INT64_FORMAT " kB\n", + stats->sort_count_on_disk, + stats->total_space_used_on_disk, + stats->max_space_used_on_disk, + stats->total_space_used_on_disk / stats->sort_count_on_disk); + } + else + { + ExplainOpenGroup("On-Disk Sorts", "On-Disk Sorts", true, es); + + ExplainPropertyInteger("Count", NULL, (int64) + stats->sort_count_on_disk, es); + + ExplainPropertyInteger("Average space", "kB", (int64) + stats->total_space_used_on_disk / stats->sort_count_on_disk, es); + + ExplainPropertyInteger("Maximum space", "kB", (int64) + stats->max_space_used_on_disk, es); + + ExplainPropertyInteger("Total space", "kB", (int64) + stats->total_space_used_on_disk, es); + + ExplainCloseGroup("On-Disk Sorts", "On-Disk Sorts", true, es); + } + } +} + /* * Likewise, for a MergeAppend node. */ diff --git a/src/backend/executor/nodeBrinSort.c b/src/backend/executor/nodeBrinSort.c index 9505eafc54..614d28c83b 100644 --- a/src/backend/executor/nodeBrinSort.c +++ b/src/backend/executor/nodeBrinSort.c @@ -450,6 +450,8 @@ brinsort_load_tuples(BrinSortState *node, bool check_watermark, bool null_proces if (null_processing && !(range->has_nulls || range->not_summarized || range->all_nulls)) return; + node->bs_stats.range_count++; + brinsort_start_tidscan(node); scan = node->ss.ss_currentScanDesc; @@ -534,7 +536,10 @@ brinsort_load_tuples(BrinSortState *node, bool check_watermark, bool null_proces /* Stash it to the tuplestore (when NULL, or ignore * it (when not-NULL). */ if (isnull) + { tuplestore_puttupleslot(node->bs_tuplestore, tmpslot); + node->bs_stats.ntuples_spilled++; + } /* NULL or not, we're done */ continue; @@ -554,7 +559,12 @@ brinsort_load_tuples(BrinSortState *node, bool check_watermark, bool null_proces &node->bs_sortsupport); if (cmp <= 0) + { tuplesort_puttupleslot(node->bs_tuplesortstate, tmpslot); + node->bs_stats.ntuples_tuplesort_direct++; + node->bs_stats.ntuples_tuplesort_all++; + node->bs_stats.ntuples_tuplesort++; + } else { /* @@ -565,6 +575,7 @@ brinsort_load_tuples(BrinSortState *node, bool check_watermark, bool null_proces * respill) and stop spilling. */ tuplestore_puttupleslot(node->bs_tuplestore, tmpslot); + node->bs_stats.ntuples_spilled++; } } @@ -633,7 +644,11 @@ brinsort_load_spill_tuples(BrinSortState *node, bool check_watermark) &node->bs_sortsupport); if (cmp <= 0) + { tuplesort_puttupleslot(node->bs_tuplesortstate, slot); + node->bs_stats.ntuples_tuplesort_all++; + node->bs_stats.ntuples_tuplesort++; + } else { /* @@ -644,6 +659,7 @@ brinsort_load_spill_tuples(BrinSortState *node, bool check_watermark) * respill) and stop spilling. */ tuplestore_puttupleslot(tupstore, slot); + node->bs_stats.ntuples_respilled++; } } @@ -933,23 +949,40 @@ IndexNext(BrinSortState *node) */ if (node->bs_tuplesortstate) { -#ifdef DEBUG_BRIN_SORT + TuplesortInstrumentation stats; + + /* + * Reset tuplesort statistics between runs, otherwise + * we'll keep re-using stats from the largest run. + */ tuplesort_reset_stats(node->bs_tuplesortstate); -#endif tuplesort_performsort(node->bs_tuplesortstate); -#ifdef DEBUG_BRIN_SORT - if (debug_brin_sort) - { - TuplesortInstrumentation stats; + node->bs_stats.sort_count++; + node->bs_stats.ntuples_tuplesort = 0; - memset(&stats, 0, sizeof(TuplesortInstrumentation)); - tuplesort_get_stats(node->bs_tuplesortstate, &stats); + tuplesort_get_stats(node->bs_tuplesortstate, &stats); - tuplesort_get_stats(node->bs_tuplesortstate, &stats); + if (stats.spaceType == SORT_SPACE_TYPE_DISK) + { + node->bs_stats.sort_count_on_disk++; + node->bs_stats.total_space_used_on_disk += stats.spaceUsed; + node->bs_stats.max_space_used_on_disk = Max(node->bs_stats.max_space_used_on_disk, + stats.spaceUsed); + } + else if (stats.spaceType == SORT_SPACE_TYPE_MEMORY) + { + node->bs_stats.sort_count_in_memory++; + node->bs_stats.total_space_used_in_memory += stats.spaceUsed; + node->bs_stats.max_space_used_in_memory = Max(node->bs_stats.max_space_used_in_memory, + stats.spaceUsed); + } - elog(WARNING, "method: %s space: %ld kB (%s)", +#ifdef DEBUG_BRIN_SORT + if (debug_brin_sort) + { + elog(WARNING, "method: %s space: " INT64_FORMAT " kB (%s)", tuplesort_method_name(stats.sortMethod), stats.spaceUsed, tuplesort_space_type_name(stats.spaceType)); @@ -1219,9 +1252,10 @@ ExecEndBrinSort(BrinSortState *node) tuplesort_end(node->bs_tuplesortstate); node->bs_tuplesortstate = NULL; - if (node->bs_scan->ranges != NULL) + if (node->bs_scan != NULL && + node->bs_scan->ranges != NULL) tuplesort_end(node->bs_scan->ranges); - node->bs_scan->ranges = NULL; + node->bs_scan = NULL; } /* ---------------------------------------------------------------- @@ -1311,6 +1345,7 @@ ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate) FmgrInfo *rangeproc; BrinRangeScanDesc *brscan; bool asc; + TimestampTz start_ts; /* BRIN Sort only allows ORDER BY using a single column */ Assert(node->numCols == 1); @@ -1355,15 +1390,18 @@ ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate) /* * Ask the opclass to produce ranges in appropriate ordering. - * - * XXX Pass info about ASC/DESC, NULLS FIRST/LAST. */ + start_ts = GetCurrentTimestamp(); + brscan = (BrinRangeScanDesc *) DatumGetPointer(FunctionCall3Coll(rangeproc, node->collations[0], PointerGetDatum(scan), Int16GetDatum(attno), BoolGetDatum(asc))); + planstate->bs_stats.ranges_build_ms + = TimestampDifferenceMilliseconds(start_ts, GetCurrentTimestamp()); + /* allocate for space, and also for the alternative ordering */ planstate->bs_scan = brscan; } diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index cbaa540768..b635a845e1 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1646,6 +1646,44 @@ typedef struct BrinRanges BrinRange ranges[FLEXIBLE_ARRAY_MEMBER]; } BrinRanges; +typedef struct BrinSortStats +{ + /* number of sorts */ + int64 sort_count; + + /* number of ranges loaded */ + int64 range_count; + + /* tuples in the current tuplesort */ + int64 ntuples_tuplesort; + + /* tuples written directly to tuplesort */ + int64 ntuples_tuplesort_direct; + + /* tuples written to tuplesort (all) */ + int64 ntuples_tuplesort_all; + + /* tuples written to tuplestore */ + int64 ntuples_spilled; + + /* tuples copied from old to new tuplestore */ + int64 ntuples_respilled; + + /* number of in-memory/on-disk sorts */ + int64 sort_count_in_memory; + int64 sort_count_on_disk; + + /* total/maximum amount of space used by either sort */ + int64 total_space_used_in_memory; + int64 total_space_used_on_disk; + int64 max_space_used_in_memory; + int64 max_space_used_on_disk; + + /* time to build ranges (milliseconds) */ + int64 ranges_build_ms; + +} BrinSortStats; + typedef struct BrinSortState { ScanState ss; /* its first field is NodeTag */ @@ -1682,6 +1720,7 @@ typedef struct BrinSortState BrinSortPhase bs_phase; SortSupportData bs_sortsupport; ProjectionInfo *bs_ProjInfo; + BrinSortStats bs_stats; /* * We need two tuplesort instances - one for current range, one for -- 2.41.0
0006-wip-multiple-watermark-steps-20230710.patch (text/x-patch, 6.3K) download From 8cc64c67e37e70ad205670d324fa5cfc0d66cf44 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Thu, 20 Oct 2022 13:03:00 +0200 Subject: [PATCH 06/11] wip: multiple watermark steps Allow incrementing the minval watermark faster, by skipping some minval values. This allows sorting more data at once (instead of many tiny sorts, which is inefficient). This also reduces the number of rows we need to spill (and possibly transfer multiple times). To use a different watermark step, use a new GUC: SET brinsort_watermark_step = 16 --- src/backend/commands/explain.c | 3 ++ src/backend/executor/nodeBrinSort.c | 59 ++++++++++++++++++++++++++--- src/backend/utils/misc/guc_tables.c | 12 ++++++ 3 files changed, 68 insertions(+), 6 deletions(-) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 6d1609011e..893bce64cb 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -47,6 +47,7 @@ ExplainOneQuery_hook_type ExplainOneQuery_hook = NULL; /* Hook for plugins to get control in explain_get_index_name() */ explain_get_index_name_hook_type explain_get_index_name_hook = NULL; +extern int brinsort_watermark_step; /* OR-able flags for ExplainXMLTag() */ #define X_OPENING 0 @@ -2471,6 +2472,8 @@ show_brinsort_stats(BrinSortState *sortstate, List *ancestors, ExplainState *es) tuplesort_space_type_name(stats.spaceType)); } + ExplainPropertyInteger("Step", NULL, (int64) brinsort_watermark_step, es); + if (stats->sort_count > 0) { ExplainPropertyInteger("Ranges Processed", NULL, (int64) diff --git a/src/backend/executor/nodeBrinSort.c b/src/backend/executor/nodeBrinSort.c index 614d28c83b..15d1579773 100644 --- a/src/backend/executor/nodeBrinSort.c +++ b/src/backend/executor/nodeBrinSort.c @@ -248,6 +248,14 @@ static void ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate); bool debug_brin_sort = false; #endif +/* + * How many distinct minval values to look forward for the next watermark? + * + * The smallest step we can do is 1, which means the immediately following + * (while distinct) minval. + */ +int brinsort_watermark_step = 1; + /* do various consistency checks */ static void AssertCheckRanges(BrinSortState *node) @@ -351,11 +359,24 @@ brinsort_end_tidscan(BrinSortState *node) * a separate "first" parameter - "set=false" has the same meaning. */ static void -brinsort_update_watermark(BrinSortState *node, bool asc) +brinsort_update_watermark(BrinSortState *node, bool first, bool asc, int steps) { int cmp; + + /* assume we haven't found a watermark */ bool found = false; + Assert(steps > 0); + + /* + * If the watermark is empty, either this is the first call (in + * which case we just use the first (or rather second) value. + * Otherwise it means we've reached the end, so no point in looking + * for more watermarks. + */ + if (node->bs_watermark_empty && !first) + return; + tuplesort_markpos(node->bs_scan->ranges); while (tuplesort_gettupleslot(node->bs_scan->ranges, true, false, node->bs_scan->slot, NULL)) @@ -381,22 +402,48 @@ brinsort_update_watermark(BrinSortState *node, bool asc) else value = slot_getattr(node->bs_scan->slot, 7, &isnull); + /* + * Has to be the first call (otherwise we would not get here, because we + * terminate after bs_watermark_set gets flipped back to false), so we + * just set the value. But we don't count this as a step, because that + * just picks the first minval value, as we certainly need to do at least + * one more step. + * + * XXX Actually, do we need to make another step? Maybe there are enough + * not-summarized ranges? Although, we don't know what values are in + * those, ranges, and with increasing data we might easily end up just + * writing all of it into the spill tuplestore. So making one more step + * seems like a better idea - we'll at least be able to produce something + * which is good for LIMIT queries. + */ if (!node->bs_watermark_set) { + Assert(first); node->bs_watermark_set = true; node->bs_watermark = value; + found = true; continue; } cmp = ApplySortComparator(node->bs_watermark, false, value, false, &node->bs_sortsupport); - if (cmp < 0) + /* + * Values should not decrease (or whatever the operator says, might + * be a DESC sort). + */ + Assert(cmp <= 0); + + if (cmp < 0) /* new watermark value */ { node->bs_watermark_set = true; node->bs_watermark = value; found = true; - break; + + steps--; + + if (steps == 0) + break; } } @@ -913,7 +960,7 @@ IndexNext(BrinSortState *node) node->bs_phase = BRINSORT_LOAD_RANGE; /* set the first watermark */ - brinsort_update_watermark(node, asc); + brinsort_update_watermark(node, true, asc, brinsort_watermark_step); } break; @@ -1034,7 +1081,7 @@ IndexNext(BrinSortState *node) { /* updte the watermark and try reading more ranges */ node->bs_phase = BRINSORT_LOAD_RANGE; - brinsort_update_watermark(node, asc); + brinsort_update_watermark(node, false, asc, brinsort_watermark_step); } break; @@ -1059,7 +1106,7 @@ IndexNext(BrinSortState *node) { brinsort_rescan(node); node->bs_phase = BRINSORT_LOAD_RANGE; - brinsort_update_watermark(node, asc); + brinsort_update_watermark(node, true, asc, brinsort_watermark_step); } else node->bs_phase = BRINSORT_FINISHED; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index e696029cf3..1b798a7886 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -97,6 +97,7 @@ extern char *default_tablespace; extern char *temp_tablespaces; extern bool ignore_checksum_failure; extern bool ignore_invalid_pages; +extern int brinsort_watermark_step; #ifdef DEBUG_BRIN_STATS extern bool debug_brin_stats; @@ -3579,6 +3580,17 @@ struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"brinsort_watermark_step", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("sets the step for brinsort watermark increments"), + NULL, + GUC_NOT_IN_SAMPLE + }, + &brinsort_watermark_step, + 1, 1, INT_MAX, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL -- 2.41.0
0007-wip-adjust-watermark-step-20230710.patch (text/x-patch, 9.1K) download From 3cce707c1ff59cbfbac18be2b70b0d1fbca93934 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sat, 22 Oct 2022 00:06:28 +0200 Subject: [PATCH 07/11] wip: adjust watermark step Look at available statistics - number of possible watermark values, number of rows, work_mem, etc. and pick a good watermark_step value. To calculate step using statistics, set the GUC to 0: SET brinsort_watermark_step = 0; --- src/backend/commands/explain.c | 6 +++ src/backend/executor/nodeBrinSort.c | 21 ++++---- src/backend/optimizer/plan/createplan.c | 70 +++++++++++++++++++++++++ src/backend/utils/misc/guc_tables.c | 2 +- src/include/nodes/execnodes.h | 5 ++ src/include/nodes/plannodes.h | 3 ++ 6 files changed, 94 insertions(+), 13 deletions(-) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 893bce64cb..020fb38cae 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -2454,6 +2454,7 @@ static void show_brinsort_stats(BrinSortState *sortstate, List *ancestors, ExplainState *es) { BrinSortStats *stats = &sortstate->bs_stats; + BrinSort *plan = (BrinSort *) sortstate->ss.ps.plan; if (sortstate->bs_scan != NULL && sortstate->bs_scan->ranges != NULL) @@ -2476,6 +2477,9 @@ show_brinsort_stats(BrinSortState *sortstate, List *ancestors, ExplainState *es) if (stats->sort_count > 0) { + ExplainPropertyInteger("Average Step", NULL, (int64) + stats->watermark_updates_steps / stats->watermark_updates_count, es); + ExplainPropertyInteger("Ranges Processed", NULL, (int64) stats->range_count, es); @@ -2517,6 +2521,8 @@ show_brinsort_stats(BrinSortState *sortstate, List *ancestors, ExplainState *es) ExplainCloseGroup("Sorts", "Sorts", true, es); } } + else + ExplainPropertyInteger("Initial Step", NULL, (int64) plan->watermark_step, es); if (stats->sort_count_in_memory > 0) { diff --git a/src/backend/executor/nodeBrinSort.c b/src/backend/executor/nodeBrinSort.c index 15d1579773..f8356202b7 100644 --- a/src/backend/executor/nodeBrinSort.c +++ b/src/backend/executor/nodeBrinSort.c @@ -248,14 +248,6 @@ static void ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate); bool debug_brin_sort = false; #endif -/* - * How many distinct minval values to look forward for the next watermark? - * - * The smallest step we can do is 1, which means the immediately following - * (while distinct) minval. - */ -int brinsort_watermark_step = 1; - /* do various consistency checks */ static void AssertCheckRanges(BrinSortState *node) @@ -359,9 +351,11 @@ brinsort_end_tidscan(BrinSortState *node) * a separate "first" parameter - "set=false" has the same meaning. */ static void -brinsort_update_watermark(BrinSortState *node, bool first, bool asc, int steps) +brinsort_update_watermark(BrinSortState *node, bool first, bool asc) { int cmp; + BrinSort *plan = (BrinSort *) node->ss.ps.plan; + int steps = plan->watermark_step; /* assume we haven't found a watermark */ bool found = false; @@ -449,6 +443,9 @@ brinsort_update_watermark(BrinSortState *node, bool first, bool asc, int steps) tuplesort_restorepos(node->bs_scan->ranges); + node->bs_stats.watermark_updates_count++; + node->bs_stats.watermark_updates_steps += plan->watermark_step; + node->bs_watermark_empty = (!found); } @@ -960,7 +957,7 @@ IndexNext(BrinSortState *node) node->bs_phase = BRINSORT_LOAD_RANGE; /* set the first watermark */ - brinsort_update_watermark(node, true, asc, brinsort_watermark_step); + brinsort_update_watermark(node, true, asc); } break; @@ -1081,7 +1078,7 @@ IndexNext(BrinSortState *node) { /* updte the watermark and try reading more ranges */ node->bs_phase = BRINSORT_LOAD_RANGE; - brinsort_update_watermark(node, false, asc, brinsort_watermark_step); + brinsort_update_watermark(node, false, asc); } break; @@ -1106,7 +1103,7 @@ IndexNext(BrinSortState *node) { brinsort_rescan(node); node->bs_phase = BRINSORT_LOAD_RANGE; - brinsort_update_watermark(node, true, asc, brinsort_watermark_step); + brinsort_update_watermark(node, true, asc); } else node->bs_phase = BRINSORT_FINISHED; diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 4eea942dbf..6a3a38dd14 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -18,6 +18,7 @@ #include <math.h> +#include "access/brin.h" #include "access/genam.h" #include "access/sysattr.h" #include "catalog/pg_class.h" @@ -323,6 +324,14 @@ static GatherMerge *create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path); +/* + * How many distinct minval values to look forward for the next watermark? + * + * The smallest step we can do is 1, which means the immediately following + * (while distinct) minval. + */ +int brinsort_watermark_step = 0; + /* * create_plan * Creates the access plan for a query by recursively processing the @@ -3401,6 +3410,67 @@ create_brinsort_plan(PlannerInfo *root, index_close(indexRel, NoLock); + /* + * determine watermark step (how fast to advance) + * + * If the brinsort_watermark_step is set to a non-zero value, we just use + * that value directly. Otherwise we pick a value using some simple + * heuristics - we don't want the rows to exceed work_mem, and we leave + * a bit slack (because we're adding batches of rows, not row by row). + * + * This has a weakness, because it assumes we incrementally add the same + * number of rows into the "sort" set - but imagine very wide overlapping + * ranges (e.g. random data on the same domain). Most of them will have + * about the same minval, so the sort grows only very slowly. Until the + * very last range, that removes the watermark and only then do most of + * the rows get to the tuplesort. + * + * XXX But maybe we can look at the other statistics we have, like number + * of overlaps and average range selectivity (% of tuples matching), and + * deduce something from that? + * + * XXX Could we maybe adjust the watermark step adaptively at runtime? + * That is, when we get to the "sort" step, maybe check how many rows + * are there, and if there are only few then try increasing the step? + */ + brinsort_plan->watermark_step = brinsort_watermark_step; + + if (brinsort_plan->watermark_step == 0) + { + BrinMinmaxStats *amstats; + + /**/ + Cardinality rows = brinsort_plan->scan.plan.plan_rows; + + /* estimate rowsize in the tuplesort */ + int width = brinsort_plan->scan.plan.plan_width; + int tupwidth = (MAXALIGN(width) + MAXALIGN(SizeofHeapTupleHeader)); + + /* Don't overflow work_mem (use only half to absorb variations. */ + int maxrows = (work_mem * 1024L / tupwidth / 2); + + /* If this is a LIMIT query, aim only for the required number of rows. */ + if (root->limit_tuples > 0) + maxrows = Min(maxrows, root->limit_tuples); + + /* Use the attnum calculated above. */ + amstats = (BrinMinmaxStats *) get_attindexam(brinsort_plan->indexid, + brinsort_plan->attnums[0]); + + if (amstats) + { + double pct_per_step = Max(amstats->minval_increment_avg, + amstats->maxval_increment_avg); + double rows_per_step = Max(1.0, pct_per_step * rows); + + brinsort_plan->watermark_step = (int) (maxrows / rows_per_step); + } + + /* some rough safety estimates */ + brinsort_plan->watermark_step = Max(brinsort_plan->watermark_step, 1); + brinsort_plan->watermark_step = Min(brinsort_plan->watermark_step, 8192); + } + return brinsort_plan; } diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 1b798a7886..5e8d4e3852 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -3587,7 +3587,7 @@ struct config_int ConfigureNamesInt[] = GUC_NOT_IN_SAMPLE }, &brinsort_watermark_step, - 1, 1, INT_MAX, + 0, 0, INT_MAX, NULL, NULL, NULL }, diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b635a845e1..ae69a99cb8 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1682,6 +1682,10 @@ typedef struct BrinSortStats /* time to build ranges (milliseconds) */ int64 ranges_build_ms; + /* number/sum of watermark update steps */ + int64 watermark_updates_steps; + int64 watermark_updates_count; + } BrinSortStats; typedef struct BrinSortState @@ -1714,6 +1718,7 @@ typedef struct BrinSortState BrinRangeScanDesc *bs_scan; BrinRange *bs_range; ExprState *bs_qual; + int bs_watermark_step; Datum bs_watermark; bool bs_watermark_set; bool bs_watermark_empty; diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index b4fa5faff5..9d784f6373 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -528,6 +528,9 @@ typedef struct BrinSort /* NULLS FIRST/LAST directions */ bool *nullsFirst pg_node_attr(array_size(numCols)); + /* number of watermark steps to make */ + int watermark_step; + } BrinSort; /* ---------------- -- 2.41.0
0008-wip-adaptive-watermark-step-20230710.patch (text/x-patch, 13.1K) download From 0b22050597a40aaf79d9b6504c095d0d08ba5a93 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sat, 22 Oct 2022 01:39:39 +0200 Subject: [PATCH 08/11] wip: adaptive watermark step Another option it to adjust the watermark step based on past tuplesort executions, and either increase or decrease the step, based on whether the sort was in-memory or on-disk, etc. To do this, set the GUC to -1: SET brinsort_watermark_step = -1; --- src/backend/access/brin/brin_minmax.c | 7 +- src/backend/executor/nodeBrinSort.c | 189 +++++++++++++++++++++++- src/backend/optimizer/plan/createplan.c | 21 +-- src/backend/utils/misc/guc_tables.c | 2 +- src/include/nodes/execnodes.h | 2 +- src/include/nodes/plannodes.h | 3 +- 6 files changed, 206 insertions(+), 18 deletions(-) diff --git a/src/backend/access/brin/brin_minmax.c b/src/backend/access/brin/brin_minmax.c index 162650cf73..6390f93a0d 100644 --- a/src/backend/access/brin/brin_minmax.c +++ b/src/backend/access/brin/brin_minmax.c @@ -47,9 +47,6 @@ static FmgrInfo *minmax_get_strategy_procinfo(BrinDesc *bdesc, uint16 attno, Oid subtype, uint16 strategynum); -/* print info about ranges */ -#define BRINSORT_DEBUG - Datum brin_minmax_opcinfo(PG_FUNCTION_ARGS) { @@ -1995,7 +1992,7 @@ brin_minmax_scan_add_tuple(BrinRangeScanDesc *scan, TupleTableSlot *slot, scan->nranges++; } -#ifdef BRINSORT_DEBUG +#ifdef BRIN_SORT_DEBUG /* * brin_minmax_scan_next * Return the next BRIN range information from the tuplestore. @@ -2204,7 +2201,7 @@ brin_minmax_ranges(PG_FUNCTION_ARGS) /* do the sort and any necessary post-processing */ brin_minmax_scan_finalize(brscan); -#ifdef BRINSORT_DEBUG +#ifdef BRIN_SORT_DEBUG brin_minmax_scan_dump(brscan); #endif diff --git a/src/backend/executor/nodeBrinSort.c b/src/backend/executor/nodeBrinSort.c index f8356202b7..08507f2b5d 100644 --- a/src/backend/executor/nodeBrinSort.c +++ b/src/backend/executor/nodeBrinSort.c @@ -218,6 +218,8 @@ * ExecBrinSortReInitializeDSM reinitialize DSM for fresh scan * ExecBrinSortInitializeWorker attach to DSM info in parallel worker */ +#include <math.h> + #include "postgres.h" #include "access/brin.h" @@ -248,6 +250,14 @@ static void ExecInitBrinSortRanges(BrinSort *node, BrinSortState *planstate); bool debug_brin_sort = false; #endif +/* + * How many distinct minval values to look forward for the next watermark? + * + * The smallest step we can do is 1, which means the immediately following + * (while distinct) minval. + */ +int brinsort_watermark_step = 0; + /* do various consistency checks */ static void AssertCheckRanges(BrinSortState *node) @@ -859,6 +869,175 @@ brinsort_rescan(BrinSortState *node) tuplesort_rescan(node->bs_scan->ranges); } +/* + * Look at the tuplesort statistics, and maybe increase or decrease the + * watermark step. If the last sort was in-memory, we decrease the step. + * If the sort was in-memory, but we used less than work_mem/3, increment + * the step value. + * + * XXX This should probably behave differently for LIMIT queries, so that + * we don't load too many rows unnecessarily. We already consider that in + * create_brinsort_plan, but maybe we should limit increments to the step + * value here too - say, by tracking how many rows are we supposed to + * produce, and limiting the watermark so that we don't process too many + * rows in future steps. + * + * XXX We might also track the number of rows in the sort and space used, + * to calculate more accurate estimate of row width. And then use that to + * calculate number of rows that fit into work_mem. But the number of rows + * that go into tuplesort (per range) added would still remain fairly + * inaccurate, so not sure how good this woud be. + */ +static void +brinsort_adjust_watermark_step(BrinSortState *node, TuplesortInstrumentation *stats) +{ + BrinSort *plan = (BrinSort *) node->ss.ps.plan; + + if (brinsort_watermark_step != -1) + return; + + if (stats->spaceType == SORT_SPACE_TYPE_DISK) + { + /* + * We don't know how much to decrease the step (hard to estimate + * due to space needed for in-memory and on-disk sorts is not + * easily comparable, so we just cut the step in half. For the + * in-memory sort, we then can do better estimate and increase + * the step more accurately. + */ + plan->watermark_step = Max(1, plan->watermark_step / 2); + } + else + { + /* + * Adjust the step based on the last sort - we shoot for 2/3 of + * work_mem, to keep some slack (and not switch to on-disk sort + * due to minor differences). We calculate the average row width + * using space used and number of rows in the tuplesort, number + * of rows we could fit into work_mem, and how many steps would + * that mean (assuming number of rows is proportional to the + * number of steps). + * + * We need to be careful about the number of rows we're supposed + * to produce (and how many we already produced). Consider for + * example a query with LIMIT 1000, and that we produce 999 rows + * in the first sort, so that we need only 1 more row. It would + * be silly to pick the steps with the goal to "fill work_mem" + * instead of just enough to produce the one row. + * + * XXX In principle, we don't know how many rows will need to be + * read from the table - there may be interesting rows already in + * the tuplestore (in which case we could do a smaller step). But + * we don't know how many such rows are there - maybe if we had + * multiple smaller tuplestores, which would also reduce the + * amount of "respill" we need to do. + */ + int nrows_remaining; + int step = plan->watermark_step; + int step_max = plan->watermark_step * 2; + + /* number of remaining rows we're expected to produce */ + nrows_remaining = Max(1.0, plan->step_maxrows - node->bs_stats.ntuples_tuplesort_all); + + /* + * If we sorted any rows, calculate how many similar rows we can fit + * into work_mem. We restrict ourselves to 2/3 of work_mem, to leave + * a bit of slack space. + * + * XXX Hopefully the average width is somewhat accurate, but maybe + * we should remember the width we originally expected, and combine + * that somehow. Maybe we should not use just the last tuplesort, + * but instead accumulate average from all preceding sorts and + * combine them somehow (say, using weighted average with older + * values having less influence). + */ + if (node->bs_stats.ntuples_tuplesort > 0) + { + int nrows_wmem; + int avgwidth; + + /* average tuple width, calculated from last sort */ + avgwidth = (stats->spaceUsed * 1024L / node->bs_stats.ntuples_tuplesort); + + /* + * Calculate the numer of rows to fit into 2/3 of work_mem, but + * cap to the number of rows we're expected to produce. + */ + nrows_wmem = Min(nrows_remaining, (2 * 1024L * work_mem / 3) / avgwidth); + + /* scale the number of steps to produce the number of rows */ + step = step * ((double) (nrows_wmem * avgwidth) / (stats->spaceUsed * 1024L)); + + /* remember this as the max, so that we don't overflow work_mem */ + step_max = Min(step, step_max); + + /* however, make sure we don't grow too fast - cap to 2x */ + step = Min(step, step_max); + } + + /* + * Now calculate average step size using data from all sorts we did + * up to now. Then we calculate the number of steps we expect to be + * necessary. + * + * If we had calculated average number of rows per step from AM stats, + * consider that too. It's possible the batch had just one row, which + * might result in very high estimate of steps - it'd be silly to + * jump e.g. from 1 to 1000 based on this unreliable statistics. To + * prevent that, we combine the two rows_per_step sources as weighted + * sum, using the observed vs. target number of rows as weight. The + * closer we're to the target, the more reliable value from past + * executions is. + * + * But we don't want to overflow work_mem, so cap by step_max. + */ + if (node->bs_stats.ntuples_tuplesort_all > 0) + { + double rows_per_step; + + /* average number of rows we produced per step so far */ + rows_per_step = (double) node->bs_stats.ntuples_tuplesort_all / node->bs_stats.watermark_updates_steps; + + /* + * If we have AM stats with average number of rows per step, consider + * that too - approximate depending on what fraction of rows we already + * produced (with higher fraction of rows produced we prefer the local + * average, as opposed to the global average from index AM stats). + */ + if (plan->rows_per_step > 0) + { + /* number of rows we already produced (as a fraction) */ + double weight = (double) node->bs_stats.ntuples_tuplesort_all / plan->step_maxrows; + + /* paranoia */ + weight = Min(1.0, weight); + + /* + * Approximate between index AM and "local" average calculated + * from past executions. The closer we get to target rows, the + * more we ignore the index AM stats. + */ + rows_per_step = weight * rows_per_step + (1 - weight) * plan->rows_per_step; + } + + /* approximate the steps between */ + step = Max(step, ceil((double) nrows_remaining / rows_per_step)); + + /* + * But don't overflow the current max (which is set either + * as 2x starting value, or from work_mem. + */ + step = Min(step, step_max); + } + + plan->watermark_step = step; + + } + + plan->watermark_step = Max(1, plan->watermark_step); + plan->watermark_step = Min(8192, plan->watermark_step); +} + /* ---------------------------------------------------------------- * IndexNext * @@ -997,13 +1176,21 @@ IndexNext(BrinSortState *node) /* * Reset tuplesort statistics between runs, otherwise - * we'll keep re-using stats from the largest run. + * we'll keep re-using stats from the largest run, which + * would then confuse the adaptive adjustment of the + * watermark step. */ tuplesort_reset_stats(node->bs_tuplesortstate); tuplesort_performsort(node->bs_tuplesortstate); node->bs_stats.sort_count++; + + memset(&stats, 0, sizeof(TuplesortInstrumentation)); + tuplesort_get_stats(node->bs_tuplesortstate, &stats); + + brinsort_adjust_watermark_step(node, &stats); + node->bs_stats.ntuples_tuplesort = 0; tuplesort_get_stats(node->bs_tuplesortstate, &stats); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 6a3a38dd14..3e406d04eb 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -324,13 +324,8 @@ static GatherMerge *create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path); -/* - * How many distinct minval values to look forward for the next watermark? - * - * The smallest step we can do is 1, which means the immediately following - * (while distinct) minval. - */ -int brinsort_watermark_step = 0; +/* defined in nodeBrinSort.c */ +extern int brinsort_watermark_step; /* * create_plan @@ -3434,8 +3429,14 @@ create_brinsort_plan(PlannerInfo *root, * are there, and if there are only few then try increasing the step? */ brinsort_plan->watermark_step = brinsort_watermark_step; + brinsort_plan->rows_per_step = -1; - if (brinsort_plan->watermark_step == 0) + if (root->limit_tuples > 0) + brinsort_plan->step_maxrows = root->limit_tuples; + else + brinsort_plan->step_maxrows = brinsort_plan->scan.plan.plan_rows; + + if (brinsort_plan->watermark_step <= 0) { BrinMinmaxStats *amstats; @@ -3463,7 +3464,9 @@ create_brinsort_plan(PlannerInfo *root, amstats->maxval_increment_avg); double rows_per_step = Max(1.0, pct_per_step * rows); - brinsort_plan->watermark_step = (int) (maxrows / rows_per_step); + brinsort_plan->rows_per_step = rows_per_step; + + brinsort_plan->watermark_step = (int) ceil(maxrows / rows_per_step); } /* some rough safety estimates */ diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 5e8d4e3852..43e73995e2 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -3587,7 +3587,7 @@ struct config_int ConfigureNamesInt[] = GUC_NOT_IN_SAMPLE }, &brinsort_watermark_step, - 0, 0, INT_MAX, + 0, -1, INT_MAX, NULL, NULL, NULL }, diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ae69a99cb8..21d75f749d 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1731,7 +1731,7 @@ typedef struct BrinSortState * We need two tuplesort instances - one for current range, one for * spill-over tuples from the overlapping ranges */ - void *bs_tuplesortstate; + Tuplesortstate *bs_tuplesortstate; Tuplestorestate *bs_tuplestore; } BrinSortState; diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index 9d784f6373..0600b1f379 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -530,7 +530,8 @@ typedef struct BrinSort /* number of watermark steps to make */ int watermark_step; - + int step_maxrows; + int rows_per_step; } BrinSort; /* ---------------- -- 2.41.0
0009-wip-add-brinsort-regression-tests-20230710.patch (text/x-patch, 179.7K) download From 6588978e12a349fcfdafd7745af01a7908ccbe01 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Fri, 3 Feb 2023 13:19:24 +0100 Subject: [PATCH 09/11] wip: add brinsort regression tests --- src/test/regress/expected/brin_sort.out | 536 ++++++++++++ src/test/regress/expected/brin_sort_exprs.out | 776 ++++++++++++++++++ src/test/regress/expected/brin_sort_multi.out | 533 ++++++++++++ .../expected/brin_sort_multi_exprs.out | 772 +++++++++++++++++ src/test/regress/parallel_schedule | 6 + src/test/regress/sql/brin_sort.sql | 238 ++++++ src/test/regress/sql/brin_sort_exprs.sql | 373 +++++++++ src/test/regress/sql/brin_sort_multi.sql | 235 ++++++ .../regress/sql/brin_sort_multi_exprs.sql | 369 +++++++++ 9 files changed, 3838 insertions(+) create mode 100644 src/test/regress/expected/brin_sort.out create mode 100644 src/test/regress/expected/brin_sort_exprs.out create mode 100644 src/test/regress/expected/brin_sort_multi.out create mode 100644 src/test/regress/expected/brin_sort_multi_exprs.out create mode 100644 src/test/regress/sql/brin_sort.sql create mode 100644 src/test/regress/sql/brin_sort_exprs.sql create mode 100644 src/test/regress/sql/brin_sort_multi.sql create mode 100644 src/test/regress/sql/brin_sort_multi_exprs.sql diff --git a/src/test/regress/expected/brin_sort.out b/src/test/regress/expected/brin_sort.out new file mode 100644 index 0000000000..350a101a5b --- /dev/null +++ b/src/test/regress/expected/brin_sort.out @@ -0,0 +1,536 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin (int_val) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin (bigint_val) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (text_val) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin (inet_val inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +drop table brin_sort_test; diff --git a/src/test/regress/expected/brin_sort_exprs.out b/src/test/regress/expected/brin_sort_exprs.out new file mode 100644 index 0000000000..9044b4991a --- /dev/null +++ b/src/test/regress/expected/brin_sort_exprs.out @@ -0,0 +1,776 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1)) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin ((bigint_val + 1)) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (('x' || text_val)) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin ((inet_val + 1) inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +drop table brin_sort_test; diff --git a/src/test/regress/expected/brin_sort_multi.out b/src/test/regress/expected/brin_sort_multi.out new file mode 100644 index 0000000000..512511f712 --- /dev/null +++ b/src/test/regress/expected/brin_sort_multi.out @@ -0,0 +1,533 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_multi_idx on brin_sort_test using brin (int_val, bigint_val, text_val, inet_val inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +drop table brin_sort_test; diff --git a/src/test/regress/expected/brin_sort_multi_exprs.out b/src/test/regress/expected/brin_sort_multi_exprs.out new file mode 100644 index 0000000000..7fade4e7c4 --- /dev/null +++ b/src/test/regress/expected/brin_sort_multi_exprs.out @@ -0,0 +1,772 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1), (bigint_val + 1), ('x' || text_val), (inet_val + 1) inet_minmax_ops) with (pages_per_range=1); +vacuum analyze brin_sort_test; +set enable_seqscan = off; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 31 at RAISE +drop table brin_sort_test; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index cf46fa3359..15ead24d4d 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -131,3 +131,9 @@ test: fast_default # run tablespace test at the end because it drops the tablespace created during # setup that other tests may use. test: tablespace + +# try sorting using BRIN index +test: brin_sort +test: brin_sort_multi +test: brin_sort_exprs +test: brin_sort_multi_exprs diff --git a/src/test/regress/sql/brin_sort.sql b/src/test/regress/sql/brin_sort.sql new file mode 100644 index 0000000000..b38bd42457 --- /dev/null +++ b/src/test/regress/sql/brin_sort.sql @@ -0,0 +1,238 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; + +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); + +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); + +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin (int_val) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin (bigint_val) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (text_val) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin (inet_val inet_minmax_ops) with (pages_per_range=1); + +-- +vacuum analyze brin_sort_test; + +set enable_seqscan = off; + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +drop table brin_sort_test; diff --git a/src/test/regress/sql/brin_sort_exprs.sql b/src/test/regress/sql/brin_sort_exprs.sql new file mode 100644 index 0000000000..c2dccf6618 --- /dev/null +++ b/src/test/regress/sql/brin_sort_exprs.sql @@ -0,0 +1,373 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; + +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); + +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); + +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1)) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin ((bigint_val + 1)) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (('x' || text_val)) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin ((inet_val + 1) inet_minmax_ops) with (pages_per_range=1); + +-- +vacuum analyze brin_sort_test; + +set enable_seqscan = off; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + + + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); + + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + + + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); + + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + + + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); + + +drop table brin_sort_test; diff --git a/src/test/regress/sql/brin_sort_multi.sql b/src/test/regress/sql/brin_sort_multi.sql new file mode 100644 index 0000000000..e5d96877d3 --- /dev/null +++ b/src/test/regress/sql/brin_sort_multi.sql @@ -0,0 +1,235 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; + +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); + +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); + +-- create brin indexes on individual columns +create index brin_sort_test_multi_idx on brin_sort_test using brin (int_val, bigint_val, text_val, inet_val inet_minmax_ops) with (pages_per_range=1); + +-- +vacuum analyze brin_sort_test; + +set enable_seqscan = off; + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +drop table brin_sort_test; diff --git a/src/test/regress/sql/brin_sort_multi_exprs.sql b/src/test/regress/sql/brin_sort_multi_exprs.sql new file mode 100644 index 0000000000..3ef7c8acee --- /dev/null +++ b/src/test/regress/sql/brin_sort_multi_exprs.sql @@ -0,0 +1,369 @@ +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; + +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); + +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); + +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1), (bigint_val + 1), ('x' || text_val), (inet_val + 1) inet_minmax_ops) with (pages_per_range=1); + +vacuum analyze brin_sort_test; + +set enable_seqscan = off; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + + + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); + + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + + + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); + + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + + + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); + + +drop table brin_sort_test; -- 2.41.0
0010-wip-add-brinsort-amstats-regression-tests-20230710.patch (text/x-patch, 179.7K) download From 323edd19b673f0ac7cf476d1ebf881b67272a7d2 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Sat, 18 Feb 2023 16:41:33 +0100 Subject: [PATCH 10/11] wip: add brinsort amstats regression tests --- .../regress/expected/brin_sort_amstats.out | 537 ++++++++++++ .../expected/brin_sort_exprs_amstats.out | 762 ++++++++++++++++++ .../expected/brin_sort_multi_amstats.out | 534 ++++++++++++ .../brin_sort_multi_exprs_amstats.out | 758 +++++++++++++++++ src/test/regress/parallel_schedule | 6 + src/test/regress/sql/brin_sort_amstats.sql | 240 ++++++ .../regress/sql/brin_sort_exprs_amstats.sql | 270 +++++++ .../regress/sql/brin_sort_multi_amstats.sql | 237 ++++++ .../sql/brin_sort_multi_exprs_amstats.sql | 265 ++++++ 9 files changed, 3609 insertions(+) create mode 100644 src/test/regress/expected/brin_sort_amstats.out create mode 100644 src/test/regress/expected/brin_sort_exprs_amstats.out create mode 100644 src/test/regress/expected/brin_sort_multi_amstats.out create mode 100644 src/test/regress/expected/brin_sort_multi_exprs_amstats.out create mode 100644 src/test/regress/sql/brin_sort_amstats.sql create mode 100644 src/test/regress/sql/brin_sort_exprs_amstats.sql create mode 100644 src/test/regress/sql/brin_sort_multi_amstats.sql create mode 100644 src/test/regress/sql/brin_sort_multi_exprs_amstats.sql diff --git a/src/test/regress/expected/brin_sort_amstats.out b/src/test/regress/expected/brin_sort_amstats.out new file mode 100644 index 0000000000..c3cba20502 --- /dev/null +++ b/src/test/regress/expected/brin_sort_amstats.out @@ -0,0 +1,537 @@ +set enable_indexam_stats = true; +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin (int_val) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin (bigint_val) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (text_val) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin (inet_val inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +drop table brin_sort_test; diff --git a/src/test/regress/expected/brin_sort_exprs_amstats.out b/src/test/regress/expected/brin_sort_exprs_amstats.out new file mode 100644 index 0000000000..b038f7ce01 --- /dev/null +++ b/src/test/regress/expected/brin_sort_exprs_amstats.out @@ -0,0 +1,762 @@ +set enable_indexam_stats = true; +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + -- needed because the p_sql query has different data types + execute 'discard plans'; + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + CLOSE v_curs; + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + CLOSE v_curs; + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1)) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin ((bigint_val + 1)) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (('x' || text_val)) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin ((inet_val + 1) inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +drop table brin_sort_test; diff --git a/src/test/regress/expected/brin_sort_multi_amstats.out b/src/test/regress/expected/brin_sort_multi_amstats.out new file mode 100644 index 0000000000..4e71f21cf3 --- /dev/null +++ b/src/test/regress/expected/brin_sort_multi_amstats.out @@ -0,0 +1,534 @@ +set enable_indexam_stats = true; +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_multi_idx on brin_sort_test using brin (int_val, bigint_val, text_val, inet_val inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +drop table brin_sort_test; diff --git a/src/test/regress/expected/brin_sort_multi_exprs_amstats.out b/src/test/regress/expected/brin_sort_multi_exprs_amstats.out new file mode 100644 index 0000000000..2b9adfce62 --- /dev/null +++ b/src/test/regress/expected/brin_sort_multi_exprs_amstats.out @@ -0,0 +1,758 @@ +set enable_indexam_stats = true; +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + -- needed because the p_sql query has different data types + execute 'discard plans'; + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + CLOSE v_curs; + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + CLOSE v_curs; + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1), (bigint_val + 1), ('x' || text_val), (inet_val + 1) inet_minmax_ops) with (pages_per_range=1); +vacuum analyze brin_sort_test; +set enable_seqscan = off; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); + brinsort_check_ordering +------------------------- + +(1 row) + +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); + brinsort_check_ordering +------------------------- + +(1 row) + +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +ERROR: BRIN Sort: not found +CONTEXT: PL/pgSQL function brinsort_check_ordering(text,integer,boolean) line 24 at RAISE +drop table brin_sort_test; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 15ead24d4d..9b36202518 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -137,3 +137,9 @@ test: brin_sort test: brin_sort_multi test: brin_sort_exprs test: brin_sort_multi_exprs + +# try sorting using BRIN index with indexam stats +test: brin_sort_amstats +test: brin_sort_multi_amstats +test: brin_sort_exprs_amstats +test: brin_sort_multi_exprs_amstats diff --git a/src/test/regress/sql/brin_sort_amstats.sql b/src/test/regress/sql/brin_sort_amstats.sql new file mode 100644 index 0000000000..c3da189675 --- /dev/null +++ b/src/test/regress/sql/brin_sort_amstats.sql @@ -0,0 +1,240 @@ +set enable_indexam_stats = true; + +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; + +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); + +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); + +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin (int_val) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin (bigint_val) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (text_val) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin (inet_val inet_minmax_ops) with (pages_per_range=1); + +-- +vacuum analyze brin_sort_test; + +set enable_seqscan = off; + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +drop table brin_sort_test; diff --git a/src/test/regress/sql/brin_sort_exprs_amstats.sql b/src/test/regress/sql/brin_sort_exprs_amstats.sql new file mode 100644 index 0000000000..02a9f23f80 --- /dev/null +++ b/src/test/regress/sql/brin_sort_exprs_amstats.sql @@ -0,0 +1,270 @@ +set enable_indexam_stats = true; +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + -- needed because the p_sql query has different data types + execute 'discard plans'; + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + CLOSE v_curs; + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + CLOSE v_curs; + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1)) with (pages_per_range=1); +create index brin_sort_test_bigint_idx on brin_sort_test using brin ((bigint_val + 1)) with (pages_per_range=1); +create index brin_sort_test_text_idx on brin_sort_test using brin (('x' || text_val)) with (pages_per_range=1); +create index brin_sort_test_inet_idx on brin_sort_test using brin ((inet_val + 1) inet_minmax_ops) with (pages_per_range=1); +-- +vacuum analyze brin_sort_test; +set enable_seqscan = off; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +drop table brin_sort_test; diff --git a/src/test/regress/sql/brin_sort_multi_amstats.sql b/src/test/regress/sql/brin_sort_multi_amstats.sql new file mode 100644 index 0000000000..1bd8c7802e --- /dev/null +++ b/src/test/regress/sql/brin_sort_multi_amstats.sql @@ -0,0 +1,237 @@ +set enable_indexam_stats = true; + +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + + -- needed because the p_sql query has different data types + execute 'discard plans'; + + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + + CLOSE v_curs; + + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + + LOOP + FETCH v_curs INTO v_row; + + IF NOT FOUND THEN + EXIT; + END IF; + + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + + CLOSE v_curs; + + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; + +end; +$$ language plpgsql; + +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); + +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); + +-- create brin indexes on individual columns +create index brin_sort_test_multi_idx on brin_sort_test using brin (int_val, bigint_val, text_val, inet_val inet_minmax_ops) with (pages_per_range=1); + +-- +vacuum analyze brin_sort_test; + +set enable_seqscan = off; + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); + +reindex table brin_sort_test; + +vacuum analyze brin_sort_test; + + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val', 1000, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc', 1000, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100', 100, true); + +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select int_val as val from brin_sort_test order by int_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val', 1000, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc', 1000, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100', 100, true); + +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select bigint_val as val from brin_sort_test order by bigint_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val', 1000, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc', 1000, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100', 100, true); + +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select inet_val as val from brin_sort_test order by inet_val desc limit 100 offset 100', 100, true); + + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val', 1000, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc', 1000, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100', 100, true); + +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val limit 100 offset 100', 100, false); +select brinsort_check_ordering('select text_val as val from brin_sort_test order by text_val desc limit 100 offset 100', 100, true); + + +drop table brin_sort_test; diff --git a/src/test/regress/sql/brin_sort_multi_exprs_amstats.sql b/src/test/regress/sql/brin_sort_multi_exprs_amstats.sql new file mode 100644 index 0000000000..d91666642c --- /dev/null +++ b/src/test/regress/sql/brin_sort_multi_exprs_amstats.sql @@ -0,0 +1,265 @@ +set enable_indexam_stats = true; +-- function to verify various sort-related data (total rows, ordering) +create or replace function brinsort_check_ordering(p_sql text, p_rows_expected int, p_desc boolean) returns void as $$ +declare + v_curs refcursor; + v_row record; + v_prev record; + v_brin_sort_found bool := false; + v_count int := 0; +begin + -- needed because the p_sql query has different data types + execute 'discard plans'; + OPEN v_curs NO SCROLL FOR EXECUTE format('explain %s', p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_row::text LIKE '%BRIN Sort%' THEN + v_brin_sort_found := true; + EXIT; + END IF; + END LOOP; + CLOSE v_curs; + IF NOT v_brin_sort_found THEN + RAISE EXCEPTION 'BRIN Sort: not found'; + END IF; + OPEN v_curs NO SCROLL FOR EXECUTE format(p_sql); + LOOP + FETCH v_curs INTO v_row; + IF NOT FOUND THEN + EXIT; + END IF; + IF v_prev IS NOT NULL THEN + IF v_prev.val > v_row.val AND NOT p_desc THEN + RAISE EXCEPTION 'ordering mismatch % > % (asc)', v_prev.val, v_row.val; + END IF; + IF v_prev.val < v_row.val AND p_desc THEN + RAISE EXCEPTION 'ordering mismatch % < % (desc)', v_prev.val, v_row.val; + END IF; + END IF; + v_prev := v_row; + v_count := v_count + 1; + END LOOP; + CLOSE v_curs; + IF v_count != p_rows_expected THEN + RAISE EXCEPTION 'count mismatch: % != %', v_count, p_rows_expected; + END IF; +end; +$$ language plpgsql; +create table brin_sort_test (int_val int, bigint_val bigint, text_val text, inet_val inet) with (fillfactor=10); +-- sequential values +insert into brin_sort_test +select + i, + -i, -- same as int, but at least opposite + lpad(i::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + i +from generate_series(1,1000) s(i); +-- create brin indexes on individual columns +create index brin_sort_test_int_idx on brin_sort_test using brin ((int_val + 1), (bigint_val + 1), ('x' || text_val), (inet_val + 1) inet_minmax_ops) with (pages_per_range=1); +vacuum analyze brin_sort_test; +set enable_seqscan = off; +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +-- semi-random data (sequential + randomness) +truncate table brin_sort_test; +insert into brin_sort_test +select + i + (100 * random())::int, + -(i + (100 * random())::int), -- same as int, but at least opposite + lpad((i + (100 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (i + 100 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +-- random data +truncate table brin_sort_test; +insert into brin_sort_test +select + (1000 * random())::int, + -((1000 * random())::int), -- same as int, but at least opposite + lpad(((1000 * random())::int)::text || md5(i::text), 40, '0'), + '10.0.0.0'::inet + (1000 * random()::int) +from generate_series(1,1000) s(i); +reindex table brin_sort_test; +vacuum analyze brin_sort_test; + +-- matching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1)', 1000, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (int_val + 1) as val from brin_sort_test order by (int_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1)', 1000, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (bigint_val + 1) as val from brin_sort_test order by (bigint_val - 1) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''x'' || text_val) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val)', 1000, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc', 1000, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100', 100, true); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (''x'' || text_val) as val from brin_sort_test order by (''y'' || text_val) desc limit 100 offset 100', 100, true); + +-- matching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val + 1) desc limit 100 offset 100', 100, true); +-- mismatching expression +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1)', 1000, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc', 1000, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100', 100, true); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) limit 100 offset 100', 100, false); +select brinsort_check_ordering('select (inet_val + 1) as val from brin_sort_test order by (inet_val - 1) desc limit 100 offset 100', 100, true); +drop table brin_sort_test; -- 2.41.0
0011-wip-test-generator-script-20230710.patch (text/x-patch, 11.2K) download From 48b4e1a8709cbb51e3e0cb4f5442df773080af49 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <tomas.vondra@postgresql.org> Date: Mon, 6 Feb 2023 03:42:52 +0100 Subject: [PATCH 11/11] wip: test generator script --- brin-test.py | 386 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 386 insertions(+) create mode 100644 brin-test.py diff --git a/brin-test.py b/brin-test.py new file mode 100644 index 0000000000..c90ec798be --- /dev/null +++ b/brin-test.py @@ -0,0 +1,386 @@ +import psycopg2 +import psycopg2.extras +import random +import sys +import time +import re + +from datetime import datetime +from statistics import mean + +cols = [('int_val', 'int4_minmax_ops'), + ('bigint_val', 'int8_minmax_ops'), + ('text_val', 'text_minmax_ops'), + ('inet_val', 'inet_minmax_ops'), + ('(int_val+1)', 'int4_minmax_ops'), + ('(bigint_val+1)', 'int8_minmax_ops'), + ("('x' || text_val)", 'text_minmax_ops'), + ('(inet_val + 1)', 'inet_minmax_ops'), + ('(int_val+2)', 'int4_minmax_ops'), + ('(bigint_val+2)', 'int8_minmax_ops'), + ("('y' || text_val)", 'text_minmax_ops'), + ('(inet_val + 2)', 'inet_minmax_ops')] + +# randomly reorder the table columns +#table_cols = [('int_val int', 'i', 'i + %(skew)d * random()', 'i + 1000000 * random()'), +# ('bigint_val bigint', '-i', '-i - 100 * random()', '-1 - 1000000 * random()'), +# ('inet_val inet', "'10.0.0.0'::inet + i", "'10.0.0.0'::inet + i * 100 * random()::int", "'10.0.0.0'::inet + i + 1000000 * random()::int"), +# ('text_val text', "lpad(i::text || md5(i::text), 40, '0')", "lpad((i + 100*random()::int)::text || md5(i::text), 40, '0')", "lpad((i + 1000000*random()::int)::text || md5(i::text), 40, '0')")] + +table_cols = [('int_val int', 'i + %(randomness)d * random()'), + ('bigint_val bigint', '-i - %(randomness)d * random()'), + ('inet_val inet', "'10.0.0.0'::inet + i + %(randomness)d * random()::int"), + ('text_val text', "lpad((i + %(randomness)d * random()::int)::text || md5(i::text), 40, '0')")] + + +def execute_query(cur, query, fetch_result = False): + + cur.execute(query) + + if fetch_result: + return cur.fetchall() + + +# recreate the table with the columns in randomized order +def recreate_table(conn, nrows, randomness, fillfactor): + + random.shuffle(table_cols) + + cur = conn.cursor() + + execute_query(cur, 'BEGIN') + + execute_query(cur, 'DROP TABLE IF EXISTS test_table') + + execute_query(cur, 'CREATE TABLE test_table (%s) with (fillfactor=%d)' % (', '.join([v[0] for v in table_cols]), fillfactor)) + print('CREATE TABLE test_table (%s) with (fillfactor=%d)' % (', '.join([v[0] for v in table_cols]), fillfactor)) + + insert_sql = 'INSERT INTO test_table SELECT %s FROM generate_series(1,%d) s(i)' % (', '.join([v[1] for v in table_cols]), nrows) + insert_sql = insert_sql % {'randomness' : int(nrows * randomness), 'rows' : nrows} + + print(insert_sql) + + execute_query(cur, insert_sql) + + execute_query(cur, 'COMMIT') + + cur.close() + + +def create_indexes(conn, pages_per_range): + + cur = conn.cursor() + + num_indexes = random.randint(1,len(cols)) + + # randomly pick columns to index + indexed = random.sample(cols, num_indexes) + + for c in indexed: + # f = random.random() + # num_pages = 1 + int(f * f * f * 256) + index_sql = 'CREATE INDEX ON test_table USING brin (%s %s) WITH (pages_per_range=%d)' % (c[0], c[1], pages_per_range) + print(index_sql) + execute_query(cur, index_sql) + + cur.close() + + return indexed + + +def brinsort_in_explain(cur, query): + + cur.execute('explain ' + query) + for r in cur.fetchall(): + if 'BRIN Sort' in r['QUERY PLAN']: + return True + + return False + + +def compare_default(a, b): + if a < b: + return -1 + elif a > b: + return 1 + return 0 + + +def compare_inet(a, b): + a = [int(v) for v in a.split('.')] + b = [int(v) for v in b.split('.')] + + for p in range(0,4): + r = compare_default(a[p], b[p]) + if r != 0: + return r + + return r + + +def check_ordering(conn, config, query, expected_rows, select_star, select_list, sort_list, is_desc): + + cur = conn.cursor() + + data = execute_query(cur, query, True) + + if len(data) != expected_rows: + print('ERROR: unexpected number of rows %s %s' % (expected_rows, len(data))) + sys.exit(1) + + # what prefix we can check ordering for (some sort columns may not be + # included in the result, and we need a continuous prefix) + prefix = [] + indexes = [] + sort_order = {} + for s in sort_list: + if select_star: + if s[0] not in [x for x in table_cols]: + break + + idx = [x for x in table_cols].index(s[0]) + else: + if s not in select_list: + break + + idx = select_list.index(s) + + if idx is None: + break + + sort_idx = sort_list.index(s) + + prefix.append(s) + indexes.append(idx) + # sort_order.update({select_list.index(s) : is_desc[sort_list.index(s)]}) + sort_order.update({idx : is_desc[sort_idx]}) + + # print("PREFIX", indexes, prefix) + + if len(prefix) != 0: + prev = None + for row in data: + if prev is not None: + + for idx in indexes: + + if select_list[idx][1] == 'inet_minmax_ops': + r = compare_inet(prev[idx], row[idx]) + else: + r = compare_default(prev[idx], row[idx]) + + if sort_order[idx]: + r = -r + + if r > 0: + print("ERROR: incorrect ordering %s > %s" % (prev[idx], row[idx])) + sys.exit(1) + + if r < 0: + break + + prev = row + + cur.close() + + +def run_queries(conn, config, indexed_cols, num_queries = 1000): + + nquery = 0 + + while nquery < num_queries: + if run_query(conn, config, indexed_cols): + nquery += 1 + + +def query_timing(cur, query): + + runs = [] + + # get explain plan and costs from the first node + r = execute_query(cur, 'explain (analyze, timing off) %s' % (query,), True) + print("") + print("\n".join([' ' + x[0] for x in r])) + print("") + + sys.stdout.flush() + + r = re.search('cost=([^\s]*)\.\.([^\s]*)', r[0][0]) + costs = [float(r.groups()[0]), float(r.groups()[1])] + + for r in range(0,1): + s = time.time() + execute_query(cur, query) + d = time.time() + runs.append(d-s) + + # print("runs %s => mean %s" % (str(runs), mean(runs))) + + sys.stdout.flush() + + return (mean(runs), costs) + + +def check_timing(conn, config, query): + + cur = conn.cursor() + + # get timing for a simple plan without a BRIN sort + + execute_query(cur, 'set enable_seqscan = on') + execute_query(cur, 'set enable_brinsort = off') + + (seqscan_time, seqscan_costs) = query_timing(cur, query) + + # get timing for a simple plan with a BRIN sort + execute_query(cur, 'set enable_seqscan = off') + execute_query(cur, 'set enable_brinsort = on') + execute_query(cur, query) + + (brinsort_time, brinsort_costs) = query_timing(cur, query) + + print ("timing", 'rows', config['nrows'], 'pages_per_range', config['pages_per_range'], 'randomness', config['randomness'], 'fillfactor', config['fillfactor'], 'work_mem', config['work_mem'], 'watermark_step', config['watermark_step'], 'limit', config['limit'], 'offset', config['offset'], "seqscan", seqscan_time, "brinsort", brinsort_time, "costs seqscan", seqscan_costs[0], seqscan_costs[1], "brinsort", brinsort_costs[0], brinsort_costs[1]) + # print ("brinsort timing", brinsort_time, "costs", brinsort_costs[0], brinsort_costs[1]) + + if (seqscan_costs[1] * 1.1 < brinsort_costs[1]) and (seqscan_time > brinsort_time * 1.1): + print ("COSTING ISSUE (%f < %f) && (%f > %f)" % (seqscan_costs[1], brinsort_costs[1], seqscan_time, brinsort_time)) + + if (seqscan_costs[1] > brinsort_costs[1] * 1.1) and (seqscan_time * 1.1 < brinsort_time): + print ("COSTING ISSUE (%f > %f) && (%f < %f)" % (seqscan_costs[1], brinsort_costs[1], seqscan_time, brinsort_time)) + + sys.stdout.flush() + + +def run_query(conn, config, indexed_cols): + + limit_rows = config['nrows'] + offset_rows = 0 + + cur = conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) + + # random columns to reference in the SELECT list, may not include sort column(s) + select_list = random.sample(cols, random.randint(1,len(cols))) + + # but maybe just do select *, so that we don't do a projection + select_star = False + if random.random() < 0.5: + select_star = True + select_list = [('*', None)] + + # random columns to reference in the ORDER BY clause + sort_list = random.sample(cols, random.randint(1,len(cols))) + + # generate random ASC / DESC modifiers + is_desc = [] + order_by = [] + for s in range(0,len(sort_list)): + desc = random.choice([True, False]) + is_desc.append(desc) + x = sort_list[s][0] + if desc: + x = x + ' DESC' + order_by.append(x) + + query = 'SELECT %s FROM test_table ORDER BY %s' % (', '.join([v[0] for v in select_list]), ', '.join(order_by)) + + # randomly add LIMIT and OFFSET clause(s) + if random.random() < 0.5: + + limit_rows = 1 + int(pow(random.random(), 3) * random.randint(1,config['nrows'])) + query = query + ' LIMIT %d' % (limit_rows,); + + if limit_rows < config['nrows'] and random.random() < 0.5: + + offset_rows = int(pow(random.random(), 3) * random.randint(1,config['nrows'] - limit_rows)) + query = query + ' OFFSET %d' % (offset_rows,); + + expected_rows = min(limit_rows, config['nrows'] - offset_rows) + + # watermark_step = random.randint(-1, 3) + watermark_step = random.choice([-1, 0, 1, 8, 32, 128]) + execute_query(cur, 'SET brinsort_watermark_step = %d' % (watermark_step,)) + + f = random.random() + #work_mem_kb = 64 + int((f * f * f) * random.randint(64, 32768)) + work_mem_kb = random.choice([64, 1024, 4096, 32768]) + + execute_query(cur, "SET work_mem = '%dkB'" % (work_mem_kb,)) + + config = config.copy() + config.update({'work_mem' : work_mem_kb}) + config.update({'watermark_step' : watermark_step}) + config.update({'limit' : limit_rows}) + config.update({'offset' : offset_rows}) + + # do we expect brinsort or not? only when the first ORDER BY is indexed + if sort_list[0] in indexed_cols: + + print('--------------', datetime.now(), '--------------') + print("SQL:", query) + print("CONFIG:", config) + + if brinsort_in_explain(cur, query): + check_ordering(conn, config, query, expected_rows, select_star, select_list, sort_list, is_desc) + check_timing(conn, config, query) + else: + print("ERROR: BRIN Sort not in plan") + sys.exit(1) + + result = True + + else: + + if brinsort_in_explain(cur, query): + print("ERROR: BRIN Sort in plan") + sys.exit(1) + + result = False + + cur.close() + + return result + + +def setup_connection(conn): + cur = conn.cursor() + + # force index access + execute_query(cur, 'SET enable_seqscan = off') + execute_query(cur, 'SET max_parallel_workers_per_gather = 0') + + cur.close() + + +run_id = 0 + +while True: + + run_id += 1 + + config = {} + + conn = psycopg2.connect('host=localhost port=5432 dbname=test user=user') + + setup_connection(conn) + + print('========== run %d ==========' % (run_id,)) + + # data distribution (1 - sequential, 3 - random) + config['randomness'] = random.choice([0, 0.05, 0.1, 0.25, 0.5, 1.0]) + + # random fillfactor, skewed closer to 10% + config['fillfactor'] = 10 + int(pow(random.random(),3) * 90) + + # random number of rows + config['nrows'] = random.choice([100000, 1000000]) + + # pages per BRIN range (for all indexes) + config['pages_per_range'] = random.choice([1, 32, 128]) + + recreate_table(conn, config['nrows'], config['randomness'], config['fillfactor']) + + indexed_cols = create_indexes(conn, config['pages_per_range']) + + run_queries(conn, config, indexed_cols) + + conn.close() -- 2.41.0

view thread (60+ messages)  latest in thread

Message-ID: <1992ff4f-bf0e-3ad0-a859-cf054f6abc74@enterprisedb.com>
Permalink:  ../1992ff4f-bf0e-3ad0-a859-cf054f6abc74@enterprisedb.com/
Also on:    postgresql.org/message-id/1992ff4f-bf0e-3ad0-a859-cf054f6abc74@enterprisedb.com

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: tomas.vondra@enterprisedb.com, boekewurm+postgres@gmail.com, pryzby@telsasoft.com, andres@anarazel.de, stark@mit.edu, pgsql-hackers@lists.postgresql.org
  Subject: Re: PATCH: Using BRIN indexes for sorted output
  In-Reply-To: <1992ff4f-bf0e-3ad0-a859-cf054f6abc74@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox