public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 8/9] Define multi-minmax oclasses for types without distance 3+ messages / 3 participants [nested] [flat]
* [PATCH 8/9] Define multi-minmax oclasses for types without distance @ 2021-02-03 18:12 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Tomas Vondra @ 2021-02-03 18:12 UTC (permalink / raw) The existing multi-minmax opclasses rely on "distance" function when deciding how to build the ranges, covering all the values. In principle, those opclasses try to minimize the lengths of ranges, i.e. maximize the lengths of "gaps" between them. For some data types it's hard to construct a distance function - types like "text" or "name" generally serve as labels, and may have ordering only. So this uses a different approach, based on the observation that it's the outliers that "break" BRIN minmax indexes, i.e. the lowest and highest values. So simply sort the values, keep the (K-2) extreme values on either tail, and build a single range representing the values in the middle. Of course, the question is whether this is actually useful, but that's hard to judge, and it's the best we can do. --- src/backend/access/brin/brin_minmax_multi.c | 75 ++++++++++++ src/include/catalog/pg_amop.dat | 119 ++++++++++++++++++++ src/include/catalog/pg_amproc.dat | 113 +++++++++++++++++++ src/include/catalog/pg_opclass.dat | 21 ++++ src/include/catalog/pg_opfamily.dat | 14 +++ 5 files changed, 342 insertions(+) diff --git a/src/backend/access/brin/brin_minmax_multi.c b/src/backend/access/brin/brin_minmax_multi.c index fd85c18d83..c77c207e94 100644 --- a/src/backend/access/brin/brin_minmax_multi.c +++ b/src/backend/access/brin/brin_minmax_multi.c @@ -1210,6 +1210,9 @@ build_distances(FmgrInfo *distanceFn, Oid colloid, Assert(ncranges >= 2); + if (!distanceFn) + return NULL; + distances = (DistanceValue *) palloc0(sizeof(DistanceValue) * (ncranges - 1)); /* @@ -1298,6 +1301,70 @@ count_values(CombineRange *cranges, int ncranges) } #endif + +static int +reduce_combine_ranges_simple(CombineRange *cranges, int ncranges, + int max_values, FmgrInfo *cmp, Oid colloid) +{ + int i; + int nvalues; + Datum *values; + + compare_context cxt; + + /* number of values to keep on each tail */ + int tail = (max_values - 2) / 2; + int m = Min(tail / 2, ncranges - 1 / 2); + + /* allocate space for the boundary values */ + nvalues = 0; + values = (Datum *) palloc(sizeof(Datum) * max_values); + + for (i = 0; i < m; i++) + { + /* head */ + values[nvalues++] = cranges[i].minval; + values[nvalues++] = cranges[i].maxval; + + /* tail */ + values[nvalues++] = cranges[ncranges - 1 - i].minval; + values[nvalues++] = cranges[ncranges - 1 - i].maxval; + } + + /* middle part */ + values[nvalues++] = cranges[m].maxval; + values[nvalues++] = cranges[ncranges - 1 - m].minval; + + /* We should have even number of range values. */ + Assert(nvalues % 2 == 0); + + /* sort the values */ + cxt.colloid = colloid; + cxt.cmpFn = cmp; + + /* + * Sort the values using the comparator function, and form ranges + * from the sorted result. + */ + qsort_arg(values, nvalues, sizeof(Datum), + compare_values, (void *) &cxt); + + /* We have nvalues boundary values, which means nvalues/2 ranges. */ + for (i = 0; i < (nvalues / 2); i++) + { + cranges[i].minval = values[2*i]; + cranges[i].maxval = values[2*i + 1]; + + /* if the boundary values are the same, it's a collapsed range */ + cranges[i].collapsed = (compare_values(&values[2*i], + &values[2*i+1], + &cxt) == 0); + } + + return (nvalues / 2); +} + + /* * reduce_combine_ranges * reduce the ranges until the number of values is low enough @@ -1366,6 +1433,14 @@ reduce_combine_ranges(CombineRange *cranges, int ncranges, if (keep >= ndistances) return ncranges; + /* + * Without distances, we use a simple approach keeping as many + * outliers as possible. + */ + if (!distances) + return reduce_combine_ranges_simple(cranges, ncranges, max_values, + cmp, colloid); + /* sort the values */ cxt.colloid = colloid; cxt.cmpFn = cmp; diff --git a/src/include/catalog/pg_amop.dat b/src/include/catalog/pg_amop.dat index 8135854163..5cf820f04b 100644 --- a/src/include/catalog/pg_amop.dat +++ b/src/include/catalog/pg_amop.dat @@ -1814,6 +1814,23 @@ amoprighttype => 'bytea', amopstrategy => '5', amopopr => '>(bytea,bytea)', amopmethod => 'brin' }, +# minmax multi bytea +{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea', + amoprighttype => 'bytea', amopstrategy => '1', amopopr => '<(bytea,bytea)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea', + amoprighttype => 'bytea', amopstrategy => '2', amopopr => '<=(bytea,bytea)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea', + amoprighttype => 'bytea', amopstrategy => '3', amopopr => '=(bytea,bytea)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea', + amoprighttype => 'bytea', amopstrategy => '4', amopopr => '>=(bytea,bytea)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea', + amoprighttype => 'bytea', amopstrategy => '5', amopopr => '>(bytea,bytea)', + amopmethod => 'brin' }, + # bloom bytea { amopfamily => 'brin/bytea_bloom_ops', amoplefttype => 'bytea', amoprighttype => 'bytea', amopstrategy => '1', amopopr => '=(bytea,bytea)', @@ -1836,6 +1853,23 @@ amoprighttype => 'char', amopstrategy => '5', amopopr => '>(char,char)', amopmethod => 'brin' }, +# minmax multi "char" +{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char', + amoprighttype => 'char', amopstrategy => '1', amopopr => '<(char,char)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char', + amoprighttype => 'char', amopstrategy => '2', amopopr => '<=(char,char)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char', + amoprighttype => 'char', amopstrategy => '3', amopopr => '=(char,char)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char', + amoprighttype => 'char', amopstrategy => '4', amopopr => '>=(char,char)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char', + amoprighttype => 'char', amopstrategy => '5', amopopr => '>(char,char)', + amopmethod => 'brin' }, + # bloom "char" { amopfamily => 'brin/char_bloom_ops', amoplefttype => 'char', amoprighttype => 'char', amopstrategy => '1', amopopr => '=(char,char)', @@ -1858,6 +1892,23 @@ amoprighttype => 'name', amopstrategy => '5', amopopr => '>(name,name)', amopmethod => 'brin' }, +# minmax multi name +{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name', + amoprighttype => 'name', amopstrategy => '1', amopopr => '<(name,name)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name', + amoprighttype => 'name', amopstrategy => '2', amopopr => '<=(name,name)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name', + amoprighttype => 'name', amopstrategy => '3', amopopr => '=(name,name)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name', + amoprighttype => 'name', amopstrategy => '4', amopopr => '>=(name,name)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name', + amoprighttype => 'name', amopstrategy => '5', amopopr => '>(name,name)', + amopmethod => 'brin' }, + # bloom name { amopfamily => 'brin/name_bloom_ops', amoplefttype => 'name', amoprighttype => 'name', amopstrategy => '1', amopopr => '=(name,name)', @@ -2186,6 +2237,23 @@ amoprighttype => 'text', amopstrategy => '5', amopopr => '>(text,text)', amopmethod => 'brin' }, +# minmax multi text +{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text', + amoprighttype => 'text', amopstrategy => '1', amopopr => '<(text,text)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text', + amoprighttype => 'text', amopstrategy => '2', amopopr => '<=(text,text)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text', + amoprighttype => 'text', amopstrategy => '3', amopopr => '=(text,text)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text', + amoprighttype => 'text', amopstrategy => '4', amopopr => '>=(text,text)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text', + amoprighttype => 'text', amopstrategy => '5', amopopr => '>(text,text)', + amopmethod => 'brin' }, + # bloom text { amopfamily => 'brin/text_bloom_ops', amoplefttype => 'text', amoprighttype => 'text', amopstrategy => '1', amopopr => '=(text,text)', @@ -2562,6 +2630,23 @@ amoprighttype => 'bpchar', amopstrategy => '5', amopopr => '>(bpchar,bpchar)', amopmethod => 'brin' }, +# minmax multi character +{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar', + amoprighttype => 'bpchar', amopstrategy => '1', amopopr => '<(bpchar,bpchar)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar', + amoprighttype => 'bpchar', amopstrategy => '2', + amopopr => '<=(bpchar,bpchar)', amopmethod => 'brin' }, +{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar', + amoprighttype => 'bpchar', amopstrategy => '3', amopopr => '=(bpchar,bpchar)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar', + amoprighttype => 'bpchar', amopstrategy => '4', + amopopr => '>=(bpchar,bpchar)', amopmethod => 'brin' }, +{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar', + amoprighttype => 'bpchar', amopstrategy => '5', amopopr => '>(bpchar,bpchar)', + amopmethod => 'brin' }, + # bloom character { amopfamily => 'brin/bpchar_bloom_ops', amoplefttype => 'bpchar', amoprighttype => 'bpchar', amopstrategy => '1', amopopr => '=(bpchar,bpchar)', @@ -3007,6 +3092,23 @@ amoprighttype => 'bit', amopstrategy => '5', amopopr => '>(bit,bit)', amopmethod => 'brin' }, +# minmax multi bit +{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit', + amoprighttype => 'bit', amopstrategy => '1', amopopr => '<(bit,bit)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit', + amoprighttype => 'bit', amopstrategy => '2', amopopr => '<=(bit,bit)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit', + amoprighttype => 'bit', amopstrategy => '3', amopopr => '=(bit,bit)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit', + amoprighttype => 'bit', amopstrategy => '4', amopopr => '>=(bit,bit)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit', + amoprighttype => 'bit', amopstrategy => '5', amopopr => '>(bit,bit)', + amopmethod => 'brin' }, + # minmax bit varying { amopfamily => 'brin/varbit_minmax_ops', amoplefttype => 'varbit', amoprighttype => 'varbit', amopstrategy => '1', amopopr => '<(varbit,varbit)', @@ -3024,6 +3126,23 @@ amoprighttype => 'varbit', amopstrategy => '5', amopopr => '>(varbit,varbit)', amopmethod => 'brin' }, +# minmax multi bit varying +{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit', + amoprighttype => 'varbit', amopstrategy => '1', amopopr => '<(varbit,varbit)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit', + amoprighttype => 'varbit', amopstrategy => '2', + amopopr => '<=(varbit,varbit)', amopmethod => 'brin' }, +{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit', + amoprighttype => 'varbit', amopstrategy => '3', amopopr => '=(varbit,varbit)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit', + amoprighttype => 'varbit', amopstrategy => '4', + amopopr => '>=(varbit,varbit)', amopmethod => 'brin' }, +{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit', + amoprighttype => 'varbit', amopstrategy => '5', amopopr => '>(varbit,varbit)', + amopmethod => 'brin' }, + # minmax numeric { amopfamily => 'brin/numeric_minmax_ops', amoplefttype => 'numeric', amoprighttype => 'numeric', amopstrategy => '1', diff --git a/src/include/catalog/pg_amproc.dat b/src/include/catalog/pg_amproc.dat index 51403716b1..69a7ed682c 100644 --- a/src/include/catalog/pg_amproc.dat +++ b/src/include/catalog/pg_amproc.dat @@ -805,6 +805,22 @@ { amprocfamily => 'brin/bytea_minmax_ops', amproclefttype => 'bytea', amprocrighttype => 'bytea', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi bytea +{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea', + amprocrighttype => 'bytea', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # bloom bytea { amprocfamily => 'brin/bytea_bloom_ops', amproclefttype => 'bytea', amprocrighttype => 'bytea', amprocnum => '1', @@ -836,6 +852,22 @@ { amprocfamily => 'brin/char_minmax_ops', amproclefttype => 'char', amprocrighttype => 'char', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi "char" +{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char', + amprocrighttype => 'char', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # bloom "char" { amprocfamily => 'brin/char_bloom_ops', amproclefttype => 'char', amprocrighttype => 'char', amprocnum => '1', @@ -867,6 +899,22 @@ { amprocfamily => 'brin/name_minmax_ops', amproclefttype => 'name', amprocrighttype => 'name', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi name +{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name', + amprocrighttype => 'name', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # bloom name { amprocfamily => 'brin/name_bloom_ops', amproclefttype => 'name', amprocrighttype => 'name', amprocnum => '1', @@ -1197,6 +1245,22 @@ { amprocfamily => 'brin/text_minmax_ops', amproclefttype => 'text', amprocrighttype => 'text', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi text +{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text', + amprocrighttype => 'text', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # bloom text { amprocfamily => 'brin/text_bloom_ops', amproclefttype => 'text', amprocrighttype => 'text', amprocnum => '1', @@ -1663,6 +1727,23 @@ amprocrighttype => 'bpchar', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi character +{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar', + amprocrighttype => 'bpchar', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # bloom character { amprocfamily => 'brin/bpchar_bloom_ops', amproclefttype => 'bpchar', amprocrighttype => 'bpchar', amprocnum => '1', @@ -2180,6 +2261,21 @@ { amprocfamily => 'brin/bit_minmax_ops', amproclefttype => 'bit', amprocrighttype => 'bit', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi bit +{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '1', amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit', + amprocrighttype => 'bit', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # minmax bit varying { amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit', amprocrighttype => 'varbit', amprocnum => '1', @@ -2194,6 +2290,23 @@ amprocrighttype => 'varbit', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi bit varying +{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit', + amprocrighttype => 'varbit', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, + # minmax numeric { amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric', amprocrighttype => 'numeric', amprocnum => '1', diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat index da25befefe..823f1b01fe 100644 --- a/src/include/catalog/pg_opclass.dat +++ b/src/include/catalog/pg_opclass.dat @@ -266,18 +266,27 @@ { opcmethod => 'brin', opcname => 'bytea_minmax_ops', opcfamily => 'brin/bytea_minmax_ops', opcintype => 'bytea', opckeytype => 'bytea' }, +{ opcmethod => 'brin', opcname => 'bytea_minmax_multi_ops', + opcfamily => 'brin/bytea_minmax_multi_ops', opcintype => 'bytea', + opckeytype => 'bytea', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'bytea_bloom_ops', opcfamily => 'brin/bytea_bloom_ops', opcintype => 'bytea', opckeytype => 'bytea', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'char_minmax_ops', opcfamily => 'brin/char_minmax_ops', opcintype => 'char', opckeytype => 'char' }, +{ opcmethod => 'brin', opcname => 'char_minmax_multi_ops', + opcfamily => 'brin/char_minmax_multi_ops', opcintype => 'char', + opckeytype => 'char', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'char_bloom_ops', opcfamily => 'brin/char_bloom_ops', opcintype => 'char', opckeytype => 'char', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'name_minmax_ops', opcfamily => 'brin/name_minmax_ops', opcintype => 'name', opckeytype => 'name' }, +{ opcmethod => 'brin', opcname => 'name_minmax_multi_ops', + opcfamily => 'brin/name_minmax_multi_ops', opcintype => 'name', + opckeytype => 'name', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'name_bloom_ops', opcfamily => 'brin/name_bloom_ops', opcintype => 'name', opckeytype => 'name', opcdefault => 'f' }, @@ -311,6 +320,9 @@ { opcmethod => 'brin', opcname => 'text_minmax_ops', opcfamily => 'brin/text_minmax_ops', opcintype => 'text', opckeytype => 'text' }, +{ opcmethod => 'brin', opcname => 'text_minmax_multi_ops', + opcfamily => 'brin/text_minmax_multi_ops', opcintype => 'text', + opckeytype => 'text', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'text_bloom_ops', opcfamily => 'brin/text_bloom_ops', opcintype => 'text', opckeytype => 'text', opcdefault => 'f' }, @@ -381,6 +393,9 @@ { opcmethod => 'brin', opcname => 'bpchar_minmax_ops', opcfamily => 'brin/bpchar_minmax_ops', opcintype => 'bpchar', opckeytype => 'bpchar' }, +{ opcmethod => 'brin', opcname => 'bpchar_minmax_multi_ops', + opcfamily => 'brin/bpchar_minmax_multi_ops', opcintype => 'bpchar', + opckeytype => 'bpchar', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'bpchar_bloom_ops', opcfamily => 'brin/bpchar_bloom_ops', opcintype => 'bpchar', opckeytype => 'bpchar', opcdefault => 'f' }, @@ -440,9 +455,15 @@ opckeytype => 'timetz', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'bit_minmax_ops', opcfamily => 'brin/bit_minmax_ops', opcintype => 'bit', opckeytype => 'bit' }, +{ opcmethod => 'brin', opcname => 'bit_minmax_multi_ops', + opcfamily => 'brin/bit_minmax_multi_ops', opcintype => 'bit', + opckeytype => 'bit', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'varbit_minmax_ops', opcfamily => 'brin/varbit_minmax_ops', opcintype => 'varbit', opckeytype => 'varbit' }, +{ opcmethod => 'brin', opcname => 'varbit_minmax_multi_ops', + opcfamily => 'brin/varbit_minmax_multi_ops', opcintype => 'varbit', + opckeytype => 'varbit', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'numeric_minmax_ops', opcfamily => 'brin/numeric_minmax_ops', opcintype => 'numeric', opckeytype => 'numeric' }, diff --git a/src/include/catalog/pg_opfamily.dat b/src/include/catalog/pg_opfamily.dat index ba9231ac8c..ffb20e72ab 100644 --- a/src/include/catalog/pg_opfamily.dat +++ b/src/include/catalog/pg_opfamily.dat @@ -192,6 +192,8 @@ opfmethod => 'brin', opfname => 'numeric_minmax_multi_ops' }, { oid => '4056', opfmethod => 'brin', opfname => 'text_minmax_ops' }, +{ oid => '9967', + opfmethod => 'brin', opfname => 'text_minmax_multi_ops' }, { oid => '9902', opfmethod => 'brin', opfname => 'text_bloom_ops' }, { oid => '9903', @@ -210,14 +212,20 @@ opfmethod => 'brin', opfname => 'datetime_bloom_ops' }, { oid => '4062', opfmethod => 'brin', opfname => 'char_minmax_ops' }, +{ oid => '9968', + opfmethod => 'brin', opfname => 'char_minmax_multi_ops' }, { oid => '9906', opfmethod => 'brin', opfname => 'char_bloom_ops' }, { oid => '4064', opfmethod => 'brin', opfname => 'bytea_minmax_ops' }, +{ oid => '9969', + opfmethod => 'brin', opfname => 'bytea_minmax_multi_ops' }, { oid => '9907', opfmethod => 'brin', opfname => 'bytea_bloom_ops' }, { oid => '4065', opfmethod => 'brin', opfname => 'name_minmax_ops' }, +{ oid => '9970', + opfmethod => 'brin', opfname => 'name_minmax_multi_ops' }, { oid => '9908', opfmethod => 'brin', opfname => 'name_bloom_ops' }, { oid => '4068', @@ -260,6 +268,8 @@ opfmethod => 'brin', opfname => 'network_bloom_ops' }, { oid => '4076', opfmethod => 'brin', opfname => 'bpchar_minmax_ops' }, +{ oid => '9971', + opfmethod => 'brin', opfname => 'bpchar_minmax_multi_ops' }, { oid => '9915', opfmethod => 'brin', opfname => 'bpchar_bloom_ops' }, { oid => '4077', @@ -276,8 +286,12 @@ opfmethod => 'brin', opfname => 'interval_bloom_ops' }, { oid => '4079', opfmethod => 'brin', opfname => 'bit_minmax_ops' }, +{ oid => '9972', + opfmethod => 'brin', opfname => 'bit_minmax_multi_ops' }, { oid => '4080', opfmethod => 'brin', opfname => 'varbit_minmax_ops' }, +{ oid => '9973', + opfmethod => 'brin', opfname => 'varbit_minmax_multi_ops' }, { oid => '4081', opfmethod => 'brin', opfname => 'uuid_minmax_ops' }, { oid => '9938', -- 2.26.2 --------------22A4B241170149838D4D1F8F Content-Type: text/x-patch; charset=UTF-8; name="0009-Ignore-correlation-for-new-BRIN-opclasses-20210215.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0009-Ignore-correlation-for-new-BRIN-opclasses-20210215.patc"; filename*1="h" ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: MergeJoin beats HashJoin in the case of multiple hash clauses @ 2023-09-11 08:04 Lepikhov Andrei <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Lepikhov Andrei @ 2023-09-11 08:04 UTC (permalink / raw) To: Andy Fan <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]> On Mon, Sep 11, 2023, at 11:51 AM, Andy Fan wrote: > Hi, > > On Thu, Jun 15, 2023 at 4:30 PM Andrey Lepikhov > <[email protected]> wrote: >> Hi, all. >> >> Some of my clients use JOIN's with three - four clauses. Quite >> frequently, I see complaints on unreasonable switch of JOIN algorithm to >> Merge Join instead of Hash Join. Quick research have shown one weak >> place - estimation of an average bucket size in final_cost_hashjoin (see >> q2.sql in attachment) with very conservative strategy. >> Unlike estimation of groups, here we use smallest ndistinct value across >> all buckets instead of multiplying them (or trying to make multivariate >> analysis). >> It works fine for the case of one clause. But if we have many clauses, >> and if each has high value of ndistinct, we will overestimate average >> size of a bucket and, as a result, prefer to use Merge Join. As the >> example in attachment shows, it leads to worse plan than possible, >> sometimes drastically worse. >> I assume, this is done with fear of functional dependencies between hash >> clause components. But as for me, here we should go the same way, as >> estimation of groups. > > I can reproduce the visitation you want to improve and verify the patch > can do it expectedly. I think this is a right thing to do. > >> The attached patch shows a sketch of the solution. > > I understand that this is a sketch of the solution, but the below > changes still > make me confused. > > + if (innerbucketsize > virtualbuckets) > + innerbucketsize = 1.0 / virtualbuckets; > > innerbucketsize is a fraction of rows in all the rows, so it is between > 0.0 and 1.0. > and virtualbuckets is the number of buckets in total (when considered > the mutli > batchs), how is it possible for 'innerbucketsize > virtualbuckets' ? > Am > I missing something? You are right here. I've made a mistake here. Changed diff is in attachment. -- Regards, Andrei Lepikhov Attachments: [application/octet-stream] fix_bucketsize_v2.diff (680B, ../../[email protected]/2-fix_bucketsize_v2.diff) download | inline diff: diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index d6ceafd51c..5c5b24a843 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -4274,11 +4274,12 @@ final_cost_hashjoin(PlannerInfo *root, HashPath *path, thismcvfreq = restrictinfo->left_mcvfreq; } - if (innerbucketsize > thisbucketsize) - innerbucketsize = thisbucketsize; - if (innermcvfreq > thismcvfreq) - innermcvfreq = thismcvfreq; + innerbucketsize *= thisbucketsize; + innermcvfreq *= thismcvfreq; } + + if (innerbucketsize < 1.0 / virtualbuckets) + innerbucketsize = 1.0 / virtualbuckets; } /* ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: MergeJoin beats HashJoin in the case of multiple hash clauses @ 2023-11-03 16:43 Tomas Vondra <[email protected]> parent: Lepikhov Andrei <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Tomas Vondra @ 2023-11-03 16:43 UTC (permalink / raw) To: Lepikhov Andrei <[email protected]>; Andy Fan <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]> On 9/11/23 10:04, Lepikhov Andrei wrote: > > > On Mon, Sep 11, 2023, at 11:51 AM, Andy Fan wrote: >> Hi, >> >> On Thu, Jun 15, 2023 at 4:30 PM Andrey Lepikhov >> <[email protected]> wrote: >>> Hi, all. >>> >>> Some of my clients use JOIN's with three - four clauses. Quite >>> frequently, I see complaints on unreasonable switch of JOIN algorithm to >>> Merge Join instead of Hash Join. Quick research have shown one weak >>> place - estimation of an average bucket size in final_cost_hashjoin (see >>> q2.sql in attachment) with very conservative strategy. >>> Unlike estimation of groups, here we use smallest ndistinct value across >>> all buckets instead of multiplying them (or trying to make multivariate >>> analysis). >>> It works fine for the case of one clause. But if we have many clauses, >>> and if each has high value of ndistinct, we will overestimate average >>> size of a bucket and, as a result, prefer to use Merge Join. As the >>> example in attachment shows, it leads to worse plan than possible, >>> sometimes drastically worse. >>> I assume, this is done with fear of functional dependencies between hash >>> clause components. But as for me, here we should go the same way, as >>> estimation of groups. >> Yes, this analysis is correct - final_cost_hashjoin assumes the clauses may be correlated (not necessarily by functional dependencies, just that the overall ndistinct is not a simple product of per-column ndistincts). And it even says so in the comment before calculating bucket size: * Determine bucketsize fraction and MCV frequency for the inner * relation. We use the smallest bucketsize or MCV frequency estimated * for any individual hashclause; this is undoubtedly conservative. I'm sure this may lead to inflated cost for "good" cases (where the actual bucket size really is a product), which may push the optimizer to use the less efficient/slower join method. Unfortunately, AFAICS the patch simply assumes the extreme in the opposite direction - it assumes each clause splits the bucket for each distinct value in the column. Which works great when it's true, but surely it'd have issues when the columns are correlated? I think this deserves more discussion, i.e. what happens if the assumptions do not hold? We know what happens for the conservative approach, but what's the worst thing that would happen for the optimistic one? I doubt e can simply switch from the conservative approach to the optimistic one. Yes, it'll make some queries faster, but for other queries it likely causes problems and slowdowns. IMHO the only principled way forward is to get a better ndistinct estimate (which this implicitly does), perhaps by using extended statistics. I haven't tried, but I guess it'd need to extract the clauses for the inner side, and call estimate_num_groups() on it. This however reminds me we don't use extended statistics for join clauses at all. Which means that even with accurate extended statistics, we can still get stuff like this for multiple join clauses: Hash Join (cost=1317.00..2386.00 rows=200 width=24) (actual time=85.781..8574.784 rows=8000000 loops=1) This is unrelated to the issue discussed here, of course, as it won't affect join method selection for that join. But it certainly will affect all estimates/costs above that join, which can be pretty disastrous. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2023-11-03 16:43 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-02-03 18:12 [PATCH 8/9] Define multi-minmax oclasses for types without distance Tomas Vondra <[email protected]> 2023-09-11 08:04 Re: MergeJoin beats HashJoin in the case of multiple hash clauses Lepikhov Andrei <[email protected]> 2023-11-03 16:43 ` Re: MergeJoin beats HashJoin in the case of multiple hash clauses Tomas Vondra <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox