public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 7/8] BRIN minmax-multi indexes 12+ messages / 7 participants [nested] [flat]
* [PATCH 7/8] BRIN minmax-multi indexes @ 2020-12-16 20:32 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Tomas Vondra @ 2020-12-16 20:32 UTC (permalink / raw) Adds a BRIN minmax-multi opclass, summarizing each range into a list of intervals, allowing more efficient handling of cases where the minmax opclasses quickly degrade. Instead of representing each range with a single interval, minmax-multi opclasses store a list of intervals and points. This allows effective handling of outliers and poorly correlated values. The number of intervals/points per range may be configured using an opclass parameter values_per_range. When building the summary, the interval are merged based on distance, determined by the new support BRIN procedure. Author: Tomas Vondra <[email protected]> Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Alexander Korotkov <[email protected]> Reviewed-by: Sokolov Yura <[email protected]> Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/5d78b774-7e9c-c94e-12cf-fef51cc89b1a%402ndquadrant.com --- doc/src/sgml/brin.sgml | 252 +- doc/src/sgml/ref/create_index.sgml | 11 + src/backend/access/brin/Makefile | 1 + src/backend/access/brin/brin_minmax_multi.c | 2389 +++++++++++++++++++ src/backend/access/brin/brin_tuple.c | 17 + src/include/access/brin.h | 1 + src/include/access/brin_internal.h | 3 + src/include/access/brin_tuple.h | 4 + src/include/access/transam.h | 2 +- src/include/catalog/pg_amop.dat | 544 +++++ src/include/catalog/pg_amproc.dat | 600 ++++- src/include/catalog/pg_opclass.dat | 57 + src/include/catalog/pg_opfamily.dat | 28 + src/include/catalog/pg_proc.dat | 80 + src/include/catalog/pg_type.dat | 7 + src/test/regress/expected/brin_multi.out | 421 ++++ src/test/regress/expected/psql.out | 13 +- src/test/regress/expected/type_sanity.out | 7 +- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/brin_multi.sql | 371 +++ 21 files changed, 4792 insertions(+), 19 deletions(-) create mode 100644 src/backend/access/brin/brin_minmax_multi.c create mode 100644 src/test/regress/expected/brin_multi.out create mode 100644 src/test/regress/sql/brin_multi.sql diff --git a/doc/src/sgml/brin.sgml b/doc/src/sgml/brin.sgml index 577dd18c49..3caa30f104 100644 --- a/doc/src/sgml/brin.sgml +++ b/doc/src/sgml/brin.sgml @@ -214,6 +214,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (date,date)</literal></entry></row> <row><entry><literal>>= (date,date)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>date_minmax_multi_ops</literal></entry> + <entry><literal>= (date,date)</literal></entry> + </row> + <row><entry><literal>< (date,date)</literal></entry></row> + <row><entry><literal><= (date,date)</literal></entry></row> + <row><entry><literal>> (date,date)</literal></entry></row> + <row><entry><literal>>= (date,date)</literal></entry></row> + <row> <entry valign="middle"><literal>float4_bloom_ops</literal></entry> <entry><literal>= (float4,float4)</literal></entry> @@ -228,6 +237,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (float4,float4)</literal></entry></row> <row><entry><literal>>= (float4,float4)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>float4_minmax_multi_ops</literal></entry> + <entry><literal>= (float4,float4)</literal></entry> + </row> + <row><entry><literal>< (float4,float4)</literal></entry></row> + <row><entry><literal>> (float4,float4)</literal></entry></row> + <row><entry><literal><= (float4,float4)</literal></entry></row> + <row><entry><literal>>= (float4,float4)</literal></entry></row> + <row> <entry valign="middle"><literal>float8_bloom_ops</literal></entry> <entry><literal>= (float8,float8)</literal></entry> @@ -242,6 +260,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (float8,float8)</literal></entry></row> <row><entry><literal>>= (float8,float8)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>float8_minmax_multi_ops</literal></entry> + <entry><literal>= (float8,float8)</literal></entry> + </row> + <row><entry><literal>< (float8,float8)</literal></entry></row> + <row><entry><literal><= (float8,float8)</literal></entry></row> + <row><entry><literal>> (float8,float8)</literal></entry></row> + <row><entry><literal>>= (float8,float8)</literal></entry></row> + <row> <entry valign="middle" morerows="5"><literal>inet_inclusion_ops</literal></entry> <entry><literal><< (inet,inet)</literal></entry> @@ -266,6 +293,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (inet,inet)</literal></entry></row> <row><entry><literal>>= (inet,inet)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>inet_minmax_multi_ops</literal></entry> + <entry><literal>= (inet,inet)</literal></entry> + </row> + <row><entry><literal>< (inet,inet)</literal></entry></row> + <row><entry><literal><= (inet,inet)</literal></entry></row> + <row><entry><literal>> (inet,inet)</literal></entry></row> + <row><entry><literal>>= (inet,inet)</literal></entry></row> + <row> <entry valign="middle"><literal>int2_bloom_ops</literal></entry> <entry><literal>= (int2,int2)</literal></entry> @@ -280,6 +316,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (int2,int2)</literal></entry></row> <row><entry><literal>>= (int2,int2)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>int2_minmax_multi_ops</literal></entry> + <entry><literal>= (int2,int2)</literal></entry> + </row> + <row><entry><literal>< (int2,int2)</literal></entry></row> + <row><entry><literal>> (int2,int2)</literal></entry></row> + <row><entry><literal><= (int2,int2)</literal></entry></row> + <row><entry><literal>>= (int2,int2)</literal></entry></row> + <row> <entry valign="middle"><literal>int4_bloom_ops</literal></entry> <entry><literal>= (int4,int4)</literal></entry> @@ -294,6 +339,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (int4,int4)</literal></entry></row> <row><entry><literal>>= (int4,int4)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>int4_minmax_multi_ops</literal></entry> + <entry><literal>= (int4,int4)</literal></entry> + </row> + <row><entry><literal>< (int4,int4)</literal></entry></row> + <row><entry><literal>> (int4,int4)</literal></entry></row> + <row><entry><literal><= (int4,int4)</literal></entry></row> + <row><entry><literal>>= (int4,int4)</literal></entry></row> + <row> <entry valign="middle"><literal>int8_bloom_ops</literal></entry> <entry><literal>= (bigint,bigint)</literal></entry> @@ -308,6 +362,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (bigint,bigint)</literal></entry></row> <row><entry><literal>>= (bigint,bigint)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>int8_minmax_multi_ops</literal></entry> + <entry><literal>= (bigint,bigint)</literal></entry> + </row> + <row><entry><literal>< (bigint,bigint)</literal></entry></row> + <row><entry><literal>> (bigint,bigint)</literal></entry></row> + <row><entry><literal><= (bigint,bigint)</literal></entry></row> + <row><entry><literal>>= (bigint,bigint)</literal></entry></row> + <row> <entry valign="middle"><literal>interval_bloom_ops</literal></entry> <entry><literal>= (interval,interval)</literal></entry> @@ -322,6 +385,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (interval,interval)</literal></entry></row> <row><entry><literal>>= (interval,interval)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>interval_minmax_multi_ops</literal></entry> + <entry><literal>= (interval,interval)</literal></entry> + </row> + <row><entry><literal>< (interval,interval)</literal></entry></row> + <row><entry><literal><= (interval,interval)</literal></entry></row> + <row><entry><literal>> (interval,interval)</literal></entry></row> + <row><entry><literal>>= (interval,interval)</literal></entry></row> + <row> <entry valign="middle"><literal>macaddr_bloom_ops</literal></entry> <entry><literal>= (macaddr,macaddr)</literal></entry> @@ -336,6 +408,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (macaddr,macaddr)</literal></entry></row> <row><entry><literal>>= (macaddr,macaddr)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>macaddr_minmax_multi_ops</literal></entry> + <entry><literal>= (macaddr,macaddr)</literal></entry> + </row> + <row><entry><literal>< (macaddr,macaddr)</literal></entry></row> + <row><entry><literal><= (macaddr,macaddr)</literal></entry></row> + <row><entry><literal>> (macaddr,macaddr)</literal></entry></row> + <row><entry><literal>>= (macaddr,macaddr)</literal></entry></row> + <row> <entry valign="middle"><literal>macaddr8_bloom_ops</literal></entry> <entry><literal>= (macaddr8,macaddr8)</literal></entry> @@ -350,6 +431,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (macaddr8,macaddr8)</literal></entry></row> <row><entry><literal>>= (macaddr8,macaddr8)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>macaddr8_minmax_multi_ops</literal></entry> + <entry><literal>= (macaddr8,macaddr8)</literal></entry> + </row> + <row><entry><literal>< (macaddr8,macaddr8)</literal></entry></row> + <row><entry><literal><= (macaddr8,macaddr8)</literal></entry></row> + <row><entry><literal>> (macaddr8,macaddr8)</literal></entry></row> + <row><entry><literal>>= (macaddr8,macaddr8)</literal></entry></row> + <row> <entry valign="middle"><literal>name_bloom_ops</literal></entry> <entry><literal>= (name,name)</literal></entry> @@ -378,6 +468,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (numeric,numeric)</literal></entry></row> <row><entry><literal>>= (numeric,numeric)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>numeric_minmax_multi_ops</literal></entry> + <entry><literal>= (numeric,numeric)</literal></entry> + </row> + <row><entry><literal>< (numeric,numeric)</literal></entry></row> + <row><entry><literal><= (numeric,numeric)</literal></entry></row> + <row><entry><literal>> (numeric,numeric)</literal></entry></row> + <row><entry><literal>>= (numeric,numeric)</literal></entry></row> + <row> <entry valign="middle"><literal>oid_bloom_ops</literal></entry> <entry><literal>= (oid,oid)</literal></entry> @@ -392,6 +491,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (oid,oid)</literal></entry></row> <row><entry><literal>>= (oid,oid)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>oid_minmax_multi_ops</literal></entry> + <entry><literal>= (oid,oid)</literal></entry> + </row> + <row><entry><literal>< (oid,oid)</literal></entry></row> + <row><entry><literal>> (oid,oid)</literal></entry></row> + <row><entry><literal><= (oid,oid)</literal></entry></row> + <row><entry><literal>>= (oid,oid)</literal></entry></row> + <row> <entry valign="middle"><literal>pg_lsn_bloom_ops</literal></entry> <entry><literal>= (pg_lsn,pg_lsn)</literal></entry> @@ -406,6 +514,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (pg_lsn,pg_lsn)</literal></entry></row> <row><entry><literal>>= (pg_lsn,pg_lsn)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>pg_lsn_minmax_multi_ops</literal></entry> + <entry><literal>= (pg_lsn,pg_lsn)</literal></entry> + </row> + <row><entry><literal>< (pg_lsn,pg_lsn)</literal></entry></row> + <row><entry><literal>> (pg_lsn,pg_lsn)</literal></entry></row> + <row><entry><literal><= (pg_lsn,pg_lsn)</literal></entry></row> + <row><entry><literal>>= (pg_lsn,pg_lsn)</literal></entry></row> + <row> <entry valign="middle" morerows="13"><literal>range_inclusion_ops</literal></entry> <entry><literal>= (anyrange,anyrange)</literal></entry> @@ -452,6 +569,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (tid,tid)</literal></entry></row> <row><entry><literal>>= (tid,tid)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>tid_minmax_multi_ops</literal></entry> + <entry><literal>= (tid,tid)</literal></entry> + </row> + <row><entry><literal>< (tid,tid)</literal></entry></row> + <row><entry><literal>> (tid,tid)</literal></entry></row> + <row><entry><literal><= (tid,tid)</literal></entry></row> + <row><entry><literal>>= (tid,tid)</literal></entry></row> + <row> <entry valign="middle"><literal>timestamp_bloom_ops</literal></entry> <entry><literal>= (timestamp,timestamp)</literal></entry> @@ -466,6 +592,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (timestamp,timestamp)</literal></entry></row> <row><entry><literal>>= (timestamp,timestamp)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>timestamp_minmax_multi_ops</literal></entry> + <entry><literal>= (timestamp,timestamp)</literal></entry> + </row> + <row><entry><literal>< (timestamp,timestamp)</literal></entry></row> + <row><entry><literal><= (timestamp,timestamp)</literal></entry></row> + <row><entry><literal>> (timestamp,timestamp)</literal></entry></row> + <row><entry><literal>>= (timestamp,timestamp)</literal></entry></row> + <row> <entry valign="middle"><literal>timestamptz_bloom_ops</literal></entry> <entry><literal>= (timestamptz,timestamptz)</literal></entry> @@ -480,6 +615,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (timestamptz,timestamptz)</literal></entry></row> <row><entry><literal>>= (timestamptz,timestamptz)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>timestamptz_minmax_multi_ops</literal></entry> + <entry><literal>= (timestamptz,timestamptz)</literal></entry> + </row> + <row><entry><literal>< (timestamptz,timestamptz)</literal></entry></row> + <row><entry><literal><= (timestamptz,timestamptz)</literal></entry></row> + <row><entry><literal>> (timestamptz,timestamptz)</literal></entry></row> + <row><entry><literal>>= (timestamptz,timestamptz)</literal></entry></row> + <row> <entry valign="middle"><literal>time_bloom_ops</literal></entry> <entry><literal>= (time,time)</literal></entry> @@ -494,6 +638,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (time,time)</literal></entry></row> <row><entry><literal>>= (time,time)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>time_minmax_multi_ops</literal></entry> + <entry><literal>= (time,time)</literal></entry> + </row> + <row><entry><literal>< (time,time)</literal></entry></row> + <row><entry><literal><= (time,time)</literal></entry></row> + <row><entry><literal>> (time,time)</literal></entry></row> + <row><entry><literal>>= (time,time)</literal></entry></row> + <row> <entry valign="middle"><literal>timetz_bloom_ops</literal></entry> <entry><literal>= (timetz,timetz)</literal></entry> @@ -508,6 +661,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal>> (timetz,timetz)</literal></entry></row> <row><entry><literal>>= (timetz,timetz)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>timetz_minmax_multi_ops</literal></entry> + <entry><literal>= (timetz,timetz)</literal></entry> + </row> + <row><entry><literal>< (timetz,timetz)</literal></entry></row> + <row><entry><literal><= (timetz,timetz)</literal></entry></row> + <row><entry><literal>> (timetz,timetz)</literal></entry></row> + <row><entry><literal>>= (timetz,timetz)</literal></entry></row> + <row> <entry valign="middle"><literal>uuid_bloom_ops</literal></entry> <entry><literal>= (uuid,uuid)</literal></entry> @@ -522,6 +684,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was <row><entry><literal><= (uuid,uuid)</literal></entry></row> <row><entry><literal>>= (uuid,uuid)</literal></entry></row> + <row> + <entry valign="middle" morerows="4"><literal>uuid_minmax_multi_ops</literal></entry> + <entry><literal>= (uuid,uuid)</literal></entry> + </row> + <row><entry><literal>< (uuid,uuid)</literal></entry></row> + <row><entry><literal>> (uuid,uuid)</literal></entry></row> + <row><entry><literal><= (uuid,uuid)</literal></entry></row> + <row><entry><literal>>= (uuid,uuid)</literal></entry></row> + <row> <entry valign="middle" morerows="4"><literal>varbit_minmax_ops</literal></entry> <entry><literal>= (varbit,varbit)</literal></entry> @@ -654,13 +825,14 @@ typedef struct BrinOpcInfo </varlistentry> </variablelist> - The core distribution includes support for two types of operator classes: - minmax and inclusion. Operator class definitions using them are shipped for - in-core data types as appropriate. Additional operator classes can be - defined by the user for other data types using equivalent definitions, - without having to write any source code; appropriate catalog entries being - declared is enough. Note that assumptions about the semantics of operator - strategies are embedded in the support functions' source code. + The core distribution includes support for four types of operator classes: + minmax, multi-minmax, inclusion and bloom. Operator class definitions + using them are shipped for in-core data types as appropriate. Additional + operator classes can be defined by the user for other data types using + equivalent definitions, without having to write any source code; + appropriate catalog entries being declared is enough. Note that + assumptions about the semantics of operator strategies are embedded in the + support functions' source code. </para> <para> @@ -957,6 +1129,72 @@ typedef struct BrinOpcInfo and return a hash of the value. </para> + <para> + The multi-minmax operator class is also intended for data types implementing + a totally ordered sets, and may be seen as a simple extension of the minmax + operator class. While minmax operator class summarizes values from each block + range into a single contiguous interval, multi-minmax allows summarization + into multiple smaller intervals to improve handling of outlier values. + It is possible to use the multi-minmax support procedures alongside the + corresponding operators, as shown in + <xref linkend="brin-extensibility-multi-minmax-table"/>. + All operator class members (procedures and operators) are mandatory. + </para> + + <table id="brin-extensibility-multi-minmax-table"> + <title>Procedure and Support Numbers for Multi-Minmax Operator Classes</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Operator class member</entry> + <entry>Object</entry> + </row> + </thead> + <tbody> + <row> + <entry>Support Procedure 1</entry> + <entry>internal function <function>brin_minmax_multi_opcinfo()</function></entry> + </row> + <row> + <entry>Support Procedure 2</entry> + <entry>internal function <function>brin_minmax_multi_add_value()</function></entry> + </row> + <row> + <entry>Support Procedure 3</entry> + <entry>internal function <function>brin_minmax_multi_consistent()</function></entry> + </row> + <row> + <entry>Support Procedure 4</entry> + <entry>internal function <function>brin_minmax_multi_union()</function></entry> + </row> + <row> + <entry>Support Procedure 11</entry> + <entry>function to compute distance between two values (length of a range)</entry> + </row> + <row> + <entry>Operator Strategy 1</entry> + <entry>operator less-than</entry> + </row> + <row> + <entry>Operator Strategy 2</entry> + <entry>operator less-than-or-equal-to</entry> + </row> + <row> + <entry>Operator Strategy 3</entry> + <entry>operator equal-to</entry> + </row> + <row> + <entry>Operator Strategy 4</entry> + <entry>operator greater-than-or-equal-to</entry> + </row> + <row> + <entry>Operator Strategy 5</entry> + <entry>operator greater-than</entry> + </row> + </tbody> + </tgroup> + </table> + <para> Both minmax and inclusion operator classes support cross-data-type operators, though with these the dependencies become more complicated. diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgml index 244e5834d8..ce61a28a72 100644 --- a/doc/src/sgml/ref/create_index.sgml +++ b/doc/src/sgml/ref/create_index.sgml @@ -590,6 +590,17 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ [ IF NOT EXISTS ] <replaceable class= </listitem> </varlistentry> + <varlistentry> + <term><literal>values_per_range</literal></term> + <listitem> + <para> + Defines the maximum number of values stored by <acronym>BRIN</acronym> + minmax indexes to summarize a block range. Each value may represent + eiter a point, or boundary of an interval. Values must be be between + 16 and 256, and the default value is 32. + </para> + </listitem> + </varlistentry> </variablelist> </refsect2> diff --git a/src/backend/access/brin/Makefile b/src/backend/access/brin/Makefile index 6b56131215..a386cb71f1 100644 --- a/src/backend/access/brin/Makefile +++ b/src/backend/access/brin/Makefile @@ -17,6 +17,7 @@ OBJS = \ brin_bloom.o \ brin_inclusion.o \ brin_minmax.o \ + brin_minmax_multi.o \ brin_pageops.o \ brin_revmap.o \ brin_tuple.o \ diff --git a/src/backend/access/brin/brin_minmax_multi.c b/src/backend/access/brin/brin_minmax_multi.c new file mode 100644 index 0000000000..f9a7ae6608 --- /dev/null +++ b/src/backend/access/brin/brin_minmax_multi.c @@ -0,0 +1,2389 @@ +/* + * brin_minmax_multi.c + * Implementation of Multi Min/Max opclass for BRIN + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * Implements a variant of minmax opclass, where the summary is composed of + * multiple smaller intervals. This allows us to handle outliers, which + * usually makes the simple minmax opclass inefficient. + * + * Consider for example page range with simple minmax interval [1000,2000], + * and assume a new row gets inserted into the range with value 1000000. + * Due to that the interval gets [1000,1000000]. I.e. the minmax interval + * got 1000x wider and won't be useful to eliminate scan keys between 2001 + * and 1000000. + * + * With multi-minmax opclass, we may have [1000,2000] interval initially, + * but after adding the new row we start tracking it as two interval: + * + * [1000,2000] and [1000000,1000000] + * + * This allow us to still eliminate the page range when the scan keys hit + * the gap between 2000 and 1000000, making it useful in cases when the + * simple minmax opclass gets inefficient. + * + * The number of intervals tracked per page range is somewhat flexible. + * What is restricted is the number of values per page range, and the limit + * is currently 64 (see values_per_range reloption). Collapsed intervals + * (with equal minimum and maximum value) are stored as a single value, + * while regular intervals require two values. + * + * When the number of values gets too high (by adding new values to the + * summary), we merge some of the intervals to free space for more values. + * This is done in a greedy way - we simply pick the two closest intervals, + * merge them, and repeat this until the number of values to store gets + * sufficiently low (below 75% of maximum values), but that is mostly + * arbitrary threshold and may be changed easily). + * + * To pick the closest intervals we use the "distance" support procedure, + * which measures space between two ranges (i.e. length of an interval). + * The computed value may be an approximation - in the worst case we will + * merge two ranges that are slightly less optimal at that step, but the + * index should still produce correct results. + * + * + * IDENTIFICATION + * src/backend/access/brin/brin_minmax_multi.c + */ +#include "postgres.h" + +#include "access/genam.h" +#include "access/brin.h" +#include "access/brin_internal.h" +#include "access/brin_tuple.h" +#include "access/reloptions.h" +#include "access/stratnum.h" +#include "access/htup_details.h" +#include "catalog/pg_type.h" +#include "catalog/pg_amop.h" +#include "utils/array.h" +#include "utils/builtins.h" +#include "utils/date.h" +#include "utils/datum.h" +#include "utils/inet.h" +#include "utils/lsyscache.h" +#include "utils/memutils.h" +#include "utils/numeric.h" +#include "utils/pg_lsn.h" +#include "utils/rel.h" +#include "utils/syscache.h" +#include "utils/uuid.h" + +/* + * Additional SQL level support functions + * + * Procedure numbers must not use values reserved for BRIN itself; see + * brin_internal.h. + */ +#define MINMAX_MAX_PROCNUMS 1 /* maximum support procs we need */ +#define PROCNUM_DISTANCE 11 /* required, distance between values*/ + +/* + * Subtract this from procnum to obtain index in MinmaxMultiOpaque arrays + * (Must be equal to minimum of private procnums). + */ +#define PROCNUM_BASE 11 + + +typedef struct MinmaxMultiOpaque +{ + FmgrInfo extra_procinfos[MINMAX_MAX_PROCNUMS]; + bool extra_proc_missing[MINMAX_MAX_PROCNUMS]; + Oid cached_subtype; + FmgrInfo strategy_procinfos[BTMaxStrategyNumber]; +} MinmaxMultiOpaque; + +/* + * Storage type for BRIN's minmax reloptions + */ +typedef struct MinMaxOptions +{ + int32 vl_len_; /* varlena header (do not touch directly!) */ + int valuesPerRange; /* number of values per range */ +} MinMaxOptions; + +#define MINMAX_DEFAULT_VALUES_PER_PAGE 32 + +#define MinMaxGetValuesPerRange(opts) \ + ((opts) && (((MinMaxOptions *) (opts))->valuesPerRange != 0) ? \ + ((MinMaxOptions *) (opts))->valuesPerRange : \ + MINMAX_DEFAULT_VALUES_PER_PAGE) + + + +/* + * The summary of multi-minmax indexes has two representations - Ranges for + * convenient processing, and SerializedRanges for storage in bytea value. + * + * The Ranges struct stores the boundary values in a single array, but we + * treat regular and single-point ranges differently to save space. For + * regular ranges (with different boundary values) we have to store both + * values, while for "single-point ranges" we only need to save one value. + * + * The 'values' array stores boundary values for regular ranges first (there + * are 2*nranges values to store), and then the nvalues boundary values for + * single-point ranges. That is, we have (2*nranges + nvalues) boundary + * values in the array. + * + * +---------------------------------+-------------------------------+ + * | ranges (sorted pairs of values) | sorted values (single points) | + * +---------------------------------+-------------------------------+ + * + * This allows us to quickly add new values, and store outliers without + * making the other ranges very wide. + * + * We never store more than maxvalues values (as set by values_per_range + * reloption). If needed we merge some of the ranges. + * + * To minimize palloc overhead, we always allocate the full array with + * space for maxvalues elements. This should be fine as long as the + * maxvalues is reasonably small (64 seems fine), which is the case + * thanks to values_per_range reloption being limited to 256. + */ +typedef struct Ranges +{ + Oid typid; + + /* (2*nranges + nvalues) <= maxvalues */ + int nranges; /* number of ranges in the array (stored) */ + int nvalues; /* number of values in the data array (all) */ + int maxvalues; /* maximum number of values (reloption) */ + + /* values stored for this range - either raw values, or ranges */ + Datum values[FLEXIBLE_ARRAY_MEMBER]; +} Ranges; + +/* + * On-disk the summary is stored as a bytea value, represented by the + * SerializedRanges structure. It has a 4B varlena header, so can treated + * as varlena directly. + * + * See range_serialize/range_deserialize methods for serialization details. + */ +typedef struct SerializedRanges +{ + /* varlena header (do not touch directly!) */ + int32 vl_len_; + + /* type of values stored in the data array */ + Oid typid; + + /* (2*nranges + nvalues) <= maxvalues */ + int nranges; /* number of ranges in the array (stored) */ + int nvalues; /* number of values in the data array (all) */ + int maxvalues; /* maximum number of values (reloption) */ + + /* contains the actual data */ + char data[FLEXIBLE_ARRAY_MEMBER]; +} SerializedRanges; + +static SerializedRanges *range_serialize(Ranges *range); + +static Ranges *range_deserialize(SerializedRanges *range); + +/* Cache for support and strategy procesures. */ + +static FmgrInfo *minmax_multi_get_procinfo(BrinDesc *bdesc, uint16 attno, + uint16 procnum); + +static FmgrInfo *minmax_multi_get_strategy_procinfo(BrinDesc *bdesc, + uint16 attno, Oid subtype, uint16 strategynum); + + +/* + * minmax_multi_init + * Initialize the deserialized range list, allocate all the memory. + * + * This is only in-memory representation of the ranges, so we allocate + * everything enough space for the maximum number of values (so as not + * to have to do repallocs as the ranges grow). + */ +static Ranges * +minmax_multi_init(int maxvalues) +{ + Size len; + Ranges * ranges; + + Assert(maxvalues > 0); + + len = offsetof(Ranges, values); /* fixed header */ + len += maxvalues * sizeof(Datum); /* Datum values */ + + ranges = (Ranges *) palloc0(len); + + ranges->maxvalues = maxvalues; + + return ranges; +} + +/* + * range_serialize + * Serialize the in-memory representation into a compact varlena value. + * + * Simply copy the header and then also the individual values, as stored + * in the in-memory value array. + */ +static SerializedRanges * +range_serialize(Ranges *range) +{ + Size len; + int nvalues; + SerializedRanges *serialized; + Oid typid; + int typlen; + bool typbyval; + + int i; + char *ptr; + + /* simple sanity checks */ + Assert(range->nranges >= 0); + Assert(range->nvalues >= 0); + Assert(range->maxvalues > 0); + + /* see how many Datum values we actually have */ + nvalues = 2*range->nranges + range->nvalues; + + Assert(2*range->nranges + range->nvalues <= range->maxvalues); + + typid = range->typid; + typbyval = get_typbyval(typid); + typlen = get_typlen(typid); + + /* header is always needed */ + len = offsetof(SerializedRanges,data); + + /* + * The space needed depends on data type - for fixed-length data types + * (by-value and some by-reference) it's pretty simple, just multiply + * (attlen * nvalues) and we're done. For variable-length by-reference + * types we need to actually walk all the values and sum the lengths. + */ + if (typlen == -1) /* varlena */ + { + int i; + for (i = 0; i < nvalues; i++) + { + len += VARSIZE_ANY(range->values[i]); + } + } + else if (typlen == -2) /* cstring */ + { + int i; + for (i = 0; i < nvalues; i++) + { + /* don't forget to include the null terminator ;-) */ + len += strlen(DatumGetPointer(range->values[i])) + 1; + } + } + else /* fixed-length types (even by-reference) */ + { + Assert(typlen > 0); + len += nvalues * typlen; + } + + /* + * Allocate the serialized object, copy the basic information. The + * serialized object is a varlena, so update the header. + */ + serialized = (SerializedRanges *) palloc0(len); + SET_VARSIZE(serialized, len); + + serialized->typid = typid; + serialized->nranges = range->nranges; + serialized->nvalues = range->nvalues; + serialized->maxvalues = range->maxvalues; + + /* + * And now copy also the boundary values (like the length calculation + * this depends on the particular data type). + */ + ptr = serialized->data; /* start of the serialized data */ + + for (i = 0; i < nvalues; i++) + { + if (typbyval) /* simple by-value data types */ + { + memcpy(ptr, &range->values[i], typlen); + ptr += typlen; + } + else if (typlen > 0) /* fixed-length by-ref types */ + { + memcpy(ptr, DatumGetPointer(range->values[i]), typlen); + ptr += typlen; + } + else if (typlen == -1) /* varlena */ + { + int tmp = VARSIZE_ANY(DatumGetPointer(range->values[i])); + memcpy(ptr, DatumGetPointer(range->values[i]), tmp); + ptr += tmp; + } + else if (typlen == -2) /* cstring */ + { + int tmp = strlen(DatumGetPointer(range->values[i])) + 1; + memcpy(ptr, DatumGetPointer(range->values[i]), tmp); + ptr += tmp; + } + + /* make sure we haven't overflown the buffer end */ + Assert(ptr <= ((char *)serialized + len)); + } + + /* exact size */ + Assert(ptr == ((char *)serialized + len)); + + return serialized; +} + +/* + * range_deserialize + * Serialize the in-memory representation into a compact varlena value. + * + * Simply copy the header and then also the individual values, as stored + * in the in-memory value array. + */ +static Ranges * +range_deserialize(SerializedRanges *serialized) +{ + int i, + nvalues; + char *ptr; + bool typbyval; + int typlen; + + Ranges *range; + + Assert(serialized->nranges >= 0); + Assert(serialized->nvalues >= 0); + Assert(serialized->maxvalues > 0); + + nvalues = 2*serialized->nranges + serialized->nvalues; + + Assert(nvalues <= serialized->maxvalues); + + range = minmax_multi_init(serialized->maxvalues); + + /* copy the header info */ + range->nranges = serialized->nranges; + range->nvalues = serialized->nvalues; + range->maxvalues = serialized->maxvalues; + range->typid = serialized->typid; + + typbyval = get_typbyval(serialized->typid); + typlen = get_typlen(serialized->typid); + + /* + * And now deconstruct the values into Datum array. We don't need + * to copy the values and will instead just point the values to the + * serialized varlena value (assuming it will be kept around). + */ + ptr = serialized->data; + + for (i = 0; i < nvalues; i++) + { + if (typbyval) /* simple by-value data types */ + { + memcpy(&range->values[i], ptr, typlen); + ptr += typlen; + } + else if (typlen > 0) /* fixed-length by-ref types */ + { + /* no copy, just set the value to the pointer */ + range->values[i] = PointerGetDatum(ptr); + ptr += typlen; + } + else if (typlen == -1) /* varlena */ + { + range->values[i] = PointerGetDatum(ptr); + ptr += VARSIZE_ANY(DatumGetPointer(range->values[i])); + } + else if (typlen == -2) /* cstring */ + { + range->values[i] = PointerGetDatum(ptr); + ptr += strlen(DatumGetPointer(range->values[i])) + 1; + } + + /* make sure we haven't overflown the buffer end */ + Assert(ptr <= ((char *)serialized + VARSIZE_ANY(serialized))); + } + + /* should have consumed the whole input value exactly */ + Assert(ptr == ((char *)serialized + VARSIZE_ANY(serialized))); + + /* return the deserialized value */ + return range; +} + +typedef struct compare_context +{ + FmgrInfo *cmpFn; + Oid colloid; +} compare_context; + +/* + * Used to represent ranges expanded during merging and combining (to + * reduce number of boundary values to store). + * + * XXX CombineRange name seems a bit weird. Consider renaming, perhaps to + * something ExpandedRange or so. + */ +typedef struct CombineRange +{ + Datum minval; /* lower boundary */ + Datum maxval; /* upper boundary */ + bool collapsed; /* true if minval==maxval */ +} CombineRange; + +/* + * compare_combine_ranges + * Compare the combine ranges - first by minimum, then by maximum. + * + * We do guarantee that ranges in a single Range object do not overlap, + * so it may seem strange that we don't order just by minimum. But when + * merging two Ranges (which happens in the union function), the ranges + * may in fact overlap. So we do compare both. + */ +static int +compare_combine_ranges(const void *a, const void *b, void *arg) +{ + CombineRange *ra = (CombineRange *)a; + CombineRange *rb = (CombineRange *)b; + Datum r; + + compare_context *cxt = (compare_context *)arg; + + /* first compare minvals */ + r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, ra->minval, rb->minval); + + if (DatumGetBool(r)) + return -1; + + r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, rb->minval, ra->minval); + + if (DatumGetBool(r)) + return 1; + + /* then compare maxvals */ + r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, ra->maxval, rb->maxval); + + if (DatumGetBool(r)) + return -1; + + r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, rb->maxval, ra->maxval); + + if (DatumGetBool(r)) + return 1; + + return 0; +} + +/* + * range_contains_value + * See if the new value is already contained in the range list. + * + * We first inspect the list of intervals. We use a small trick - we check + * the value against min/max of the whole range (min of the first interval, + * max of the last one) first, and only inspect the individual intervals if + * this passes. + * + * If the value matches none of the intervals, we check the exact values. + * We simply loop through them and invoke equality operator on them. + * + * XXX This might benefit from the fact that both the intervals and exact + * values are sorted - we might do bsearch or something. Currently that + * does not make much difference (there are only ~32 intervals), but if + * this gets increased and/or the comparator function is more expensive, + * it might be a huge win. + */ +static bool +range_contains_value(BrinDesc *bdesc, Oid colloid, + AttrNumber attno, Form_pg_attribute attr, + Ranges *ranges, Datum newval) +{ + int i; + FmgrInfo *cmpLessFn; + FmgrInfo *cmpGreaterFn; + FmgrInfo *cmpEqualFn; + Oid typid = attr->atttypid; + + /* + * First inspect the ranges, if there are any. We first check the whole + * range, and only when there's still a chance of getting a match we + * inspect the individual ranges. + */ + if (ranges->nranges > 0) + { + Datum compar; + bool match = true; + + Datum minvalue = ranges->values[0]; + Datum maxvalue = ranges->values[2*ranges->nranges - 1]; + + /* + * Otherwise, need to compare the new value with boundaries of all + * the ranges. First check if it's less than the absolute minimum, + * which is the first value in the array. + */ + cmpLessFn = minmax_multi_get_strategy_procinfo(bdesc, attno, typid, + BTLessStrategyNumber); + compar = FunctionCall2Coll(cmpLessFn, colloid, newval, minvalue); + + /* smaller than the smallest value in the range list */ + if (DatumGetBool(compar)) + match = false; + + /* + * And now compare it to the existing maximum (last value in the + * data array). But only if we haven't already ruled out a possible + * match in the minvalue check. + */ + if (match) + { + cmpGreaterFn = minmax_multi_get_strategy_procinfo(bdesc, attno, typid, + BTGreaterStrategyNumber); + compar = FunctionCall2Coll(cmpGreaterFn, colloid, newval, maxvalue); + + if (DatumGetBool(compar)) + match = false; + } + + /* + * So it's in the general range, but is it actually covered by any + * of the ranges? Repeat the check for each range. + * + * XXX We simply walk the ranges sequentially, but maybe we could + * further leverage the ordering and non-overlap and use bsearch to + * speed this up a bit. + */ + for (i = 0; i < ranges->nranges && match; i++) + { + /* copy the min/max values from the ranges */ + minvalue = ranges->values[2*i]; + maxvalue = ranges->values[2*i+1]; + + /* + * Otherwise, need to compare the new value with boundaries of all + * the ranges. First check if it's less than the absolute minimum, + * which is the first value in the array. + */ + compar = FunctionCall2Coll(cmpLessFn, colloid, newval, minvalue); + + /* smaller than the smallest value in this range */ + if (DatumGetBool(compar)) + continue; + + compar = FunctionCall2Coll(cmpGreaterFn, colloid, newval, maxvalue); + + /* larger than the largest value in this range */ + if (DatumGetBool(compar)) + continue; + + /* hey, we found a matching row */ + return true; + } + } + + cmpEqualFn = minmax_multi_get_strategy_procinfo(bdesc, attno, typid, + BTEqualStrategyNumber); + + /* + * We're done with the ranges, now let's inspect the exact values. + * + * XXX Again, we do sequentially search the values - consider leveraging + * the ordering of values to improve performance. + */ + for (i = 2*ranges->nranges; i < 2*ranges->nranges + ranges->nvalues; i++) + { + Datum compar; + + compar = FunctionCall2Coll(cmpEqualFn, colloid, newval, ranges->values[i]); + + /* found an exact match */ + if (DatumGetBool(compar)) + return true; + } + + /* the value is not covered by this BRIN tuple */ + return false; +} + +/* + * insert_value + * Adds a new value into the single-point part, while maintaining ordering. + * + * The function inserts the new value to the right place in the single-point + * part of the range. It assumes there's enough free space, and then does + * essentially an insert-sort. + * + * XXX Assumes the 'values' array has space for (nvalues+1) entries, and that + * only the first nvalues are used. + */ +static void +insert_value(FmgrInfo *cmp, Oid colloid, Datum *values, int nvalues, + Datum newvalue) +{ + int i; + Datum lt; + + /* If there are no values yet, store the new one and we're done. */ + if (!nvalues) + { + values[0] = newvalue; + return; + } + + /* + * A common case is that the new value is entirely out of the existing + * range, i.e. it's either smaller or larger than all previous values. + * So we check and handle this case first - first we check the larger + * case, because in that case we can just append the value to the end + * of the array and we're done. + */ + + /* Is it greater than all existing values in the array? */ + lt = FunctionCall2Coll(cmp, colloid, values[nvalues-1], newvalue); + if (DatumGetBool(lt)) + { + /* just copy it in-place and we're done */ + values[nvalues] = newvalue; + return; + } + + /* + * OK, I lied a bit - we won't check the smaller case explicitly, but + * we'll just compare the value to all existing values in the array. + * But we happen to start with the smallest value, so we're actually + * doing the check anyway. + * + * XXX We do walk the values sequentially. Perhaps we could/should be + * smarter and do some sort of bisection, to improve performance? + */ + for (i = 0; i < nvalues; i++) + { + lt = FunctionCall2Coll(cmp, colloid, newvalue, values[i]); + if (DatumGetBool(lt)) + { + /* + * Move values to make space for the new entry, which should go + * to index 'i'. Entries 0 ... (i-1) should stay where they are. + */ + memmove(&values[i+1], &values[i], (nvalues-i) * sizeof(Datum)); + values[i] = newvalue; + return; + } + } + + /* We should never really get here. */ + Assert(false); +} + +/* + * Check that the order of the array values is correct, using the cmp + * function (which should be BTLessStrategyNumber). + */ +static void +AssertArrayOrder(FmgrInfo *cmp, Oid colloid, Datum *values, int nvalues) +{ +#ifdef USE_ASSERT_CHECKING + int i; + Datum lt; + + for (i = 0; i < (nvalues-1); i++) + { + lt = FunctionCall2Coll(cmp, colloid, values[i], values[i+1]); + Assert(DatumGetBool(lt)); + } +#endif +} + +/* + * Check ordering of boundary values in both parts of the ranges, using + * the cmp function (which should be BTLessStrategyNumber). + * + * XXX We might also check that the ranges and points do not overlap. But + * that will break this check sooner or later anyway. + */ +static void +AssertRangeOrdering(FmgrInfo *cmpFn, Oid colloid, Ranges *ranges) +{ +#ifdef USE_ASSERT_CHECKING + /* first the ranges (there are 2*nranges boundary values) */ + AssertArrayOrder(cmpFn, colloid, ranges->values, 2*ranges->nranges); + + /* then the single-point ranges (with nvalues boundar values ) */ + AssertArrayOrder(cmpFn, colloid, &ranges->values[2*ranges->nranges], + ranges->nvalues); +#endif +} + +/* + * Check that the expanded ranges (built when reducing the number of ranges + * by combining some of them) are correctly sorted and do not overlap. + */ +static void +AssertValidCombineRanges(BrinDesc *bdesc, Oid colloid, AttrNumber attno, + Form_pg_attribute attr, CombineRange *ranges, + int nranges) +{ +#ifdef USE_ASSERT_CHECKING + int i; + FmgrInfo *eq; + FmgrInfo *lt; + + eq = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid, + BTEqualStrategyNumber); + + lt = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid, + BTLessStrategyNumber); + + /* + * Each range independently should be valid, i.e. that for the boundary + * values (lower <= upper). + */ + for (i = 0; i < nranges; i++) + { + Datum r; + Datum minval = ranges[i].minval; + Datum maxval = ranges[i].maxval; + + if (ranges[i].collapsed) /* collapsed: minval == maxval */ + r = FunctionCall2Coll(eq, colloid, minval, maxval); + else /* non-collapsed: minval < maxval */ + r = FunctionCall2Coll(lt, colloid, minval, maxval); + + Assert(DatumGetBool(r)); + } + + /* + * And the ranges should be ordered and must nor overlap, i.e. + * upper < lower for boundaries of consecutive ranges. + */ + for (i = 0; i < nranges-1; i++) + { + Datum r; + Datum maxval = ranges[i].maxval; + Datum minval = ranges[i+1].minval; + + r = FunctionCall2Coll(lt, colloid, maxval, minval); + + Assert(DatumGetBool(r)); + } +#endif +} + +/* + * Expand ranges from Ranges into CombineRange array. This expects the + * cranges to be pre-allocated and sufficiently large (there needs to be + * at least (nranges + nvalues) slots). + */ +static void +fill_combine_ranges(CombineRange *cranges, int ncranges, Ranges *ranges) +{ + int idx; + int i; + + idx = 0; + for (i = 0; i < ranges->nranges; i++) + { + cranges[idx].minval = ranges->values[2*i]; + cranges[idx].maxval = ranges->values[2*i+1]; + cranges[idx].collapsed = false; + idx++; + + Assert(idx <= ncranges); + } + + for (i = 0; i < ranges->nvalues; i++) + { + cranges[idx].minval = ranges->values[2*ranges->nranges + i]; + cranges[idx].maxval = ranges->values[2*ranges->nranges + i]; + cranges[idx].collapsed = true; + idx++; + + Assert(idx <= ncranges); + } + + Assert(idx == ncranges); + + return; +} + +/* + * Sort combine ranges using qsort (with BTLessStrategyNumber function). + */ +static void +sort_combine_ranges(FmgrInfo *cmp, Oid colloid, + CombineRange *cranges, int ncranges) +{ + compare_context cxt; + + /* sort the values */ + cxt.colloid = colloid; + cxt.cmpFn = cmp; + + qsort_arg(cranges, ncranges, sizeof(CombineRange), + compare_combine_ranges, (void *) &cxt); +} + +/* + * When combining multiple Range values (in union function), some of the + * ranges may overlap. We simply merge the overlapping ranges to fix that. + * + * XXX This assumes the combine ranges were previously sorted (by minval + * and then maxval). We leverage this when detecting overlap. + */ +static int +merge_combine_ranges(FmgrInfo *cmp, Oid colloid, + CombineRange *cranges, int ncranges) +{ + int idx; + + /* TODO: add assert checking the ordering of input ranges */ + + /* try merging ranges (idx) and (idx+1) if they overlap */ + idx = 0; + while (idx < (ncranges-1)) + { + Datum r; + + /* + * comparing [?,maxval] vs. [minval,?] - the ranges overlap + * if (minval < maxval) + */ + r = FunctionCall2Coll(cmp, colloid, + cranges[idx].maxval, + cranges[idx+1].minval); + + /* + * Nope, maxval < minval, so no overlap. And we know the ranges + * are ordered, so there are no more overlaps, because all the + * remaining ranges have greater or equal minval. + */ + if (DatumGetBool(r)) + { + /* proceed to the next range */ + idx += 1; + continue; + } + + /* + * So ranges 'idx' and 'idx+1' do overlap, but we don't know if + * 'idx+1' is contained in 'idx', or if they only overlap only + * partially. So compare the upper bounds and keep the larger one. + */ + r = FunctionCall2Coll(cmp, colloid, + cranges[idx].maxval, + cranges[idx+1].maxval); + + if (DatumGetBool(r)) + cranges[idx].maxval = cranges[idx+1].maxval; + + /* + * The range certainly is no longer collapsed (irrespectedly of + * the previous state). + */ + cranges[idx].collapsed = false; + + /* + * Now get rid of the (idx+1) range entirely by shifting the + * remaining ranges by 1. There are ncranges elements, and we + * need to move elements from (idx+2). That means the number + * of elements to move is [ncranges - (idx+2)]. + */ + memmove(&cranges[idx+1], &cranges[idx+2], + (ncranges - (idx + 2)) * sizeof(CombineRange)); + + /* + * Decrease the number of ranges, and repeat (with the same range, + * as it might overlap with additional ranges thanks to the merge). + */ + ncranges--; + } + + /* TODO: add assert checking the ordering etc. of produced ranges */ + + return ncranges; +} + +/* + * Represents a distance between two ranges (identified by index into + * an array of combine ranges). + */ +typedef struct DistanceValue +{ + int index; + double value; +} DistanceValue; + +/* + * Simple comparator for distance values, comparing the double value. + */ +static int +compare_distances(const void *a, const void *b) +{ + DistanceValue *da = (DistanceValue *)a; + DistanceValue *db = (DistanceValue *)b; + + if (da->value < db->value) + return -1; + else if (da->value > db->value) + return 1; + + return 0; +} + +/* + * Given an array of combine ranges, compute distance of the gaps betwen + * the ranges - for ncranges there are (ncranges-1) gaps. + * + * We simply call the "distance" function to compute the (min-max) for pairs + * of consecutive ganges. The function may be fairly expensive, so we do that + * just once (and then use it to pick as many ranges to merge as possible). + * + * See reduce_combine_ranges for details. + */ +static DistanceValue * +build_distances(FmgrInfo *distanceFn, Oid colloid, + CombineRange *cranges, int ncranges) +{ + int i; + DistanceValue *distances; + + Assert(ncranges >= 2); + + distances = (DistanceValue *) palloc0(sizeof(DistanceValue) * (ncranges - 1)); + + /* + * Walk though the ranges once and compute distance between the ranges + * so that we can sort them once. + */ + for (i = 0; i < (ncranges-1); i++) + { + Datum a1, a2, r; + + a1 = cranges[i].maxval; + a2 = cranges[i+1].minval; + + /* compute length of the empty gap (distance between max/min) */ + r = FunctionCall2Coll(distanceFn, colloid, a1, a2); + + /* remember the index */ + distances[i].index = i; + distances[i].value = DatumGetFloat8(r); + } + + /* sort the distances in ascending order */ + pg_qsort(distances, (ncranges-1), sizeof(DistanceValue), compare_distances); + + return distances; +} + +/* + * Builds combine ranges for the existing ranges (and single-point ranges), + * and also the new value (which did not fit into the array). + * + * XXX We do perform qsort on all the values, but we could also leverage + * the fact that the input data is already sorted and do merge sort. + */ +static CombineRange * +build_combine_ranges(FmgrInfo *cmp, Oid colloid, Ranges *ranges, + Datum newvalue, int *nranges) +{ + int ncranges; + CombineRange *cranges; + + /* now do the actual merge sort */ + ncranges = ranges->nranges + ranges->nvalues + 1; + cranges = (CombineRange *) palloc0(ncranges * sizeof(CombineRange)); + *nranges = ncranges; + + /* put the new value at the beginning */ + cranges[0].minval = newvalue; + cranges[0].maxval = newvalue; + cranges[0].collapsed = true; + + /* then the regular and collapsed ranges */ + fill_combine_ranges(&cranges[1], ncranges-1, ranges); + + /* and sort the ranges */ + sort_combine_ranges(cmp, colloid, cranges, ncranges); + + return cranges; +} + +/* + * Counts bondary values needed to store the ranges. Each single-point + * range is stored using a single value, each regular range needs two. + */ +static int +count_values(CombineRange *cranges, int ncranges) +{ + int i; + int count; + + count = 0; + for (i = 0; i < ncranges; i++) + { + if (cranges[i].collapsed) + count += 1; + else + count += 2; + } + + return count; +} + +/* + * Combines ranges until the number of boundary values drops below 75% + * of the capacity (as set by values_per_range reloption). + * + * XXX The ranges to merge are selected solely using the distance. But + * that may not be the best strategy, for example when multiple gaps + * are of equal (or very similar) length. + * + * Consider for example points 1, 2, 3, .., 64, which have gaps of the + * same length 1 of course. In that case we tend to pick the first + * gap of that length, which leads to this: + * + * step 1: [1, 2], 3, 4, 5, .., 64 + * step 2: [1, 3], 4, 5, .., 64 + * step 3: [1, 4], 5, .., 64 + * ... + * + * So in the end we'll have one "large" range and multiple small points. + * That may be fine, but it seems a bit strange and non-optimal. Maybe + * we should consider other things when picking ranges to merge - e.g. + * length of the ranges? Or perhaps randomize the choice of ranges, with + * probability inversely proportional to the distance (the gap lengths + * may be very close, but not exactly the same). + */ +static int +reduce_combine_ranges(CombineRange *cranges, int ncranges, + DistanceValue *distances, int max_values) +{ + int i; + int ndistances = (ncranges - 1); + int count = count_values(cranges, ncranges); + + /* + * We have one fewer 'gaps' than the ranges. We'll be decrementing + * the number of combine ranges (reduction is the primary goal of + * this function), so we must use a separate value. + */ + for (i = 0; i < ndistances; i++) + { + int j; + int shortest; + + if (count <= max_values * 0.75) + break; + + shortest = distances[i].index; + + /* + * The index must be still valid with respect to the current size + * of cranges array (and it always points to the first range, so + * never to the last one - hence the -1 in the condition). + */ + Assert(shortest < (ncranges - 1)); + + if (!cranges[shortest].collapsed && !cranges[shortest+1].collapsed) + count -= 2; + else if (!cranges[shortest].collapsed || !cranges[shortest+1].collapsed) + count -= 1; + + /* + * Move the values to join the two selected ranges. The new range is + * definiely not collapsed but a regular range. + */ + cranges[shortest].maxval = cranges[shortest+1].maxval; + cranges[shortest].collapsed = false; + + /* shuffle the subsequent combine ranges */ + memmove(&cranges[shortest+1], &cranges[shortest+2], + (ncranges - shortest - 2) * sizeof(CombineRange)); + + /* also, shuffle all higher indexes (we've just moved the ranges) */ + for (j = i; j < ndistances; j++) + { + if (distances[j].index > shortest) + distances[j].index--; + } + + ncranges--; + + Assert(ncranges > 0); + } + + return ncranges; +} + +/* + * Store the boundary values from CombineRanges back into Range (using + * only the minimal number of values needed). + */ +static void +store_combine_ranges(Ranges *ranges, CombineRange *cranges, int ncranges) +{ + int i; + int idx = 0; + + /* first copy in the regular ranges */ + ranges->nranges = 0; + for (i = 0; i < ncranges; i++) + { + if (!cranges[i].collapsed) + { + ranges->values[idx++] = cranges[i].minval; + ranges->values[idx++] = cranges[i].maxval; + ranges->nranges++; + } + } + + /* now copy in the collapsed ones */ + ranges->nvalues = 0; + for (i = 0; i < ncranges; i++) + { + if (cranges[i].collapsed) + { + ranges->values[idx++] = cranges[i].minval; + ranges->nvalues++; + } + } +} + +/* + * range_add_value + * Add the new value to the multi-minmax range. + */ +static bool +range_add_value(BrinDesc *bdesc, Oid colloid, + AttrNumber attno, Form_pg_attribute attr, + Ranges *ranges, Datum newval) +{ + FmgrInfo *cmpFn, + *distanceFn; + + /* combine ranges */ + CombineRange *cranges; + int ncranges; + DistanceValue *distances; + + MemoryContext ctx; + MemoryContext oldctx; + + Assert(2*ranges->nranges + ranges->nvalues <= ranges->maxvalues); + + /* + * Bail out if the value already is covered by the range. + * + * We could also add values until we hit values_per_range, and then + * do the deduplication in a batch, hoping for better efficiency. But + * that would mean we actually modify the range every time, which means + * having to serialize the value, which does palloc, walks the values, + * copies them, etc. Not exactly cheap. + * + * So instead we do the check, which should be fairly cheap - assuming + * the comparator function is not very expensive. + * + * This also implies means the values array can't contain duplicities. + */ + if (range_contains_value(bdesc, colloid, attno, attr, ranges, newval)) + return false; + + /* we'll certainly need the comparator, so just look it up now */ + cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid, + BTLessStrategyNumber); + + /* + * If there's space in the values array, copy it in and we're done. + * + * We do want to keep the values sorted (to speed up searches), so we + * do a simple insertion sort. We could do something more elaborate, + * e.g. by sorting the values only now and then, but for small counts + * (e.g. when maxvalues is 64) this should be fine. + */ + if (2*ranges->nranges + ranges->nvalues < ranges->maxvalues) + { + Datum *values; + + /* beginning of the 'single value' part (for convenience) */ + values = &ranges->values[2*ranges->nranges]; + + insert_value(cmpFn, colloid, values, ranges->nvalues, newval); + + ranges->nvalues++; + + /* + * Check we haven't broken the ordering of boundary values (checks + * both parts, but that doesn't hurt). + */ + AssertRangeOrdering(cmpFn, colloid, ranges); + + /* yep, we've modified the range */ + return true; + } + + /* + * Damn - the new value is not in the range yet, but we don't have space + * to just insert it. So we need to combine some of the existing ranges, + * to reduce the number of values we need to store (joining two intervals + * reduces the number of boundaries to store by 2). + * + * To do that we first construct an array of CombineRange items - each + * combine range tracks if it's a regular range or collapsed range, where + * "collapsed" means "single point." + * + * Existing ranges (we have ranges->nranges of them) map to combine ranges + * directly, while single points (ranges->nvalues of them) have to be + * expanded. We neet the combine ranges to be sorted, and we do that by + * performing a merge sort of ranges, values and new value. + */ + + /* Check the ordering invariants are not violated (for both parts). */ + AssertArrayOrder(cmpFn, colloid, ranges->values, ranges->nranges*2); + AssertArrayOrder(cmpFn, colloid, &ranges->values[ranges->nranges*2], + ranges->nvalues); + + /* and we'll also need the 'distance' procedure */ + distanceFn = minmax_multi_get_procinfo(bdesc, attno, PROCNUM_DISTANCE); + + /* + * The distanceFn calls (which may internally call e.g. numeric_le) may + * allocate quite a bit of memory, and we must not leak it. Otherwise + * we'd have problems e.g. when building indexes. So we create a local + * memory context and make sure we free the memory before leaving this + * function (not after every call). + */ + ctx = AllocSetContextCreate(CurrentMemoryContext, + "minmax-multi context", + ALLOCSET_DEFAULT_SIZES); + + oldctx = MemoryContextSwitchTo(ctx); + + /* OK build the combine ranges */ + cranges = build_combine_ranges(cmpFn, colloid, ranges, newval, &ncranges); + + /* build array of gap distances and sort them in ascending order */ + distances = build_distances(distanceFn, colloid, cranges, ncranges); + + /* + * Combine ranges until we release at least 25% of the space. This + * threshold is somewhat arbitrary, perhaps needs tuning. We must not + * use too low or high value. + */ + ncranges = reduce_combine_ranges(cranges, ncranges, distances, + ranges->maxvalues); + + Assert(count_values(cranges, ncranges) <= ranges->maxvalues * 0.75); + + /* decompose the combine ranges into regular ranges and single values */ + store_combine_ranges(ranges, cranges, ncranges); + + MemoryContextSwitchTo(oldctx); + MemoryContextDelete(ctx); + + /* Check the ordering invariants are not violated (for both parts). */ + AssertArrayOrder(cmpFn, colloid, ranges->values, ranges->nranges*2); + AssertArrayOrder(cmpFn, colloid, &ranges->values[ranges->nranges*2], + ranges->nvalues); + + return true; +} + +Datum +brin_minmax_multi_opcinfo(PG_FUNCTION_ARGS) +{ + BrinOpcInfo *result; + + /* + * opaque->strategy_procinfos is initialized lazily; here it is set to + * all-uninitialized by palloc0 which sets fn_oid to InvalidOid. + */ + + result = palloc0(MAXALIGN(SizeofBrinOpcInfo(1)) + + sizeof(MinmaxMultiOpaque)); + result->oi_nstored = 1; + result->oi_regular_nulls = true; + result->oi_opaque = (MinmaxMultiOpaque *) + MAXALIGN((char *) result + SizeofBrinOpcInfo(1)); + result->oi_typcache[0] = lookup_type_cache(PG_BRIN_MINMAX_MULTI_SUMMARYOID, 0); + + PG_RETURN_POINTER(result); +} + +/* + * Compute distance between two float4 values (plain subtraction). + */ +Datum +brin_minmax_multi_distance_float4(PG_FUNCTION_ARGS) +{ + float a1 = PG_GETARG_FLOAT4(0); + float a2 = PG_GETARG_FLOAT4(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(a1 <= a2); + + PG_RETURN_FLOAT8((double) a2 - (double) a1); +} + +/* + * Compute distance between two float8 values (plain subtraction). + */ +Datum +brin_minmax_multi_distance_float8(PG_FUNCTION_ARGS) +{ + double a1 = PG_GETARG_FLOAT8(0); + double a2 = PG_GETARG_FLOAT8(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(a1 <= a2); + + PG_RETURN_FLOAT8(a2 - a1); +} + +/* + * Compute distance between two int2 values (plain subtraction). + */ +Datum +brin_minmax_multi_distance_int2(PG_FUNCTION_ARGS) +{ + int16 a1 = PG_GETARG_INT16(0); + int16 a2 = PG_GETARG_INT16(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(a1 <= a2); + + PG_RETURN_FLOAT8((double) a2 - (double) a1); +} + +/* + * Compute distance between two int4 values (plain subtraction). + */ +Datum +brin_minmax_multi_distance_int4(PG_FUNCTION_ARGS) +{ + int32 a1 = PG_GETARG_INT32(0); + int32 a2 = PG_GETARG_INT32(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(a1 <= a2); + + PG_RETURN_FLOAT8((double) a2 - (double) a1); +} + +/* + * Compute distance between two int8 values (plain subtraction). + */ +Datum +brin_minmax_multi_distance_int8(PG_FUNCTION_ARGS) +{ + int64 a1 = PG_GETARG_INT64(0); + int64 a2 = PG_GETARG_INT64(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(a1 <= a2); + + PG_RETURN_FLOAT8((double) a2 - (double) a1); +} + +/* + * Compute distance between two tid values (by mapping them to float8 + * and then subtracting them). + */ +Datum +brin_minmax_multi_distance_tid(PG_FUNCTION_ARGS) +{ + double da1, + da2; + + ItemPointer pa1 = (ItemPointer) PG_GETARG_DATUM(0); + ItemPointer pa2 = (ItemPointer) PG_GETARG_DATUM(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(ItemPointerCompare(pa1, pa2) <= 0); + + da1 = ItemPointerGetBlockNumber(pa1) * MaxHeapTuplesPerPage + + ItemPointerGetOffsetNumber(pa1); + + da2 = ItemPointerGetBlockNumber(pa2) * MaxHeapTuplesPerPage + + ItemPointerGetOffsetNumber(pa2); + + PG_RETURN_FLOAT8(da2 - da1); +} + +/* + * Comutes distance between two numeric values (plain subtraction). + */ +Datum +brin_minmax_multi_distance_numeric(PG_FUNCTION_ARGS) +{ + Datum d; + Datum a1 = PG_GETARG_DATUM(0); + Datum a2 = PG_GETARG_DATUM(1); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(DatumGetBool(DirectFunctionCall2(numeric_le, a1, a2))); + + d = DirectFunctionCall2(numeric_sub, a2, a1); /* a2 - a1 */ + + PG_RETURN_FLOAT8(DirectFunctionCall1(numeric_float8, d)); +} + +/* + * Computes approximate distance between two UUID values. + * + * XXX We do not need a perfectly accurate value, so we approximate the + * deltas (which would have to be 128-bit integers) with a 64-bit float. + * The small inaccuracies do not matter in practice, in the worst case + * we'll decide to merge ranges that are not the closest ones. + */ +Datum +brin_minmax_multi_distance_uuid(PG_FUNCTION_ARGS) +{ + int i; + double delta = 0; + + Datum a1 = PG_GETARG_DATUM(0); + Datum a2 = PG_GETARG_DATUM(1); + + pg_uuid_t *u1 = DatumGetUUIDP(a1); + pg_uuid_t *u2 = DatumGetUUIDP(a2); + + /* + * We know the values are range boundaries, but the range may be + * collapsed (i.e. single points), with equal values. + */ + Assert(DatumGetBool(DirectFunctionCall2(uuid_le, a1, a2))); + + /* compute approximate delta as a double precision value */ + for (i = UUID_LEN-1; i >= 0; i--) + { + delta += (int) u2->data[i] - (int) u1->data[i]; + delta /= 256; + } + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Computes approximate distance between two time (without tz) values. + * + * TimeADT is just an int64, so we simply subtract the values directly. + */ +Datum +brin_minmax_multi_distance_time(PG_FUNCTION_ARGS) +{ + double delta = 0; + + TimeADT ta = PG_GETARG_TIMEADT(0); + TimeADT tb = PG_GETARG_TIMEADT(1); + + delta = (tb - ta); + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Computes approximate distance between two timetz values. + * + * Simply subtracts the TimeADT (int64) values embedded in TimeTzADT. + * + * XXX Does this need to consider the time zones? + */ +Datum +brin_minmax_multi_distance_timetz(PG_FUNCTION_ARGS) +{ + double delta = 0; + + TimeTzADT *ta = PG_GETARG_TIMETZADT_P(0); + TimeTzADT *tb = PG_GETARG_TIMETZADT_P(1); + + delta = tb->time - ta->time; + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Computes distance between two interval values. + * + * Intervals are internally just 'time' values, so use the same approach + * as for in brin_minmax_multi_distance_time. + * + * XXX Do we actually need two separate functions, then? + */ +Datum +brin_minmax_multi_distance_interval(PG_FUNCTION_ARGS) +{ + double delta = 0; + + TimeADT ia = PG_GETARG_TIMEADT(0); + TimeADT ib = PG_GETARG_TIMEADT(1); + + delta = (ib - ia); + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Compute distance between two pg_lsn values. + * + * LSN is just an int64 encoding position in the stream, so just subtract + * those int64 values directly. + */ +Datum +brin_minmax_multi_distance_pg_lsn(PG_FUNCTION_ARGS) +{ + double delta = 0; + + XLogRecPtr lsna = PG_GETARG_LSN(0); + XLogRecPtr lsnb = PG_GETARG_LSN(1); + + delta = (lsnb - lsna); + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Compute distance between two macaddr values. + * + * mac addresses are treated as 6 unsigned chars, so do the same thing we + * already do for UUID values. + */ +Datum +brin_minmax_multi_distance_macaddr(PG_FUNCTION_ARGS) +{ + double delta; + + macaddr *a = PG_GETARG_MACADDR_P(0); + macaddr *b = PG_GETARG_MACADDR_P(1); + + delta = ((double)b->f - (double)a->f); + delta /= 256; + + delta += ((double)b->e - (double)a->e); + delta /= 256; + + delta += ((double)b->d - (double)a->d); + delta /= 256; + + delta += ((double)b->c - (double)a->c); + delta /= 256; + + delta += ((double)b->b - (double)a->b); + delta /= 256; + + delta += ((double)b->a - (double)a->a); + delta /= 256; + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Compute distance between two macaddr8 values. + * + * macaddr8 addresses are 8 unsigned chars, so do the same thing we + * already do for UUID values. + */ +Datum +brin_minmax_multi_distance_macaddr8(PG_FUNCTION_ARGS) +{ + double delta; + + macaddr8 *a = PG_GETARG_MACADDR8_P(0); + macaddr8 *b = PG_GETARG_MACADDR8_P(1); + + delta = ((double)b->h - (double)a->h); + delta /= 256; + + delta += ((double)b->g - (double)a->g); + delta /= 256; + + delta += ((double)b->f - (double)a->f); + delta /= 256; + + delta += ((double)b->e - (double)a->e); + delta /= 256; + + delta += ((double)b->d - (double)a->d); + delta /= 256; + + delta += ((double)b->c - (double)a->c); + delta /= 256; + + delta += ((double)b->b - (double)a->b); + delta /= 256; + + delta += ((double)b->a - (double)a->a); + delta /= 256; + + Assert(delta >= 0); + + PG_RETURN_FLOAT8(delta); +} + +/* + * Compute distance between two inet values. + * + * The distance is defined as difference between 32-bit/128-bit values, + * depending on the IP version. The distance is computed by subtracting + * the bytes and normalizing it to [0,1] range for each IP family. + * Addresses from difference families are consider to be in maximum + * distance, which is 1.0. + * + * XXX Does this need to consider the mask (bits)? For now it's ignored. + */ +Datum +brin_minmax_multi_distance_inet(PG_FUNCTION_ARGS) +{ + double delta; + int i; + int len; + unsigned char *addra, + *addrb; + + inet *ipa = PG_GETARG_INET_PP(0); + inet *ipb = PG_GETARG_INET_PP(1); + + /* + * If the addresses are from different families, consider them to be + * in maximal possible distance (which is 1.0). + */ + if (ip_family(ipa) != ip_family(ipb)) + return 1.0; + + /* ipv4 or ipv6 */ + if (ip_family(ipa) == PGSQL_AF_INET) + len = 4; + else + len = 16; /* NS_IN6ADDRSZ */ + + addra = ip_addr(ipa); + addrb = ip_addr(ipb); + + delta = 0; + for (i = len-1; i >= 0; i--) + { + delta += (double)addrb[i] - (double)addra[i]; + delta /= 256; + } + + Assert((delta >= 0) && (delta <= 1)); + + PG_RETURN_FLOAT8(delta); +} + +static void +brin_minmax_multi_serialize(Datum src, Datum *dst) +{ + Ranges *ranges = (Ranges *) DatumGetPointer(src); + SerializedRanges *s = range_serialize(ranges); + dst[0] = PointerGetDatum(s); +} + +static int +brin_minmax_multi_get_values(BrinDesc *bdesc, MinMaxOptions *opts) +{ + return MinMaxGetValuesPerRange(opts); +} + +/* + * Examine the given index tuple (which contains partial status of a certain + * page range) by comparing it to the given value that comes from another heap + * tuple. If the new value is outside the min/max range specified by the + * existing tuple values, update the index tuple and return true. Otherwise, + * return false and do not modify in this case. + */ +Datum +brin_minmax_multi_add_value(PG_FUNCTION_ARGS) +{ + BrinDesc *bdesc = (BrinDesc *) PG_GETARG_POINTER(0); + BrinValues *column = (BrinValues *) PG_GETARG_POINTER(1); + Datum newval = PG_GETARG_DATUM(2); + bool isnull PG_USED_FOR_ASSERTS_ONLY = PG_GETARG_DATUM(3); + MinMaxOptions *opts = (MinMaxOptions *) PG_GET_OPCLASS_OPTIONS(); + Oid colloid = PG_GET_COLLATION(); + bool modified = false; + Form_pg_attribute attr; + AttrNumber attno; + Ranges *ranges; + SerializedRanges *serialized = NULL; + + Assert(!isnull); + + attno = column->bv_attno; + attr = TupleDescAttr(bdesc->bd_tupdesc, attno - 1); + + /* use the already deserialized value, is possible */ + ranges = (Ranges *) DatumGetPointer(column->bv_mem_value); + + /* + * If this is the first non-null value, we need to initialize the range + * list. Otherwise just extract the existing range list from BrinValues. + */ + if (column->bv_allnulls) + { + MemoryContext oldctx; + + oldctx = MemoryContextSwitchTo(column->bv_context); + + ranges = minmax_multi_init(brin_minmax_multi_get_values(bdesc, opts)); + ranges->typid = attr->atttypid; + + MemoryContextSwitchTo(oldctx); + + column->bv_allnulls = false; + modified = true; + + column->bv_mem_value = PointerGetDatum(ranges); + column->bv_serialize = brin_minmax_multi_serialize; + } + else if (!ranges) + { + MemoryContext oldctx; + + oldctx = MemoryContextSwitchTo(column->bv_context); + + serialized = (SerializedRanges *) PG_DETOAST_DATUM(column->bv_values[0]); + ranges = range_deserialize(serialized); + + column->bv_mem_value = PointerGetDatum(ranges); + column->bv_serialize = brin_minmax_multi_serialize; + + MemoryContextSwitchTo(oldctx); + } + + /* + * Try to add the new value to the range. We need to update the modified + * flag, so that we serialize the correct value. + */ + modified |= range_add_value(bdesc, colloid, attno, attr, ranges, newval); + + + PG_RETURN_BOOL(modified); +} + +/* + * Given an index tuple corresponding to a certain page range and a scan key, + * return whether the scan key is consistent with the index tuple's min/max + * values. Return true if so, false otherwise. + */ +Datum +brin_minmax_multi_consistent(PG_FUNCTION_ARGS) +{ + BrinDesc *bdesc = (BrinDesc *) PG_GETARG_POINTER(0); + BrinValues *column = (BrinValues *) PG_GETARG_POINTER(1); + ScanKey *keys = (ScanKey *) PG_GETARG_POINTER(2); + int nkeys = PG_GETARG_INT32(3); + // MinMaxOptions *opts = (MinMaxOptions *) PG_GET_OPCLASS_OPTIONS(); + Oid colloid = PG_GET_COLLATION(), + subtype; + AttrNumber attno; + Datum value; + FmgrInfo *finfo; + SerializedRanges *serialized; + Ranges *ranges; + int keyno; + int rangeno; + int i; + + attno = column->bv_attno; + + serialized = (SerializedRanges *) PG_DETOAST_DATUM(column->bv_values[0]); + ranges = range_deserialize(serialized); + + /* inspect the ranges, and for each one evaluate the scan keys */ + for (rangeno = 0; rangeno < ranges->nranges; rangeno++) + { + Datum minval = ranges->values[2*rangeno]; + Datum maxval = ranges->values[2*rangeno+1]; + + /* assume the range is matching, and we'll try to prove otherwise */ + bool matching = true; + + for (keyno = 0; keyno < nkeys; keyno++) + { + Datum matches; + ScanKey key = keys[keyno]; + + /* NULL keys are handled and filtered-out in bringetbitmap */ + Assert(!(key->sk_flags & SK_ISNULL)); + + attno = key->sk_attno; + subtype = key->sk_subtype; + value = key->sk_argument; + switch (key->sk_strategy) + { + case BTLessStrategyNumber: + case BTLessEqualStrategyNumber: + finfo = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype, + key->sk_strategy); + /* first value from the array */ + matches = FunctionCall2Coll(finfo, colloid, minval, value); + break; + + case BTEqualStrategyNumber: + { + Datum compar; + FmgrInfo *cmpFn; + + /* by default this range does not match */ + matches = false; + + /* + * Otherwise, need to compare the new value with boundaries of all + * the ranges. First check if it's less than the absolute minimum, + * which is the first value in the array. + */ + cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype, + BTLessStrategyNumber); + compar = FunctionCall2Coll(cmpFn, colloid, value, minval); + + /* smaller than the smallest value in this range */ + if (DatumGetBool(compar)) + break; + + cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype, + BTGreaterStrategyNumber); + compar = FunctionCall2Coll(cmpFn, colloid, value, maxval); + + /* larger than the largest value in this range */ + if (DatumGetBool(compar)) + break; + + /* haven't managed to eliminate this range, so consider it matching */ + matches = true; + + break; + } + case BTGreaterEqualStrategyNumber: + case BTGreaterStrategyNumber: + finfo = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype, + key->sk_strategy); + /* last value from the array */ + matches = FunctionCall2Coll(finfo, colloid, maxval, value); + break; + + default: + /* shouldn't happen */ + elog(ERROR, "invalid strategy number %d", key->sk_strategy); + matches = 0; + break; + } + + /* the range has to match all the scan keys */ + matching &= DatumGetBool(matches); + + /* once we find a non-matching key, we're done */ + if (! matching) + break; + } + + /* have we found a range matching all scan keys? if yes, we're + * done */ + if (matching) + PG_RETURN_DATUM(BoolGetDatum(true)); + } + + /* and now inspect the values */ + for (i = 0; i < ranges->nvalues; i++) + { + Datum val = ranges->values[2*ranges->nranges + i]; + + /* assume the range is matching, and we'll try to prove otherwise */ + bool matching = true; + + for (keyno = 0; keyno < nkeys; keyno++) + { + Datum matches; + ScanKey key = keys[keyno]; + + /* we've already dealt with NULL keys at the beginning */ + if (key->sk_flags & SK_ISNULL) + continue; + + attno = key->sk_attno; + subtype = key->sk_subtype; + value = key->sk_argument; + switch (key->sk_strategy) + { + case BTLessStrategyNumber: + case BTLessEqualStrategyNumber: + case BTEqualStrategyNumber: + case BTGreaterEqualStrategyNumber: + case BTGreaterStrategyNumber: + + finfo = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype, + key->sk_strategy); + matches = FunctionCall2Coll(finfo, colloid, val, value); + break; + + default: + /* shouldn't happen */ + elog(ERROR, "invalid strategy number %d", key->sk_strategy); + matches = 0; + break; + } + + /* the range has to match all the scan keys */ + matching &= DatumGetBool(matches); + + /* once we find a non-matching key, we're done */ + if (! matching) + break; + } + + /* have we found a range matching all scan keys? if yes, we're + * done */ + if (matching) + PG_RETURN_DATUM(BoolGetDatum(true)); + } + + PG_RETURN_DATUM(BoolGetDatum(false)); +} + +/* + * Given two BrinValues, update the first of them as a union of the summary + * values contained in both. The second one is untouched. + */ +Datum +brin_minmax_multi_union(PG_FUNCTION_ARGS) +{ + BrinDesc *bdesc = (BrinDesc *) PG_GETARG_POINTER(0); + BrinValues *col_a = (BrinValues *) PG_GETARG_POINTER(1); + BrinValues *col_b = (BrinValues *) PG_GETARG_POINTER(2); + // MinMaxOptions *opts = (MinMaxOptions *) PG_GET_OPCLASS_OPTIONS(); + Oid colloid = PG_GET_COLLATION(); + SerializedRanges *serialized_a; + SerializedRanges *serialized_b; + Ranges *ranges_a; + Ranges *ranges_b; + AttrNumber attno; + Form_pg_attribute attr; + CombineRange *cranges; + int ncranges; + FmgrInfo *cmpFn, + *distanceFn; + DistanceValue *distances; + MemoryContext ctx; + MemoryContext oldctx; + + Assert(col_a->bv_attno == col_b->bv_attno); + Assert(!col_a->bv_allnulls && !col_b->bv_allnulls); + + attno = col_a->bv_attno; + attr = TupleDescAttr(bdesc->bd_tupdesc, attno - 1); + + serialized_a = (SerializedRanges *) PG_DETOAST_DATUM(col_a->bv_values[0]); + serialized_b = (SerializedRanges *) PG_DETOAST_DATUM(col_b->bv_values[0]); + + ranges_a = range_deserialize(serialized_a); + ranges_b = range_deserialize(serialized_b); + + /* make sure neither of the ranges is NULL */ + Assert(ranges_a && ranges_b); + + ncranges = (ranges_a->nranges + ranges_a->nvalues) + + (ranges_b->nranges + ranges_b->nvalues); + + /* + * The distanceFn calls (which may internally call e.g. numeric_le) may + * allocate quite a bit of memory, and we must not leak it. Otherwise + * we'd have problems e.g. when building indexes. So we create a local + * memory context and make sure we free the memory before leaving this + * function (not after every call). + */ + ctx = AllocSetContextCreate(CurrentMemoryContext, + "minmax-multi context", + ALLOCSET_DEFAULT_SIZES); + + oldctx = MemoryContextSwitchTo(ctx); + + /* allocate and fill */ + cranges = (CombineRange *)palloc0(ncranges * sizeof(CombineRange)); + + /* fill the combine ranges with entries for the first range */ + fill_combine_ranges(cranges, ranges_a->nranges + ranges_a->nvalues, + ranges_a); + + /* and now add combine ranges for the second range */ + fill_combine_ranges(&cranges[ranges_a->nranges + ranges_a->nvalues], + ranges_b->nranges + ranges_b->nvalues, + ranges_b); + + cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid, + BTLessStrategyNumber); + + /* sort the combine ranges */ + sort_combine_ranges(cmpFn, colloid, cranges, ncranges); + + /* + * We've merged two different lists of ranges, so some of them may be + * overlapping. So walk through them and merge them. + */ + ncranges = merge_combine_ranges(cmpFn, colloid, cranges, ncranges); + + /* check that the combine ranges are correct (no overlaps, ordering) */ + AssertValidCombineRanges(bdesc, colloid, attno, attr, cranges, ncranges); + + /* build array of gap distances and sort them in ascending order */ + distanceFn = minmax_multi_get_procinfo(bdesc, attno, PROCNUM_DISTANCE); + distances = build_distances(distanceFn, colloid, cranges, ncranges); + + /* + * See how many values would be needed to store the current ranges, and if + * needed combine as many off them to get below the maxvalues threshold. + * The collapsed ranges will be stored as a single value. + * + * XXX The maxvalues may be different, so perhaps use Max? + */ + ncranges = reduce_combine_ranges(cranges, ncranges, distances, + ranges_a->maxvalues); + + /* update the first range summary */ + store_combine_ranges(ranges_a, cranges, ncranges); + + MemoryContextSwitchTo(oldctx); + MemoryContextDelete(ctx); + + /* cleanup and update the serialized value */ + pfree(serialized_a); + col_a->bv_values[0] = PointerGetDatum(range_serialize(ranges_a)); + + PG_RETURN_VOID(); +} + +/* + * Cache and return minmax multi opclass support procedure + * + * Return the procedure corresponding to the given function support number + * or null if it is not exists. + */ +static FmgrInfo * +minmax_multi_get_procinfo(BrinDesc *bdesc, uint16 attno, uint16 procnum) +{ + MinmaxMultiOpaque *opaque; + uint16 basenum = procnum - PROCNUM_BASE; + + /* + * We cache these in the opaque struct, to avoid repetitive syscache + * lookups. + */ + opaque = (MinmaxMultiOpaque *) bdesc->bd_info[attno - 1]->oi_opaque; + + /* + * If we already searched for this proc and didn't find it, don't bother + * searching again. + */ + if (opaque->extra_proc_missing[basenum]) + return NULL; + + if (opaque->extra_procinfos[basenum].fn_oid == InvalidOid) + { + if (RegProcedureIsValid(index_getprocid(bdesc->bd_index, attno, + procnum))) + { + fmgr_info_copy(&opaque->extra_procinfos[basenum], + index_getprocinfo(bdesc->bd_index, attno, procnum), + bdesc->bd_context); + } + else + { + opaque->extra_proc_missing[basenum] = true; + return NULL; + } + } + + return &opaque->extra_procinfos[basenum]; +} + +/* + * Cache and return the procedure for the given strategy. + * + * Note: this function mirrors minmax_multi_get_strategy_procinfo; see notes + * there. If changes are made here, see that function too. + */ +static FmgrInfo * +minmax_multi_get_strategy_procinfo(BrinDesc *bdesc, uint16 attno, Oid subtype, + uint16 strategynum) +{ + MinmaxMultiOpaque *opaque; + + Assert(strategynum >= 1 && + strategynum <= BTMaxStrategyNumber); + + opaque = (MinmaxMultiOpaque *) bdesc->bd_info[attno - 1]->oi_opaque; + + /* + * We cache the procedures for the previous subtype in the opaque struct, + * to avoid repetitive syscache lookups. If the subtype changed, + * invalidate all the cached entries. + */ + if (opaque->cached_subtype != subtype) + { + uint16 i; + + for (i = 1; i <= BTMaxStrategyNumber; i++) + opaque->strategy_procinfos[i - 1].fn_oid = InvalidOid; + opaque->cached_subtype = subtype; + } + + if (opaque->strategy_procinfos[strategynum - 1].fn_oid == InvalidOid) + { + Form_pg_attribute attr; + HeapTuple tuple; + Oid opfamily, + oprid; + bool isNull; + + opfamily = bdesc->bd_index->rd_opfamily[attno - 1]; + attr = TupleDescAttr(bdesc->bd_tupdesc, attno - 1); + tuple = SearchSysCache4(AMOPSTRATEGY, ObjectIdGetDatum(opfamily), + ObjectIdGetDatum(attr->atttypid), + ObjectIdGetDatum(subtype), + Int16GetDatum(strategynum)); + + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "missing operator %d(%u,%u) in opfamily %u", + strategynum, attr->atttypid, subtype, opfamily); + + oprid = DatumGetObjectId(SysCacheGetAttr(AMOPSTRATEGY, tuple, + Anum_pg_amop_amopopr, &isNull)); + ReleaseSysCache(tuple); + Assert(!isNull && RegProcedureIsValid(oprid)); + + fmgr_info_cxt(get_opcode(oprid), + &opaque->strategy_procinfos[strategynum - 1], + bdesc->bd_context); + } + + return &opaque->strategy_procinfos[strategynum - 1]; +} + +Datum +brin_minmax_multi_options(PG_FUNCTION_ARGS) +{ + local_relopts *relopts = (local_relopts *) PG_GETARG_POINTER(0); + + init_local_reloptions(relopts, sizeof(MinMaxOptions)); + + add_local_int_reloption(relopts, "values_per_range", "desc", + 32, 16, 256, offsetof(MinMaxOptions, valuesPerRange)); + + PG_RETURN_VOID(); +} + +/* + * brin_minmax_multi_summary_in + * - input routine for type brin_minmax_multi_summary. + * + * brin_minmax_multi_summary is only used internally to represent summaries + * in BRIN minmax-multi indexes, so it has no operations of its own, and we + * disallow input too. + */ +Datum +brin_minmax_multi_summary_in(PG_FUNCTION_ARGS) +{ + /* + * brin_minmax_multi_summary stores the data in binary form and parsing + * text input is not needed, so disallow this. + */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot accept a value of type %s", "brin_minmax_multi_summary"))); + + PG_RETURN_VOID(); /* keep compiler quiet */ +} + + +/* + * brin_minmax_multi_summary_out + * - output routine for type brin_minmax_multi_summary. + * + * BRIN minmax-multi summaries are serialized into a bytea value, but we + * want to output something nicer humans can understand. + */ +Datum +brin_minmax_multi_summary_out(PG_FUNCTION_ARGS) +{ + int i; + int idx; + SerializedRanges *ranges; + Ranges *ranges_deserialized; + StringInfoData str; + bool isvarlena; + Oid outfunc; + FmgrInfo fmgrinfo; + ArrayBuildState *astate_values = NULL; + + initStringInfo(&str); + appendStringInfoChar(&str, '{'); + + /* + * XXX not sure the detoasting is necessary (probably not, this + * can only be in an index). + */ + ranges = (SerializedRanges *) PG_DETOAST_DATUM(PG_GETARG_BYTEA_PP(0)); + + /* lookup output func for the type */ + getTypeOutputInfo(ranges->typid, &outfunc, &isvarlena); + fmgr_info(outfunc, &fmgrinfo); + + /* deserialize the range info easy-to-process pieces */ + ranges_deserialized = range_deserialize(ranges); + + appendStringInfo(&str, "nranges: %u nvalues: %u maxvalues: %u", + ranges_deserialized->nranges, + ranges_deserialized->nvalues, + ranges_deserialized->maxvalues); + + /* serialize ranges */ + idx = 0; + for (i = 0; i < ranges_deserialized->nranges; i++) + { + Datum a, b; + text *c; + StringInfoData str; + + initStringInfo(&str); + + a = FunctionCall1(&fmgrinfo, ranges_deserialized->values[idx++]); + b = FunctionCall1(&fmgrinfo, ranges_deserialized->values[idx++]); + + appendStringInfo(&str, "%s ... %s", + DatumGetPointer(a), + DatumGetPointer(b)); + + c = cstring_to_text(str.data); + + astate_values = accumArrayResult(astate_values, + PointerGetDatum(c), + false, + TEXTOID, + CurrentMemoryContext); + } + + if (ranges_deserialized->nranges > 0) + { + Oid typoutput; + bool typIsVarlena; + Datum val; + char *extval; + + getTypeOutputInfo(ANYARRAYOID, &typoutput, &typIsVarlena); + + val = PointerGetDatum(makeArrayResult(astate_values, CurrentMemoryContext)); + + extval = OidOutputFunctionCall(typoutput, val); + + appendStringInfo(&str, " ranges: %s", extval); + } + + /* serialize individual values */ + astate_values = NULL; + + for (i = 0; i < ranges_deserialized->nvalues; i++) + { + Datum a; + text *b; + StringInfoData str; + + initStringInfo(&str); + + a = FunctionCall1(&fmgrinfo, ranges_deserialized->values[idx++]); + + appendStringInfo(&str, "%s", DatumGetPointer(a)); + + b = cstring_to_text(str.data); + + astate_values = accumArrayResult(astate_values, + PointerGetDatum(b), + false, + TEXTOID, + CurrentMemoryContext); + } + + if (ranges_deserialized->nvalues > 0) + { + Oid typoutput; + bool typIsVarlena; + Datum val; + char *extval; + + getTypeOutputInfo(ANYARRAYOID, &typoutput, &typIsVarlena); + + val = PointerGetDatum(makeArrayResult(astate_values, CurrentMemoryContext)); + + extval = OidOutputFunctionCall(typoutput, val); + + appendStringInfo(&str, " values: %s", extval); + } + + + appendStringInfoChar(&str, '}'); + + PG_RETURN_CSTRING(str.data); +} + +/* + * brin_minmax_multi_summary_recv + * - binary input routine for type brin_minmax_multi_summary. + */ +Datum +brin_minmax_multi_summary_recv(PG_FUNCTION_ARGS) +{ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot accept a value of type %s", "brin_minmax_multi_summary"))); + + PG_RETURN_VOID(); /* keep compiler quiet */ +} + +/* + * brin_minmax_multi_summary_send + * - binary output routine for type brin_minmax_multi_summary. + * + * BRIN minmax-multi summaries are serialized in a bytea value (although + * the type is named differently), so let's just send that. + */ +Datum +brin_minmax_multi_summary_send(PG_FUNCTION_ARGS) +{ + return byteasend(fcinfo); +} diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..df820db029 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -159,6 +159,14 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, if (tuple->bt_columns[keyno].bv_hasnulls) anynulls = true; + /* if needed, serialize the values before forming the on-disk tuple */ + if (tuple->bt_columns[keyno].bv_serialize) + { + tuple->bt_columns[keyno].bv_serialize( + tuple->bt_columns[keyno].bv_mem_value, + tuple->bt_columns[keyno].bv_values); + } + /* * Now obtain the values of each stored datum. Note that some values * might be toasted, and we cannot rely on the original heap values @@ -495,6 +503,11 @@ brin_memtuple_initialize(BrinMemTuple *dtuple, BrinDesc *brdesc) dtuple->bt_columns[i].bv_allnulls = true; dtuple->bt_columns[i].bv_hasnulls = false; dtuple->bt_columns[i].bv_values = (Datum *) currdatum; + + dtuple->bt_columns[i].bv_mem_value = PointerGetDatum(NULL); + dtuple->bt_columns[i].bv_serialize = NULL; + dtuple->bt_columns[i].bv_context = dtuple->bt_context; + currdatum += sizeof(Datum) * brdesc->bd_info[i]->oi_nstored; } @@ -574,6 +587,10 @@ brin_deform_tuple(BrinDesc *brdesc, BrinTuple *tuple, BrinMemTuple *dMemtuple) dtup->bt_columns[keyno].bv_hasnulls = hasnulls[keyno]; dtup->bt_columns[keyno].bv_allnulls = false; + + dtup->bt_columns[keyno].bv_mem_value = PointerGetDatum(NULL); + dtup->bt_columns[keyno].bv_serialize = NULL; + dtup->bt_columns[keyno].bv_context = dtup->bt_context; } MemoryContextSwitchTo(oldcxt); diff --git a/src/include/access/brin.h b/src/include/access/brin.h index 0e52d75457..9cd5fa9f62 100644 --- a/src/include/access/brin.h +++ b/src/include/access/brin.h @@ -24,6 +24,7 @@ typedef struct BrinOptions bool autosummarize; double nDistinctPerRange; /* number of distinct values per range */ double falsePositiveRate; /* false positive for bloom filter */ + int valuesPerRange; /* number of values per range */ } BrinOptions; diff --git a/src/include/access/brin_internal.h b/src/include/access/brin_internal.h index d114ac58cd..e7ce4d0209 100644 --- a/src/include/access/brin_internal.h +++ b/src/include/access/brin_internal.h @@ -62,6 +62,9 @@ typedef struct BrinDesc double bd_nDistinctPerRange; double bd_falsePositiveRange; + /* parameters for multi-minmax indexes */ + int bd_valuesPerRange; + /* per-column info; bd_tupdesc->natts entries long */ BrinOpcInfo *bd_info[FLEXIBLE_ARRAY_MEMBER]; } BrinDesc; diff --git a/src/include/access/brin_tuple.h b/src/include/access/brin_tuple.h index 5715bd58df..66a6c3c792 100644 --- a/src/include/access/brin_tuple.h +++ b/src/include/access/brin_tuple.h @@ -14,6 +14,7 @@ #include "access/brin_internal.h" #include "access/tupdesc.h" +typedef void (*brin_serialize_callback_type) (Datum src, Datum * dst); /* * A BRIN index stores one index tuple per page range. Each index tuple @@ -27,6 +28,9 @@ typedef struct BrinValues bool bv_hasnulls; /* are there any nulls in the page range? */ bool bv_allnulls; /* are all values nulls in the page range? */ Datum *bv_values; /* current accumulated values */ + Datum bv_mem_value; /* expanded accumulated values */ + MemoryContext bv_context; + brin_serialize_callback_type bv_serialize; } BrinValues; /* diff --git a/src/include/access/transam.h b/src/include/access/transam.h index e07dd83550..e2a04d9cb0 100644 --- a/src/include/access/transam.h +++ b/src/include/access/transam.h @@ -186,7 +186,7 @@ FullTransactionIdAdvance(FullTransactionId *dest) * ---------- */ #define FirstGenbkiObjectId 10000 -#define FirstBootstrapObjectId 12000 +#define FirstBootstrapObjectId 13000 #define FirstNormalObjectId 16384 /* diff --git a/src/include/catalog/pg_amop.dat b/src/include/catalog/pg_amop.dat index c48d07f8fc..791c755492 100644 --- a/src/include/catalog/pg_amop.dat +++ b/src/include/catalog/pg_amop.dat @@ -2009,6 +2009,152 @@ amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int4,int8)', amopmethod => 'brin' }, +# minmax multi integer + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int8', amopstrategy => '1', amopopr => '<(int8,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int8', amopstrategy => '2', amopopr => '<=(int8,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int8', amopstrategy => '3', amopopr => '=(int8,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int8', amopstrategy => '4', amopopr => '>=(int8,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int8,int8)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int2', amopstrategy => '1', amopopr => '<(int8,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int2', amopstrategy => '2', amopopr => '<=(int8,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int2', amopstrategy => '3', amopopr => '=(int8,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int2', amopstrategy => '4', amopopr => '>=(int8,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int2', amopstrategy => '5', amopopr => '>(int8,int2)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int4', amopstrategy => '1', amopopr => '<(int8,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int4', amopstrategy => '2', amopopr => '<=(int8,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int4', amopstrategy => '3', amopopr => '=(int8,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int4', amopstrategy => '4', amopopr => '>=(int8,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8', + amoprighttype => 'int4', amopstrategy => '5', amopopr => '>(int8,int4)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int2', amopstrategy => '1', amopopr => '<(int2,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int2', amopstrategy => '2', amopopr => '<=(int2,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int2', amopstrategy => '3', amopopr => '=(int2,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int2', amopstrategy => '4', amopopr => '>=(int2,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int2', amopstrategy => '5', amopopr => '>(int2,int2)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int8', amopstrategy => '1', amopopr => '<(int2,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int8', amopstrategy => '2', amopopr => '<=(int2,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int8', amopstrategy => '3', amopopr => '=(int2,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int8', amopstrategy => '4', amopopr => '>=(int2,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int2,int8)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int4', amopstrategy => '1', amopopr => '<(int2,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int4', amopstrategy => '2', amopopr => '<=(int2,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int4', amopstrategy => '3', amopopr => '=(int2,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int4', amopstrategy => '4', amopopr => '>=(int2,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2', + amoprighttype => 'int4', amopstrategy => '5', amopopr => '>(int2,int4)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int4', amopstrategy => '1', amopopr => '<(int4,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int4', amopstrategy => '2', amopopr => '<=(int4,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int4', amopstrategy => '3', amopopr => '=(int4,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int4', amopstrategy => '4', amopopr => '>=(int4,int4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int4', amopstrategy => '5', amopopr => '>(int4,int4)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int2', amopstrategy => '1', amopopr => '<(int4,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int2', amopstrategy => '2', amopopr => '<=(int4,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int2', amopstrategy => '3', amopopr => '=(int4,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int2', amopstrategy => '4', amopopr => '>=(int4,int2)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int2', amopstrategy => '5', amopopr => '>(int4,int2)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int8', amopstrategy => '1', amopopr => '<(int4,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int8', amopstrategy => '2', amopopr => '<=(int4,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int8', amopstrategy => '3', amopopr => '=(int4,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int8', amopstrategy => '4', amopopr => '>=(int4,int8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4', + amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int4,int8)', + amopmethod => 'brin' }, + # bloom integer { amopfamily => 'brin/integer_bloom_ops', amoplefttype => 'int8', @@ -2086,6 +2232,23 @@ amoprighttype => 'oid', amopstrategy => '5', amopopr => '>(oid,oid)', amopmethod => 'brin' }, +# minmax multi oid +{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid', + amoprighttype => 'oid', amopstrategy => '1', amopopr => '<(oid,oid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid', + amoprighttype => 'oid', amopstrategy => '2', amopopr => '<=(oid,oid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid', + amoprighttype => 'oid', amopstrategy => '3', amopopr => '=(oid,oid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid', + amoprighttype => 'oid', amopstrategy => '4', amopopr => '>=(oid,oid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid', + amoprighttype => 'oid', amopstrategy => '5', amopopr => '>(oid,oid)', + amopmethod => 'brin' }, + # bloom oid { amopfamily => 'brin/oid_bloom_ops', amoplefttype => 'oid', amoprighttype => 'oid', amopstrategy => '1', amopopr => '=(oid,oid)', @@ -2112,6 +2275,22 @@ { amopfamily => 'brin/tid_bloom_ops', amoplefttype => 'tid', amoprighttype => 'tid', amopstrategy => '1', amopopr => '=(tid,tid)', amopmethod => 'brin' }, +# minmax multi tid +{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid', + amoprighttype => 'tid', amopstrategy => '1', amopopr => '<(tid,tid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid', + amoprighttype => 'tid', amopstrategy => '2', amopopr => '<=(tid,tid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid', + amoprighttype => 'tid', amopstrategy => '3', amopopr => '=(tid,tid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid', + amoprighttype => 'tid', amopstrategy => '4', amopopr => '>=(tid,tid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid', + amoprighttype => 'tid', amopstrategy => '5', amopopr => '>(tid,tid)', + amopmethod => 'brin' }, # minmax float (float4, float8) @@ -2179,6 +2358,72 @@ amoprighttype => 'float8', amopstrategy => '5', amopopr => '>(float8,float8)', amopmethod => 'brin' }, +# minmax multi float (float4, float8) + +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float4', amopstrategy => '1', amopopr => '<(float4,float4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float4', amopstrategy => '2', + amopopr => '<=(float4,float4)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float4', amopstrategy => '3', amopopr => '=(float4,float4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float4', amopstrategy => '4', + amopopr => '>=(float4,float4)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float4', amopstrategy => '5', amopopr => '>(float4,float4)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float8', amopstrategy => '1', amopopr => '<(float4,float8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float8', amopstrategy => '2', + amopopr => '<=(float4,float8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float8', amopstrategy => '3', amopopr => '=(float4,float8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float8', amopstrategy => '4', + amopopr => '>=(float4,float8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4', + amoprighttype => 'float8', amopstrategy => '5', amopopr => '>(float4,float8)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float4', amopstrategy => '1', amopopr => '<(float8,float4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float4', amopstrategy => '2', + amopopr => '<=(float8,float4)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float4', amopstrategy => '3', amopopr => '=(float8,float4)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float4', amopstrategy => '4', + amopopr => '>=(float8,float4)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float4', amopstrategy => '5', amopopr => '>(float8,float4)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float8', amopstrategy => '1', amopopr => '<(float8,float8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float8', amopstrategy => '2', + amopopr => '<=(float8,float8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float8', amopstrategy => '3', amopopr => '=(float8,float8)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float8', amopstrategy => '4', + amopopr => '>=(float8,float8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8', + amoprighttype => 'float8', amopstrategy => '5', amopopr => '>(float8,float8)', + amopmethod => 'brin' }, + # bloom float { amopfamily => 'brin/float_bloom_ops', amoplefttype => 'float4', amoprighttype => 'float4', amopstrategy => '1', amopopr => '=(float4,float4)', @@ -2210,6 +2455,23 @@ amoprighttype => 'macaddr', amopstrategy => '5', amopopr => '>(macaddr,macaddr)', amopmethod => 'brin' }, +# minmax multi macaddr +{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr', + amoprighttype => 'macaddr', amopstrategy => '1', + amopopr => '<(macaddr,macaddr)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr', + amoprighttype => 'macaddr', amopstrategy => '2', + amopopr => '<=(macaddr,macaddr)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr', + amoprighttype => 'macaddr', amopstrategy => '3', + amopopr => '=(macaddr,macaddr)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr', + amoprighttype => 'macaddr', amopstrategy => '4', + amopopr => '>=(macaddr,macaddr)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr', + amoprighttype => 'macaddr', amopstrategy => '5', + amopopr => '>(macaddr,macaddr)', amopmethod => 'brin' }, + # bloom macaddr { amopfamily => 'brin/macaddr_bloom_ops', amoplefttype => 'macaddr', amoprighttype => 'macaddr', amopstrategy => '1', @@ -2232,6 +2494,23 @@ amoprighttype => 'macaddr8', amopstrategy => '5', amopopr => '>(macaddr8,macaddr8)', amopmethod => 'brin' }, +# minmax multi macaddr8 +{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8', + amoprighttype => 'macaddr8', amopstrategy => '1', + amopopr => '<(macaddr8,macaddr8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8', + amoprighttype => 'macaddr8', amopstrategy => '2', + amopopr => '<=(macaddr8,macaddr8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8', + amoprighttype => 'macaddr8', amopstrategy => '3', + amopopr => '=(macaddr8,macaddr8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8', + amoprighttype => 'macaddr8', amopstrategy => '4', + amopopr => '>=(macaddr8,macaddr8)', amopmethod => 'brin' }, +{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8', + amoprighttype => 'macaddr8', amopstrategy => '5', + amopopr => '>(macaddr8,macaddr8)', amopmethod => 'brin' }, + # bloom macaddr8 { amopfamily => 'brin/macaddr8_bloom_ops', amoplefttype => 'macaddr8', amoprighttype => 'macaddr8', amopstrategy => '1', @@ -2254,6 +2533,23 @@ amoprighttype => 'inet', amopstrategy => '5', amopopr => '>(inet,inet)', amopmethod => 'brin' }, +# minmax multi inet +{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet', + amoprighttype => 'inet', amopstrategy => '1', amopopr => '<(inet,inet)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet', + amoprighttype => 'inet', amopstrategy => '2', amopopr => '<=(inet,inet)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet', + amoprighttype => 'inet', amopstrategy => '3', amopopr => '=(inet,inet)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet', + amoprighttype => 'inet', amopstrategy => '4', amopopr => '>=(inet,inet)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet', + amoprighttype => 'inet', amopstrategy => '5', amopopr => '>(inet,inet)', + amopmethod => 'brin' }, + # bloom inet { amopfamily => 'brin/network_bloom_ops', amoplefttype => 'inet', amoprighttype => 'inet', amopstrategy => '1', amopopr => '=(inet,inet)', @@ -2318,6 +2614,23 @@ amoprighttype => 'time', amopstrategy => '5', amopopr => '>(time,time)', amopmethod => 'brin' }, +# minmax multi time without time zone +{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time', + amoprighttype => 'time', amopstrategy => '1', amopopr => '<(time,time)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time', + amoprighttype => 'time', amopstrategy => '2', amopopr => '<=(time,time)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time', + amoprighttype => 'time', amopstrategy => '3', amopopr => '=(time,time)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time', + amoprighttype => 'time', amopstrategy => '4', amopopr => '>=(time,time)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time', + amoprighttype => 'time', amopstrategy => '5', amopopr => '>(time,time)', + amopmethod => 'brin' }, + # bloom time without time zone { amopfamily => 'brin/time_bloom_ops', amoplefttype => 'time', amoprighttype => 'time', amopstrategy => '1', amopopr => '=(time,time)', @@ -2469,6 +2782,152 @@ amoprighttype => 'timestamptz', amopstrategy => '5', amopopr => '>(timestamptz,timestamptz)', amopmethod => 'brin' }, +# minmax multi datetime (date, timestamp, timestamptz) + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamp', amopstrategy => '1', + amopopr => '<(timestamp,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamp', amopstrategy => '2', + amopopr => '<=(timestamp,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamp', amopstrategy => '3', + amopopr => '=(timestamp,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamp', amopstrategy => '4', + amopopr => '>=(timestamp,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamp', amopstrategy => '5', + amopopr => '>(timestamp,timestamp)', amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'date', amopstrategy => '1', amopopr => '<(timestamp,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'date', amopstrategy => '2', amopopr => '<=(timestamp,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'date', amopstrategy => '3', amopopr => '=(timestamp,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'date', amopstrategy => '4', amopopr => '>=(timestamp,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'date', amopstrategy => '5', amopopr => '>(timestamp,date)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamptz', amopstrategy => '1', + amopopr => '<(timestamp,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamptz', amopstrategy => '2', + amopopr => '<=(timestamp,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamptz', amopstrategy => '3', + amopopr => '=(timestamp,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamptz', amopstrategy => '4', + amopopr => '>=(timestamp,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp', + amoprighttype => 'timestamptz', amopstrategy => '5', + amopopr => '>(timestamp,timestamptz)', amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'date', amopstrategy => '1', amopopr => '<(date,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'date', amopstrategy => '2', amopopr => '<=(date,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'date', amopstrategy => '3', amopopr => '=(date,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'date', amopstrategy => '4', amopopr => '>=(date,date)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'date', amopstrategy => '5', amopopr => '>(date,date)', + amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamp', amopstrategy => '1', + amopopr => '<(date,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamp', amopstrategy => '2', + amopopr => '<=(date,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamp', amopstrategy => '3', + amopopr => '=(date,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamp', amopstrategy => '4', + amopopr => '>=(date,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamp', amopstrategy => '5', + amopopr => '>(date,timestamp)', amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamptz', amopstrategy => '1', + amopopr => '<(date,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamptz', amopstrategy => '2', + amopopr => '<=(date,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamptz', amopstrategy => '3', + amopopr => '=(date,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamptz', amopstrategy => '4', + amopopr => '>=(date,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date', + amoprighttype => 'timestamptz', amopstrategy => '5', + amopopr => '>(date,timestamptz)', amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'date', amopstrategy => '1', + amopopr => '<(timestamptz,date)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'date', amopstrategy => '2', + amopopr => '<=(timestamptz,date)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'date', amopstrategy => '3', + amopopr => '=(timestamptz,date)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'date', amopstrategy => '4', + amopopr => '>=(timestamptz,date)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'date', amopstrategy => '5', + amopopr => '>(timestamptz,date)', amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamp', amopstrategy => '1', + amopopr => '<(timestamptz,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamp', amopstrategy => '2', + amopopr => '<=(timestamptz,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamp', amopstrategy => '3', + amopopr => '=(timestamptz,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamp', amopstrategy => '4', + amopopr => '>=(timestamptz,timestamp)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamp', amopstrategy => '5', + amopopr => '>(timestamptz,timestamp)', amopmethod => 'brin' }, + +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamptz', amopstrategy => '1', + amopopr => '<(timestamptz,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamptz', amopstrategy => '2', + amopopr => '<=(timestamptz,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamptz', amopstrategy => '3', + amopopr => '=(timestamptz,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamptz', amopstrategy => '4', + amopopr => '>=(timestamptz,timestamptz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz', + amoprighttype => 'timestamptz', amopstrategy => '5', + amopopr => '>(timestamptz,timestamptz)', amopmethod => 'brin' }, + # bloom datetime (date, timestamp, timestamptz) { amopfamily => 'brin/datetime_bloom_ops', amoplefttype => 'timestamp', @@ -2524,6 +2983,23 @@ amoprighttype => 'interval', amopstrategy => '5', amopopr => '>(interval,interval)', amopmethod => 'brin' }, +# minmax multi interval +{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval', + amoprighttype => 'interval', amopstrategy => '1', + amopopr => '<(interval,interval)', amopmethod => 'brin' }, +{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval', + amoprighttype => 'interval', amopstrategy => '2', + amopopr => '<=(interval,interval)', amopmethod => 'brin' }, +{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval', + amoprighttype => 'interval', amopstrategy => '3', + amopopr => '=(interval,interval)', amopmethod => 'brin' }, +{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval', + amoprighttype => 'interval', amopstrategy => '4', + amopopr => '>=(interval,interval)', amopmethod => 'brin' }, +{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval', + amoprighttype => 'interval', amopstrategy => '5', + amopopr => '>(interval,interval)', amopmethod => 'brin' }, + # bloom interval { amopfamily => 'brin/interval_bloom_ops', amoplefttype => 'interval', amoprighttype => 'interval', amopstrategy => '1', @@ -2546,6 +3022,23 @@ amoprighttype => 'timetz', amopstrategy => '5', amopopr => '>(timetz,timetz)', amopmethod => 'brin' }, +# minmax multi time with time zone +{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz', + amoprighttype => 'timetz', amopstrategy => '1', amopopr => '<(timetz,timetz)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz', + amoprighttype => 'timetz', amopstrategy => '2', + amopopr => '<=(timetz,timetz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz', + amoprighttype => 'timetz', amopstrategy => '3', amopopr => '=(timetz,timetz)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz', + amoprighttype => 'timetz', amopstrategy => '4', + amopopr => '>=(timetz,timetz)', amopmethod => 'brin' }, +{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz', + amoprighttype => 'timetz', amopstrategy => '5', amopopr => '>(timetz,timetz)', + amopmethod => 'brin' }, + # bloom time with time zone { amopfamily => 'brin/timetz_bloom_ops', amoplefttype => 'timetz', amoprighttype => 'timetz', amopstrategy => '1', amopopr => '=(timetz,timetz)', @@ -2602,6 +3095,23 @@ amoprighttype => 'numeric', amopstrategy => '5', amopopr => '>(numeric,numeric)', amopmethod => 'brin' }, +# minmax multi numeric +{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric', + amoprighttype => 'numeric', amopstrategy => '1', + amopopr => '<(numeric,numeric)', amopmethod => 'brin' }, +{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric', + amoprighttype => 'numeric', amopstrategy => '2', + amopopr => '<=(numeric,numeric)', amopmethod => 'brin' }, +{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric', + amoprighttype => 'numeric', amopstrategy => '3', + amopopr => '=(numeric,numeric)', amopmethod => 'brin' }, +{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric', + amoprighttype => 'numeric', amopstrategy => '4', + amopopr => '>=(numeric,numeric)', amopmethod => 'brin' }, +{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric', + amoprighttype => 'numeric', amopstrategy => '5', + amopopr => '>(numeric,numeric)', amopmethod => 'brin' }, + # bloom numeric { amopfamily => 'brin/numeric_bloom_ops', amoplefttype => 'numeric', amoprighttype => 'numeric', amopstrategy => '1', @@ -2624,6 +3134,23 @@ amoprighttype => 'uuid', amopstrategy => '5', amopopr => '>(uuid,uuid)', amopmethod => 'brin' }, +# minmax multi uuid +{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid', + amoprighttype => 'uuid', amopstrategy => '1', amopopr => '<(uuid,uuid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid', + amoprighttype => 'uuid', amopstrategy => '2', amopopr => '<=(uuid,uuid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid', + amoprighttype => 'uuid', amopstrategy => '3', amopopr => '=(uuid,uuid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid', + amoprighttype => 'uuid', amopstrategy => '4', amopopr => '>=(uuid,uuid)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid', + amoprighttype => 'uuid', amopstrategy => '5', amopopr => '>(uuid,uuid)', + amopmethod => 'brin' }, + # bloom uuid { amopfamily => 'brin/uuid_bloom_ops', amoplefttype => 'uuid', amoprighttype => 'uuid', amopstrategy => '1', amopopr => '=(uuid,uuid)', @@ -2690,6 +3217,23 @@ amoprighttype => 'pg_lsn', amopstrategy => '5', amopopr => '>(pg_lsn,pg_lsn)', amopmethod => 'brin' }, +# minmax multi pg_lsn +{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn', + amoprighttype => 'pg_lsn', amopstrategy => '1', amopopr => '<(pg_lsn,pg_lsn)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn', + amoprighttype => 'pg_lsn', amopstrategy => '2', + amopopr => '<=(pg_lsn,pg_lsn)', amopmethod => 'brin' }, +{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn', + amoprighttype => 'pg_lsn', amopstrategy => '3', amopopr => '=(pg_lsn,pg_lsn)', + amopmethod => 'brin' }, +{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn', + amoprighttype => 'pg_lsn', amopstrategy => '4', + amopopr => '>=(pg_lsn,pg_lsn)', amopmethod => 'brin' }, +{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn', + amoprighttype => 'pg_lsn', amopstrategy => '5', amopopr => '>(pg_lsn,pg_lsn)', + amopmethod => 'brin' }, + # bloom pg_lsn { amopfamily => 'brin/pg_lsn_bloom_ops', amoplefttype => 'pg_lsn', amoprighttype => 'pg_lsn', amopstrategy => '1', amopopr => '=(pg_lsn,pg_lsn)', diff --git a/src/include/catalog/pg_amproc.dat b/src/include/catalog/pg_amproc.dat index d9308c1337..41bd834770 100644 --- a/src/include/catalog/pg_amproc.dat +++ b/src/include/catalog/pg_amproc.dat @@ -982,6 +982,152 @@ { amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4', amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi integer: int2, int4, int8 +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int2', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int2', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int2', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int2', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int4', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int4', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int4', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int4', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8', + amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int2' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int8', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int8', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int8', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int8', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int4', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int4', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int4', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int4', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2', + amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int8', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int8', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int8', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int8', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int2', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int2', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int2', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int2', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4', + amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' }, + # bloom integer: int2, int4, int8 { amprocfamily => 'brin/integer_bloom_ops', amproclefttype => 'int8', amprocrighttype => 'int8', amprocnum => '1', @@ -1077,6 +1223,23 @@ { amprocfamily => 'brin/oid_minmax_ops', amproclefttype => 'oid', amprocrighttype => 'oid', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi oid +{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '1', amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid', + amprocrighttype => 'oid', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' }, + # bloom oid { amprocfamily => 'brin/oid_bloom_ops', amproclefttype => 'oid', amprocrighttype => 'oid', amprocnum => '1', amproc => 'brin_bloom_opcinfo' }, @@ -1123,6 +1286,23 @@ { amprocfamily => 'brin/tid_bloom_ops', amproclefttype => 'tid', amprocrighttype => 'tid', amprocnum => '11', amproc => 'hashtid' }, +# minmax multi tid +{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '1', amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid', + amprocrighttype => 'tid', amprocnum => '11', amproc => 'brin_minmax_multi_distance_tid' }, + # minmax float { amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float4', amprocrighttype => 'float4', amprocnum => '1', @@ -1173,6 +1353,80 @@ amprocrighttype => 'float4', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi float +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float4', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float4' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float8', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float8', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float8', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float8', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float8', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4', + amprocrighttype => 'float8', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float8', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float4', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float4', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float4', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float4', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float4', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8', + amprocrighttype => 'float4', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, + # bloom float { amprocfamily => 'brin/float_bloom_ops', amproclefttype => 'float4', amprocrighttype => 'float4', amprocnum => '1', @@ -1226,6 +1480,26 @@ amprocrighttype => 'macaddr', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi macaddr +{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr', + amprocrighttype => 'macaddr', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_macaddr' }, + # bloom macaddr { amprocfamily => 'brin/macaddr_bloom_ops', amproclefttype => 'macaddr', amprocrighttype => 'macaddr', amprocnum => '1', @@ -1260,6 +1534,26 @@ amprocrighttype => 'macaddr8', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi macaddr8 +{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8', + amprocrighttype => 'macaddr8', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_macaddr8' }, + # bloom macaddr8 { amprocfamily => 'brin/macaddr8_bloom_ops', amproclefttype => 'macaddr8', amprocrighttype => 'macaddr8', amprocnum => '1', @@ -1293,6 +1587,26 @@ { amprocfamily => 'brin/network_minmax_ops', amproclefttype => 'inet', amprocrighttype => 'inet', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi inet +{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet', + amprocrighttype => 'inet', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_inet' }, + # bloom inet { amprocfamily => 'brin/network_bloom_ops', amproclefttype => 'inet', amprocrighttype => 'inet', amprocnum => '1', @@ -1378,6 +1692,25 @@ { amprocfamily => 'brin/time_minmax_ops', amproclefttype => 'time', amprocrighttype => 'time', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi time without time zone +{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '4', amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time', + amprocrighttype => 'time', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_time' }, + # bloom time without time zone { amprocfamily => 'brin/time_bloom_ops', amproclefttype => 'time', amprocrighttype => 'time', amprocnum => '1', @@ -1503,6 +1836,170 @@ amprocrighttype => 'timestamptz', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi datetime (date, timestamp, timestamptz) +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamp', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamptz', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamptz', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamptz', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamptz', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamptz', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'timestamptz', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'date', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'date', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'date', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'date', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'date', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp', + amprocrighttype => 'date', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamptz', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamp', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamp', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamp', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamp', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamp', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'timestamp', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'date', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'date', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'date', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'date', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'date', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz', + amprocrighttype => 'date', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'date', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamp', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamp', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamp', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamp', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamp', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamp', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamptz', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamptz', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamptz', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamptz', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamptz', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date', + amprocrighttype => 'timestamptz', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_float8' }, + # bloom datetime (date, timestamp, timestamptz) { amprocfamily => 'brin/datetime_bloom_ops', amproclefttype => 'timestamp', amprocrighttype => 'timestamp', amprocnum => '1', @@ -1573,6 +2070,26 @@ amprocrighttype => 'interval', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi interval +{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval', + amprocrighttype => 'interval', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_interval' }, + # bloom interval { amprocfamily => 'brin/interval_bloom_ops', amproclefttype => 'interval', amprocrighttype => 'interval', amprocnum => '1', @@ -1607,6 +2124,26 @@ amprocrighttype => 'timetz', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi time with time zone +{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz', + amprocrighttype => 'timetz', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_timetz' }, + # bloom time with time zone { amprocfamily => 'brin/timetz_bloom_ops', amproclefttype => 'timetz', amprocrighttype => 'timetz', amprocnum => '1', @@ -1667,6 +2204,26 @@ amprocrighttype => 'numeric', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi numeric +{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric', + amprocrighttype => 'numeric', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_numeric' }, + # bloom numeric { amprocfamily => 'brin/numeric_bloom_ops', amproclefttype => 'numeric', amprocrighttype => 'numeric', amprocnum => '1', @@ -1698,7 +2255,28 @@ amprocrighttype => 'uuid', amprocnum => '3', amproc => 'brin_minmax_consistent' }, { amprocfamily => 'brin/uuid_minmax_ops', amproclefttype => 'uuid', - amprocrighttype => 'uuid', amprocnum => '4', amproc => 'brin_minmax_union' }, + amprocrighttype => 'uuid', amprocnum => '4', + amproc => 'brin_minmax_union' }, + +# minmax multi uuid +{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid', + amprocrighttype => 'uuid', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_uuid' }, # bloom uuid { amprocfamily => 'brin/uuid_bloom_ops', amproclefttype => 'uuid', @@ -1754,6 +2332,26 @@ amprocrighttype => 'pg_lsn', amprocnum => '4', amproc => 'brin_minmax_union' }, +# minmax multi pg_lsn +{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '1', + amproc => 'brin_minmax_multi_opcinfo' }, +{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '2', + amproc => 'brin_minmax_multi_add_value' }, +{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '3', + amproc => 'brin_minmax_multi_consistent' }, +{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '4', + amproc => 'brin_minmax_multi_union' }, +{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '5', + amproc => 'brin_minmax_multi_options' }, +{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn', + amprocrighttype => 'pg_lsn', amprocnum => '11', + amproc => 'brin_minmax_multi_distance_pg_lsn' }, + # bloom pg_lsn { amprocfamily => 'brin/pg_lsn_bloom_ops', amproclefttype => 'pg_lsn', amprocrighttype => 'pg_lsn', amprocnum => '1', diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat index b9e96bf27e..e0e9651cb1 100644 --- a/src/include/catalog/pg_opclass.dat +++ b/src/include/catalog/pg_opclass.dat @@ -283,18 +283,27 @@ { opcmethod => 'brin', opcname => 'int8_minmax_ops', opcfamily => 'brin/integer_minmax_ops', opcintype => 'int8', opckeytype => 'int8' }, +{ opcmethod => 'brin', opcname => 'int8_minmax_multi_ops', + opcfamily => 'brin/integer_minmax_multi_ops', opcintype => 'int8', + opckeytype => 'int8', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'int8_bloom_ops', opcfamily => 'brin/integer_bloom_ops', opcintype => 'int8', opckeytype => 'int8', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'int2_minmax_ops', opcfamily => 'brin/integer_minmax_ops', opcintype => 'int2', opckeytype => 'int2' }, +{ opcmethod => 'brin', opcname => 'int2_minmax_multi_ops', + opcfamily => 'brin/integer_minmax_multi_ops', opcintype => 'int2', + opckeytype => 'int2', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'int2_bloom_ops', opcfamily => 'brin/integer_bloom_ops', opcintype => 'int2', opckeytype => 'int2', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'int4_minmax_ops', opcfamily => 'brin/integer_minmax_ops', opcintype => 'int4', opckeytype => 'int4' }, +{ opcmethod => 'brin', opcname => 'int4_minmax_multi_ops', + opcfamily => 'brin/integer_minmax_multi_ops', opcintype => 'int4', + opckeytype => 'int4', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'int4_bloom_ops', opcfamily => 'brin/integer_bloom_ops', opcintype => 'int4', opckeytype => 'int4', opcdefault => 'f' }, @@ -306,6 +315,9 @@ opckeytype => 'text', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'oid_minmax_ops', opcfamily => 'brin/oid_minmax_ops', opcintype => 'oid', opckeytype => 'oid' }, +{ opcmethod => 'brin', opcname => 'oid_minmax_multi_ops', + opcfamily => 'brin/oid_minmax_multi_ops', opcintype => 'oid', + opckeytype => 'oid', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'oid_bloom_ops', opcfamily => 'brin/oid_bloom_ops', opcintype => 'oid', opckeytype => 'oid', opcdefault => 'f' }, @@ -314,33 +326,51 @@ { opcmethod => 'brin', opcname => 'tid_bloom_ops', opcfamily => 'brin/tid_bloom_ops', opcintype => 'tid', opckeytype => 'tid', opcdefault => 'f'}, +{ opcmethod => 'brin', opcname => 'tid_minmax_multi_ops', + opcfamily => 'brin/tid_minmax_multi_ops', opcintype => 'tid', + opckeytype => 'tid', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'float4_minmax_ops', opcfamily => 'brin/float_minmax_ops', opcintype => 'float4', opckeytype => 'float4' }, +{ opcmethod => 'brin', opcname => 'float4_minmax_multi_ops', + opcfamily => 'brin/float_minmax_multi_ops', opcintype => 'float4', + opckeytype => 'float4', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'float4_bloom_ops', opcfamily => 'brin/float_bloom_ops', opcintype => 'float4', opckeytype => 'float4', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'float8_minmax_ops', opcfamily => 'brin/float_minmax_ops', opcintype => 'float8', opckeytype => 'float8' }, +{ opcmethod => 'brin', opcname => 'float8_minmax_multi_ops', + opcfamily => 'brin/float_minmax_multi_ops', opcintype => 'float8', + opckeytype => 'float8', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'float8_bloom_ops', opcfamily => 'brin/float_bloom_ops', opcintype => 'float8', opckeytype => 'float8', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'macaddr_minmax_ops', opcfamily => 'brin/macaddr_minmax_ops', opcintype => 'macaddr', opckeytype => 'macaddr' }, +{ opcmethod => 'brin', opcname => 'macaddr_minmax_multi_ops', + opcfamily => 'brin/macaddr_minmax_multi_ops', opcintype => 'macaddr', + opckeytype => 'macaddr', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'macaddr_bloom_ops', opcfamily => 'brin/macaddr_bloom_ops', opcintype => 'macaddr', opckeytype => 'macaddr', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'macaddr8_minmax_ops', opcfamily => 'brin/macaddr8_minmax_ops', opcintype => 'macaddr8', opckeytype => 'macaddr8' }, +{ opcmethod => 'brin', opcname => 'macaddr8_minmax_multi_ops', + opcfamily => 'brin/macaddr8_minmax_multi_ops', opcintype => 'macaddr8', + opckeytype => 'macaddr8', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'macaddr8_bloom_ops', opcfamily => 'brin/macaddr8_bloom_ops', opcintype => 'macaddr8', opckeytype => 'macaddr8', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'inet_minmax_ops', opcfamily => 'brin/network_minmax_ops', opcintype => 'inet', opcdefault => 'f', opckeytype => 'inet' }, +{ opcmethod => 'brin', opcname => 'inet_minmax_multi_ops', + opcfamily => 'brin/network_minmax_multi_ops', opcintype => 'inet', + opcdefault => 'f', opckeytype => 'inet', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'inet_bloom_ops', opcfamily => 'brin/network_bloom_ops', opcintype => 'inet', opcdefault => 'f', opckeytype => 'inet', opcdefault => 'f' }, @@ -356,36 +386,54 @@ { opcmethod => 'brin', opcname => 'time_minmax_ops', opcfamily => 'brin/time_minmax_ops', opcintype => 'time', opckeytype => 'time' }, +{ opcmethod => 'brin', opcname => 'time_minmax_multi_ops', + opcfamily => 'brin/time_minmax_multi_ops', opcintype => 'time', + opckeytype => 'time', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'time_bloom_ops', opcfamily => 'brin/time_bloom_ops', opcintype => 'time', opckeytype => 'time', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'date_minmax_ops', opcfamily => 'brin/datetime_minmax_ops', opcintype => 'date', opckeytype => 'date' }, +{ opcmethod => 'brin', opcname => 'date_minmax_multi_ops', + opcfamily => 'brin/datetime_minmax_multi_ops', opcintype => 'date', + opckeytype => 'date', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'date_bloom_ops', opcfamily => 'brin/datetime_bloom_ops', opcintype => 'date', opckeytype => 'date', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'timestamp_minmax_ops', opcfamily => 'brin/datetime_minmax_ops', opcintype => 'timestamp', opckeytype => 'timestamp' }, +{ opcmethod => 'brin', opcname => 'timestamp_minmax_multi_ops', + opcfamily => 'brin/datetime_minmax_multi_ops', opcintype => 'timestamp', + opckeytype => 'timestamp', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'timestamp_bloom_ops', opcfamily => 'brin/datetime_bloom_ops', opcintype => 'timestamp', opckeytype => 'timestamp', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'timestamptz_minmax_ops', opcfamily => 'brin/datetime_minmax_ops', opcintype => 'timestamptz', opckeytype => 'timestamptz' }, +{ opcmethod => 'brin', opcname => 'timestamptz_minmax_multi_ops', + opcfamily => 'brin/datetime_minmax_multi_ops', opcintype => 'timestamptz', + opckeytype => 'timestamptz', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'timestamptz_bloom_ops', opcfamily => 'brin/datetime_bloom_ops', opcintype => 'timestamptz', opckeytype => 'timestamptz', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'interval_minmax_ops', opcfamily => 'brin/interval_minmax_ops', opcintype => 'interval', opckeytype => 'interval' }, +{ opcmethod => 'brin', opcname => 'interval_minmax_multi_ops', + opcfamily => 'brin/interval_minmax_multi_ops', opcintype => 'interval', + opckeytype => 'interval', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'interval_bloom_ops', opcfamily => 'brin/interval_bloom_ops', opcintype => 'interval', opckeytype => 'interval', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'timetz_minmax_ops', opcfamily => 'brin/timetz_minmax_ops', opcintype => 'timetz', opckeytype => 'timetz' }, +{ opcmethod => 'brin', opcname => 'timetz_minmax_multi_ops', + opcfamily => 'brin/timetz_minmax_multi_ops', opcintype => 'timetz', + opckeytype => 'timetz', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'timetz_bloom_ops', opcfamily => 'brin/timetz_bloom_ops', opcintype => 'timetz', opckeytype => 'timetz', opcdefault => 'f' }, @@ -397,6 +445,9 @@ { opcmethod => 'brin', opcname => 'numeric_minmax_ops', opcfamily => 'brin/numeric_minmax_ops', opcintype => 'numeric', opckeytype => 'numeric' }, +{ opcmethod => 'brin', opcname => 'numeric_minmax_multi_ops', + opcfamily => 'brin/numeric_minmax_multi_ops', opcintype => 'numeric', + opckeytype => 'numeric', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'numeric_bloom_ops', opcfamily => 'brin/numeric_bloom_ops', opcintype => 'numeric', opckeytype => 'numeric', opcdefault => 'f' }, @@ -406,6 +457,9 @@ { opcmethod => 'brin', opcname => 'uuid_minmax_ops', opcfamily => 'brin/uuid_minmax_ops', opcintype => 'uuid', opckeytype => 'uuid' }, +{ opcmethod => 'brin', opcname => 'uuid_minmax_multi_ops', + opcfamily => 'brin/uuid_minmax_multi_ops', opcintype => 'uuid', + opckeytype => 'uuid', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'uuid_bloom_ops', opcfamily => 'brin/uuid_bloom_ops', opcintype => 'uuid', opckeytype => 'uuid', opcdefault => 'f' }, @@ -415,6 +469,9 @@ { opcmethod => 'brin', opcname => 'pg_lsn_minmax_ops', opcfamily => 'brin/pg_lsn_minmax_ops', opcintype => 'pg_lsn', opckeytype => 'pg_lsn' }, +{ opcmethod => 'brin', opcname => 'pg_lsn_minmax_multi_ops', + opcfamily => 'brin/pg_lsn_minmax_multi_ops', opcintype => 'pg_lsn', + opckeytype => 'pg_lsn', opcdefault => 'f' }, { opcmethod => 'brin', opcname => 'pg_lsn_bloom_ops', opcfamily => 'brin/pg_lsn_bloom_ops', opcintype => 'pg_lsn', opckeytype => 'pg_lsn', opcdefault => 'f' }, diff --git a/src/include/catalog/pg_opfamily.dat b/src/include/catalog/pg_opfamily.dat index 5c294bac89..c00d6d6500 100644 --- a/src/include/catalog/pg_opfamily.dat +++ b/src/include/catalog/pg_opfamily.dat @@ -182,10 +182,14 @@ opfmethod => 'gin', opfname => 'jsonb_path_ops' }, { oid => '4054', opfmethod => 'brin', opfname => 'integer_minmax_ops' }, +{ oid => '9015', + opfmethod => 'brin', opfname => 'integer_minmax_multi_ops' }, { oid => '8100', opfmethod => 'brin', opfname => 'integer_bloom_ops' }, { oid => '4055', opfmethod => 'brin', opfname => 'numeric_minmax_ops' }, +{ oid => '9016', + opfmethod => 'brin', opfname => 'numeric_minmax_multi_ops' }, { oid => '4056', opfmethod => 'brin', opfname => 'text_minmax_ops' }, { oid => '8101', @@ -194,10 +198,14 @@ opfmethod => 'brin', opfname => 'numeric_bloom_ops' }, { oid => '4058', opfmethod => 'brin', opfname => 'timetz_minmax_ops' }, +{ oid => '9017', + opfmethod => 'brin', opfname => 'timetz_minmax_multi_ops' }, { oid => '8103', opfmethod => 'brin', opfname => 'timetz_bloom_ops' }, { oid => '4059', opfmethod => 'brin', opfname => 'datetime_minmax_ops' }, +{ oid => '9018', + opfmethod => 'brin', opfname => 'datetime_minmax_multi_ops' }, { oid => '8104', opfmethod => 'brin', opfname => 'datetime_bloom_ops' }, { oid => '4062', @@ -214,26 +222,38 @@ opfmethod => 'brin', opfname => 'name_bloom_ops' }, { oid => '4068', opfmethod => 'brin', opfname => 'oid_minmax_ops' }, +{ oid => '9019', + opfmethod => 'brin', opfname => 'oid_minmax_multi_ops' }, { oid => '8108', opfmethod => 'brin', opfname => 'oid_bloom_ops' }, { oid => '4069', opfmethod => 'brin', opfname => 'tid_minmax_ops' }, { oid => '8123', opfmethod => 'brin', opfname => 'tid_bloom_ops' }, +{ oid => '9020', + opfmethod => 'brin', opfname => 'tid_minmax_multi_ops' }, { oid => '4070', opfmethod => 'brin', opfname => 'float_minmax_ops' }, +{ oid => '9021', + opfmethod => 'brin', opfname => 'float_minmax_multi_ops' }, { oid => '8109', opfmethod => 'brin', opfname => 'float_bloom_ops' }, { oid => '4074', opfmethod => 'brin', opfname => 'macaddr_minmax_ops' }, +{ oid => '9022', + opfmethod => 'brin', opfname => 'macaddr_minmax_multi_ops' }, { oid => '8110', opfmethod => 'brin', opfname => 'macaddr_bloom_ops' }, { oid => '4109', opfmethod => 'brin', opfname => 'macaddr8_minmax_ops' }, +{ oid => '9023', + opfmethod => 'brin', opfname => 'macaddr8_minmax_multi_ops' }, { oid => '8111', opfmethod => 'brin', opfname => 'macaddr8_bloom_ops' }, { oid => '4075', opfmethod => 'brin', opfname => 'network_minmax_ops' }, +{ oid => '9024', + opfmethod => 'brin', opfname => 'network_minmax_multi_ops' }, { oid => '4102', opfmethod => 'brin', opfname => 'network_inclusion_ops' }, { oid => '8112', @@ -244,10 +264,14 @@ opfmethod => 'brin', opfname => 'bpchar_bloom_ops' }, { oid => '4077', opfmethod => 'brin', opfname => 'time_minmax_ops' }, +{ oid => '9025', + opfmethod => 'brin', opfname => 'time_minmax_multi_ops' }, { oid => '8114', opfmethod => 'brin', opfname => 'time_bloom_ops' }, { oid => '4078', opfmethod => 'brin', opfname => 'interval_minmax_ops' }, +{ oid => '9026', + opfmethod => 'brin', opfname => 'interval_minmax_multi_ops' }, { oid => '8115', opfmethod => 'brin', opfname => 'interval_bloom_ops' }, { oid => '4079', @@ -256,12 +280,16 @@ opfmethod => 'brin', opfname => 'varbit_minmax_ops' }, { oid => '4081', opfmethod => 'brin', opfname => 'uuid_minmax_ops' }, +{ oid => '9027', + opfmethod => 'brin', opfname => 'uuid_minmax_multi_ops' }, { oid => '8116', opfmethod => 'brin', opfname => 'uuid_bloom_ops' }, { oid => '4103', opfmethod => 'brin', opfname => 'range_inclusion_ops' }, { oid => '4082', opfmethod => 'brin', opfname => 'pg_lsn_minmax_ops' }, +{ oid => '9028', + opfmethod => 'brin', opfname => 'pg_lsn_minmax_multi_ops' }, { oid => '8117', opfmethod => 'brin', opfname => 'pg_lsn_bloom_ops' }, { oid => '4104', diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c8c4f449d4..41f76e6104 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -8149,6 +8149,71 @@ proname => 'brin_minmax_union', prorettype => 'bool', proargtypes => 'internal internal internal', prosrc => 'brin_minmax_union' }, +# BRIN minmax multi +{ oid => '9029', descr => 'BRIN multi minmax support', + proname => 'brin_minmax_multi_opcinfo', prorettype => 'internal', + proargtypes => 'internal', prosrc => 'brin_minmax_multi_opcinfo' }, +{ oid => '9030', descr => 'BRIN multi minmax support', + proname => 'brin_minmax_multi_add_value', prorettype => 'bool', + proargtypes => 'internal internal internal internal', + prosrc => 'brin_minmax_multi_add_value' }, +{ oid => '9031', descr => 'BRIN multi minmax support', + proname => 'brin_minmax_multi_consistent', prorettype => 'bool', + proargtypes => 'internal internal internal int4', + prosrc => 'brin_minmax_multi_consistent' }, +{ oid => '9032', descr => 'BRIN multi minmax support', + proname => 'brin_minmax_multi_union', prorettype => 'bool', + proargtypes => 'internal internal internal', prosrc => 'brin_minmax_multi_union' }, +{ oid => '9033', descr => 'BRIN multi minmax support', + proname => 'brin_minmax_multi_options', prorettype => 'void', proisstrict => 'f', + proargtypes => 'internal', prosrc => 'brin_minmax_multi_options' }, + +{ oid => '9000', descr => 'BRIN multi minmax int2 distance', + proname => 'brin_minmax_multi_distance_int2', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_int2' }, +{ oid => '9001', descr => 'BRIN multi minmax int4 distance', + proname => 'brin_minmax_multi_distance_int4', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_int4' }, +{ oid => '9002', descr => 'BRIN multi minmax int8 distance', + proname => 'brin_minmax_multi_distance_int8', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_int8' }, +{ oid => '9003', descr => 'BRIN multi minmax float4 distance', + proname => 'brin_minmax_multi_distance_float4', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_float4' }, +{ oid => '9004', descr => 'BRIN multi minmax float8 distance', + proname => 'brin_minmax_multi_distance_float8', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_float8' }, +{ oid => '9005', descr => 'BRIN multi minmax numeric distance', + proname => 'brin_minmax_multi_distance_numeric', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_numeric' }, +{ oid => '9006', descr => 'BRIN multi minmax tid distance', + proname => 'brin_minmax_multi_distance_tid', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_tid' }, +{ oid => '9007', descr => 'BRIN multi minmax uuid distance', + proname => 'brin_minmax_multi_distance_uuid', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_uuid' }, +{ oid => '9008', descr => 'BRIN multi minmax time distance', + proname => 'brin_minmax_multi_distance_time', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_time' }, +{ oid => '9009', descr => 'BRIN multi minmax interval distance', + proname => 'brin_minmax_multi_distance_interval', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_interval' }, +{ oid => '9010', descr => 'BRIN multi minmax timetz distance', + proname => 'brin_minmax_multi_distance_timetz', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_timetz' }, +{ oid => '9011', descr => 'BRIN multi minmax pg_lsn distance', + proname => 'brin_minmax_multi_distance_pg_lsn', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_pg_lsn' }, +{ oid => '9012', descr => 'BRIN multi minmax macaddr distance', + proname => 'brin_minmax_multi_distance_macaddr', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_macaddr' }, +{ oid => '9013', descr => 'BRIN multi minmax macaddr8 distance', + proname => 'brin_minmax_multi_distance_macaddr8', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_macaddr8' }, +{ oid => '9014', descr => 'BRIN multi minmax inet distance', + proname => 'brin_minmax_multi_distance_inet', prorettype => 'float8', + proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_inet' }, + # BRIN inclusion { oid => '4105', descr => 'BRIN inclusion support', proname => 'brin_inclusion_opcinfo', prorettype => 'internal', @@ -11349,3 +11414,18 @@ { oid => '9038', descr => 'I/O', proname => 'brin_bloom_summary_send', provolatile => 's', prorettype => 'bytea', proargtypes => 'pg_brin_bloom_summary', prosrc => 'brin_bloom_summary_send' }, + + +{ oid => '9040', descr => 'I/O', + proname => 'brin_minmax_multi_summary_in', prorettype => 'pg_brin_minmax_multi_summary', + proargtypes => 'cstring', prosrc => 'brin_minmax_multi_summary_in' }, +{ oid => '9041', descr => 'I/O', + proname => 'brin_minmax_multi_summary_out', prorettype => 'cstring', + proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_out' }, +{ oid => '9042', descr => 'I/O', + proname => 'brin_minmax_multi_summary_recv', provolatile => 's', + prorettype => 'pg_brin_minmax_multi_summary', proargtypes => 'internal', + prosrc => 'brin_minmax_multi_summary_recv' }, +{ oid => '9043', descr => 'I/O', + proname => 'brin_minmax_multi_summary_send', provolatile => 's', prorettype => 'bytea', + proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_send' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 3275719896..93b745757a 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -680,3 +680,10 @@ typinput => 'brin_bloom_summary_in', typoutput => 'brin_bloom_summary_out', typreceive => 'brin_bloom_summary_recv', typsend => 'brin_bloom_summary_send', typalign => 'i', typstorage => 'x', typcollation => 'default' }, + +{ oid => '9039', + descr => 'BRIN minmax-multi summary', + typname => 'pg_brin_minmax_multi_summary', typlen => '-1', typbyval => 'f', typcategory => 'S', + typinput => 'brin_minmax_multi_summary_in', typoutput => 'brin_minmax_multi_summary_out', + typreceive => 'brin_minmax_multi_summary_recv', typsend => 'brin_minmax_multi_summary_send', + typalign => 'i', typstorage => 'x', typcollation => 'default' }, diff --git a/src/test/regress/expected/brin_multi.out b/src/test/regress/expected/brin_multi.out new file mode 100644 index 0000000000..966dc4aadc --- /dev/null +++ b/src/test/regress/expected/brin_multi.out @@ -0,0 +1,421 @@ +CREATE TABLE brintest_multi ( + int8col bigint, + int2col smallint, + int4col integer, + oidcol oid, + tidcol tid, + float4col real, + float8col double precision, + macaddrcol macaddr, + inetcol inet, + cidrcol cidr, + datecol date, + timecol time without time zone, + timestampcol timestamp without time zone, + timestamptzcol timestamp with time zone, + intervalcol interval, + timetzcol time with time zone, + numericcol numeric, + uuidcol uuid, + lsncol pg_lsn +) WITH (fillfactor=10); +INSERT INTO brintest_multi SELECT + 142857 * tenthous, + thousand, + twothousand, + unique1::oid, + format('(%s,%s)', tenthous, twenty)::tid, + (four + 1.0)/(hundred+1), + odd::float8 / (tenthous + 1), + format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, + inet '10.2.3.4/24' + tenthous, + cidr '10.2.3/24' + tenthous, + date '1995-08-15' + tenthous, + time '01:20:30' + thousand * interval '18.5 second', + timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours', + timestamptz '1972-10-10 03:00' + thousand * interval '1 hour', + justify_days(justify_hours(tenthous * interval '12 minutes')), + timetz '01:30:20+02' + hundred * interval '15 seconds', + tenthous::numeric(36,30) * fivethous * even / (hundred + 1), + format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid, + format('%s/%s%s', odd, even, tenthous)::pg_lsn +FROM tenk1 ORDER BY unique2 LIMIT 100; +-- throw in some NULL's and different values +INSERT INTO brintest_multi (inetcol, cidrcol) SELECT + inet 'fe80::6e40:8ff:fea9:8c46' + tenthous, + cidr 'fe80::6e40:8ff:fea9:8c46' + tenthous +FROM tenk1 ORDER BY thousand, tenthous LIMIT 25; +-- test minmax-multi specific index options +-- number of values must be >= 16 +CREATE INDEX brinidx_multi ON brintest_multi USING brin ( + int8col int8_minmax_multi_ops(values_per_range = 15) +); +ERROR: value 15 out of bounds for option "values_per_range" +DETAIL: Valid values are between "16" and "256". +-- number of values must be <= 256 +CREATE INDEX brinidx_multi ON brintest_multi USING brin ( + int8col int8_minmax_multi_ops(values_per_range = 257) +); +ERROR: value 257 out of bounds for option "values_per_range" +DETAIL: Valid values are between "16" and "256". +CREATE INDEX brinidx_multi ON brintest_multi USING brin ( + int8col int8_minmax_multi_ops, + int2col int2_minmax_multi_ops, + int4col int4_minmax_multi_ops, + oidcol oid_minmax_multi_ops, + tidcol tid_minmax_multi_ops, + float4col float4_minmax_multi_ops, + float8col float8_minmax_multi_ops, + macaddrcol macaddr_minmax_multi_ops, + inetcol inet_minmax_multi_ops, + cidrcol inet_minmax_multi_ops, + datecol date_minmax_multi_ops, + timecol time_minmax_multi_ops, + timestampcol timestamp_minmax_multi_ops, + timestamptzcol timestamptz_minmax_multi_ops, + intervalcol interval_minmax_multi_ops, + timetzcol timetz_minmax_multi_ops, + numericcol numeric_minmax_multi_ops, + uuidcol uuid_minmax_multi_ops, + lsncol pg_lsn_minmax_multi_ops +) with (pages_per_range = 1); +CREATE TABLE brinopers_multi (colname name, typ text, + op text[], value text[], matches int[], + check (cardinality(op) = cardinality(value)), + check (cardinality(op) = cardinality(matches))); +INSERT INTO brinopers_multi VALUES + ('int2col', 'int2', + '{>, >=, =, <=, <}', + '{0, 0, 800, 999, 999}', + '{100, 100, 1, 100, 100}'), + ('int2col', 'int4', + '{>, >=, =, <=, <}', + '{0, 0, 800, 999, 1999}', + '{100, 100, 1, 100, 100}'), + ('int2col', 'int8', + '{>, >=, =, <=, <}', + '{0, 0, 800, 999, 1428427143}', + '{100, 100, 1, 100, 100}'), + ('int4col', 'int2', + '{>, >=, =, <=, <}', + '{0, 0, 800, 1999, 1999}', + '{100, 100, 1, 100, 100}'), + ('int4col', 'int4', + '{>, >=, =, <=, <}', + '{0, 0, 800, 1999, 1999}', + '{100, 100, 1, 100, 100}'), + ('int4col', 'int8', + '{>, >=, =, <=, <}', + '{0, 0, 800, 1999, 1428427143}', + '{100, 100, 1, 100, 100}'), + ('int8col', 'int2', + '{>, >=}', + '{0, 0}', + '{100, 100}'), + ('int8col', 'int4', + '{>, >=}', + '{0, 0}', + '{100, 100}'), + ('int8col', 'int8', + '{>, >=, =, <=, <}', + '{0, 0, 1257141600, 1428427143, 1428427143}', + '{100, 100, 1, 100, 100}'), + ('oidcol', 'oid', + '{>, >=, =, <=, <}', + '{0, 0, 8800, 9999, 9999}', + '{100, 100, 1, 100, 100}'), + ('tidcol', 'tid', + '{>, >=, =, <=, <}', + '{"(0,0)", "(0,0)", "(8800,0)", "(9999,19)", "(9999,19)"}', + '{100, 100, 1, 100, 100}'), + ('float4col', 'float4', + '{>, >=, =, <=, <}', + '{0.0103093, 0.0103093, 1, 1, 1}', + '{100, 100, 4, 100, 96}'), + ('float4col', 'float8', + '{>, >=, =, <=, <}', + '{0.0103093, 0.0103093, 1, 1, 1}', + '{100, 100, 4, 100, 96}'), + ('float8col', 'float4', + '{>, >=, =, <=, <}', + '{0, 0, 0, 1.98, 1.98}', + '{99, 100, 1, 100, 100}'), + ('float8col', 'float8', + '{>, >=, =, <=, <}', + '{0, 0, 0, 1.98, 1.98}', + '{99, 100, 1, 100, 100}'), + ('macaddrcol', 'macaddr', + '{>, >=, =, <=, <}', + '{00:00:01:00:00:00, 00:00:01:00:00:00, 2c:00:2d:00:16:00, ff:fe:00:00:00:00, ff:fe:00:00:00:00}', + '{99, 100, 2, 100, 100}'), + ('inetcol', 'inet', + '{=, <, <=, >, >=}', + '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{1, 100, 100, 125, 125}'), + ('inetcol', 'cidr', + '{<, <=, >, >=}', + '{255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{100, 100, 125, 125}'), + ('cidrcol', 'inet', + '{=, <, <=, >, >=}', + '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{2, 100, 100, 125, 125}'), + ('cidrcol', 'cidr', + '{=, <, <=, >, >=}', + '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{2, 100, 100, 125, 125}'), + ('datecol', 'date', + '{>, >=, =, <=, <}', + '{1995-08-15, 1995-08-15, 2009-12-01, 2022-12-30, 2022-12-30}', + '{100, 100, 1, 100, 100}'), + ('timecol', 'time', + '{>, >=, =, <=, <}', + '{01:20:30, 01:20:30, 02:28:57, 06:28:31.5, 06:28:31.5}', + '{100, 100, 1, 100, 100}'), + ('timestampcol', 'timestamp', + '{>, >=, =, <=, <}', + '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}', + '{100, 100, 1, 100, 100}'), + ('timestampcol', 'timestamptz', + '{>, >=, =, <=, <}', + '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}', + '{100, 100, 1, 100, 100}'), + ('timestamptzcol', 'timestamptz', + '{>, >=, =, <=, <}', + '{1972-10-10 03:00:00-04, 1972-10-10 03:00:00-04, 1972-10-19 09:00:00-07, 1972-11-20 19:00:00-03, 1972-11-20 19:00:00-03}', + '{100, 100, 1, 100, 100}'), + ('intervalcol', 'interval', + '{>, >=, =, <=, <}', + '{00:00:00, 00:00:00, 1 mons 13 days 12:24, 2 mons 23 days 07:48:00, 1 year}', + '{100, 100, 1, 100, 100}'), + ('timetzcol', 'timetz', + '{>, >=, =, <=, <}', + '{01:30:20+02, 01:30:20+02, 01:35:50+02, 23:55:05+02, 23:55:05+02}', + '{99, 100, 2, 100, 100}'), + ('numericcol', 'numeric', + '{>, >=, =, <=, <}', + '{0.00, 0.01, 2268164.347826086956521739130434782609, 99470151.9, 99470151.9}', + '{100, 100, 1, 100, 100}'), + ('uuidcol', 'uuid', + '{>, >=, =, <=, <}', + '{00040004-0004-0004-0004-000400040004, 00040004-0004-0004-0004-000400040004, 52225222-5222-5222-5222-522252225222, 99989998-9998-9998-9998-999899989998, 99989998-9998-9998-9998-999899989998}', + '{100, 100, 1, 100, 100}'), + ('lsncol', 'pg_lsn', + '{>, >=, =, <=, <, IS, IS NOT}', + '{0/1200, 0/1200, 44/455222, 198/1999799, 198/1999799, NULL, NULL}', + '{100, 100, 1, 100, 100, 25, 100}'); +DO $x$ +DECLARE + r record; + r2 record; + cond text; + idx_ctids tid[]; + ss_ctids tid[]; + count int; + plan_ok bool; + plan_line text; +BEGIN + FOR r IN SELECT colname, oper, typ, value[ordinality], matches[ordinality] FROM brinopers_multi, unnest(op) WITH ORDINALITY AS oper LOOP + + -- prepare the condition + IF r.value IS NULL THEN + cond := format('%I %s %L', r.colname, r.oper, r.value); + ELSE + cond := format('%I %s %L::%s', r.colname, r.oper, r.value, r.typ); + END IF; + + -- run the query using the brin index + SET enable_seqscan = 0; + SET enable_bitmapscan = 1; + + plan_ok := false; + FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP + IF plan_line LIKE '%Bitmap Heap Scan on brintest_multi%' THEN + plan_ok := true; + END IF; + END LOOP; + IF NOT plan_ok THEN + RAISE WARNING 'did not get bitmap indexscan plan for %', r; + END IF; + + EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) + INTO idx_ctids; + + -- run the query using a seqscan + SET enable_seqscan = 1; + SET enable_bitmapscan = 0; + + plan_ok := false; + FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP + IF plan_line LIKE '%Seq Scan on brintest_multi%' THEN + plan_ok := true; + END IF; + END LOOP; + IF NOT plan_ok THEN + RAISE WARNING 'did not get seqscan plan for %', r; + END IF; + + EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) + INTO ss_ctids; + + -- make sure both return the same results + count := array_length(idx_ctids, 1); + + IF NOT (count = array_length(ss_ctids, 1) AND + idx_ctids @> ss_ctids AND + idx_ctids <@ ss_ctids) THEN + -- report the results of each scan to make the differences obvious + RAISE WARNING 'something not right in %: count %', r, count; + SET enable_seqscan = 1; + SET enable_bitmapscan = 0; + FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP + RAISE NOTICE 'seqscan: %', r2; + END LOOP; + + SET enable_seqscan = 0; + SET enable_bitmapscan = 1; + FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP + RAISE NOTICE 'bitmapscan: %', r2; + END LOOP; + END IF; + + -- make sure we found expected number of matches + IF count != r.matches THEN RAISE WARNING 'unexpected number of results % for %', count, r; END IF; + END LOOP; +END; +$x$; +RESET enable_seqscan; +RESET enable_bitmapscan; +INSERT INTO brintest_multi SELECT + 142857 * tenthous, + thousand, + twothousand, + unique1::oid, + format('(%s,%s)', tenthous, twenty)::tid, + (four + 1.0)/(hundred+1), + odd::float8 / (tenthous + 1), + format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, + inet '10.2.3.4' + tenthous, + cidr '10.2.3/24' + tenthous, + date '1995-08-15' + tenthous, + time '01:20:30' + thousand * interval '18.5 second', + timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours', + timestamptz '1972-10-10 03:00' + thousand * interval '1 hour', + justify_days(justify_hours(tenthous * interval '12 minutes')), + timetz '01:30:20' + hundred * interval '15 seconds', + tenthous::numeric(36,30) * fivethous * even / (hundred + 1), + format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid, + format('%s/%s%s', odd, even, tenthous)::pg_lsn +FROM tenk1 ORDER BY unique2 LIMIT 5 OFFSET 5; +SELECT brin_desummarize_range('brinidx_multi', 0); + brin_desummarize_range +------------------------ + +(1 row) + +VACUUM brintest_multi; -- force a summarization cycle in brinidx +UPDATE brintest_multi SET int8col = int8col * int4col; +-- Tests for brin_summarize_new_values +SELECT brin_summarize_new_values('brintest_multi'); -- error, not an index +ERROR: "brintest_multi" is not an index +SELECT brin_summarize_new_values('tenk1_unique1'); -- error, not a BRIN index +ERROR: "tenk1_unique1" is not a BRIN index +SELECT brin_summarize_new_values('brinidx_multi'); -- ok, no change expected + brin_summarize_new_values +--------------------------- + 0 +(1 row) + +-- Tests for brin_desummarize_range +SELECT brin_desummarize_range('brinidx_multi', -1); -- error, invalid range +ERROR: block number out of range: -1 +SELECT brin_desummarize_range('brinidx_multi', 0); + brin_desummarize_range +------------------------ + +(1 row) + +SELECT brin_desummarize_range('brinidx_multi', 0); + brin_desummarize_range +------------------------ + +(1 row) + +SELECT brin_desummarize_range('brinidx_multi', 100000000); + brin_desummarize_range +------------------------ + +(1 row) + +-- Test brin_summarize_range +CREATE TABLE brin_summarize_multi ( + value int +) WITH (fillfactor=10, autovacuum_enabled=false); +CREATE INDEX brin_summarize_multi_idx ON brin_summarize_multi USING brin (value) WITH (pages_per_range=2); +-- Fill a few pages +DO $$ +DECLARE curtid tid; +BEGIN + LOOP + INSERT INTO brin_summarize_multi VALUES (1) RETURNING ctid INTO curtid; + EXIT WHEN curtid > tid '(2, 0)'; + END LOOP; +END; +$$; +-- summarize one range +SELECT brin_summarize_range('brin_summarize_multi_idx', 0); + brin_summarize_range +---------------------- + 0 +(1 row) + +-- nothing: already summarized +SELECT brin_summarize_range('brin_summarize_multi_idx', 1); + brin_summarize_range +---------------------- + 0 +(1 row) + +-- summarize one range +SELECT brin_summarize_range('brin_summarize_multi_idx', 2); + brin_summarize_range +---------------------- + 1 +(1 row) + +-- nothing: page doesn't exist in table +SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967295); + brin_summarize_range +---------------------- + 0 +(1 row) + +-- invalid block number values +SELECT brin_summarize_range('brin_summarize_multi_idx', -1); +ERROR: block number out of range: -1 +SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967296); +ERROR: block number out of range: 4294967296 +-- test brin cost estimates behave sanely based on correlation of values +CREATE TABLE brin_test_multi (a INT, b INT); +INSERT INTO brin_test_multi SELECT x/100,x%100 FROM generate_series(1,10000) x(x); +CREATE INDEX brin_test_multi_a_idx ON brin_test_multi USING brin (a) WITH (pages_per_range = 2); +CREATE INDEX brin_test_multi_b_idx ON brin_test_multi USING brin (b) WITH (pages_per_range = 2); +VACUUM ANALYZE brin_test_multi; +-- Ensure brin index is used when columns are perfectly correlated +EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE a = 1; + QUERY PLAN +-------------------------------------------------- + Bitmap Heap Scan on brin_test_multi + Recheck Cond: (a = 1) + -> Bitmap Index Scan on brin_test_multi_a_idx + Index Cond: (a = 1) +(4 rows) + +-- Ensure brin index is not used when values are not correlated +EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE b = 1; + QUERY PLAN +----------------------------- + Seq Scan on brin_test_multi + Filter: (b = 1) +(2 rows) + diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index a7a5b22d4f..76050af797 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4990,12 +4990,13 @@ List of access methods (0 rows) \dAc brin pg*.oid* - List of operator classes - AM | Input type | Storage type | Operator class | Default? -------+------------+--------------+----------------+---------- - brin | oid | | oid_bloom_ops | no - brin | oid | | oid_minmax_ops | yes -(2 rows) + List of operator classes + AM | Input type | Storage type | Operator class | Default? +------+------------+--------------+----------------------+---------- + brin | oid | | oid_bloom_ops | no + brin | oid | | oid_minmax_multi_ops | no + brin | oid | | oid_minmax_ops | yes +(3 rows) \dAf spgist List of operator families diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out index a44bde2e6e..0ca2671910 100644 --- a/src/test/regress/expected/type_sanity.out +++ b/src/test/regress/expected/type_sanity.out @@ -67,14 +67,15 @@ WHERE p1.typtype not in ('p') AND p1.typname NOT LIKE E'\\_%' WHERE p2.typname = ('_' || p1.typname)::name AND p2.typelem = p1.oid and p1.typarray = p2.oid) ORDER BY p1.oid; - oid | typname -------+----------------------- + oid | typname +------+------------------------------ 194 | pg_node_tree 3361 | pg_ndistinct 3402 | pg_dependencies 5017 | pg_mcv_list 9034 | pg_brin_bloom_summary -(5 rows) + 9039 | pg_brin_minmax_multi_summary +(6 rows) -- Make sure typarray points to a "true" array type of our own base SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype, diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index ef372281ef..db078e5201 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -80,7 +80,7 @@ test: brin gin gist spgist privileges init_privs security_label collate matview # ---------- # Additional BRIN tests # ---------- -test: brin_bloom +test: brin_bloom brin_multi # ---------- # Another group of parallel tests diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index fdddc48f62..daf2e898cb 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -110,6 +110,7 @@ test: namespace test: prepared_xacts test: brin test: brin_bloom +test: brin_multi test: gin test: gist test: spgist diff --git a/src/test/regress/sql/brin_multi.sql b/src/test/regress/sql/brin_multi.sql new file mode 100644 index 0000000000..9de6ddc6d7 --- /dev/null +++ b/src/test/regress/sql/brin_multi.sql @@ -0,0 +1,371 @@ +CREATE TABLE brintest_multi ( + int8col bigint, + int2col smallint, + int4col integer, + oidcol oid, + tidcol tid, + float4col real, + float8col double precision, + macaddrcol macaddr, + inetcol inet, + cidrcol cidr, + datecol date, + timecol time without time zone, + timestampcol timestamp without time zone, + timestamptzcol timestamp with time zone, + intervalcol interval, + timetzcol time with time zone, + numericcol numeric, + uuidcol uuid, + lsncol pg_lsn +) WITH (fillfactor=10); + +INSERT INTO brintest_multi SELECT + 142857 * tenthous, + thousand, + twothousand, + unique1::oid, + format('(%s,%s)', tenthous, twenty)::tid, + (four + 1.0)/(hundred+1), + odd::float8 / (tenthous + 1), + format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, + inet '10.2.3.4/24' + tenthous, + cidr '10.2.3/24' + tenthous, + date '1995-08-15' + tenthous, + time '01:20:30' + thousand * interval '18.5 second', + timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours', + timestamptz '1972-10-10 03:00' + thousand * interval '1 hour', + justify_days(justify_hours(tenthous * interval '12 minutes')), + timetz '01:30:20+02' + hundred * interval '15 seconds', + tenthous::numeric(36,30) * fivethous * even / (hundred + 1), + format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid, + format('%s/%s%s', odd, even, tenthous)::pg_lsn +FROM tenk1 ORDER BY unique2 LIMIT 100; + +-- throw in some NULL's and different values +INSERT INTO brintest_multi (inetcol, cidrcol) SELECT + inet 'fe80::6e40:8ff:fea9:8c46' + tenthous, + cidr 'fe80::6e40:8ff:fea9:8c46' + tenthous +FROM tenk1 ORDER BY thousand, tenthous LIMIT 25; + +-- test minmax-multi specific index options +-- number of values must be >= 16 +CREATE INDEX brinidx_multi ON brintest_multi USING brin ( + int8col int8_minmax_multi_ops(values_per_range = 15) +); +-- number of values must be <= 256 +CREATE INDEX brinidx_multi ON brintest_multi USING brin ( + int8col int8_minmax_multi_ops(values_per_range = 257) +); + +CREATE INDEX brinidx_multi ON brintest_multi USING brin ( + int8col int8_minmax_multi_ops, + int2col int2_minmax_multi_ops, + int4col int4_minmax_multi_ops, + oidcol oid_minmax_multi_ops, + tidcol tid_minmax_multi_ops, + float4col float4_minmax_multi_ops, + float8col float8_minmax_multi_ops, + macaddrcol macaddr_minmax_multi_ops, + inetcol inet_minmax_multi_ops, + cidrcol inet_minmax_multi_ops, + datecol date_minmax_multi_ops, + timecol time_minmax_multi_ops, + timestampcol timestamp_minmax_multi_ops, + timestamptzcol timestamptz_minmax_multi_ops, + intervalcol interval_minmax_multi_ops, + timetzcol timetz_minmax_multi_ops, + numericcol numeric_minmax_multi_ops, + uuidcol uuid_minmax_multi_ops, + lsncol pg_lsn_minmax_multi_ops +) with (pages_per_range = 1); + +CREATE TABLE brinopers_multi (colname name, typ text, + op text[], value text[], matches int[], + check (cardinality(op) = cardinality(value)), + check (cardinality(op) = cardinality(matches))); + +INSERT INTO brinopers_multi VALUES + ('int2col', 'int2', + '{>, >=, =, <=, <}', + '{0, 0, 800, 999, 999}', + '{100, 100, 1, 100, 100}'), + ('int2col', 'int4', + '{>, >=, =, <=, <}', + '{0, 0, 800, 999, 1999}', + '{100, 100, 1, 100, 100}'), + ('int2col', 'int8', + '{>, >=, =, <=, <}', + '{0, 0, 800, 999, 1428427143}', + '{100, 100, 1, 100, 100}'), + ('int4col', 'int2', + '{>, >=, =, <=, <}', + '{0, 0, 800, 1999, 1999}', + '{100, 100, 1, 100, 100}'), + ('int4col', 'int4', + '{>, >=, =, <=, <}', + '{0, 0, 800, 1999, 1999}', + '{100, 100, 1, 100, 100}'), + ('int4col', 'int8', + '{>, >=, =, <=, <}', + '{0, 0, 800, 1999, 1428427143}', + '{100, 100, 1, 100, 100}'), + ('int8col', 'int2', + '{>, >=}', + '{0, 0}', + '{100, 100}'), + ('int8col', 'int4', + '{>, >=}', + '{0, 0}', + '{100, 100}'), + ('int8col', 'int8', + '{>, >=, =, <=, <}', + '{0, 0, 1257141600, 1428427143, 1428427143}', + '{100, 100, 1, 100, 100}'), + ('oidcol', 'oid', + '{>, >=, =, <=, <}', + '{0, 0, 8800, 9999, 9999}', + '{100, 100, 1, 100, 100}'), + ('tidcol', 'tid', + '{>, >=, =, <=, <}', + '{"(0,0)", "(0,0)", "(8800,0)", "(9999,19)", "(9999,19)"}', + '{100, 100, 1, 100, 100}'), + ('float4col', 'float4', + '{>, >=, =, <=, <}', + '{0.0103093, 0.0103093, 1, 1, 1}', + '{100, 100, 4, 100, 96}'), + ('float4col', 'float8', + '{>, >=, =, <=, <}', + '{0.0103093, 0.0103093, 1, 1, 1}', + '{100, 100, 4, 100, 96}'), + ('float8col', 'float4', + '{>, >=, =, <=, <}', + '{0, 0, 0, 1.98, 1.98}', + '{99, 100, 1, 100, 100}'), + ('float8col', 'float8', + '{>, >=, =, <=, <}', + '{0, 0, 0, 1.98, 1.98}', + '{99, 100, 1, 100, 100}'), + ('macaddrcol', 'macaddr', + '{>, >=, =, <=, <}', + '{00:00:01:00:00:00, 00:00:01:00:00:00, 2c:00:2d:00:16:00, ff:fe:00:00:00:00, ff:fe:00:00:00:00}', + '{99, 100, 2, 100, 100}'), + ('inetcol', 'inet', + '{=, <, <=, >, >=}', + '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{1, 100, 100, 125, 125}'), + ('inetcol', 'cidr', + '{<, <=, >, >=}', + '{255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{100, 100, 125, 125}'), + ('cidrcol', 'inet', + '{=, <, <=, >, >=}', + '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{2, 100, 100, 125, 125}'), + ('cidrcol', 'cidr', + '{=, <, <=, >, >=}', + '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', + '{2, 100, 100, 125, 125}'), + ('datecol', 'date', + '{>, >=, =, <=, <}', + '{1995-08-15, 1995-08-15, 2009-12-01, 2022-12-30, 2022-12-30}', + '{100, 100, 1, 100, 100}'), + ('timecol', 'time', + '{>, >=, =, <=, <}', + '{01:20:30, 01:20:30, 02:28:57, 06:28:31.5, 06:28:31.5}', + '{100, 100, 1, 100, 100}'), + ('timestampcol', 'timestamp', + '{>, >=, =, <=, <}', + '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}', + '{100, 100, 1, 100, 100}'), + ('timestampcol', 'timestamptz', + '{>, >=, =, <=, <}', + '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}', + '{100, 100, 1, 100, 100}'), + ('timestamptzcol', 'timestamptz', + '{>, >=, =, <=, <}', + '{1972-10-10 03:00:00-04, 1972-10-10 03:00:00-04, 1972-10-19 09:00:00-07, 1972-11-20 19:00:00-03, 1972-11-20 19:00:00-03}', + '{100, 100, 1, 100, 100}'), + ('intervalcol', 'interval', + '{>, >=, =, <=, <}', + '{00:00:00, 00:00:00, 1 mons 13 days 12:24, 2 mons 23 days 07:48:00, 1 year}', + '{100, 100, 1, 100, 100}'), + ('timetzcol', 'timetz', + '{>, >=, =, <=, <}', + '{01:30:20+02, 01:30:20+02, 01:35:50+02, 23:55:05+02, 23:55:05+02}', + '{99, 100, 2, 100, 100}'), + ('numericcol', 'numeric', + '{>, >=, =, <=, <}', + '{0.00, 0.01, 2268164.347826086956521739130434782609, 99470151.9, 99470151.9}', + '{100, 100, 1, 100, 100}'), + ('uuidcol', 'uuid', + '{>, >=, =, <=, <}', + '{00040004-0004-0004-0004-000400040004, 00040004-0004-0004-0004-000400040004, 52225222-5222-5222-5222-522252225222, 99989998-9998-9998-9998-999899989998, 99989998-9998-9998-9998-999899989998}', + '{100, 100, 1, 100, 100}'), + ('lsncol', 'pg_lsn', + '{>, >=, =, <=, <, IS, IS NOT}', + '{0/1200, 0/1200, 44/455222, 198/1999799, 198/1999799, NULL, NULL}', + '{100, 100, 1, 100, 100, 25, 100}'); + +DO $x$ +DECLARE + r record; + r2 record; + cond text; + idx_ctids tid[]; + ss_ctids tid[]; + count int; + plan_ok bool; + plan_line text; +BEGIN + FOR r IN SELECT colname, oper, typ, value[ordinality], matches[ordinality] FROM brinopers_multi, unnest(op) WITH ORDINALITY AS oper LOOP + + -- prepare the condition + IF r.value IS NULL THEN + cond := format('%I %s %L', r.colname, r.oper, r.value); + ELSE + cond := format('%I %s %L::%s', r.colname, r.oper, r.value, r.typ); + END IF; + + -- run the query using the brin index + SET enable_seqscan = 0; + SET enable_bitmapscan = 1; + + plan_ok := false; + FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP + IF plan_line LIKE '%Bitmap Heap Scan on brintest_multi%' THEN + plan_ok := true; + END IF; + END LOOP; + IF NOT plan_ok THEN + RAISE WARNING 'did not get bitmap indexscan plan for %', r; + END IF; + + EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) + INTO idx_ctids; + + -- run the query using a seqscan + SET enable_seqscan = 1; + SET enable_bitmapscan = 0; + + plan_ok := false; + FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP + IF plan_line LIKE '%Seq Scan on brintest_multi%' THEN + plan_ok := true; + END IF; + END LOOP; + IF NOT plan_ok THEN + RAISE WARNING 'did not get seqscan plan for %', r; + END IF; + + EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) + INTO ss_ctids; + + -- make sure both return the same results + count := array_length(idx_ctids, 1); + + IF NOT (count = array_length(ss_ctids, 1) AND + idx_ctids @> ss_ctids AND + idx_ctids <@ ss_ctids) THEN + -- report the results of each scan to make the differences obvious + RAISE WARNING 'something not right in %: count %', r, count; + SET enable_seqscan = 1; + SET enable_bitmapscan = 0; + FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP + RAISE NOTICE 'seqscan: %', r2; + END LOOP; + + SET enable_seqscan = 0; + SET enable_bitmapscan = 1; + FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP + RAISE NOTICE 'bitmapscan: %', r2; + END LOOP; + END IF; + + -- make sure we found expected number of matches + IF count != r.matches THEN RAISE WARNING 'unexpected number of results % for %', count, r; END IF; + END LOOP; +END; +$x$; + +RESET enable_seqscan; +RESET enable_bitmapscan; + +INSERT INTO brintest_multi SELECT + 142857 * tenthous, + thousand, + twothousand, + unique1::oid, + format('(%s,%s)', tenthous, twenty)::tid, + (four + 1.0)/(hundred+1), + odd::float8 / (tenthous + 1), + format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, + inet '10.2.3.4' + tenthous, + cidr '10.2.3/24' + tenthous, + date '1995-08-15' + tenthous, + time '01:20:30' + thousand * interval '18.5 second', + timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours', + timestamptz '1972-10-10 03:00' + thousand * interval '1 hour', + justify_days(justify_hours(tenthous * interval '12 minutes')), + timetz '01:30:20' + hundred * interval '15 seconds', + tenthous::numeric(36,30) * fivethous * even / (hundred + 1), + format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid, + format('%s/%s%s', odd, even, tenthous)::pg_lsn +FROM tenk1 ORDER BY unique2 LIMIT 5 OFFSET 5; + +SELECT brin_desummarize_range('brinidx_multi', 0); +VACUUM brintest_multi; -- force a summarization cycle in brinidx + +UPDATE brintest_multi SET int8col = int8col * int4col; + +-- Tests for brin_summarize_new_values +SELECT brin_summarize_new_values('brintest_multi'); -- error, not an index +SELECT brin_summarize_new_values('tenk1_unique1'); -- error, not a BRIN index +SELECT brin_summarize_new_values('brinidx_multi'); -- ok, no change expected + +-- Tests for brin_desummarize_range +SELECT brin_desummarize_range('brinidx_multi', -1); -- error, invalid range +SELECT brin_desummarize_range('brinidx_multi', 0); +SELECT brin_desummarize_range('brinidx_multi', 0); +SELECT brin_desummarize_range('brinidx_multi', 100000000); + +-- Test brin_summarize_range +CREATE TABLE brin_summarize_multi ( + value int +) WITH (fillfactor=10, autovacuum_enabled=false); +CREATE INDEX brin_summarize_multi_idx ON brin_summarize_multi USING brin (value) WITH (pages_per_range=2); +-- Fill a few pages +DO $$ +DECLARE curtid tid; +BEGIN + LOOP + INSERT INTO brin_summarize_multi VALUES (1) RETURNING ctid INTO curtid; + EXIT WHEN curtid > tid '(2, 0)'; + END LOOP; +END; +$$; + +-- summarize one range +SELECT brin_summarize_range('brin_summarize_multi_idx', 0); +-- nothing: already summarized +SELECT brin_summarize_range('brin_summarize_multi_idx', 1); +-- summarize one range +SELECT brin_summarize_range('brin_summarize_multi_idx', 2); +-- nothing: page doesn't exist in table +SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967295); +-- invalid block number values +SELECT brin_summarize_range('brin_summarize_multi_idx', -1); +SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967296); + + +-- test brin cost estimates behave sanely based on correlation of values +CREATE TABLE brin_test_multi (a INT, b INT); +INSERT INTO brin_test_multi SELECT x/100,x%100 FROM generate_series(1,10000) x(x); +CREATE INDEX brin_test_multi_a_idx ON brin_test_multi USING brin (a) WITH (pages_per_range = 2); +CREATE INDEX brin_test_multi_b_idx ON brin_test_multi USING brin (b) WITH (pages_per_range = 2); +VACUUM ANALYZE brin_test_multi; + +-- Ensure brin index is used when columns are perfectly correlated +EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE a = 1; +-- Ensure brin index is not used when values are not correlated +EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE b = 1; -- 2.26.2 --------------CF71AF65F7C337C37C24B045 Content-Type: text/x-patch; charset=UTF-8; name="0008-Ignore-correlation-for-new-BRIN-opclasses-20210112.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Ignore-correlation-for-new-BRIN-opclasses-20210112.patc"; filename*1="h" ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-09-24 19:43 Noah Misch <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Noah Misch @ 2024-09-24 19:43 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Heikki Linnakangas <[email protected]>; Robert Haas <[email protected]>; Thomas Munro <[email protected]> On Tue, Sep 24, 2024 at 11:55:08AM -0400, Andres Freund wrote: > So far the AIO patchset has solved this by introducing a set of "bounce > buffers", which can be acquired and used as the source/target of IO when doing > it in-place into shared buffers isn't viable. > > I am worried about that solution however, as either acquisition of bounce > buffers becomes a performance issue (that's how I did it at first, it was hard > to avoid regressions) or we reserve bounce buffers for each backend, in which > case the memory overhead for instances with relatively small amount of > shared_buffers and/or many connections can be significant. > But: We can address this and improve performance over the status quo! Today we > determine tuple visiblity determination one-by-one, even when checking the > visibility of an entire page worth of tuples. That's not exactly free. I've > prototyped checking visibility of an entire page of tuples at once and it > indeed speeds up visibility checks substantially (in some cases seqscans are > over 20% faster!). Nice! It sounds like you refactored the relationship between heap_prepare_pagescan() and HeapTupleSatisfiesVisibility() to move the hint bit setting upward or the iterate-over-tuples downward. Is that about right? > Once we have page-level visibility checks we can get the right to set hint > bits once for an entire page instead of doing it for every tuple - with that > in place the "new approach" of setting hint bits only with BM_SETTING_HINTS > wins. How did page-level+BM_SETTING_HINTS performance compare to performance of the page-level change w/o the BM_SETTING_HINTS change? > Having a page level approach to setting hint bits has other advantages: > > E.g. today, with wal_log_hints, we'll log hint bits on the first hint bit set > on the page and we don't mark a page dirty on hot standby. Which often will > result in hint bits notpersistently set on replicas until the page is frozen. Nice way to improve that. > Does this sound like a reasonable idea? Counterpoints? I guess the main part left to discuss is index scans or other scan types where we'd either not do page-level visibility or we'd do page-level visibility including tuples we wouldn't otherwise use. BM_SETTING_HINTS likely won't show up so readily in index scan profiles, but the cost is still there. How should we think about comparing the distributed cost of the buffer header manipulations during index scans vs. the costs of bounce buffers? Thanks, nm ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-09-24 20:30 Andres Freund <[email protected]> parent: Noah Misch <[email protected]> 0 siblings, 3 replies; 12+ messages in thread From: Andres Freund @ 2024-09-24 20:30 UTC (permalink / raw) To: Noah Misch <[email protected]>; +Cc: pgsql-hackers; Heikki Linnakangas <[email protected]>; Robert Haas <[email protected]>; Thomas Munro <[email protected]> Hi, On 2024-09-24 12:43:40 -0700, Noah Misch wrote: > On Tue, Sep 24, 2024 at 11:55:08AM -0400, Andres Freund wrote: > > So far the AIO patchset has solved this by introducing a set of "bounce > > buffers", which can be acquired and used as the source/target of IO when doing > > it in-place into shared buffers isn't viable. > > > > I am worried about that solution however, as either acquisition of bounce > > buffers becomes a performance issue (that's how I did it at first, it was hard > > to avoid regressions) or we reserve bounce buffers for each backend, in which > > case the memory overhead for instances with relatively small amount of > > shared_buffers and/or many connections can be significant. > > > But: We can address this and improve performance over the status quo! Today we > > determine tuple visiblity determination one-by-one, even when checking the > > visibility of an entire page worth of tuples. That's not exactly free. I've > > prototyped checking visibility of an entire page of tuples at once and it > > indeed speeds up visibility checks substantially (in some cases seqscans are > > over 20% faster!). > > Nice! It sounds like you refactored the relationship between > heap_prepare_pagescan() and HeapTupleSatisfiesVisibility() to move the hint > bit setting upward or the iterate-over-tuples downward. Is that about right? I've tried about five variations, so I don't have one answer to this yet :). One problem is that having repeated loops doing PageGetItemId(), PageGetItem(), ItemIdGetLength() isn't exactly free. To some degree it can be hidden by allowing for better superscalar execution, but not entirely. I've been somewhat confused by the compiler generated code around ItemId handling for a while, it looks way more expensive than it should - it regularly is a bottleneck due to the sheer number of instructions being executed leading to being frontend bound. But never quite looked into it deeply enough to figure out what's causing it / how to fix it. > > Once we have page-level visibility checks we can get the right to set hint > > bits once for an entire page instead of doing it for every tuple - with that > > in place the "new approach" of setting hint bits only with BM_SETTING_HINTS > > wins. > > How did page-level+BM_SETTING_HINTS performance compare to performance of the > page-level change w/o the BM_SETTING_HINTS change? Just ran that. There probably is a performance difference, but it's small (<0.5%) making it somewhat hard to be certain. It looks like the main reason for that is ConditionVariableBroadcast() on the iocv shows up even though nobody is waiting. I've been fighting that with AIO as well, so maybe it's time to figure out the memory ordering rules that'd allow to check that without a full spinlock acquisition. If we figure it out, we perhaps should use the chance to get rid of BM_PIN_COUNT_WAITER... > > Does this sound like a reasonable idea? Counterpoints? > > I guess the main part left to discuss is index scans or other scan types where > we'd either not do page-level visibility or we'd do page-level visibility > including tuples we wouldn't otherwise use. BM_SETTING_HINTS likely won't > show up so readily in index scan profiles, but the cost is still there. I could indeed not make it show up in some simple index lookup heavy workloads. I need to try some more extreme cases though (e.g. fetching all tuples in a table via an index or having very long HOT chains). If it's not visible cost-wise compared to all the other costs of index scans - does it matter? If it's not visible it's either because it proportionally is very small or because it's completely hidden by superscalar execution. Another thing I forgot to mention that probably fits into the "tradeoffs" bucket: Because BM_SETTING_HINTS would be just a single bit, one backend setting hint bits would block out another backend setting hint bits. In most situations that'll be fine, or even faster than not doing so due to reducing cache line ping-pong, but in cases of multiple backends doing index lookups to different unhinted tuples on the same page it could be a bit worse. But I suspect that's fine because it's hard to believe that you could have enough index lookups to unhinted tuples for that to be a bottleneck - something has to produce all those unhinted tuples after all, and that's rather more expensive. And for single-tuple visibility checks the window in which hint bits are set is very small. > How should we think about comparing the distributed cost of the buffer > header manipulations during index scans vs. the costs of bounce buffers? Well, the cost of bounce buffers would be born as long as postgres is up, whereas a not-measurable (if it indeed isn't) cost during index scans wouldn't really show up. If there are cases where the cost of the more expensive hint bit logic does show up, it'll get a lot harder to weigh. So far my prototype uses the path that avoids hint bits being set while IO is going on all the time, not just when checksums are enabled. We could change that, I guess. However, our habit of modifying buffers while IO is going on is causing issues with filesystem level checksums as well, as evidenced by the fact that debug_io_direct = data on btrfs causes filesystem corruption. So I tend to think it'd be better to just stop doing that alltogether (we also do that for WAL, when writing out a partial page, but a potential fix there would be different, I think). A thing I forgot to mention: Bounce buffers are kind of an architectural dead end, in that we wouln't need them in that form if we get to a threaded postgres. Zooming out (a lot) more: I like the idea of having a way to get the permission to perform some kinds of modifications on a page without an exlusive lock. While obviously a lot more work, I do think there's some potential to have some fast-paths that perform work on a page level without blocking out readers. E.g. some simple cases of insert could correctly be done without blocking out readers (by ordering the update of the max page offset Greetings, Andres Freund ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-09-25 00:45 Thomas Munro <[email protected]> parent: Andres Freund <[email protected]> 2 siblings, 1 reply; 12+ messages in thread From: Thomas Munro @ 2024-09-25 00:45 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers; Heikki Linnakangas <[email protected]>; Robert Haas <[email protected]> On Wed, Sep 25, 2024 at 8:30 AM Andres Freund <[email protected]> wrote: > Just ran that. There probably is a performance difference, but it's small > (<0.5%) making it somewhat hard to be certain. It looks like the main reason > for that is ConditionVariableBroadcast() on the iocv shows up even though > nobody is waiting. . o O { Gotta fix that. Memory barriers might be enough to check for empty wait list?, and even in the slow path, atomic wait lists or something better than spinlocks... } > However, our habit of modifying buffers while IO is going on is > causing issues with filesystem level checksums as well, as evidenced by the > fact that debug_io_direct = data on btrfs causes filesystem corruption. So I > tend to think it'd be better to just stop doing that alltogether (we also do > that for WAL, when writing out a partial page, but a potential fix there would > be different, I think). +many. Interesting point re the WAL variant. For the record, here's some discussion and a repro for that problem, which Andrew currently works around in a build farm animal with mount options: https://www.postgresql.org/message-id/CA%2BhUKGKSBaz78Fw3WTF3Q8ArqKCz1GgsTfRFiDPbu-j9OFz-jw%40mail.g... ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-09-25 02:00 Noah Misch <[email protected]> parent: Andres Freund <[email protected]> 2 siblings, 0 replies; 12+ messages in thread From: Noah Misch @ 2024-09-25 02:00 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Heikki Linnakangas <[email protected]>; Robert Haas <[email protected]>; Thomas Munro <[email protected]> On Tue, Sep 24, 2024 at 04:30:25PM -0400, Andres Freund wrote: > On 2024-09-24 12:43:40 -0700, Noah Misch wrote: > > On Tue, Sep 24, 2024 at 11:55:08AM -0400, Andres Freund wrote: > > > Besides that, the need to copy the buffers makes checkpoints with AIO > > > noticeably slower when checksums are enabled - it's not the checksum but the > > > copy that's the biggest source of the slowdown. How big is that copy's contribution to the slowdown there? A measurable CPU overhead on writes likely does outweigh the unmeasurable overhead on index scans, but ... > > > Does this sound like a reasonable idea? Counterpoints? > > How should we think about comparing the distributed cost of the buffer > > header manipulations during index scans vs. the costs of bounce buffers? > > Well, the cost of bounce buffers would be born as long as postgres is up, > whereas a not-measurable (if it indeed isn't) cost during index scans wouldn't > really show up. ... neither BM_SETTING_HINTS nor keeping bounce buffers looks like a bad decision. From what I've heard so far of the performance effects, if it were me, I would keep the bounce buffers. I'd pursue BM_SETTING_HINTS and bounce buffer removal as a distinct project after the main AIO capability. Bounce buffers have an implementation. They aren't harming other design decisions. The AIO project is big, so I'd want to err on the side of not designating other projects as its prerequisites. > Zooming out (a lot) more: I like the idea of having a way to get the > permission to perform some kinds of modifications on a page without an > exlusive lock. While obviously a lot more work, I do think there's some > potential to have some fast-paths that perform work on a page level without > blocking out readers. E.g. some simple cases of insert could correctly be done > without blocking out readers (by ordering the update of the max page offset True. ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-09-26 21:56 Thomas Munro <[email protected]> parent: Thomas Munro <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Thomas Munro @ 2024-09-26 21:56 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers; Heikki Linnakangas <[email protected]>; Robert Haas <[email protected]> On Wed, Sep 25, 2024 at 12:45 PM Thomas Munro <[email protected]> wrote: > On Wed, Sep 25, 2024 at 8:30 AM Andres Freund <[email protected]> wrote: > > However, our habit of modifying buffers while IO is going on is > > causing issues with filesystem level checksums as well, as evidenced by the > > fact that debug_io_direct = data on btrfs causes filesystem corruption. So I > > tend to think it'd be better to just stop doing that alltogether (we also do > > that for WAL, when writing out a partial page, but a potential fix there would > > be different, I think). > > +many. Interesting point re the WAL variant. For the record, here's > some discussion and a repro for that problem, which Andrew currently > works around in a build farm animal with mount options: > > https://www.postgresql.org/message-id/CA%2BhUKGKSBaz78Fw3WTF3Q8ArqKCz1GgsTfRFiDPbu-j9OFz-jw%40mail.g... Here's an interesting new development in that area, this time from OpenZFS, which committed its long awaited O_DIRECT support a couple of weeks ago[1] and seems to have taken a different direction since that last discussion. Clearly it has the same checksum stability problem as BTRFS and PostgreSQL itself, so an O_DIRECT mode with the goal of avoiding copying and caching must confront that and break *something*, or accept something like bounce buffers and give up the zero-copy goal. Curiously, they seem to have landed on two different solutions with three different possible behaviours: (1) On FreeBSD, temporarily make the memory non-writeable, (2) On Linux, they couldn't do that so they have an extra checksum verification on write. I haven't fully grokked all this yet, or even tried it, and it's not released or anything, but it looks a bit like all three behaviours are bad for our current hint bit design: on FreeBSD, setting a hint bit might crash (?) if a write is in progress in another process, and on Linux, depending on zfs_vdev_direct_write_verify, either the concurrent write might fail (= checkpointer failing on EIO because someone concurrently set a hint bit) or a later read might fail (= file is permanently corrupted and you don't find out until later, like btrfs). I plan to look more closely soon and see if I understood that right... [1] https://github.com/openzfs/zfs/pull/10018/commits/d7b861e7cfaea867ae28ab46ab11fba89a5a1fda ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-10-30 02:21 Andres Freund <[email protected]> parent: Andres Freund <[email protected]> 2 siblings, 1 reply; 12+ messages in thread From: Andres Freund @ 2024-10-30 02:21 UTC (permalink / raw) To: Noah Misch <[email protected]>; +Cc: pgsql-hackers; Heikki Linnakangas <[email protected]>; Robert Haas <[email protected]>; Thomas Munro <[email protected]> Hi, Attached is a, unfortunately long, series of patches implementing what I described upthread. 0001 Add very basic test for kill_prior_tuples We currently don't exercise this patch for gist and hash, which seems somewhat criminal. 0002 heapam: Move logic to handle HEAP_MOVED into a helper function A prep patch, which consolidates HEAP_MOVED handling into a helper. This isn't strictly necessary, but I got bothered by it while writing this patch series. 0003 bufmgr: Add BufferLockHeldByMe() Useful for writing assertions in later patches. 0004 heapam: Use exclusive lock on old page in CLUSTER When I started looking into this I wanted to make sure that it's actually safe to skip setting hint bits in all places. One place that's *not* safe is cluster - we assume that all hint bits are set in rewrite_heap_tuple(). This might not strictly be required, but it seemed very fragile as is. While using an exclusive lock does seem like it makes it a bit safer, it's not a very good fix, as we still dirty the old buffer, which seems like a shame when one uses VACUUM FULL. 0005 heapam: Only set tuple's block once per page in pagemode Minor, but measurable, optimization. 0006 heapam: Add batch mode mvcc check and use it in page mode This is a good bit faster on its own, but is required to avoid a performance regression later, when setting hint bits only only when no IO going on at the same. 0007 bufmgr: Make it easier to change number of buffer state bits This just makes it easier to change the division of bits in BufferDesc->state. 0008 bufmgr: Add interface to acquire right to set hint bits The main change. Adds BufferPrepareToSetHintBits() which, if true is returned, gives the rigth to modify a buffer without an exclusive lock. Once the modification is done BufferFinishSetHintBits() needs to be called. I dithered a bit on the name, we don't really call them hint bits in all the places that modify buffers without an exlusive lock. The new infra is unused as of this commit. 0009 heapam: Acquire right to set hint bits Moves over heapam to the new interface. 0010 Acquire right to set hint bits in the remaining places Moves indexams and freespace/ over to the new interface. 0011 bufmgr: Detect some missing BufferPrepareToSetHintBits() calls Now that we don't set hint bits without calling BufferPrepareToSetHintBits() first, we can detect many missing calls by adding an assertion to MarkBufferDirtyHint(). 0012 bufmgr: Don't copy pages while writing out Finally remove the copy of the buffer while writing it out. 0013 WIP: bufmgr: Detect some bad buffer accesses This is a lot more thorough infrastructure for detecting modifications of buffers without appropriate locks, by using mprotect() to detect such cases. Only works when not using huge pages, and not on windows (not sure how mprotect() is spelled there) and not arm macos (due to 16kB pages). Has two modes, one just disallows modifications, and is reasonably cheap. The more expensive mode disallows any access when the buffer is unpinned. The commit message has some performance numbers. This has proven to be a good investment as far as this thread goes - I wasn't initially aware that freespace modified buffers without a lock and this quickly found that. Greetings, Andres Freund Attachments: [text/x-diff] v1-0001-Add-very-basic-test-for-kill_prior_tuples.patch (26.3K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/2-v1-0001-Add-very-basic-test-for-kill_prior_tuples.patch) download | inline diff: From 13ce1a3c5840e3d76a25fdcbeeda24fe4d2d4017 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 28 Oct 2024 15:48:50 -0400 Subject: [PATCH v1 01/14] Add very basic test for kill_prior_tuples Previously our tests did not exercise kill_prior_tuples for hash and gist. The latter only because it does not work if all the dead tuples are on one page (or something like that). Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- .../isolation/expected/index-killtuples.out | 338 ++++++++++++++++++ .../isolation/specs/index-killtuples.spec | 124 +++++++ 2 files changed, 462 insertions(+) create mode 100644 src/test/isolation/expected/index-killtuples.out create mode 100644 src/test/isolation/specs/index-killtuples.spec diff --git a/src/test/isolation/expected/index-killtuples.out b/src/test/isolation/expected/index-killtuples.out new file mode 100644 index 00000000000..1361b60c267 --- /dev/null +++ b/src/test/isolation/expected/index-killtuples.out @@ -0,0 +1,338 @@ +unused step name: delete_1 +Parsed test spec with 1 sessions + +starting permutation: create_table fill_500 create_btree flush disable_seq disable_bitmap measure access flush result measure access flush result delete flush measure access flush result measure access flush result drop_table +step create_table: CREATE TEMPORARY TABLE kill_prior_tuple(key int not null, cat text not null); +step fill_500: INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 500) g(i); +step create_btree: CREATE INDEX kill_prior_tuple_btree ON kill_prior_tuple USING btree (key); +step flush: SELECT FROM pg_stat_force_next_flush(); +step disable_seq: SET enable_seqscan = false; +step disable_bitmap: SET enable_bitmapscan = false; +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +----------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_btree on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +----------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_btree on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step delete: DELETE FROM kill_prior_tuple; +step flush: SELECT FROM pg_stat_force_next_flush(); +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +----------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_btree on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +----------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_btree on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 0 +(1 row) + +step drop_table: DROP TABLE IF EXISTS kill_prior_tuple; + +starting permutation: create_table fill_500 create_ext_btree_gist create_gist flush disable_seq disable_bitmap measure access flush result measure access flush result delete flush measure access flush result measure access flush result drop_table drop_ext_btree_gist +step create_table: CREATE TEMPORARY TABLE kill_prior_tuple(key int not null, cat text not null); +step fill_500: INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 500) g(i); +step create_ext_btree_gist: CREATE EXTENSION btree_gist; +step create_gist: CREATE INDEX kill_prior_tuple_gist ON kill_prior_tuple USING gist (key); +step flush: SELECT FROM pg_stat_force_next_flush(); +step disable_seq: SET enable_seqscan = false; +step disable_bitmap: SET enable_bitmapscan = false; +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step delete: DELETE FROM kill_prior_tuple; +step flush: SELECT FROM pg_stat_force_next_flush(); +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 0 +(1 row) + +step drop_table: DROP TABLE IF EXISTS kill_prior_tuple; +step drop_ext_btree_gist: DROP EXTENSION btree_gist; + +starting permutation: create_table fill_10 create_ext_btree_gist create_gist flush disable_seq disable_bitmap measure access flush result measure access flush result delete flush measure access flush result measure access flush result drop_table drop_ext_btree_gist +step create_table: CREATE TEMPORARY TABLE kill_prior_tuple(key int not null, cat text not null); +step fill_10: INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 10) g(i); +step create_ext_btree_gist: CREATE EXTENSION btree_gist; +step create_gist: CREATE INDEX kill_prior_tuple_gist ON kill_prior_tuple USING gist (key); +step flush: SELECT FROM pg_stat_force_next_flush(); +step disable_seq: SET enable_seqscan = false; +step disable_bitmap: SET enable_bitmapscan = false; +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step delete: DELETE FROM kill_prior_tuple; +step flush: SELECT FROM pg_stat_force_next_flush(); +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_gist on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step drop_table: DROP TABLE IF EXISTS kill_prior_tuple; +step drop_ext_btree_gist: DROP EXTENSION btree_gist; + +starting permutation: create_table fill_500 create_hash flush disable_seq disable_bitmap measure access flush result measure access flush result delete flush measure access flush result measure access flush result drop_table +step create_table: CREATE TEMPORARY TABLE kill_prior_tuple(key int not null, cat text not null); +step fill_500: INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 500) g(i); +step create_hash: CREATE INDEX kill_prior_tuple_hash ON kill_prior_tuple USING hash (key); +step flush: SELECT FROM pg_stat_force_next_flush(); +step disable_seq: SET enable_seqscan = false; +step disable_bitmap: SET enable_bitmapscan = false; +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_hash on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_hash on kill_prior_tuple (actual rows=1 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step delete: DELETE FROM kill_prior_tuple; +step flush: SELECT FROM pg_stat_force_next_flush(); +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_hash on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +---------------------------------------------------------------------------------- +Index Scan using kill_prior_tuple_hash on kill_prior_tuple (actual rows=0 loops=1) + Index Cond: (key = 1) +(2 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 0 +(1 row) + +step drop_table: DROP TABLE IF EXISTS kill_prior_tuple; + +starting permutation: create_table fill_500 create_ext_btree_gin create_gin flush disable_seq delete flush measure access flush result measure access flush result drop_table drop_ext_btree_gin +step create_table: CREATE TEMPORARY TABLE kill_prior_tuple(key int not null, cat text not null); +step fill_500: INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 500) g(i); +step create_ext_btree_gin: CREATE EXTENSION btree_gin; +step create_gin: CREATE INDEX kill_prior_tuple_gin ON kill_prior_tuple USING gin (key); +step flush: SELECT FROM pg_stat_force_next_flush(); +step disable_seq: SET enable_seqscan = false; +step delete: DELETE FROM kill_prior_tuple; +step flush: SELECT FROM pg_stat_force_next_flush(); +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +----------------------------------------------------------------------- +Bitmap Heap Scan on kill_prior_tuple (actual rows=0 loops=1) + Recheck Cond: (key = 1) + Heap Blocks: exact=1 + -> Bitmap Index Scan on kill_prior_tuple_gin (actual rows=1 loops=1) + Index Cond: (key = 1) +(5 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step measure: UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); +step access: EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; +QUERY PLAN +----------------------------------------------------------------------- +Bitmap Heap Scan on kill_prior_tuple (actual rows=0 loops=1) + Recheck Cond: (key = 1) + Heap Blocks: exact=1 + -> Bitmap Index Scan on kill_prior_tuple_gin (actual rows=1 loops=1) + Index Cond: (key = 1) +(5 rows) + +step flush: SELECT FROM pg_stat_force_next_flush(); +step result: SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; +new_heap_accesses +----------------- + 1 +(1 row) + +step drop_table: DROP TABLE IF EXISTS kill_prior_tuple; +step drop_ext_btree_gin: DROP EXTENSION btree_gin; diff --git a/src/test/isolation/specs/index-killtuples.spec b/src/test/isolation/specs/index-killtuples.spec new file mode 100644 index 00000000000..d597be02c23 --- /dev/null +++ b/src/test/isolation/specs/index-killtuples.spec @@ -0,0 +1,124 @@ +# Basic testing of killtuples / kill_prior_tuples / all_dead testing +# for various index AMs +# +# This tests just enough to ensure that the kill* routines are actually +# executed and does something approximately reasonble. It's *not* sufficient +# testing for adding killitems support to a new AM! + +setup +{ + CREATE TABLE counter(heap_accesses int); + INSERT INTO counter(heap_accesses) VALUES (0); +} + +teardown +{ + DROP TABLE counter; +} + +session s1 +# to ensure GUCs are reset +setup { RESET ALL; } + +step disable_seq { SET enable_seqscan = false; } + +step disable_bitmap { SET enable_bitmapscan = false; } + +# use a temporary table to make sure no other session can interfere with +# visibility determinations +step create_table { CREATE TEMPORARY TABLE kill_prior_tuple(key int not null, cat text not null); } + +step fill_10 { INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 10) g(i); } + +step fill_500 { INSERT INTO kill_prior_tuple(key, cat) SELECT g.i, 'a' FROM generate_series(1, 500) g(i); } + +# column-less select to make output easier to read +step flush { SELECT FROM pg_stat_force_next_flush(); } + +step measure { UPDATE counter SET heap_accesses = (SELECT heap_blks_read + heap_blks_hit FROM pg_statio_all_tables WHERE relname = 'kill_prior_tuple'); } + +step result { SELECT heap_blks_read + heap_blks_hit - counter.heap_accesses AS new_heap_accesses FROM counter, pg_statio_all_tables WHERE relname = 'kill_prior_tuple'; } + +step access { EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT * FROM kill_prior_tuple WHERE key = 1; } + +step delete { DELETE FROM kill_prior_tuple; } +step delete_1 { DELETE FROM kill_prior_tuple WHERE cat = 1; } + +step drop_table { DROP TABLE IF EXISTS kill_prior_tuple; } + +### steps for testing btree indexes ### +step create_btree { CREATE INDEX kill_prior_tuple_btree ON kill_prior_tuple USING btree (key); } + +### steps for testing gist indexes ### +# Creating the extensions takes time, so we don't want to do so when testing +# other AMs +step create_ext_btree_gist { CREATE EXTENSION btree_gist; } +step drop_ext_btree_gist { DROP EXTENSION btree_gist; } +step create_gist { CREATE INDEX kill_prior_tuple_gist ON kill_prior_tuple USING gist (key); } + +### steps for testing gin indexes ### +# See create_ext_btree_gist +step create_ext_btree_gin { CREATE EXTENSION btree_gin; } +step drop_ext_btree_gin { DROP EXTENSION btree_gin; } +step create_gin { CREATE INDEX kill_prior_tuple_gin ON kill_prior_tuple USING gin (key); } + +### steps for testing hash indexes ### +step create_hash { CREATE INDEX kill_prior_tuple_hash ON kill_prior_tuple USING hash (key); } + + +# test killtuples with btree inex +permutation + create_table fill_500 create_btree flush + disable_seq disable_bitmap + # show each access to non-deleted tuple increments heap_blks_* + measure access flush result + measure access flush result + delete flush + # first access after accessing deleted tuple still needs to access heap + measure access flush result + # but after kill_prior_tuple did its thing, we shouldn't access heap anymore + measure access flush result + drop_table + +# Same as first permutation, except testing gist +permutation + create_table fill_500 create_ext_btree_gist create_gist flush + disable_seq disable_bitmap + measure access flush result + measure access flush result + delete flush + measure access flush result + measure access flush result + drop_table drop_ext_btree_gist + +# Test gist, but with fewer rows - shows that killitems doesn't work anymore! +permutation + create_table fill_10 create_ext_btree_gist create_gist flush + disable_seq disable_bitmap + measure access flush result + measure access flush result + delete flush + measure access flush result + measure access flush result + drop_table drop_ext_btree_gist + +# Same as first permutation, except testing hash +permutation + create_table fill_500 create_hash flush + disable_seq disable_bitmap + measure access flush result + measure access flush result + delete flush + measure access flush result + measure access flush result + drop_table + +# # Similar to first permutation, except that gin does not have killtuples support +permutation + create_table fill_500 create_ext_btree_gin create_gin flush + disable_seq + delete flush + measure access flush result + # will still fetch from heap + measure access flush result + drop_table drop_ext_btree_gin -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0002-heapam-Move-logic-to-handle-HEAP_MOVED-into-a-hel.patch (11.5K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/3-v1-0002-heapam-Move-logic-to-handle-HEAP_MOVED-into-a-hel.patch) download | inline diff: From cddf2ae3d5d4300a99e8bff6a98ac8fee82fe169 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 23 Sep 2024 12:23:33 -0400 Subject: [PATCH v1 02/14] heapam: Move logic to handle HEAP_MOVED into a helper function Before we dealt with this in 6 near identical and one very similar copy. The helper function errors out when encountering a HEAP_MOVED_IN/HEAP_MOVED_OUT tuple with xvac considered current or in-progress. It'd be preferrable to do that change separately, but otherwise it'd not be possible to deduplicate the handling in HeapTupleSatisfiesVacuum(). Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/backend/access/heap/heapam_visibility.c | 307 ++++---------------- 1 file changed, 61 insertions(+), 246 deletions(-) diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c index 9243feed01f..057b88767db 100644 --- a/src/backend/access/heap/heapam_visibility.c +++ b/src/backend/access/heap/heapam_visibility.c @@ -144,6 +144,55 @@ HeapTupleSetHintBits(HeapTupleHeader tuple, Buffer buffer, SetHintBits(tuple, buffer, infomask, xid); } +/* + * If HEAP_MOVED_OFF or HEAP_MOVED_IN are set on the tuple, remove them and + * adjust hint bits. See the comment for SetHintBits() for more background. + * + * This helper returns false if the row ought to be invisible, true otherwise. + */ +static inline bool +HeapTupleCleanMoved(HeapTupleHeader tuple, Buffer buffer) +{ + TransactionId xvac; + + /* only used by pre-9.0 binary upgrades */ + if (likely(!(tuple->t_infomask & (HEAP_MOVED_OFF | HEAP_MOVED_IN)))) + return true; + + xvac = HeapTupleHeaderGetXvac(tuple); + + if (TransactionIdIsCurrentTransactionId(xvac)) + elog(ERROR, "encountered tuple with HEAP_MOVED considered current"); + + if (TransactionIdIsInProgress(xvac)) + elog(ERROR, "encountered tuple with HEAP_MOVED considered in-progress"); + + if (tuple->t_infomask & HEAP_MOVED_OFF) + { + if (TransactionIdDidCommit(xvac)) + { + SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, + InvalidTransactionId); + return false; + } + SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, + InvalidTransactionId); + } + else if (tuple->t_infomask & HEAP_MOVED_IN) + { + if (TransactionIdDidCommit(xvac)) + SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, + InvalidTransactionId); + else + { + SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, + InvalidTransactionId); + return false; + } + } + + return true; +} /* * HeapTupleSatisfiesSelf @@ -179,45 +228,8 @@ HeapTupleSatisfiesSelf(HeapTuple htup, Snapshot snapshot, Buffer buffer) if (HeapTupleHeaderXminInvalid(tuple)) return false; - /* Used by pre-9.0 binary upgrades */ - if (tuple->t_infomask & HEAP_MOVED_OFF) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return false; - if (!TransactionIdIsInProgress(xvac)) - { - if (TransactionIdDidCommit(xvac)) - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - } - } - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_IN) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (!TransactionIdIsCurrentTransactionId(xvac)) - { - if (TransactionIdIsInProgress(xvac)) - return false; - if (TransactionIdDidCommit(xvac)) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - else - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - } - } + if (!HeapTupleCleanMoved(tuple, buffer)) + return false; else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple))) { if (tuple->t_infomask & HEAP_XMAX_INVALID) /* xid invalid */ @@ -372,45 +384,8 @@ HeapTupleSatisfiesToast(HeapTuple htup, Snapshot snapshot, if (HeapTupleHeaderXminInvalid(tuple)) return false; - /* Used by pre-9.0 binary upgrades */ - if (tuple->t_infomask & HEAP_MOVED_OFF) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return false; - if (!TransactionIdIsInProgress(xvac)) - { - if (TransactionIdDidCommit(xvac)) - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - } - } - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_IN) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (!TransactionIdIsCurrentTransactionId(xvac)) - { - if (TransactionIdIsInProgress(xvac)) - return false; - if (TransactionIdDidCommit(xvac)) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - else - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - } - } + if (!HeapTupleCleanMoved(tuple, buffer)) + return false; /* * An invalid Xmin can be left behind by a speculative insertion that @@ -468,45 +443,8 @@ HeapTupleSatisfiesUpdate(HeapTuple htup, CommandId curcid, if (HeapTupleHeaderXminInvalid(tuple)) return TM_Invisible; - /* Used by pre-9.0 binary upgrades */ - if (tuple->t_infomask & HEAP_MOVED_OFF) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return TM_Invisible; - if (!TransactionIdIsInProgress(xvac)) - { - if (TransactionIdDidCommit(xvac)) - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return TM_Invisible; - } - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - } - } - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_IN) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (!TransactionIdIsCurrentTransactionId(xvac)) - { - if (TransactionIdIsInProgress(xvac)) - return TM_Invisible; - if (TransactionIdDidCommit(xvac)) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - else - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return TM_Invisible; - } - } - } + else if (!HeapTupleCleanMoved(tuple, buffer)) + return false; else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple))) { if (HeapTupleHeaderGetCmin(tuple) >= curcid) @@ -756,45 +694,8 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot, if (HeapTupleHeaderXminInvalid(tuple)) return false; - /* Used by pre-9.0 binary upgrades */ - if (tuple->t_infomask & HEAP_MOVED_OFF) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return false; - if (!TransactionIdIsInProgress(xvac)) - { - if (TransactionIdDidCommit(xvac)) - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - } - } - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_IN) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (!TransactionIdIsCurrentTransactionId(xvac)) - { - if (TransactionIdIsInProgress(xvac)) - return false; - if (TransactionIdDidCommit(xvac)) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - else - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - } - } + if (!HeapTupleCleanMoved(tuple, buffer)) + return false; else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple))) { if (tuple->t_infomask & HEAP_XMAX_INVALID) /* xid invalid */ @@ -970,45 +871,8 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot, if (HeapTupleHeaderXminInvalid(tuple)) return false; - /* Used by pre-9.0 binary upgrades */ - if (tuple->t_infomask & HEAP_MOVED_OFF) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return false; - if (!XidInMVCCSnapshot(xvac, snapshot)) - { - if (TransactionIdDidCommit(xvac)) - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - } - } - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_IN) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (!TransactionIdIsCurrentTransactionId(xvac)) - { - if (XidInMVCCSnapshot(xvac, snapshot)) - return false; - if (TransactionIdDidCommit(xvac)) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - else - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return false; - } - } - } + if (!HeapTupleCleanMoved(tuple, buffer)) + return false; else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple))) { if (HeapTupleHeaderGetCmin(tuple) >= snapshot->curcid) @@ -1213,57 +1077,8 @@ HeapTupleSatisfiesVacuumHorizon(HeapTuple htup, Buffer buffer, TransactionId *de { if (HeapTupleHeaderXminInvalid(tuple)) return HEAPTUPLE_DEAD; - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_OFF) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return HEAPTUPLE_DELETE_IN_PROGRESS; - if (TransactionIdIsInProgress(xvac)) - return HEAPTUPLE_DELETE_IN_PROGRESS; - if (TransactionIdDidCommit(xvac)) - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return HEAPTUPLE_DEAD; - } - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - } - /* Used by pre-9.0 binary upgrades */ - else if (tuple->t_infomask & HEAP_MOVED_IN) - { - TransactionId xvac = HeapTupleHeaderGetXvac(tuple); - - if (TransactionIdIsCurrentTransactionId(xvac)) - return HEAPTUPLE_INSERT_IN_PROGRESS; - if (TransactionIdIsInProgress(xvac)) - return HEAPTUPLE_INSERT_IN_PROGRESS; - if (TransactionIdDidCommit(xvac)) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - InvalidTransactionId); - else - { - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); - return HEAPTUPLE_DEAD; - } - } - else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple))) - { - if (tuple->t_infomask & HEAP_XMAX_INVALID) /* xid invalid */ - return HEAPTUPLE_INSERT_IN_PROGRESS; - /* only locked? run infomask-only check first, for performance */ - if (HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask) || - HeapTupleHeaderIsOnlyLocked(tuple)) - return HEAPTUPLE_INSERT_IN_PROGRESS; - /* inserted and then deleted by same xact */ - if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetUpdateXid(tuple))) - return HEAPTUPLE_DELETE_IN_PROGRESS; - /* deleting subtransaction must have aborted */ - return HEAPTUPLE_INSERT_IN_PROGRESS; - } + else if (!HeapTupleCleanMoved(tuple, buffer)) + return HEAPTUPLE_DEAD; else if (TransactionIdIsInProgress(HeapTupleHeaderGetRawXmin(tuple))) { /* -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0003-bufmgr-Add-BufferLockHeldByMe.patch (2.3K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/4-v1-0003-bufmgr-Add-BufferLockHeldByMe.patch) download | inline diff: From aa054c44b41669c44f238f127c20bd48dff3af11 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 23 Sep 2024 12:51:37 -0400 Subject: [PATCH v1 03/14] bufmgr: Add BufferLockHeldByMe() Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/include/storage/bufmgr.h | 1 + src/backend/storage/buffer/bufmgr.c | 41 +++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index eb0fba4230b..ded46a57889 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -291,6 +291,7 @@ extern void LockBufferForCleanup(Buffer buffer); extern bool ConditionalLockBufferForCleanup(Buffer buffer); extern bool IsBufferCleanupOK(Buffer buffer); extern bool HoldingBufferPinThatDelaysRecovery(void); +extern bool BufferLockHeldByMe(Buffer buffer, int mode); extern bool BgBufferSync(struct WritebackContext *wb_context); diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 0f02bf62fa3..88d18e85d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -5492,6 +5492,47 @@ IsBufferCleanupOK(Buffer buffer) return false; } +/* + * BufferLockHeldByMe - does the backend have the buffer locked? + * + * This likely should only be used for asserts etc. + * + * Note that this can only be called for non-temp buffers - there is no + * correct value to return for temporary buffers. One might think that just + * returning true for temp buffers would work, but the caller might assert + * that a lock is *not* held. + */ +bool +BufferLockHeldByMe(Buffer buffer, int mode) +{ + BufferDesc *buf; + LWLockMode lwmode; + + /* + * Can't hold a lock without a pin, there never should be uncertainty + * about having a pin. + */ + Assert(BufferIsPinned(buffer)); + + /* there'd be no correct value to return */ + Assert(!BufferIsLocal(buffer)); + + buf = GetBufferDescriptor(buffer - 1); + + if (mode == BUFFER_LOCK_EXCLUSIVE) + lwmode = LW_EXCLUSIVE; + else if (mode == BUFFER_LOCK_SHARE) + lwmode = LW_SHARED; + else + { + Assert(false); + pg_unreachable(); + lwmode = LW_EXCLUSIVE; /* assuage compiler */ + } + + return LWLockHeldByMeInMode(BufferDescriptorGetContentLock(buf), lwmode); +} + /* * Functions for buffer I/O handling -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0004-heapam-Use-exclusive-lock-on-old-page-in-CLUSTER.patch (3.2K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/5-v1-0004-heapam-Use-exclusive-lock-on-old-page-in-CLUSTER.patch) download | inline diff: From 08c1b45cfd7c223b146d228e997d16624274d0b4 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 17 Oct 2024 11:45:36 -0400 Subject: [PATCH v1 04/14] heapam: Use exclusive lock on old page in CLUSTER To be able to guarantee that we can set the hint bit, acquire an exclusive lock on the old buffer. We need the hint bits to be set as otherwise reform_and_rewrite_tuple() -> rewrite_heap_tuple() -> heap_freeze_tuple() will get confused. It'd be better if we somehow coulda void setting hint bits on the old page. One reason to use VACUUM FULL are very bloated tables - rewriting most of the old table before during VACUUM FULL doesn't. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/backend/access/heap/heapam_handler.c | 13 ++++++++++++- src/backend/access/heap/heapam_visibility.c | 7 +++++++ src/backend/storage/buffer/bufmgr.c | 2 +- 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index a8d95e0f1c1..009601445aa 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -833,7 +833,18 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tuple = ExecFetchSlotHeapTuple(slot, false, NULL); buf = hslot->buffer; - LockBuffer(buf, BUFFER_LOCK_SHARE); + /* + * To be able to guarantee that we can set the hint bit, acquire an + * exclusive lock on the old buffer. We need the hint bits to be set + * as otherwise reform_and_rewrite_tuple() -> rewrite_heap_tuple() -> + * heap_freeze_tuple() will get confused. + * + * It'd be better if we somehow could avoid setting hint bits on the + * old page. One reason to use VACUUM FULL are very bloated tables - + * rewriting most of the old table before during VACUUM FULL doesn't + * exactly help... + */ + LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); switch (HeapTupleSatisfiesVacuum(tuple, OldestXmin, buf)) { diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c index 057b88767db..b7aa8bb7a52 100644 --- a/src/backend/access/heap/heapam_visibility.c +++ b/src/backend/access/heap/heapam_visibility.c @@ -141,6 +141,13 @@ void HeapTupleSetHintBits(HeapTupleHeader tuple, Buffer buffer, uint16 infomask, TransactionId xid) { + /* + * The uses from heapam.c rely on being able to perform the hint bit + * updates, which can only be guaranteed if we are holding an exclusive + * lock on the buffer - which all callers are doing. + */ + Assert(BufferIsLocal(buffer) || BufferLockHeldByMe(buffer, BUFFER_LOCK_EXCLUSIVE)); + SetHintBits(tuple, buffer, infomask, xid); } diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 88d18e85d64..f56f7f9f441 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -5527,7 +5527,7 @@ BufferLockHeldByMe(Buffer buffer, int mode) { Assert(false); pg_unreachable(); - lwmode = LW_EXCLUSIVE; /* assuage compiler */ + lwmode = LW_EXCLUSIVE; /* assuage compiler */ } return LWLockHeldByMeInMode(BufferDescriptorGetContentLock(buf), lwmode); -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0005-heapam-Only-set-tuple-s-block-once-per-page-in-pa.patch (1.6K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/6-v1-0005-heapam-Only-set-tuple-s-block-once-per-page-in-pa.patch) download | inline diff: From 9b56778c495de3ed7b179083c926598ab5fbaf57 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 17 Oct 2024 11:50:13 -0400 Subject: [PATCH v1 05/14] heapam: Only set tuple's block once per page in pagemode Due to splitting the block id into two 16 bit integers, BlockIdSet() is more expensive than one might think. Doing it once per returned shows up as small but relibly reproducible cost. It's simple enough to just set the block number so once per block in pagemode, so do so. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/backend/access/heap/heapam.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index 1748eafa100..d0cb4b1e29b 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -981,6 +981,9 @@ heapgettup_pagemode(HeapScanDesc scan, linesleft = scan->rs_ntuples; lineindex = ScanDirectionIsForward(dir) ? 0 : linesleft - 1; + /* set block once per page, instead of doing so for every tuple */ + BlockIdSet(&tuple->t_self.ip_blkid, scan->rs_cblock); + /* lineindex now references the next or previous visible tid */ continue_page: @@ -995,7 +998,7 @@ continue_page: tuple->t_data = (HeapTupleHeader) PageGetItem(page, lpp); tuple->t_len = ItemIdGetLength(lpp); - ItemPointerSet(&(tuple->t_self), scan->rs_cblock, lineoff); + tuple->t_self.ip_posid = lineoff; /* skip any tuples that don't match the scan key */ if (key != NULL && -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0006-heapam-Add-batch-mode-mvcc-check-and-use-it-in-pa.patch (8.0K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/7-v1-0006-heapam-Add-batch-mode-mvcc-check-and-use-it-in-pa.patch) download | inline diff: From ed74f0f2c6dfc4e4afe31ac162200c0783bf2240 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 17 Oct 2024 13:16:36 -0400 Subject: [PATCH v1 06/14] heapam: Add batch mode mvcc check and use it in page mode There are two reasons for doing so: 1) It is generally faster to perform checks in a batched fashion and making sequential scans faster is nice. 2) We would like to stop setting hint bits while pages are being written out. The necessary locking becomes visible for page mode scans if done for every tuple. With batching the overhead can be amortized to only happen once per page. There are substantial further optimization opportunities along these lines: - Right now HeapTupleSatisfiesMVCCBatch() simply uses the single-tuple HeapTupleSatisfiesMVCC(), relying on the compiler to inline it. We could instead write an explicitly optimized version that avoids repeated xid tests. - Introduce batched version of the serializability test - Introduce batched version of HeapTupleSatisfiesVacuum Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/include/access/heapam.h | 28 +++++ src/backend/access/heap/heapam.c | 110 ++++++++++++++++---- src/backend/access/heap/heapam_visibility.c | 50 +++++++++ src/tools/pgindent/typedefs.list | 1 + 4 files changed, 166 insertions(+), 23 deletions(-) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 96cf82f97b7..d4a790251cc 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -417,6 +417,34 @@ extern bool HeapTupleHeaderIsOnlyLocked(HeapTupleHeader tuple); extern bool HeapTupleIsSurelyDead(HeapTuple htup, struct GlobalVisState *vistest); +/* + * FIXME: define to be removed + * + * Without this I see worse performance. But it's a bit ugly, so I thought + * it'd be useful to leave a way in for others to experiment with this. + */ +#define BATCHMVCC_FEWER_ARGS + +#ifdef BATCHMVCC_FEWER_ARGS +typedef struct BatchMVCCState +{ + HeapTupleData tuples[MaxHeapTuplesPerPage]; + bool visible[MaxHeapTuplesPerPage]; +} BatchMVCCState; +#endif + +extern int HeapTupleSatisfiesMVCCBatch(Snapshot snapshot, Buffer buffer, + int ntups, +#ifdef BATCHMVCC_FEWER_ARGS + BatchMVCCState *batchmvcc, +#else + HeapTupleData *tuples, + bool *visible, +#endif + OffsetNumber *vistuples_dense); + + + /* * To avoid leaking too much knowledge about reorderbuffer implementation * details this is implemented in reorderbuffer.c not heapam_visibility.c diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index d0cb4b1e29b..99656a1aac5 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -437,42 +437,106 @@ page_collect_tuples(HeapScanDesc scan, Snapshot snapshot, BlockNumber block, int lines, bool all_visible, bool check_serializable) { + Oid relid = RelationGetRelid(scan->rs_base.rs_rd); +#ifdef BATCHMVCC_FEWER_ARGS + BatchMVCCState batchmvcc; +#else + HeapTupleData tuples[MaxHeapTuplesPerPage]; + bool visible[MaxHeapTuplesPerPage]; +#endif int ntup = 0; - OffsetNumber lineoff; + int nvis = 0; - for (lineoff = FirstOffsetNumber; lineoff <= lines; lineoff++) + /* page at a time should have been disabled otherwise */ + Assert(IsMVCCSnapshot(snapshot)); + + /* first find all tuples on the page */ + for (OffsetNumber lineoff = FirstOffsetNumber; lineoff <= lines; lineoff++) { ItemId lpp = PageGetItemId(page, lineoff); - HeapTupleData loctup; - bool valid; + HeapTuple tup; - if (!ItemIdIsNormal(lpp)) + if (unlikely(!ItemIdIsNormal(lpp))) continue; - loctup.t_data = (HeapTupleHeader) PageGetItem(page, lpp); - loctup.t_len = ItemIdGetLength(lpp); - loctup.t_tableOid = RelationGetRelid(scan->rs_base.rs_rd); - ItemPointerSet(&(loctup.t_self), block, lineoff); - - if (all_visible) - valid = true; - else - valid = HeapTupleSatisfiesVisibility(&loctup, snapshot, buffer); - - if (check_serializable) - HeapCheckForSerializableConflictOut(valid, scan->rs_base.rs_rd, - &loctup, buffer, snapshot); - - if (valid) + /* + * If the page is not all-visible or we need to check serializability, + * maintain enough state to be able to refind the tuple efficiently, + * without again needing to extract it from the page. + */ + if (!all_visible || check_serializable) { - scan->rs_vistuples[ntup] = lineoff; - ntup++; +#ifdef BATCHMVCC_FEWER_ARGS + tup = &batchmvcc.tuples[ntup]; +#else + tup = &tuples[ntup]; +#endif + + tup->t_data = (HeapTupleHeader) PageGetItem(page, lpp); + tup->t_len = ItemIdGetLength(lpp); + tup->t_tableOid = relid; + ItemPointerSet(&(tup->t_self), block, lineoff); } + + /* + * If the page is all visible, these fields won'otherwise wont be + * populated in loop below. + */ + if (all_visible) + { + if (check_serializable) + { +#ifdef BATCHMVCC_FEWER_ARGS + batchmvcc.visible[ntup] = true; +#else + visible[ntup] = true; +#endif + } + scan->rs_vistuples[ntup] = lineoff; + } + + ntup++; } Assert(ntup <= MaxHeapTuplesPerPage); - return ntup; + /* unless the page is all visible, test visibility for all tuples one go */ + if (all_visible) + nvis = ntup; + else + nvis = HeapTupleSatisfiesMVCCBatch(snapshot, buffer, + ntup, +#ifdef BATCHMVCC_FEWER_ARGS + &batchmvcc, +#else + tuples, visible, +#endif + scan->rs_vistuples + ); + + /* + * So far we don't have batch API for testing serializabilty, so do so + * one-by-one. + */ + if (check_serializable) + { + for (int i = 0; i < ntup; i++) + { +#ifdef BATCHMVCC_FEWER_ARGS + HeapCheckForSerializableConflictOut(batchmvcc.visible[i], + scan->rs_base.rs_rd, + &batchmvcc.tuples[i], + buffer, snapshot); +#else + HeapCheckForSerializableConflictOut(visible[i], + scan->rs_base.rs_rd, + &tuples[i], + buffer, snapshot); +#endif + } + } + + return nvis; } /* diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c index b7aa8bb7a52..4baafc01a2c 100644 --- a/src/backend/access/heap/heapam_visibility.c +++ b/src/backend/access/heap/heapam_visibility.c @@ -1575,6 +1575,56 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot, return true; } +/* + * Perform HeaptupleSatisfiesMVCC() on each passed in tuple. This is more + * efficient than doing HeapTupleSatisfiesMVCC() one-by-one. + * + * To be checked tuples are passed via BatchMVCCState->tuples. Each tuple's + * visibility is set in batchmvcc->visible[]. In addition, ->vistuples_dense + * is set to contain the offsets of visible tuples. + * + * Returns the number of visible tuples. + */ +int +HeapTupleSatisfiesMVCCBatch(Snapshot snapshot, Buffer buffer, + int ntups, +#ifdef BATCHMVCC_FEWER_ARGS + BatchMVCCState *batchmvcc, +#else + HeapTupleData *tuples, + bool *visible, +#endif + OffsetNumber *vistuples_dense) +{ + int nvis = 0; + + Assert(IsMVCCSnapshot(snapshot)); + + for (int i = 0; i < ntups; i++) + { + bool valid; +#ifdef BATCHMVCC_FEWER_ARGS + HeapTuple tup = &batchmvcc->tuples[i]; +#else + HeapTuple tup = &tuples[i]; +#endif + + valid = HeapTupleSatisfiesMVCC(tup, snapshot, buffer); +#ifdef BATCHMVCC_FEWER_ARGS + batchmvcc->visible[i] = valid; +#else + visible[i] = valid; +#endif + if (likely(valid)) + { + vistuples_dense[nvis] = tup->t_self.ip_posid; + nvis++; + } + } + + return nvis; +} + /* * HeapTupleSatisfiesVisibility * True iff heap tuple satisfies a time qual. diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 171a7dd5d2b..15c7673465b 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -244,6 +244,7 @@ Barrier BaseBackupCmd BaseBackupTargetHandle BaseBackupTargetType +BatchMVCCState BeginDirectModify_function BeginForeignInsert_function BeginForeignModify_function -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0007-bufmgr-Make-it-easier-to-change-number-of-buffer-.patch (2.2K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/8-v1-0007-bufmgr-Make-it-easier-to-change-number-of-buffer-.patch) download | inline diff: From 7a3db51e47865fb2d3691e02f39cda790c9f6d10 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 28 Oct 2024 18:01:14 -0400 Subject: [PATCH v1 07/14] bufmgr: Make it easier to change number of buffer state bits In an upcoming commit I'd like to change the number of bits for the usage count (the current max is 5, fitting in three bits, but we resere four bits). Until now that required adjusting a bunch of magic constants, now the constants are defined based on the number of bits reserved. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/include/storage/buf_internals.h | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index eda6c699212..d8444651736 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -39,12 +39,19 @@ * * The definition of buffer state components is below. */ +#define BUF_REFCOUNT_BITS 18 +#define BUF_USAGECOUNT_BITS 4 +#define BUF_FLAG_BITS 10 + +StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, + "buffer bit counts need to equal 32"); + #define BUF_REFCOUNT_ONE 1 -#define BUF_REFCOUNT_MASK ((1U << 18) - 1) -#define BUF_USAGECOUNT_MASK 0x003C0000U -#define BUF_USAGECOUNT_ONE (1U << 18) -#define BUF_USAGECOUNT_SHIFT 18 -#define BUF_FLAG_MASK 0xFFC00000U +#define BUF_REFCOUNT_MASK ((1U << BUF_REFCOUNT_BITS) - 1) +#define BUF_USAGECOUNT_MASK (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) +#define BUF_USAGECOUNT_ONE (1U << BUF_REFCOUNT_BITS) +#define BUF_USAGECOUNT_SHIFT BUF_REFCOUNT_BITS +#define BUF_FLAG_MASK (((1U << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) /* Get refcount and usagecount from buffer state */ #define BUF_STATE_GET_REFCOUNT(state) ((state) & BUF_REFCOUNT_MASK) @@ -77,6 +84,9 @@ */ #define BM_MAX_USAGE_COUNT 5 +StaticAssertDecl(BM_MAX_USAGE_COUNT < ((1 << BUF_USAGECOUNT_BITS) - 1), + "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); + /* * Buffer tag identifies which disk block the buffer contains. * -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0008-bufmgr-Add-interface-to-acquire-right-to-set-hint.patch (15.4K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/9-v1-0008-bufmgr-Add-interface-to-acquire-right-to-set-hint.patch) download | inline diff: From ab2a7ea025a32053f6f201d14ee764a235c480ce Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 17 Oct 2024 13:59:13 -0400 Subject: [PATCH v1 08/14] bufmgr: Add interface to acquire right to set hint bits At the moment hint bits can be set with just a share lock on a page (and in one place even without any lock). Because of this we need to copy pages while writing them out, as otherwise the checksum could be corrupted. The need to copy the page is problematic for the AIO patchset: 1) Instead of just needing a single buffer for a copied page we need one for each page that's potentially undergoing IO 2) To be able to use the "worker" AIO implementation the copied page needs to reside in shared memory. Even without AIO copying the page isn't free... This commit starts to address that by adding BufferPrepareToSetHintBits(), which needs to be called before setting hint bits on a buffer. BufferPrepareToSetHintBits() only allows hint bit writes if there's no ongoing IO and while hint bits are being set no IO is allowed to be started. To know that a buffer is undergoing IO a new BufferDesc state flag is used, BM_SETTING_HINTS. Theoretically it'd be possible to reuse BM_IO_IN_PROGRESS, but that'd make it harder to debug the system. The new interface is not yet used, that will happen in subsequent commits, to make review a bit easier. Therefore we cannot yet rely on not needing a copy of the buffer during IO. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/include/storage/buf_internals.h | 22 ++- src/include/storage/bufmgr.h | 3 + src/backend/storage/buffer/README | 6 +- src/backend/storage/buffer/bufmgr.c | 177 +++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 5 files changed, 198 insertions(+), 11 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index d8444651736..1f8a0482e90 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -31,8 +31,8 @@ * Buffer state is a single 32-bit variable where following data is combined. * * - 18 bits refcount - * - 4 bits usage count - * - 10 bits of flags + * - 3 bits usage count + * - 11 bits of flags * * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. @@ -40,8 +40,8 @@ * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 -#define BUF_USAGECOUNT_BITS 4 -#define BUF_FLAG_BITS 10 +#define BUF_USAGECOUNT_BITS 3 +#define BUF_FLAG_BITS 11 StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, "buffer bit counts need to equal 32"); @@ -63,6 +63,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, * Note: BM_TAG_VALID essentially means that there is a buffer hashtable * entry associated with the buffer's tag. */ +#define BM_SETTING_HINTS (1U << 21) /* hint bits are being set */ #define BM_LOCKED (1U << 22) /* buffer header is locked */ #define BM_DIRTY (1U << 23) /* data needs writing */ #define BM_VALID (1U << 24) /* data is valid */ @@ -396,6 +397,7 @@ extern PGDLLIMPORT CkptSortItem *CkptBufferIds; /* ResourceOwner callbacks to hold buffer I/Os and pins */ extern PGDLLIMPORT const ResourceOwnerDesc buffer_io_resowner_desc; extern PGDLLIMPORT const ResourceOwnerDesc buffer_pin_resowner_desc; +extern PGDLLIMPORT const ResourceOwnerDesc buffer_setting_hints_resowner_desc; /* Convenience wrappers over ResourceOwnerRemember/Forget */ static inline void @@ -418,6 +420,18 @@ ResourceOwnerForgetBufferIO(ResourceOwner owner, Buffer buffer) { ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_io_resowner_desc); } +static inline void +ResourceOwnerRememberBufferSettingHints(ResourceOwner owner, Buffer buffer) +{ + ResourceOwnerRemember(owner, Int32GetDatum(buffer), + &buffer_setting_hints_resowner_desc); +} +static inline void +ResourceOwnerForgetBufferSettingHints(ResourceOwner owner, Buffer buffer) +{ + ResourceOwnerForget(owner, Int32GetDatum(buffer), + &buffer_setting_hints_resowner_desc); +} /* * Internal buffer management routines diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index ded46a57889..a37872377bc 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -300,6 +300,9 @@ extern void LimitAdditionalLocalPins(uint32 *additional_pins); extern bool EvictUnpinnedBuffer(Buffer buf); +extern bool BufferPrepareToSetHintBits(Buffer buf); +extern void BufferFinishSetHintBits(Buffer buf); + /* in buf_init.c */ extern void BufferManagerShmemInit(void); extern Size BufferManagerShmemSize(void); diff --git a/src/backend/storage/buffer/README b/src/backend/storage/buffer/README index 011af7aff3e..756ad542910 100644 --- a/src/backend/storage/buffer/README +++ b/src/backend/storage/buffer/README @@ -58,7 +58,8 @@ tuple while they are doing visibility checks. 4. It is considered OK to update tuple commit status bits (ie, OR the values HEAP_XMIN_COMMITTED, HEAP_XMIN_INVALID, HEAP_XMAX_COMMITTED, or HEAP_XMAX_INVALID into t_infomask) while holding only a shared lock and -pin on a buffer. This is OK because another backend looking at the tuple +pin on a buffer after calling BufferPrepareToSetHintBits(). +This is OK because another backend looking at the tuple at about the same time would OR the same bits into the field, so there is little or no risk of conflicting update; what's more, if there did manage to be a conflict it would merely mean that one bit-update would @@ -68,6 +69,9 @@ great harm is done if they get reset to zero by conflicting updates. Note, however, that a tuple is frozen by setting both HEAP_XMIN_INVALID and HEAP_XMIN_COMMITTED; this is a critical update and accordingly requires an exclusive buffer lock (and it must also be WAL-logged). +Calling BufferPrepareToSetHintBits() ensures that write IO cannot happen at +the same time as modifying hint bits, which can lead the checksum computed at +the start of the write to not match anymore. 5. To physically remove a tuple or compact free space on a page, one must hold a pin and an exclusive lock, *and* observe while holding the diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index f56f7f9f441..ee8b898121c 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -222,6 +222,8 @@ static void ResOwnerReleaseBufferIO(Datum res); static char *ResOwnerPrintBufferIO(Datum res); static void ResOwnerReleaseBufferPin(Datum res); static char *ResOwnerPrintBufferPin(Datum res); +static void ResOwnerReleaseBufferSettingHints(Datum res); +static char *ResOwnerPrintBufferSettingHints(Datum res); const ResourceOwnerDesc buffer_io_resowner_desc = { @@ -241,6 +243,15 @@ const ResourceOwnerDesc buffer_pin_resowner_desc = .DebugPrint = ResOwnerPrintBufferPin }; +const ResourceOwnerDesc buffer_setting_hints_resowner_desc = +{ + .name = "buffer setting hints", + .release_phase = RESOURCE_RELEASE_BEFORE_LOCKS, + .release_priority = RELEASE_PRIO_BUFFER_IOS, + .ReleaseResource = ResOwnerReleaseBufferSettingHints, + .DebugPrint = ResOwnerPrintBufferSettingHints +}; + /* * Ensure that the PrivateRefCountArray has sufficient space to store one more * entry. This has to be called before using NewPrivateRefCountEntry() to fill @@ -1737,7 +1748,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, /* some sanity checks while we hold the buffer header lock */ Assert(BUF_STATE_GET_REFCOUNT(victim_buf_state) == 1); - Assert(!(victim_buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY | BM_IO_IN_PROGRESS))); + Assert(!(victim_buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY + | BM_IO_IN_PROGRESS | BM_SETTING_HINTS))); victim_buf_hdr->tag = newTag; @@ -2092,7 +2104,8 @@ again: buf_state = pg_atomic_read_u32(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); - Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); + Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY + | BM_IO_IN_PROGRESS | BM_SETTING_HINTS))); CheckBufferIsPinnedOnce(buf); #endif @@ -5544,7 +5557,8 @@ BufferLockHeldByMe(Buffer buffer, int mode) */ /* - * WaitIO -- Block until the IO_IN_PROGRESS flag on 'buf' is cleared. + * WaitIO -- Block until the IO_IN_PROGRESS and SETTING_HINTS flags on 'buf' + * are cleared. */ static void WaitIO(BufferDesc *buf) @@ -5555,6 +5569,7 @@ WaitIO(BufferDesc *buf) for (;;) { uint32 buf_state; + uint32 wait_event; /* * It may not be necessary to acquire the spinlock to check the flag @@ -5564,9 +5579,21 @@ WaitIO(BufferDesc *buf) buf_state = LockBufHdr(buf); UnlockBufHdr(buf, buf_state); - if (!(buf_state & BM_IO_IN_PROGRESS)) + if (likely((buf_state & (BM_IO_IN_PROGRESS | BM_SETTING_HINTS)) == 0)) break; - ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO); + else if (buf_state & BM_IO_IN_PROGRESS) + { + Assert(!(buf_state & BM_SETTING_HINTS)); + wait_event = WAIT_EVENT_BUFFER_IO; + } + else + { + Assert(buf_state & BM_SETTING_HINTS); + Assert(!(buf_state & BM_IO_IN_PROGRESS)); + wait_event = WAIT_EVENT_BUFFER_SETTING_HINTS; + } + + ConditionVariableSleep(cv, wait_event); } ConditionVariableCancelSleep(); } @@ -5606,7 +5633,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { buf_state = LockBufHdr(buf); - if (!(buf_state & BM_IO_IN_PROGRESS)) + if (!(buf_state & (BM_IO_IN_PROGRESS | BM_SETTING_HINTS))) break; UnlockBufHdr(buf, buf_state); if (nowait) @@ -5623,6 +5650,8 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) return false; } + Assert(!(buf_state & BM_SETTING_HINTS)); + buf_state |= BM_IO_IN_PROGRESS; UnlockBufHdr(buf, buf_state); @@ -5661,6 +5690,7 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, buf_state = LockBufHdr(buf); Assert(buf_state & BM_IO_IN_PROGRESS); + Assert(!(buf_state & BM_SETTING_HINTS)); buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR); if (clear_dirty && !(buf_state & BM_JUST_DIRTIED)) @@ -5697,6 +5727,7 @@ AbortBufferIO(Buffer buffer) buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); + Assert(!(buf_state & BM_SETTING_HINTS)); if (!(buf_state & BM_VALID)) { @@ -6122,6 +6153,23 @@ ResOwnerPrintBufferPin(Datum res) return DebugPrintBufferRefcount(DatumGetInt32(res)); } + +static void +ResOwnerReleaseBufferSettingHints(Datum res) +{ + Buffer buffer = DatumGetInt32(res); + + BufferFinishSetHintBits(buffer); +} + +static char * +ResOwnerPrintBufferSettingHints(Datum res) +{ + Buffer buffer = DatumGetInt32(res); + + return psprintf("lost track of setting hint bits on buffer %d", buffer); +} + /* * Try to evict the current block in a shared buffer. * @@ -6184,3 +6232,120 @@ EvictUnpinnedBuffer(Buffer buf) return result; } + +/* + * Try to acquire the right to set hint bits for buf. + * + * It's only permitted to set hint bits in a buffer if the buffer is not + * undergoing IO. Otherwise the page level checksum could be corrupted. This + * could happen both for PG's checksum and on the OS/filesystem + * level. E.g. btrfs with direct-io relies on the page to not change while IO + * is going on. + * + * We can't just check if IO going on at the time BufferPrepareToSetHintBits() + * is called, we also need to block IO from starting before we're done setting + * hints. This is achieved by setting BM_SETTING_HINTS for the buffer and + * having StartBufferIO()/WaitIO() wait for that. We could combine + * BM_SETTING_HINTS and BM_IO_IN_PROGRESS into one, as they can never be set + * at the same time, but that seems unnecessarily confusing. + * + * Because we use only a single bit (BM_SETTING_HINTS) to track whether hint + * bits are currently being set, we cannot allow multiple backends to set + * hints at the same time - it'd be unknown whether BM_SETTING_HINTS would + * need to be remain set when a backend finishes setting hint bits. In almost + * all situations the two backends would just set the same hint bits anyway, + * so this is unlikely to be a problem. + * + * If allowed to set hint bits, the caller needs to call + * BufferFinishSetHintBits() once done setting hint bits. In case of an error + * occuring before BufferFinishSetHintBits() is reached, the cleanup will be + * done via resowner.c. + * + * It can make sense to amortize the cost of BufferPrepareToSetHintBits() + + * BufferFinishSetHintBits() over multiple hint bit sets on a page. + * + * Returns whether caller is allowed to set hint bits. + */ +bool +BufferPrepareToSetHintBits(Buffer buf) +{ + BufferDesc *desc; + uint32 old_buf_state; + + Assert(BufferIsPinned(buf)); + + if (BufferIsLocal(buf)) + return true; + + ResourceOwnerEnlarge(CurrentResourceOwner); + + desc = GetBufferDescriptor(buf - 1); + + /* + * We could use LockBufHdr() instead, but a CAS loop turns out to be + * slightly faster and has better concurrency behaviour due to not + * blocking other backends. + */ + old_buf_state = pg_atomic_read_u32(&desc->state); + + for (;;) + { + uint32 new_buf_state; + + if (unlikely(old_buf_state & BM_LOCKED)) + old_buf_state = WaitBufHdrUnlocked(desc); + + if (unlikely((old_buf_state & (BM_SETTING_HINTS | BM_IO_IN_PROGRESS)) != 0)) + return false; + + new_buf_state = old_buf_state; + + new_buf_state |= BM_SETTING_HINTS; + + if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + new_buf_state)) + { + ResourceOwnerRememberBufferSettingHints(CurrentResourceOwner, buf); + return true; + } + } +} + +/* + * Release the permission to set hint bits on the current page. + */ +void +BufferFinishSetHintBits(Buffer buf) +{ + BufferDesc *desc; + uint32 buf_state; + uint32 new_buf_state; + + if (BufferIsLocal(buf)) + return; + + desc = GetBufferDescriptor(buf - 1); + + buf_state = LockBufHdr(desc); + new_buf_state = buf_state; + new_buf_state &= ~BM_SETTING_HINTS; + UnlockBufHdr(desc, new_buf_state); + + Assert(buf_state & (BM_TAG_VALID)); + Assert(buf_state & (BM_VALID)); + Assert(buf_state & (BM_SETTING_HINTS)); + Assert(!(buf_state & (BM_IO_IN_PROGRESS))); + + ResourceOwnerForgetBufferSettingHints(CurrentResourceOwner, buf); + + /* + * Wake everyone that might be waiting for a chance to perform IO (i.e. + * WaitIO()). + * + * XXX: Right now this is somewhat expensive, due to + * ConditionVariableBroadcast() acquiring its spinlock unconditionally. I + * don't see a good way to avoid that from the bufmgr.c side, we don't + * know if there is somebody waiting. + */ + ConditionVariableBroadcast(BufferDescriptorGetIOCV(desc)); +} diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 8efb4044d6f..a1222a1cfbb 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -111,6 +111,7 @@ BGWORKER_SHUTDOWN "Waiting for background worker to shut down." BGWORKER_STARTUP "Waiting for background worker to start up." BTREE_PAGE "Waiting for the page number needed to continue a parallel B-tree scan to become available." BUFFER_IO "Waiting for buffer I/O to complete." +BUFFER_SETTING_HINTS "Waiting for hint bit setting to complete." CHECKPOINT_DELAY_COMPLETE "Waiting for a backend that blocks a checkpoint from completing." CHECKPOINT_DELAY_START "Waiting for a backend that blocks a checkpoint from starting." CHECKPOINT_DONE "Waiting for a checkpoint to complete." -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0009-heapam-Acquire-right-to-set-hint-bits.patch (8.3K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/10-v1-0009-heapam-Acquire-right-to-set-hint-bits.patch) download | inline diff: From 1271da941a9b093abb4565ddb2893f23eafbb33b Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 17 Oct 2024 14:05:15 -0400 Subject: [PATCH v1 09/14] heapam: Acquire right to set hint bits This commit starts to use BufferPrepareToSetHintBits(), introduced in a previous commit, for the reasons explained in said commit. To amortize the cost of BufferPrepareToSetHintBits() for cases where hint bits are set at a high frequency, HeapTupleSatisfiesMVCCBatch() uses the new SetHintBitsExt() which defers BufferFinishSetHintBits() until all hint bits on a page have been set. It's likely worth introducing additional batch visibility routines, e.g. for vacuuming, but I did not find a regression with the state as of this commit. So that's left for later. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/backend/access/heap/heapam_visibility.c | 116 +++++++++++++++++--- src/tools/pgindent/typedefs.list | 1 + 2 files changed, 99 insertions(+), 18 deletions(-) diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c index 4baafc01a2c..61fbac6e7f5 100644 --- a/src/backend/access/heap/heapam_visibility.c +++ b/src/backend/access/heap/heapam_visibility.c @@ -80,10 +80,35 @@ /* - * SetHintBits() + * To be allowed to set hint bits SetHintBits() needs to call + * BufferPrepareToSetHintBits(). However, that's not free, and some callsites + * call SetHintBits() on numerous tuples in a row. For those it makes sense to + * amortize the cost of BufferPrepareToSetHintBits(). Additionally it's + * desirable to defer the cost of BufferPrepareToSetHintBits() until a hint + * bit needs to actually be set. This enum serves as the necessary state space + * passed to SetHintbitsExt(). + */ +typedef enum SetHintBitsState +{ + /* not yet checked if hint bits may be set */ + SHB_INITIAL, + /* failed to get permission to set hint bits, don't check again */ + SHB_DISABLED, + /* allowed to set hint bits */ + SHB_ENABLED, +} SetHintBitsState; + +/* + * SetHintBitsExt() * * Set commit/abort hint bits on a tuple, if appropriate at this time. * + * To be allowed to set a hint bit on a tuple, the page must not be undergoing + * IO at this time (otherwise we e.g. could corrupt PG's page checksum or even + * the filesystem's, as is known to happen with btrfs). The right to set a + * hint bit is acquired on a page level with BufferPrepareToSetHintBits(). + * Only a single backend gets the right to set hint bits at a time. + * * It is only safe to set a transaction-committed hint bit if we know the * transaction's commit record is guaranteed to be flushed to disk before the * buffer, or if the table is temporary or unlogged and will be obliterated by @@ -111,9 +136,45 @@ * InvalidTransactionId if no check is needed. */ static inline void -SetHintBits(HeapTupleHeader tuple, Buffer buffer, - uint16 infomask, TransactionId xid) +SetHintBitsExt(HeapTupleHeader tuple, Buffer buffer, + uint16 infomask, TransactionId xid, SetHintBitsState *state) { + /* + * In batched mode and we previously did not get permission to set hint + * bits. Don't try again, in all likelihood IO is still going on. + */ + if (state && *state == SHB_DISABLED) + return; + + /* + * If not batching or this is the first hint that we'd like to set on the + * page, check if we are allowed to do so. + * + * XXX: Should we defer this until after the TransactionIdIsValid() check + * below? + */ + if ((!state || *state == SHB_INITIAL)) + { + if (BufferPrepareToSetHintBits(buffer)) + { + if (state) + *state = SHB_ENABLED; + } + else + { + /* + * If we hold an exclusive lock nobody else should be able to + * prevent us from setting hint bits. This is important as there + * are a few hint bit sets that are important for correctness - + * all those happen with the buffer exclusively locked. + */ + Assert(!BufferLockHeldByMe(buffer, BUFFER_LOCK_EXCLUSIVE)); + if (state) + *state = SHB_DISABLED; + return; + } + } + if (TransactionIdIsValid(xid)) { /* NB: xid must be known committed here! */ @@ -123,12 +184,27 @@ SetHintBits(HeapTupleHeader tuple, Buffer buffer, BufferGetLSNAtomic(buffer) < commitLSN) { /* not flushed and no LSN interlock, so don't set hint */ - return; + goto out; } } tuple->t_infomask |= infomask; MarkBufferDirtyHint(buffer, true); + +out: + if (!state) + BufferFinishSetHintBits(buffer); +} + +/* + * Simple wrapper around SetHintBitExt(), use when operating on a single + * tuple. + */ +static inline void +SetHintBits(HeapTupleHeader tuple, Buffer buffer, + uint16 infomask, TransactionId xid) +{ + SetHintBitsExt(tuple, buffer, infomask, xid, NULL); } /* @@ -864,9 +940,9 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot, * inserting/deleting transaction was still running --- which was more cycles * and more contention on ProcArrayLock. */ -static bool +static inline bool HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot, - Buffer buffer) + Buffer buffer, SetHintBitsState *state) { HeapTupleHeader tuple = htup->t_data; @@ -912,8 +988,8 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot, if (!TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmax(tuple))) { /* deleting subtransaction must have aborted */ - SetHintBits(tuple, buffer, HEAP_XMAX_INVALID, - InvalidTransactionId); + SetHintBitsExt(tuple, buffer, HEAP_XMAX_INVALID, + InvalidTransactionId, state); return true; } @@ -925,13 +1001,13 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot, else if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmin(tuple), snapshot)) return false; else if (TransactionIdDidCommit(HeapTupleHeaderGetRawXmin(tuple))) - SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED, - HeapTupleHeaderGetRawXmin(tuple)); + SetHintBitsExt(tuple, buffer, HEAP_XMIN_COMMITTED, + HeapTupleHeaderGetRawXmin(tuple), state); else { /* it must have aborted or crashed */ - SetHintBits(tuple, buffer, HEAP_XMIN_INVALID, - InvalidTransactionId); + SetHintBitsExt(tuple, buffer, HEAP_XMIN_INVALID, + InvalidTransactionId, state); return false; } } @@ -994,14 +1070,14 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot, if (!TransactionIdDidCommit(HeapTupleHeaderGetRawXmax(tuple))) { /* it must have aborted or crashed */ - SetHintBits(tuple, buffer, HEAP_XMAX_INVALID, - InvalidTransactionId); + SetHintBitsExt(tuple, buffer, HEAP_XMAX_INVALID, + InvalidTransactionId, state); return true; } /* xmax transaction committed */ - SetHintBits(tuple, buffer, HEAP_XMAX_COMMITTED, - HeapTupleHeaderGetRawXmax(tuple)); + SetHintBitsExt(tuple, buffer, HEAP_XMAX_COMMITTED, + HeapTupleHeaderGetRawXmax(tuple), state); } else { @@ -1597,6 +1673,7 @@ HeapTupleSatisfiesMVCCBatch(Snapshot snapshot, Buffer buffer, OffsetNumber *vistuples_dense) { int nvis = 0; + SetHintBitsState state = SHB_INITIAL; Assert(IsMVCCSnapshot(snapshot)); @@ -1609,7 +1686,7 @@ HeapTupleSatisfiesMVCCBatch(Snapshot snapshot, Buffer buffer, HeapTuple tup = &tuples[i]; #endif - valid = HeapTupleSatisfiesMVCC(tup, snapshot, buffer); + valid = HeapTupleSatisfiesMVCC(tup, snapshot, buffer, &state); #ifdef BATCHMVCC_FEWER_ARGS batchmvcc->visible[i] = valid; #else @@ -1622,6 +1699,9 @@ HeapTupleSatisfiesMVCCBatch(Snapshot snapshot, Buffer buffer, } } + if (state == SHB_ENABLED) + BufferFinishSetHintBits(buffer); + return nvis; } @@ -1641,7 +1721,7 @@ HeapTupleSatisfiesVisibility(HeapTuple htup, Snapshot snapshot, Buffer buffer) switch (snapshot->snapshot_type) { case SNAPSHOT_MVCC: - return HeapTupleSatisfiesMVCC(htup, snapshot, buffer); + return HeapTupleSatisfiesMVCC(htup, snapshot, buffer, NULL); case SNAPSHOT_SELF: return HeapTupleSatisfiesSelf(htup, snapshot, buffer); case SNAPSHOT_ANY: diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 15c7673465b..04b4f5cb240 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2604,6 +2604,7 @@ SetConstraintStateData SetConstraintTriggerData SetExprState SetFunctionReturnMode +SetHintBitsState SetOp SetOpCmd SetOpPath -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0010-Acquire-right-to-set-hint-bits-in-the-remaining-p.patch (6.6K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/11-v1-0010-Acquire-right-to-set-hint-bits-in-the-remaining-p.patch) download | inline diff: From 853d9e57765e89db711ac6d9751061a2327a571f Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 28 Oct 2024 16:44:11 -0400 Subject: [PATCH v1 10/14] Acquire right to set hint bits in the remaining places Use BufferPrepareToSetHintBits() in the remaining places that modify buffers without an exclusive lock. The remaining places are indexes with support for kill_prior_tuples and the freespace code. After this we do not need to copy buffers to write them out anymore. That change is done separately however. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/backend/access/gist/gistget.c | 12 +++++---- src/backend/access/hash/hashutil.c | 6 +++++ src/backend/access/nbtree/nbtinsert.c | 31 +++++++++++++++-------- src/backend/access/nbtree/nbtutils.c | 7 +++++ src/backend/storage/freespace/freespace.c | 14 ++++++---- src/backend/storage/freespace/fsmpage.c | 8 +++++- 6 files changed, 57 insertions(+), 21 deletions(-) diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c index b35b8a97577..c0a39768fdf 100644 --- a/src/backend/access/gist/gistget.c +++ b/src/backend/access/gist/gistget.c @@ -57,17 +57,16 @@ gistkillitems(IndexScanDesc scan) gistcheckpage(scan->indexRelation, buffer); page = BufferGetPage(buffer); + if (!BufferPrepareToSetHintBits(buffer)) + goto unlock; + /* * If page LSN differs it means that the page was modified since the last * read. killedItems could be not valid so LP_DEAD hints applying is not * safe. */ if (BufferGetLSNAtomic(buffer) != so->curPageLSN) - { - UnlockReleaseBuffer(buffer); - so->numKilled = 0; /* reset counter */ - return; - } + goto unlock; Assert(GistPageIsLeaf(page)); @@ -89,6 +88,9 @@ gistkillitems(IndexScanDesc scan) MarkBufferDirtyHint(buffer, true); } + BufferFinishSetHintBits(buffer); + +unlock: UnlockReleaseBuffer(buffer); /* diff --git a/src/backend/access/hash/hashutil.c b/src/backend/access/hash/hashutil.c index 20028f5cd14..a9adf95bafa 100644 --- a/src/backend/access/hash/hashutil.c +++ b/src/backend/access/hash/hashutil.c @@ -572,6 +572,9 @@ _hash_kill_items(IndexScanDesc scan) else buf = _hash_getbuf(rel, blkno, HASH_READ, LH_OVERFLOW_PAGE); + if (!BufferPrepareToSetHintBits(so->currPos.buf)) + goto unlock_page; + page = BufferGetPage(buf); opaque = HashPageGetOpaque(page); maxoff = PageGetMaxOffsetNumber(page); @@ -613,6 +616,9 @@ _hash_kill_items(IndexScanDesc scan) MarkBufferDirtyHint(buf, true); } + BufferFinishSetHintBits(so->currPos.buf); + +unlock_page: if (so->hashso_bucket_buf == so->currPos.buf || havePin) LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK); diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c index 99043da8412..eef498ada7c 100644 --- a/src/backend/access/nbtree/nbtinsert.c +++ b/src/backend/access/nbtree/nbtinsert.c @@ -679,20 +679,31 @@ _bt_check_unique(Relation rel, BTInsertState insertstate, Relation heapRel, { /* * The conflicting tuple (or all HOT chains pointed to by - * all posting list TIDs) is dead to everyone, so mark the - * index entry killed. + * all posting list TIDs) is dead to everyone, so try to + * mark the index entry killed. It's ok if we're not + * allowed to, this isn't required for correctness. */ - ItemIdMarkDead(curitemid); - opaque->btpo_flags |= BTP_HAS_GARBAGE; + Buffer buf; - /* - * Mark buffer with a dirty hint, since state is not - * crucial. Be sure to mark the proper buffer dirty. - */ + /* Be sure to operate on the proper buffer */ if (nbuf != InvalidBuffer) - MarkBufferDirtyHint(nbuf, true); + buf = nbuf; else - MarkBufferDirtyHint(insertstate->buf, true); + buf = insertstate->buf; + + if (BufferPrepareToSetHintBits(buf)) + { + ItemIdMarkDead(curitemid); + opaque->btpo_flags |= BTP_HAS_GARBAGE; + + /* + * Mark buffer with a dirty hint, since state is not + * crucial. + */ + MarkBufferDirtyHint(buf, true); + + BufferFinishSetHintBits(buf); + } } /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 76be65123c8..ea618a5565d 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -4210,6 +4210,10 @@ _bt_killitems(IndexScanDesc scan) } } + /* if we're not able to set hint bits, there's no point continuing */ + if (!BufferPrepareToSetHintBits(so->currPos.buf)) + goto unlock_page; + opaque = BTPageGetOpaque(page); minoff = P_FIRSTDATAKEY(opaque); maxoff = PageGetMaxOffsetNumber(page); @@ -4329,6 +4333,9 @@ _bt_killitems(IndexScanDesc scan) MarkBufferDirtyHint(so->currPos.buf, true); } + BufferFinishSetHintBits(so->currPos.buf); + +unlock_page: _bt_unlockbuf(scan->indexRelation, so->currPos.buf); } diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c index b1262ac3b69..864c7e231ee 100644 --- a/src/backend/storage/freespace/freespace.c +++ b/src/backend/storage/freespace/freespace.c @@ -904,12 +904,16 @@ fsm_vacuum_page(Relation rel, FSMAddress addr, max_avail = fsm_get_max_avail(page); /* - * Reset the next slot pointer. This encourages the use of low-numbered - * pages, increasing the chances that a later vacuum can truncate the - * relation. We don't bother with a lock here, nor with marking the page - * dirty if it wasn't already, since this is just a hint. + * Try to reset the next slot pointer. This encourages the use of + * low-numbered pages, increasing the chances that a later vacuum can + * truncate the relation. We don't bother with a lock here, nor with + * marking the page dirty if it wasn't already, since this is just a hint. */ - ((FSMPage) PageGetContents(page))->fp_next_slot = 0; + if (BufferPrepareToSetHintBits(buf)) + { + ((FSMPage) PageGetContents(page))->fp_next_slot = 0; + BufferFinishSetHintBits(buf); + } ReleaseBuffer(buf); diff --git a/src/backend/storage/freespace/fsmpage.c b/src/backend/storage/freespace/fsmpage.c index ba85cad0697..84bfa410dc0 100644 --- a/src/backend/storage/freespace/fsmpage.c +++ b/src/backend/storage/freespace/fsmpage.c @@ -300,7 +300,13 @@ restart: * * Wrap-around is handled at the beginning of this function. */ - fsmpage->fp_next_slot = slot + (advancenext ? 1 : 0); + if (exclusive_lock_held || BufferPrepareToSetHintBits(buf)) + { + fsmpage->fp_next_slot = slot + (advancenext ? 1 : 0); + + if (!exclusive_lock_held) + BufferFinishSetHintBits(buf); + } return slot; } -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0011-bufmgr-Detect-some-missing-BufferPrepareToSetHint.patch (1.5K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/12-v1-0011-bufmgr-Detect-some-missing-BufferPrepareToSetHint.patch) download | inline diff: From 1199f480b9d4f4be9095307bbd41dee3d92d338e Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 28 Oct 2024 18:11:23 -0400 Subject: [PATCH v1 11/14] bufmgr: Detect some missing BufferPrepareToSetHintBits() calls Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/backend/storage/buffer/bufmgr.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index ee8b898121c..5c7aa6662c5 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -5018,6 +5018,23 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) /* here, either share or exclusive lock is OK */ Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr))); + /* + * A share-locked buffer may only be marked dirty for hints after having + * gotten the permission to do so with BufferPrepareToSetHintBits(). It'd + * perhaps be cheap enough to test this even outside of assert enabled + * builds, but LWLockHeldByMeInMode() says "debug support only". + */ +#ifdef USE_ASSERT_CHECKING + { + uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + + if (!LWLockHeldByMeInMode(BufferDescriptorGetContentLock(bufHdr), LW_EXCLUSIVE)) + { + Assert(buf_state & BM_SETTING_HINTS); + } + } +#endif + /* * This routine might get called many times on the same page, if we are * making the first scan after commit of an xact that added/deleted many -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0012-bufmgr-Don-t-copy-pages-while-writing-out.patch (8.2K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/13-v1-0012-bufmgr-Don-t-copy-pages-while-writing-out.patch) download | inline diff: From c8f3dd69d09a1e6bb2f393e091aff02edc47a5f2 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 17 Oct 2024 14:14:35 -0400 Subject: [PATCH v1 12/14] bufmgr: Don't copy pages while writing out After the series of preceding commits introducing and using BufferPrepareToSetHintBits() hint bits cannot be set while IO is going on. Therefore we do not need to copy pages while they are being written out anymore. Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/include/storage/bufpage.h | 3 +- src/backend/access/hash/hashpage.c | 2 +- src/backend/storage/buffer/bufmgr.c | 23 +++++-------- src/backend/storage/buffer/localbuf.c | 2 +- src/backend/storage/page/bufpage.c | 48 ++++----------------------- src/backend/storage/smgr/bulk_write.c | 2 +- 6 files changed, 18 insertions(+), 62 deletions(-) diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h index 6222d46e535..e8cadc73aad 100644 --- a/src/include/storage/bufpage.h +++ b/src/include/storage/bufpage.h @@ -507,7 +507,6 @@ extern void PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems); extern void PageIndexTupleDeleteNoCompact(Page page, OffsetNumber offnum); extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum, Item newtup, Size newsize); -extern char *PageSetChecksumCopy(Page page, BlockNumber blkno); -extern void PageSetChecksumInplace(Page page, BlockNumber blkno); +extern void PageSetChecksum(Page page, BlockNumber blkno); #endif /* BUFPAGE_H */ diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c index d09c349e28f..2becf48bf06 100644 --- a/src/backend/access/hash/hashpage.c +++ b/src/backend/access/hash/hashpage.c @@ -1029,7 +1029,7 @@ _hash_alloc_buckets(Relation rel, BlockNumber firstblock, uint32 nblocks) zerobuf.data, true); - PageSetChecksumInplace(page, lastblock); + PageSetChecksum(page, lastblock); smgrextend(RelationGetSmgr(rel), MAIN_FORKNUM, lastblock, zerobuf.data, false); diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 5c7aa6662c5..a123e4e7f40 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -1996,9 +1996,8 @@ again: /* * If the buffer was dirty, try to write it out. There is a race * condition here, in that someone might dirty it after we released the - * buffer header lock above, or even while we are writing it out (since - * our share-lock won't prevent hint-bit updates). We will recheck the - * dirty bit after re-locking the buffer header. + * buffer header lock above. We will recheck the dirty bit after + * re-locking the buffer header. */ if (buf_state & BM_DIRTY) { @@ -3786,9 +3785,8 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum, * * The caller must hold a pin on the buffer and have share-locked the * buffer contents. (Note: a share-lock does not prevent updates of - * hint bits in the buffer, so the page could change while the write - * is in progress, but we assume that that will not invalidate the data - * written.) + * hint bits in the buffer, but hint bits are not allowed to be set after + * StartBufferIO() succeeded.) * * If the caller has an smgr reference for the buffer's relation, pass it * as the second parameter. If not, pass NULL. @@ -3801,7 +3799,6 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, ErrorContextCallback errcallback; instr_time io_start; Block bufBlock; - char *bufToWrite; uint32 buf_state; /* @@ -3867,12 +3864,8 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, */ bufBlock = BufHdrGetBlock(buf); - /* - * Update page checksum if desired. Since we have only shared lock on the - * buffer, other processes might be updating hint bits in it, so we must - * copy the page to private storage if we do checksumming. - */ - bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum); + /* Update page checksum if desired. */ + PageSetChecksum((Page) bufBlock, buf->tag.blockNum); io_start = pgstat_prepare_io_time(track_io_timing); @@ -3882,7 +3875,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, smgrwrite(reln, BufTagGetForkNum(&buf->tag), buf->tag.blockNum, - bufToWrite, + bufBlock, false); /* @@ -4531,7 +4524,7 @@ FlushRelationBuffers(Relation rel) errcallback.previous = error_context_stack; error_context_stack = &errcallback; - PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); + PageSetChecksum(localpage, bufHdr->tag.blockNum); io_start = pgstat_prepare_io_time(track_io_timing); diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 8da7dd6c98a..e75a1960d10 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -242,7 +242,7 @@ GetLocalVictimBuffer(void) /* Find smgr relation for buffer */ oreln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag), MyProcNumber); - PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); + PageSetChecksum(localpage, bufHdr->tag.blockNum); io_start = pgstat_prepare_io_time(track_io_timing); diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c index be6f1f62d29..2930fd936cd 100644 --- a/src/backend/storage/page/bufpage.c +++ b/src/backend/storage/page/bufpage.c @@ -1495,51 +1495,15 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum, /* * Set checksum for a page in shared buffers. * - * If checksums are disabled, or if the page is not initialized, just return - * the input. Otherwise, we must make a copy of the page before calculating - * the checksum, to prevent concurrent modifications (e.g. setting hint bits) - * from making the final checksum invalid. It doesn't matter if we include or - * exclude hints during the copy, as long as we write a valid page and - * associated checksum. + * If checksums are disabled, or if the page is not initialized, just + * return. Otherwise compute and set the checksum. * - * Returns a pointer to the block-sized data that needs to be written. Uses - * statically-allocated memory, so the caller must immediately write the - * returned page and not refer to it again. - */ -char * -PageSetChecksumCopy(Page page, BlockNumber blkno) -{ - static char *pageCopy = NULL; - - /* If we don't need a checksum, just return the passed-in data */ - if (PageIsNew(page) || !DataChecksumsEnabled()) - return (char *) page; - - /* - * We allocate the copy space once and use it over on each subsequent - * call. The point of palloc'ing here, rather than having a static char - * array, is first to ensure adequate alignment for the checksumming code - * and second to avoid wasting space in processes that never call this. - */ - if (pageCopy == NULL) - pageCopy = MemoryContextAllocAligned(TopMemoryContext, - BLCKSZ, - PG_IO_ALIGN_SIZE, - 0); - - memcpy(pageCopy, (char *) page, BLCKSZ); - ((PageHeader) pageCopy)->pd_checksum = pg_checksum_page(pageCopy, blkno); - return pageCopy; -} - -/* - * Set checksum for a page in private memory. - * - * This must only be used when we know that no other process can be modifying - * the page buffer. + * In the past this needed to be done on a copy of the page, due to the + * possibility of e.g. hint bits being set concurrently. However, this is not + * necessary anymore as hint bits won't be set while IO is going on. */ void -PageSetChecksumInplace(Page page, BlockNumber blkno) +PageSetChecksum(Page page, BlockNumber blkno) { /* If we don't need a checksum, just return */ if (PageIsNew(page) || !DataChecksumsEnabled()) diff --git a/src/backend/storage/smgr/bulk_write.c b/src/backend/storage/smgr/bulk_write.c index f0a65bfe242..2c294ddab1f 100644 --- a/src/backend/storage/smgr/bulk_write.c +++ b/src/backend/storage/smgr/bulk_write.c @@ -277,7 +277,7 @@ smgr_bulk_flush(BulkWriteState *bulkstate) BlockNumber blkno = pending_writes[i].blkno; Page page = pending_writes[i].buf->data; - PageSetChecksumInplace(page, blkno); + PageSetChecksum(page, blkno); if (blkno >= bulkstate->pages_written) { -- 2.46.0.519.g2e7b89e038 [text/x-diff] v1-0013-WIP-bufmgr-Detect-some-bad-buffer-accesses.patch (13.6K, ../../lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt6/14-v1-0013-WIP-bufmgr-Detect-some-bad-buffer-accesses.patch) download | inline diff: From d4e63de7705ec094affa0f877c905713a7dc6c32 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 28 Oct 2024 18:03:53 -0400 Subject: [PATCH v1 13/14] WIP: bufmgr: Detect some bad buffer accesses I wrote this mainly to ensure that I did not miss converting any hint bit sets to BufferPrepareToSetHintBits(), but it seems like it might be more generally useful. If we do want to include it, it needs a bit more polishing. On my workstation, the performance effect of this test infrastructure is as follows: base: real 1m4.613s user 4m31.409s sys 3m20.445s ENFORCE_BUFFER_PROT real 1m11.912s user 4m27.332s sys 3m28.063s ENFORCE_BUFFER_PROT + ENFORCE_BUFFER_PROT_READ real 1m33.455s user 4m32.188s sys 3m41.275s Author: Reviewed-by: Discussion: https://postgr.es/m/ Backpatch: --- src/include/pg_config_manual.h | 24 ++++ src/include/storage/buf_internals.h | 9 ++ src/include/storage/pg_shmem.h | 2 + src/backend/storage/buffer/buf_init.c | 10 ++ src/backend/storage/buffer/bufmgr.c | 154 ++++++++++++++++++++++++-- src/backend/utils/misc/guc_tables.c | 2 +- 6 files changed, 191 insertions(+), 10 deletions(-) diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index e49eb13e43c..8ea96bd1f1b 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -361,3 +361,27 @@ * Enable tracing of syncscan operations (see also the trace_syncscan GUC var). */ /* #define TRACE_SYNCSCAN */ + + +/* + * On some operating systems we know how to change whether pages are + * readable/writeable. We can use that to verify that we are following buffer + * locking rules, we can make pages inaccessible or read-only when we don't + * have sufficient locks etc. This obviously is fairly expensive, so by + * default we only so in assert enabled builds. + */ +#if defined(USE_ASSERT_CHECKING) && !defined(WIN32) +#define ENFORCE_BUFFER_PROT +#endif + +/* + * Protecting pages against being modified without an exclusive lock / + * BufferPrepareToSetHintBits() is reasonably cheap, neither happens *that* + * often. Pinning/unpinning buffers is a lot more common, making it more + * expensive to call mprotect() that often. + * + * Therefore disable this by default, even in assert enabled builds. + */ +#ifdef ENFORCE_BUFFER_PROT +/* #define ENFORCE_BUFFER_PROT_READ */ +#endif diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 1f8a0482e90..f542c245e27 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -457,6 +457,15 @@ extern Size StrategyShmemSize(void); extern void StrategyInitialize(bool init); extern bool have_free_buffer(void); +#ifdef ENFORCE_BUFFER_PROT +extern void SetBufferProtection(Buffer buf, bool allow_reads, bool allow_writes); +#else +static inline void +SetBufferProtection(Buffer buf, bool allow_reads, bool allow_writes) +{ +} +#endif + /* buf_table.c */ extern Size BufTableShmemSize(int size); extern void InitBufTable(int size); diff --git a/src/include/storage/pg_shmem.h b/src/include/storage/pg_shmem.h index 3065ff5be71..42a12688b9a 100644 --- a/src/include/storage/pg_shmem.h +++ b/src/include/storage/pg_shmem.h @@ -63,6 +63,8 @@ typedef enum SHMEM_TYPE_MMAP, } PGShmemType; +extern PGDLLIMPORT int huge_pages_status; + #ifndef WIN32 extern PGDLLIMPORT unsigned long UsedShmemSegID; #else diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 56761a8eedc..0e02d8e1e9b 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -135,6 +135,16 @@ BufferManagerShmemInit(void) LWTRANCHE_BUFFER_CONTENT); ConditionVariableInit(BufferDescriptorGetIOCV(buf)); + + /* + * Unused buffers are inaccessible. But if we're not enforcing + * making buffers inaccessible without a pin, we won't mark + * buffers as accessible during pinning, therefore we better don't + * make them initially inaccessible. + */ +#if defined(ENFORCE_BUFFER_PROT) && defined(ENFORCE_BUFFER_PROT_READ) + SetBufferProtection(i + 1, false, false); +#endif } /* Correct last entry of linked list */ diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index a123e4e7f40..50947b1021f 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -37,6 +37,10 @@ #include <sys/file.h> #include <unistd.h> +#ifdef ENFORCE_BUFFER_PROT +#include <sys/mman.h> /* for mprotect() */ +#endif /* ENFORCE_BUFFER_PROT */ + #include "access/tableam.h" #include "access/xloginsert.h" #include "access/xlogutils.h" @@ -53,6 +57,7 @@ #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" +#include "storage/pg_shmem.h" #include "storage/proc.h" #include "storage/read_stream.h" #include "storage/smgr.h" @@ -1065,8 +1070,6 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid) if (need_to_zero) { - memset(BufferGetPage(buffer), 0, BLCKSZ); - /* * Grab the buffer content lock before marking the page as valid, to * make sure that no other backend sees the zeroed page before the @@ -1079,7 +1082,12 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid) * already valid.) */ if (!isLocalBuf) + { LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_EXCLUSIVE); + SetBufferProtection(buffer, true, true); + } + + memset(BufferGetPage(buffer), 0, BLCKSZ); if (isLocalBuf) { @@ -1505,6 +1513,9 @@ WaitReadBuffers(ReadBuffersOperation *operation) io_first_block = blocknum + i; io_buffers_len = 1; + if (persistence != RELPERSISTENCE_TEMP) + SetBufferProtection(io_buffers[0], true, true); + /* * How many neighboring-on-disk blocks can we scatter-read into other * buffers at the same time? In this case we don't wait if we see an @@ -1520,7 +1531,13 @@ WaitReadBuffers(ReadBuffersOperation *operation) BufferGetBlockNumber(buffers[i]) + 1); io_buffers[io_buffers_len] = buffers[++i]; - io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]); + io_pages[io_buffers_len] = BufferGetBlock(buffers[i]); + + /* smgrreadv() needs to modify the buffer */ + if (persistence != RELPERSISTENCE_TEMP) + SetBufferProtection(io_buffers[io_buffers_len], true, true); + + io_buffers_len++; } io_start = pgstat_prepare_io_time(track_io_timing); @@ -1580,6 +1597,9 @@ WaitReadBuffers(ReadBuffersOperation *operation) TerminateBufferIO(bufHdr, false, BM_VALID, true); } + if (persistence != RELPERSISTENCE_TEMP) + SetBufferProtection(io_buffers[j], true, false); + /* Report I/Os as completing individually. */ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j, operation->smgr->smgr_rlocator.locator.spcOid, @@ -2233,7 +2253,9 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, buf_block = BufHdrGetBlock(GetBufferDescriptor(buffers[i] - 1)); /* new buffers are zero-filled */ + SetBufferProtection(buffers[i], true, true); MemSet((char *) buf_block, 0, BLCKSZ); + SetBufferProtection(buffers[i], true, false); } /* @@ -2460,7 +2482,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, } if (lock) - LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE); + LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); TerminateBufferIO(buf_hdr, false, BM_VALID, true); } @@ -2719,6 +2741,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * non-accessible in any case. */ VALGRIND_MAKE_MEM_DEFINED(BufHdrGetBlock(buf), BLCKSZ); + +#if defined(ENFORCE_BUFFER_PROT) && defined(ENFORCE_BUFFER_PROT_READ) + SetBufferProtection(BufferDescriptorGetBuffer(buf), true, false); +#endif break; } } @@ -2791,6 +2817,10 @@ PinBuffer_Locked(BufferDesc *buf) */ VALGRIND_MAKE_MEM_DEFINED(BufHdrGetBlock(buf), BLCKSZ); +#if defined(ENFORCE_BUFFER_PROT) && defined(ENFORCE_BUFFER_PROT_READ) + SetBufferProtection(BufferDescriptorGetBuffer(buf), true, false); +#endif + /* * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. @@ -2850,6 +2880,10 @@ UnpinBufferNoOwner(BufferDesc *buf) */ VALGRIND_MAKE_MEM_NOACCESS(BufHdrGetBlock(buf), BLCKSZ); +#if defined(ENFORCE_BUFFER_PROT) && defined(ENFORCE_BUFFER_PROT_READ) + SetBufferProtection(BufferDescriptorGetBuffer(buf), false, false); +#endif + /* I'd better not still hold the buffer content lock */ Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); @@ -3670,6 +3704,69 @@ CheckForBufferLeaks(void) #endif } +/* + * To verify that we are following buffer locking rules, we can make pages + * inaccessible or read-only when we don't have sufficient locks etc. + * + * XXX: It might be possible to fold the VALGRIND_MAKE_MEM_NOACCESS() / + * VALGRIND_MAKE_MEM_DEFINED() calls into this. + */ +#ifdef ENFORCE_BUFFER_PROT +void +SetBufferProtection(Buffer buf, bool allow_reads, bool allow_writes) +{ + static long pagesz = 0; + int prot = PROT_NONE; + int rc; + + Assert(huge_pages_status != HUGE_PAGES_UNKNOWN && + huge_pages_status != HUGE_PAGES_TRY); + StaticAssertStmt(PROT_NONE == 0, "that can't be right"); + + if (unlikely(pagesz == 0)) + { + pagesz = sysconf(_SC_PAGESIZE); + + elog(DEBUG1, "sysconf(_SC_PAGESIZE) = %ld", pagesz); + if (pagesz == -1) + { + elog(ERROR, "sysconf(_SC_PAGESIZE) failed: %m"); + } + else if (pagesz > BLCKSZ) + { + elog(DEBUG1, "pagesz > BLCKSZ, disabling buffer protection mode"); + pagesz = -1; + } + else if(BLCKSZ % pagesz != 0) + { + elog(DEBUG1, "BLCKSZ %% pagesz != 0, disabling buffer protection mode"); + pagesz = -1; + } + else if (huge_pages_status == HUGE_PAGES_ON) + { + /* can't set status in a granular enough way */ + elog(DEBUG1, "huge pages enabled, disabling buffer protection mode"); + pagesz = -1; + } + } + + /* disabled */ + if (pagesz == -1) + return; + + if (allow_reads) + prot |= PROT_READ; + + if (allow_writes) + prot |= PROT_WRITE; + + rc = mprotect(BufferGetBlock(buf), BLCKSZ, prot); + + if (rc != 0) + elog(ERROR, "mprotect(%d, %d) failed: %m", buf, prot); +} +#endif /* ENFORCE_BUFFER_PROT */ + /* * Helper routine to issue warnings when a buffer is unexpectedly pinned */ @@ -3865,7 +3962,10 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, bufBlock = BufHdrGetBlock(buf); /* Update page checksum if desired. */ + SetBufferProtection(BufferDescriptorGetBuffer(buf), true, true); PageSetChecksum((Page) bufBlock, buf->tag.blockNum); + /* FIXME: could theoretically be exclusively locked */ + SetBufferProtection(BufferDescriptorGetBuffer(buf), true, false); io_start = pgstat_prepare_io_time(track_io_timing); @@ -5181,19 +5281,38 @@ void LockBuffer(Buffer buffer, int mode) { BufferDesc *buf; + LWLock *content_lock; Assert(BufferIsPinned(buffer)); if (BufferIsLocal(buffer)) return; /* local buffers need no lock */ buf = GetBufferDescriptor(buffer - 1); + content_lock = BufferDescriptorGetContentLock(buf); if (mode == BUFFER_LOCK_UNLOCK) - LWLockRelease(BufferDescriptorGetContentLock(buf)); + { +#ifdef ENFORCE_BUFFER_PROT + bool was_exclusive; + + was_exclusive = LWLockHeldByMeInMode(content_lock, LW_EXCLUSIVE); +#endif /* ENFORCE_BUFFER_PROT */ + + LWLockRelease(content_lock); + +#ifdef ENFORCE_BUFFER_PROT + if (was_exclusive) + SetBufferProtection(buffer, true, false); +#endif /* ENFORCE_BUFFER_PROT */ + } else if (mode == BUFFER_LOCK_SHARE) - LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED); + LWLockAcquire(content_lock, LW_SHARED); else if (mode == BUFFER_LOCK_EXCLUSIVE) - LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE); + { + LWLockAcquire(content_lock, LW_EXCLUSIVE); + + SetBufferProtection(buffer, true, true); + } else elog(ERROR, "unrecognized buffer lock mode: %d", mode); } @@ -5207,6 +5326,7 @@ bool ConditionalLockBuffer(Buffer buffer) { BufferDesc *buf; + bool ret; Assert(BufferIsPinned(buffer)); if (BufferIsLocal(buffer)) @@ -5214,8 +5334,13 @@ ConditionalLockBuffer(Buffer buffer) buf = GetBufferDescriptor(buffer - 1); - return LWLockConditionalAcquire(BufferDescriptorGetContentLock(buf), - LW_EXCLUSIVE); + ret = LWLockConditionalAcquire(BufferDescriptorGetContentLock(buf), + LW_EXCLUSIVE); + + if (ret) + SetBufferProtection(buffer, true, true); + + return ret; } /* @@ -6316,6 +6441,12 @@ BufferPrepareToSetHintBits(Buffer buf) new_buf_state)) { ResourceOwnerRememberBufferSettingHints(CurrentResourceOwner, buf); + +#ifdef ENFORCE_BUFFER_PROT + if (!LWLockHeldByMeInMode(BufferDescriptorGetContentLock(desc), LW_EXCLUSIVE)) + SetBufferProtection(buf, true, true); +#endif /* ENFORCE_BUFFER_PROT */ + return true; } } @@ -6358,4 +6489,9 @@ BufferFinishSetHintBits(Buffer buf) * know if there is somebody waiting. */ ConditionVariableBroadcast(BufferDescriptorGetIOCV(desc)); + +#ifdef ENFORCE_BUFFER_PROT + if (!LWLockHeldByMeInMode(BufferDescriptorGetContentLock(desc), LW_EXCLUSIVE)) + SetBufferProtection(buf, true, false); +#endif /* ENFORCE_BUFFER_PROT */ } diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 8a67f01200c..4c94c20f0cb 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -561,7 +561,7 @@ static int ssl_renegotiation_limit; */ int huge_pages = HUGE_PAGES_TRY; int huge_page_size; -static int huge_pages_status = HUGE_PAGES_UNKNOWN; +int huge_pages_status = HUGE_PAGES_UNKNOWN; /* * These variables are all dummies that don't do anything, except in some -- 2.46.0.519.g2e7b89e038 ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-10-30 11:29 Heikki Linnakangas <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Heikki Linnakangas @ 2024-10-30 11:29 UTC (permalink / raw) To: Andres Freund <[email protected]>; Noah Misch <[email protected]>; +Cc: pgsql-hackers; Robert Haas <[email protected]>; Thomas Munro <[email protected]> On 30/10/2024 04:21, Andres Freund wrote: > Attached is a, unfortunately long, series of patches implementing what I > described upthread. Review of the preparatory patches: > 0001 Add very basic test for kill_prior_tuples > > We currently don't exercise this patch for gist and hash, which seems > somewhat criminal. Interesting to use the isolationtester for this. There's just one session, so you're just using it to define reusable steps with handy names. I'm fine with that, but please add a comment to explain it. I wonder if it'd be more straightforward to make it a regular pg_regress test though. There would be some repetition, but would it be so bad? You forgot to add the new test to 'isolation_schedule'. typos: "inex" -> "index" "does something approximately reasonble" -> "do something approximately reasonable" > 0002 heapam: Move logic to handle HEAP_MOVED into a helper function > > A prep patch, which consolidates HEAP_MOVED handling into a helper. This > isn't strictly necessary, but I got bothered by it while writing this patch > series. +1 > +/* > + * If HEAP_MOVED_OFF or HEAP_MOVED_IN are set on the tuple, remove them and > + * adjust hint bits. See the comment for SetHintBits() for more background. > + * > + * This helper returns false if the row ought to be invisible, true otherwise. > + */ > +static inline bool > +HeapTupleCleanMoved(HeapTupleHeader tuple, Buffer buffer) > +{ > + TransactionId xvac; > + > + /* only used by pre-9.0 binary upgrades */ > + if (likely(!(tuple->t_infomask & (HEAP_MOVED_OFF | HEAP_MOVED_IN)))) > + return true; > + > + ... > +} This is so unlikely these days that this function probably should not be inlined anymore. > 0003 bufmgr: Add BufferLockHeldByMe() > > Useful for writing assertions in later patches. > + if (mode == BUFFER_LOCK_EXCLUSIVE) > + lwmode = LW_EXCLUSIVE; > + else if (mode == BUFFER_LOCK_SHARE) > + lwmode = LW_SHARED; > + else > + { > + Assert(false); > + pg_unreachable(); > + lwmode = LW_EXCLUSIVE; /* assuage compiler */ > + } I just learned a new word :-). This is fine, but to avoid the assuaging, you could also do this: if (mode == BUFFER_LOCK_EXCLUSIVE) lwmode = LW_EXCLUSIVE; else { Assert(mode == BUFFER_LOCK_SHARE); lwmode = LW_SHARED; } > 0004 heapam: Use exclusive lock on old page in CLUSTER > > When I started looking into this I wanted to make sure that it's actually > safe to skip setting hint bits in all places. One place that's *not* safe > is cluster - we assume that all hint bits are set in rewrite_heap_tuple(). > > This might not strictly be required, but it seemed very fragile as > is. While using an exclusive lock does seem like it makes it a bit safer, > it's not a very good fix, as we still dirty the old buffer, which seems > like a shame when one uses VACUUM FULL. > + /* > + * To be able to guarantee that we can set the hint bit, acquire an > + * exclusive lock on the old buffer. We need the hint bits to be set > + * as otherwise reform_and_rewrite_tuple() -> rewrite_heap_tuple() -> > + * heap_freeze_tuple() will get confused. > + * Hmm, I don't see any comments in the reform_and_rewrite_tuple() -> rewrite_heap_tuple() -> heap_freeze_tuple() path about requiring hint bits to be set. How does it get confused? > 0005 heapam: Only set tuple's block once per page in pagemode > > Minor, but measurable, optimization. +1, this makes sense independently of all the other patches. > diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c > index 1748eafa100..d0cb4b1e29b 100644 > --- a/src/backend/access/heap/heapam.c > +++ b/src/backend/access/heap/heapam.c > @@ -981,6 +981,9 @@ heapgettup_pagemode(HeapScanDesc scan, > linesleft = scan->rs_ntuples; > lineindex = ScanDirectionIsForward(dir) ? 0 : linesleft - 1; > > + /* set block once per page, instead of doing so for every tuple */ > + BlockIdSet(&tuple->t_self.ip_blkid, scan->rs_cblock); > + > /* lineindex now references the next or previous visible tid */ > continue_page: > > @@ -995,7 +998,7 @@ continue_page: > > tuple->t_data = (HeapTupleHeader) PageGetItem(page, lpp); > tuple->t_len = ItemIdGetLength(lpp); > - ItemPointerSet(&(tuple->t_self), scan->rs_cblock, lineoff); > + tuple->t_self.ip_posid = lineoff; > > /* skip any tuples that don't match the scan key */ > if (key != NULL && Should probably use ItemPointerSetBlockNumber and ItemPointerSetOffsetNumber. > 0006 heapam: Add batch mode mvcc check and use it in page mode > > This is a good bit faster on its own, but is required to avoid a > performance regression later, when setting hint bits only only when no IO > going on at the same. The BATCHMVCC_FEWER_ARGS stuff is pretty weird. Hard to imagine that it'd make any difference how you pass those arguments. > + if (likely(valid)) > + { > + vistuples_dense[nvis] = tup->t_self.ip_posid; > + nvis++; > + } Is the likely(valid) hint important for performance here? It would be a bad hint for a table with lots of dead tuples. I feel we'd better rely on the branch predictor for this one. > 0007 bufmgr: Make it easier to change number of buffer state bits > > This just makes it easier to change the division of bits in > BufferDesc->state. Looks good to me. -- Heikki Linnakangas Neon (https://neon.tech) ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2024-10-30 16:45 Andres Freund <[email protected]> parent: Heikki Linnakangas <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Andres Freund @ 2024-10-30 16:45 UTC (permalink / raw) To: Heikki Linnakangas <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>; Thomas Munro <[email protected]> Hi, On 2024-10-30 13:29:01 +0200, Heikki Linnakangas wrote: > On 30/10/2024 04:21, Andres Freund wrote: > > Attached is a, unfortunately long, series of patches implementing what I > > described upthread. > > Review of the preparatory patches: > > > 0001 Add very basic test for kill_prior_tuples > > > > We currently don't exercise this patch for gist and hash, which seems > > somewhat criminal. > > Interesting to use the isolationtester for this. There's just one session, > so you're just using it to define reusable steps with handy names. Yea. I had started out writing it as a pg_regress style test and it quickly got very hard to understand. > I'm fine with that, but please add a comment to explain it. Makes sense. > I wonder if it'd be more straightforward to make it a regular pg_regress > test though. There would be some repetition, but would it be so bad? I found it to be quite bad. If testing just one AM it's ok-ish, but once you test 2-3 it gets very long and repetitive. I guess we could use functions or such to make it a bit less painful - but that point, is it actually simpler? > You forgot to add the new test to 'isolation_schedule'. Oops. > typos: > "inex" -> "index" > "does something approximately reasonble" -> "do something approximately > reasonable" Oops^2. > > +/* > > + * If HEAP_MOVED_OFF or HEAP_MOVED_IN are set on the tuple, remove them and > > + * adjust hint bits. See the comment for SetHintBits() for more background. > > + * > > + * This helper returns false if the row ought to be invisible, true otherwise. > > + */ > > +static inline bool > > +HeapTupleCleanMoved(HeapTupleHeader tuple, Buffer buffer) > > +{ > > + TransactionId xvac; > > + > > + /* only used by pre-9.0 binary upgrades */ > > + if (likely(!(tuple->t_infomask & (HEAP_MOVED_OFF | HEAP_MOVED_IN)))) > > + return true; > > + > > + ... > > +} > > This is so unlikely these days that this function probably should not be > inlined anymore. Because I put the check for moved in the function it is important that it's inlined. The compiler can understand that the rest of the function is unlikely to be executed and put it together with other less likely to be executed code. > > 0003 bufmgr: Add BufferLockHeldByMe() > > > > Useful for writing assertions in later patches. > > > + if (mode == BUFFER_LOCK_EXCLUSIVE) > > + lwmode = LW_EXCLUSIVE; > > + else if (mode == BUFFER_LOCK_SHARE) > > + lwmode = LW_SHARED; > > + else > > + { > > + Assert(false); > > + pg_unreachable(); > > + lwmode = LW_EXCLUSIVE; /* assuage compiler */ > > + } > > I just learned a new word :-). Heh. > This is fine, but to avoid the assuaging, you could also do this: > > if (mode == BUFFER_LOCK_EXCLUSIVE) > lwmode = LW_EXCLUSIVE; > else > { > Assert(mode == BUFFER_LOCK_SHARE); > lwmode = LW_SHARED; > } I have no opinion about which version is better... > > 0004 heapam: Use exclusive lock on old page in CLUSTER > > > > When I started looking into this I wanted to make sure that it's actually > > safe to skip setting hint bits in all places. One place that's *not* safe > > is cluster - we assume that all hint bits are set in rewrite_heap_tuple(). > > > > This might not strictly be required, but it seemed very fragile as > > is. While using an exclusive lock does seem like it makes it a bit safer, > > it's not a very good fix, as we still dirty the old buffer, which seems > > like a shame when one uses VACUUM FULL. > > > + /* > > + * To be able to guarantee that we can set the hint bit, acquire an > > + * exclusive lock on the old buffer. We need the hint bits to be set > > + * as otherwise reform_and_rewrite_tuple() -> rewrite_heap_tuple() -> > > + * heap_freeze_tuple() will get confused. > > + * > > Hmm, I don't see any comments in the reform_and_rewrite_tuple() -> > rewrite_heap_tuple() -> heap_freeze_tuple() path about requiring hint bits > to be set. There's this comment is in heap_prepare_freeze_tuple(): * It is assumed that the caller has checked the tuple with * HeapTupleSatisfiesVacuum() and determined that it is not HEAPTUPLE_DEAD * (else we should be removing the tuple, not freezing it). I don't remember the precise details of what all goes wrong, I think there was more than one avenue. One example is stuff like this: /* * If the tuple has been updated, check the old-to-new mapping hash table. */ if (!((old_tuple->t_data->t_infomask & HEAP_XMAX_INVALID) || HeapTupleHeaderIsOnlyLocked(old_tuple->t_data)) && !HeapTupleHeaderIndicatesMovedPartitions(old_tuple->t_data) && !(ItemPointerEquals(&(old_tuple->t_self), &(old_tuple->t_data->t_ctid)))) if the original tuple had an xmax but we didn't unset it in HeapTupleSatisfiesVacuum(), due to not being allowed to write a hint bit, we'll behave differently here. But at the same time heapam_relation_copy_for_cluster() will treat the tuple as dead and call rewrite_heap_dead_tuple(). > How does it get confused? Corrupted visibility information after a CLUSTER. > > 0006 heapam: Add batch mode mvcc check and use it in page mode > > > > This is a good bit faster on its own, but is required to avoid a > > performance regression later, when setting hint bits only only when no IO > > going on at the same. > > The BATCHMVCC_FEWER_ARGS stuff is pretty weird. Hard to imagine that it'd > make any difference how you pass those arguments. I unfortunately found it to make a difference :(, even after trying to account for binary layout difference by randomizing section order via linker flags. > > + if (likely(valid)) > > + { > > + vistuples_dense[nvis] = tup->t_self.ip_posid; > > + nvis++; > > + } > > Is the likely(valid) hint important for performance here? It would be a bad > hint for a table with lots of dead tuples. I feel we'd better rely on the > branch predictor for this one. It did :( Greetings, Andres Freund ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2025-08-13 19:38 Andres Freund <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Andres Freund @ 2025-08-13 19:38 UTC (permalink / raw) To: Heikki Linnakangas <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>; Thomas Munro <[email protected]> Hi, On 2024-10-30 12:45:27 -0400, Andres Freund wrote: > On 2024-10-30 13:29:01 +0200, Heikki Linnakangas wrote: > > On 30/10/2024 04:21, Andres Freund wrote: > > > Attached is a, unfortunately long, series of patches implementing what I > > > described upthread. > > > > Review of the preparatory patches: > > > > > 0001 Add very basic test for kill_prior_tuples > > > > > > We currently don't exercise this patch for gist and hash, which seems > > > somewhat criminal. > > > > Interesting to use the isolationtester for this. There's just one session, > > so you're just using it to define reusable steps with handy names. > > Yea. I had started out writing it as a pg_regress style test and it quickly got very > hard to understand. > > > > I'm fine with that, but please add a comment to explain it. > > Makes sense. > > > > I wonder if it'd be more straightforward to make it a regular pg_regress > > test though. There would be some repetition, but would it be so bad? > > I found it to be quite bad. If testing just one AM it's ok-ish, but once you > test 2-3 it gets very long and repetitive. I guess we could use functions or > such to make it a bit less painful - but that point, is it actually simpler? > > > > You forgot to add the new test to 'isolation_schedule'. > > Oops. > > > typos: > > "inex" -> "index" > > "does something approximately reasonble" -> "do something approximately > > reasonable" > > Oops^2. Pushed the test with these changes. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2025-10-27 11:48 Maxim Orlov <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Maxim Orlov @ 2025-10-27 11:48 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Noah Misch <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>; Thomas Munro <[email protected]> On Wed, 13 Aug 2025 at 22:38, Andres Freund <[email protected]> wrote: > > Pushed the test with these changes. I think there is one thing that could be improved here. The isolation test index-killtuples (377b7ab1452) adds a new dependency: contrib/btree_gin. contrib/btree_gist. This must be taken into account in the makefile. Prior to this patch, isolation tests could be carried out independently using "make check -C src/test/isolation", but they now require make check-world to execute successfully, otherwise this test fails with: ==== step create_ext_btree_gist: CREATE EXTENSION btree_gist; +ERROR: extension "btree_gist" is not available step create_gist: CREATE INDEX kill_prior_tuple_gist ON kill_prior_tuple USING gist (key); +ERROR: data type integer has no default operator class for access method "gist" step flush: SELECT FROM pg_stat_force_next_flush(); ==== -- Best regards, Maxim Orlov. Attachments: [application/octet-stream] 0001-Fix-isolation-tests-run-after-adding-kill_prior_tupl.patch (758B, ../../CACG=ezYQf0Np-DN0DYehYoisVRiRK1RnmZFkr-0d8RoD1NUBXQ@mail.gmail.com/3-0001-Fix-isolation-tests-run-after-adding-kill_prior_tupl.patch) download | inline diff: From d161c5d493b8851da9141282350ce6d77bb235a5 Mon Sep 17 00:00:00 2001 From: Maxim Orlov <[email protected]> Date: Mon, 27 Oct 2025 14:28:31 +0300 Subject: [PATCH] Fix isolation tests run after adding kill_prior_tuples test --- src/test/isolation/Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile index ade2256ed3a..63594a18030 100644 --- a/src/test/isolation/Makefile +++ b/src/test/isolation/Makefile @@ -5,6 +5,8 @@ PGFILEDESC = "pg_isolation_regress/isolationtester - multi-client test driver" PGAPPICON = win32 +EXTRA_INSTALL=contrib/btree_gin contrib/btree_gist + subdir = src/test/isolation top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -- 2.43.0 ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: AIO writes vs hint bits vs checksums @ 2025-10-28 04:06 Michael Paquier <[email protected]> parent: Maxim Orlov <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Michael Paquier @ 2025-10-28 04:06 UTC (permalink / raw) To: Maxim Orlov <[email protected]>; +Cc: Andres Freund <[email protected]>; Heikki Linnakangas <[email protected]>; Noah Misch <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>; Thomas Munro <[email protected]> On Mon, Oct 27, 2025 at 02:48:43PM +0300, Maxim Orlov wrote: > I think there is one thing that could be improved here. The isolation > test index-killtuples (377b7ab1452) adds a new dependency: > contrib/btree_gin. contrib/btree_gist. This must be taken into > account in the makefile. Also mentioned here: https://postgr.es/m/[email protected] It looks that the conclusion would be to move this test into its own test module to avoid the external contrib/ dependency in src/test/isolation/, as duplicated Makefile rules in src/test/isolation/Makefile are unpleasant. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2025-10-28 04:06 UTC | newest] Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-12-16 20:32 [PATCH 7/8] BRIN minmax-multi indexes Tomas Vondra <[email protected]> 2024-09-24 19:43 Re: AIO writes vs hint bits vs checksums Noah Misch <[email protected]> 2024-09-24 20:30 ` Re: AIO writes vs hint bits vs checksums Andres Freund <[email protected]> 2024-09-25 00:45 ` Re: AIO writes vs hint bits vs checksums Thomas Munro <[email protected]> 2024-09-26 21:56 ` Re: AIO writes vs hint bits vs checksums Thomas Munro <[email protected]> 2024-09-25 02:00 ` Re: AIO writes vs hint bits vs checksums Noah Misch <[email protected]> 2024-10-30 02:21 ` Re: AIO writes vs hint bits vs checksums Andres Freund <[email protected]> 2024-10-30 11:29 ` Re: AIO writes vs hint bits vs checksums Heikki Linnakangas <[email protected]> 2024-10-30 16:45 ` Re: AIO writes vs hint bits vs checksums Andres Freund <[email protected]> 2025-08-13 19:38 ` Re: AIO writes vs hint bits vs checksums Andres Freund <[email protected]> 2025-10-27 11:48 ` Re: AIO writes vs hint bits vs checksums Maxim Orlov <[email protected]> 2025-10-28 04:06 ` Re: AIO writes vs hint bits vs checksums Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox