public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 5/6] BRIN minmax-multi indexes
59+ messages / 12 participants
[nested] [flat]
* [PATCH 5/6] BRIN minmax-multi indexes
@ 2021-02-03 18:00 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2021-02-03 18:00 UTC (permalink / raw)
Adds a BRIN minmax-multi opclass, summarizing each range into a list of
intervals, allowing more efficient handling of cases where the minmax
opclasses quickly degrade.
Instead of representing each range with a single interval, minmax-multi
opclasses store a list of intervals and points. This allows effective
handling of outliers and poorly correlated values.
The number of intervals/points per range may be configured using an
opclass parameter values_per_range. When building the summary, the
interval are merged based on distance, determined by the new support
BRIN procedure.
Author: Tomas Vondra <[email protected]>
Reviewed-by: Alvaro Herrera <[email protected]>
Reviewed-by: Alexander Korotkov <[email protected]>
Reviewed-by: Sokolov Yura <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/5d78b774-7e9c-c94e-12cf-fef51cc89b1a%402ndquadrant.com
---
doc/src/sgml/brin.sgml | 252 +-
doc/src/sgml/ref/create_index.sgml | 11 +
src/backend/access/brin/Makefile | 1 +
src/backend/access/brin/brin_minmax_multi.c | 2989 +++++++++++++++++++
src/backend/access/brin/brin_tuple.c | 33 +-
src/include/access/brin.h | 1 +
src/include/access/brin_internal.h | 3 +
src/include/access/brin_tuple.h | 8 +
src/include/access/transam.h | 2 +-
src/include/catalog/pg_amop.dat | 544 ++++
src/include/catalog/pg_amproc.dat | 600 +++-
src/include/catalog/pg_opclass.dat | 57 +
src/include/catalog/pg_opfamily.dat | 28 +
src/include/catalog/pg_proc.dat | 85 +
src/include/catalog/pg_type.dat | 6 +
src/test/regress/expected/brin_multi.out | 445 +++
src/test/regress/expected/psql.out | 13 +-
src/test/regress/expected/type_sanity.out | 7 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/brin_multi.sql | 397 +++
21 files changed, 5458 insertions(+), 27 deletions(-)
create mode 100644 src/backend/access/brin/brin_minmax_multi.c
create mode 100644 src/test/regress/expected/brin_multi.out
create mode 100644 src/test/regress/sql/brin_multi.sql
diff --git a/doc/src/sgml/brin.sgml b/doc/src/sgml/brin.sgml
index 577dd18c49..3caa30f104 100644
--- a/doc/src/sgml/brin.sgml
+++ b/doc/src/sgml/brin.sgml
@@ -214,6 +214,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (date,date)</literal></entry></row>
<row><entry><literal>>= (date,date)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>date_minmax_multi_ops</literal></entry>
+ <entry><literal>= (date,date)</literal></entry>
+ </row>
+ <row><entry><literal>< (date,date)</literal></entry></row>
+ <row><entry><literal><= (date,date)</literal></entry></row>
+ <row><entry><literal>> (date,date)</literal></entry></row>
+ <row><entry><literal>>= (date,date)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>float4_bloom_ops</literal></entry>
<entry><literal>= (float4,float4)</literal></entry>
@@ -228,6 +237,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (float4,float4)</literal></entry></row>
<row><entry><literal>>= (float4,float4)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>float4_minmax_multi_ops</literal></entry>
+ <entry><literal>= (float4,float4)</literal></entry>
+ </row>
+ <row><entry><literal>< (float4,float4)</literal></entry></row>
+ <row><entry><literal>> (float4,float4)</literal></entry></row>
+ <row><entry><literal><= (float4,float4)</literal></entry></row>
+ <row><entry><literal>>= (float4,float4)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>float8_bloom_ops</literal></entry>
<entry><literal>= (float8,float8)</literal></entry>
@@ -242,6 +260,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (float8,float8)</literal></entry></row>
<row><entry><literal>>= (float8,float8)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>float8_minmax_multi_ops</literal></entry>
+ <entry><literal>= (float8,float8)</literal></entry>
+ </row>
+ <row><entry><literal>< (float8,float8)</literal></entry></row>
+ <row><entry><literal><= (float8,float8)</literal></entry></row>
+ <row><entry><literal>> (float8,float8)</literal></entry></row>
+ <row><entry><literal>>= (float8,float8)</literal></entry></row>
+
<row>
<entry valign="middle" morerows="5"><literal>inet_inclusion_ops</literal></entry>
<entry><literal><< (inet,inet)</literal></entry>
@@ -266,6 +293,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (inet,inet)</literal></entry></row>
<row><entry><literal>>= (inet,inet)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>inet_minmax_multi_ops</literal></entry>
+ <entry><literal>= (inet,inet)</literal></entry>
+ </row>
+ <row><entry><literal>< (inet,inet)</literal></entry></row>
+ <row><entry><literal><= (inet,inet)</literal></entry></row>
+ <row><entry><literal>> (inet,inet)</literal></entry></row>
+ <row><entry><literal>>= (inet,inet)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>int2_bloom_ops</literal></entry>
<entry><literal>= (int2,int2)</literal></entry>
@@ -280,6 +316,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (int2,int2)</literal></entry></row>
<row><entry><literal>>= (int2,int2)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>int2_minmax_multi_ops</literal></entry>
+ <entry><literal>= (int2,int2)</literal></entry>
+ </row>
+ <row><entry><literal>< (int2,int2)</literal></entry></row>
+ <row><entry><literal>> (int2,int2)</literal></entry></row>
+ <row><entry><literal><= (int2,int2)</literal></entry></row>
+ <row><entry><literal>>= (int2,int2)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>int4_bloom_ops</literal></entry>
<entry><literal>= (int4,int4)</literal></entry>
@@ -294,6 +339,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (int4,int4)</literal></entry></row>
<row><entry><literal>>= (int4,int4)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>int4_minmax_multi_ops</literal></entry>
+ <entry><literal>= (int4,int4)</literal></entry>
+ </row>
+ <row><entry><literal>< (int4,int4)</literal></entry></row>
+ <row><entry><literal>> (int4,int4)</literal></entry></row>
+ <row><entry><literal><= (int4,int4)</literal></entry></row>
+ <row><entry><literal>>= (int4,int4)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>int8_bloom_ops</literal></entry>
<entry><literal>= (bigint,bigint)</literal></entry>
@@ -308,6 +362,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (bigint,bigint)</literal></entry></row>
<row><entry><literal>>= (bigint,bigint)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>int8_minmax_multi_ops</literal></entry>
+ <entry><literal>= (bigint,bigint)</literal></entry>
+ </row>
+ <row><entry><literal>< (bigint,bigint)</literal></entry></row>
+ <row><entry><literal>> (bigint,bigint)</literal></entry></row>
+ <row><entry><literal><= (bigint,bigint)</literal></entry></row>
+ <row><entry><literal>>= (bigint,bigint)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>interval_bloom_ops</literal></entry>
<entry><literal>= (interval,interval)</literal></entry>
@@ -322,6 +385,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (interval,interval)</literal></entry></row>
<row><entry><literal>>= (interval,interval)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>interval_minmax_multi_ops</literal></entry>
+ <entry><literal>= (interval,interval)</literal></entry>
+ </row>
+ <row><entry><literal>< (interval,interval)</literal></entry></row>
+ <row><entry><literal><= (interval,interval)</literal></entry></row>
+ <row><entry><literal>> (interval,interval)</literal></entry></row>
+ <row><entry><literal>>= (interval,interval)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>macaddr_bloom_ops</literal></entry>
<entry><literal>= (macaddr,macaddr)</literal></entry>
@@ -336,6 +408,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (macaddr,macaddr)</literal></entry></row>
<row><entry><literal>>= (macaddr,macaddr)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>macaddr_minmax_multi_ops</literal></entry>
+ <entry><literal>= (macaddr,macaddr)</literal></entry>
+ </row>
+ <row><entry><literal>< (macaddr,macaddr)</literal></entry></row>
+ <row><entry><literal><= (macaddr,macaddr)</literal></entry></row>
+ <row><entry><literal>> (macaddr,macaddr)</literal></entry></row>
+ <row><entry><literal>>= (macaddr,macaddr)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>macaddr8_bloom_ops</literal></entry>
<entry><literal>= (macaddr8,macaddr8)</literal></entry>
@@ -350,6 +431,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (macaddr8,macaddr8)</literal></entry></row>
<row><entry><literal>>= (macaddr8,macaddr8)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>macaddr8_minmax_multi_ops</literal></entry>
+ <entry><literal>= (macaddr8,macaddr8)</literal></entry>
+ </row>
+ <row><entry><literal>< (macaddr8,macaddr8)</literal></entry></row>
+ <row><entry><literal><= (macaddr8,macaddr8)</literal></entry></row>
+ <row><entry><literal>> (macaddr8,macaddr8)</literal></entry></row>
+ <row><entry><literal>>= (macaddr8,macaddr8)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>name_bloom_ops</literal></entry>
<entry><literal>= (name,name)</literal></entry>
@@ -378,6 +468,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (numeric,numeric)</literal></entry></row>
<row><entry><literal>>= (numeric,numeric)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>numeric_minmax_multi_ops</literal></entry>
+ <entry><literal>= (numeric,numeric)</literal></entry>
+ </row>
+ <row><entry><literal>< (numeric,numeric)</literal></entry></row>
+ <row><entry><literal><= (numeric,numeric)</literal></entry></row>
+ <row><entry><literal>> (numeric,numeric)</literal></entry></row>
+ <row><entry><literal>>= (numeric,numeric)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>oid_bloom_ops</literal></entry>
<entry><literal>= (oid,oid)</literal></entry>
@@ -392,6 +491,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (oid,oid)</literal></entry></row>
<row><entry><literal>>= (oid,oid)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>oid_minmax_multi_ops</literal></entry>
+ <entry><literal>= (oid,oid)</literal></entry>
+ </row>
+ <row><entry><literal>< (oid,oid)</literal></entry></row>
+ <row><entry><literal>> (oid,oid)</literal></entry></row>
+ <row><entry><literal><= (oid,oid)</literal></entry></row>
+ <row><entry><literal>>= (oid,oid)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>pg_lsn_bloom_ops</literal></entry>
<entry><literal>= (pg_lsn,pg_lsn)</literal></entry>
@@ -406,6 +514,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (pg_lsn,pg_lsn)</literal></entry></row>
<row><entry><literal>>= (pg_lsn,pg_lsn)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>pg_lsn_minmax_multi_ops</literal></entry>
+ <entry><literal>= (pg_lsn,pg_lsn)</literal></entry>
+ </row>
+ <row><entry><literal>< (pg_lsn,pg_lsn)</literal></entry></row>
+ <row><entry><literal>> (pg_lsn,pg_lsn)</literal></entry></row>
+ <row><entry><literal><= (pg_lsn,pg_lsn)</literal></entry></row>
+ <row><entry><literal>>= (pg_lsn,pg_lsn)</literal></entry></row>
+
<row>
<entry valign="middle" morerows="13"><literal>range_inclusion_ops</literal></entry>
<entry><literal>= (anyrange,anyrange)</literal></entry>
@@ -452,6 +569,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (tid,tid)</literal></entry></row>
<row><entry><literal>>= (tid,tid)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>tid_minmax_multi_ops</literal></entry>
+ <entry><literal>= (tid,tid)</literal></entry>
+ </row>
+ <row><entry><literal>< (tid,tid)</literal></entry></row>
+ <row><entry><literal>> (tid,tid)</literal></entry></row>
+ <row><entry><literal><= (tid,tid)</literal></entry></row>
+ <row><entry><literal>>= (tid,tid)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>timestamp_bloom_ops</literal></entry>
<entry><literal>= (timestamp,timestamp)</literal></entry>
@@ -466,6 +592,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (timestamp,timestamp)</literal></entry></row>
<row><entry><literal>>= (timestamp,timestamp)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>timestamp_minmax_multi_ops</literal></entry>
+ <entry><literal>= (timestamp,timestamp)</literal></entry>
+ </row>
+ <row><entry><literal>< (timestamp,timestamp)</literal></entry></row>
+ <row><entry><literal><= (timestamp,timestamp)</literal></entry></row>
+ <row><entry><literal>> (timestamp,timestamp)</literal></entry></row>
+ <row><entry><literal>>= (timestamp,timestamp)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>timestamptz_bloom_ops</literal></entry>
<entry><literal>= (timestamptz,timestamptz)</literal></entry>
@@ -480,6 +615,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (timestamptz,timestamptz)</literal></entry></row>
<row><entry><literal>>= (timestamptz,timestamptz)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>timestamptz_minmax_multi_ops</literal></entry>
+ <entry><literal>= (timestamptz,timestamptz)</literal></entry>
+ </row>
+ <row><entry><literal>< (timestamptz,timestamptz)</literal></entry></row>
+ <row><entry><literal><= (timestamptz,timestamptz)</literal></entry></row>
+ <row><entry><literal>> (timestamptz,timestamptz)</literal></entry></row>
+ <row><entry><literal>>= (timestamptz,timestamptz)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>time_bloom_ops</literal></entry>
<entry><literal>= (time,time)</literal></entry>
@@ -494,6 +638,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (time,time)</literal></entry></row>
<row><entry><literal>>= (time,time)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>time_minmax_multi_ops</literal></entry>
+ <entry><literal>= (time,time)</literal></entry>
+ </row>
+ <row><entry><literal>< (time,time)</literal></entry></row>
+ <row><entry><literal><= (time,time)</literal></entry></row>
+ <row><entry><literal>> (time,time)</literal></entry></row>
+ <row><entry><literal>>= (time,time)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>timetz_bloom_ops</literal></entry>
<entry><literal>= (timetz,timetz)</literal></entry>
@@ -508,6 +661,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal>> (timetz,timetz)</literal></entry></row>
<row><entry><literal>>= (timetz,timetz)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>timetz_minmax_multi_ops</literal></entry>
+ <entry><literal>= (timetz,timetz)</literal></entry>
+ </row>
+ <row><entry><literal>< (timetz,timetz)</literal></entry></row>
+ <row><entry><literal><= (timetz,timetz)</literal></entry></row>
+ <row><entry><literal>> (timetz,timetz)</literal></entry></row>
+ <row><entry><literal>>= (timetz,timetz)</literal></entry></row>
+
<row>
<entry valign="middle"><literal>uuid_bloom_ops</literal></entry>
<entry><literal>= (uuid,uuid)</literal></entry>
@@ -522,6 +684,15 @@ LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was
<row><entry><literal><= (uuid,uuid)</literal></entry></row>
<row><entry><literal>>= (uuid,uuid)</literal></entry></row>
+ <row>
+ <entry valign="middle" morerows="4"><literal>uuid_minmax_multi_ops</literal></entry>
+ <entry><literal>= (uuid,uuid)</literal></entry>
+ </row>
+ <row><entry><literal>< (uuid,uuid)</literal></entry></row>
+ <row><entry><literal>> (uuid,uuid)</literal></entry></row>
+ <row><entry><literal><= (uuid,uuid)</literal></entry></row>
+ <row><entry><literal>>= (uuid,uuid)</literal></entry></row>
+
<row>
<entry valign="middle" morerows="4"><literal>varbit_minmax_ops</literal></entry>
<entry><literal>= (varbit,varbit)</literal></entry>
@@ -654,13 +825,14 @@ typedef struct BrinOpcInfo
</varlistentry>
</variablelist>
- The core distribution includes support for two types of operator classes:
- minmax and inclusion. Operator class definitions using them are shipped for
- in-core data types as appropriate. Additional operator classes can be
- defined by the user for other data types using equivalent definitions,
- without having to write any source code; appropriate catalog entries being
- declared is enough. Note that assumptions about the semantics of operator
- strategies are embedded in the support functions' source code.
+ The core distribution includes support for four types of operator classes:
+ minmax, multi-minmax, inclusion and bloom. Operator class definitions
+ using them are shipped for in-core data types as appropriate. Additional
+ operator classes can be defined by the user for other data types using
+ equivalent definitions, without having to write any source code;
+ appropriate catalog entries being declared is enough. Note that
+ assumptions about the semantics of operator strategies are embedded in the
+ support functions' source code.
</para>
<para>
@@ -957,6 +1129,72 @@ typedef struct BrinOpcInfo
and return a hash of the value.
</para>
+ <para>
+ The multi-minmax operator class is also intended for data types implementing
+ a totally ordered sets, and may be seen as a simple extension of the minmax
+ operator class. While minmax operator class summarizes values from each block
+ range into a single contiguous interval, multi-minmax allows summarization
+ into multiple smaller intervals to improve handling of outlier values.
+ It is possible to use the multi-minmax support procedures alongside the
+ corresponding operators, as shown in
+ <xref linkend="brin-extensibility-multi-minmax-table"/>.
+ All operator class members (procedures and operators) are mandatory.
+ </para>
+
+ <table id="brin-extensibility-multi-minmax-table">
+ <title>Procedure and Support Numbers for Multi-Minmax Operator Classes</title>
+ <tgroup cols="2">
+ <thead>
+ <row>
+ <entry>Operator class member</entry>
+ <entry>Object</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Support Procedure 1</entry>
+ <entry>internal function <function>brin_minmax_multi_opcinfo()</function></entry>
+ </row>
+ <row>
+ <entry>Support Procedure 2</entry>
+ <entry>internal function <function>brin_minmax_multi_add_value()</function></entry>
+ </row>
+ <row>
+ <entry>Support Procedure 3</entry>
+ <entry>internal function <function>brin_minmax_multi_consistent()</function></entry>
+ </row>
+ <row>
+ <entry>Support Procedure 4</entry>
+ <entry>internal function <function>brin_minmax_multi_union()</function></entry>
+ </row>
+ <row>
+ <entry>Support Procedure 11</entry>
+ <entry>function to compute distance between two values (length of a range)</entry>
+ </row>
+ <row>
+ <entry>Operator Strategy 1</entry>
+ <entry>operator less-than</entry>
+ </row>
+ <row>
+ <entry>Operator Strategy 2</entry>
+ <entry>operator less-than-or-equal-to</entry>
+ </row>
+ <row>
+ <entry>Operator Strategy 3</entry>
+ <entry>operator equal-to</entry>
+ </row>
+ <row>
+ <entry>Operator Strategy 4</entry>
+ <entry>operator greater-than-or-equal-to</entry>
+ </row>
+ <row>
+ <entry>Operator Strategy 5</entry>
+ <entry>operator greater-than</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
<para>
Both minmax and inclusion operator classes support cross-data-type
operators, though with these the dependencies become more complicated.
diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgml
index a45691873c..e1f0f4c0a0 100644
--- a/doc/src/sgml/ref/create_index.sgml
+++ b/doc/src/sgml/ref/create_index.sgml
@@ -612,6 +612,17 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ [ IF NOT EXISTS ] <replaceable class=
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>values_per_range</literal></term>
+ <listitem>
+ <para>
+ Defines the maximum number of values stored by <acronym>BRIN</acronym>
+ minmax indexes to summarize a block range. Each value may represent
+ eiter a point, or boundary of an interval. Values must be be between
+ 8 and 256, and the default value is 32.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</refsect2>
diff --git a/src/backend/access/brin/Makefile b/src/backend/access/brin/Makefile
index 6b56131215..a386cb71f1 100644
--- a/src/backend/access/brin/Makefile
+++ b/src/backend/access/brin/Makefile
@@ -17,6 +17,7 @@ OBJS = \
brin_bloom.o \
brin_inclusion.o \
brin_minmax.o \
+ brin_minmax_multi.o \
brin_pageops.o \
brin_revmap.o \
brin_tuple.o \
diff --git a/src/backend/access/brin/brin_minmax_multi.c b/src/backend/access/brin/brin_minmax_multi.c
new file mode 100644
index 0000000000..c522bf2285
--- /dev/null
+++ b/src/backend/access/brin/brin_minmax_multi.c
@@ -0,0 +1,2989 @@
+/*
+ * brin_minmax_multi.c
+ * Implementation of Multi Min/Max opclass for BRIN
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * Implements a variant of minmax opclass, where the summary is composed of
+ * multiple smaller intervals. This allows us to handle outliers, which
+ * usually makes the simple minmax opclass inefficient.
+ *
+ * Consider for example page range with simple minmax interval [1000,2000],
+ * and assume a new row gets inserted into the range with value 1000000.
+ * Due to that the interval gets [1000,1000000]. I.e. the minmax interval
+ * got 1000x wider and won't be useful to eliminate scan keys between 2001
+ * and 1000000.
+ *
+ * With multi-minmax opclass, we may have [1000,2000] interval initially,
+ * but after adding the new row we start tracking it as two interval:
+ *
+ * [1000,2000] and [1000000,1000000]
+ *
+ * This allow us to still eliminate the page range when the scan keys hit
+ * the gap between 2000 and 1000000, making it useful in cases when the
+ * simple minmax opclass gets inefficient.
+ *
+ * The number of intervals tracked per page range is somewhat flexible.
+ * What is restricted is the number of values per page range, and the limit
+ * is currently 64 (see values_per_range reloption). Collapsed intervals
+ * (with equal minimum and maximum value) are stored as a single value,
+ * while regular intervals require two values.
+ *
+ * When the number of values gets too high (by adding new values to the
+ * summary), we merge some of the intervals to free space for more values.
+ * This is done in a greedy way - we simply pick the two closest intervals,
+ * merge them, and repeat this until the number of values to store gets
+ * sufficiently low (below 75% of maximum values), but that is mostly
+ * arbitrary threshold and may be changed easily).
+ *
+ * To pick the closest intervals we use the "distance" support procedure,
+ * which measures space between two ranges (i.e. length of an interval).
+ * The computed value may be an approximation - in the worst case we will
+ * merge two ranges that are slightly less optimal at that step, but the
+ * index should still produce correct results.
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/brin/brin_minmax_multi.c
+ */
+#include "postgres.h"
+
+/* needef for PGSQL_AF_INET */
+#include <sys/socket.h>
+
+#include "access/genam.h"
+#include "access/brin.h"
+#include "access/brin_internal.h"
+#include "access/brin_tuple.h"
+#include "access/reloptions.h"
+#include "access/stratnum.h"
+#include "access/htup_details.h"
+#include "catalog/pg_type.h"
+#include "catalog/pg_am.h"
+#include "catalog/pg_amop.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
+#include "utils/date.h"
+#include "utils/datum.h"
+#include "utils/inet.h"
+#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+#include "utils/numeric.h"
+#include "utils/pg_lsn.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
+#include "utils/timestamp.h"
+#include "utils/uuid.h"
+
+/*
+ * Additional SQL level support functions
+ *
+ * Procedure numbers must not use values reserved for BRIN itself; see
+ * brin_internal.h.
+ */
+#define MINMAX_MAX_PROCNUMS 1 /* maximum support procs we need */
+#define PROCNUM_DISTANCE 11 /* required, distance between values */
+
+/*
+ * Subtract this from procnum to obtain index in MinmaxMultiOpaque arrays
+ * (Must be equal to minimum of private procnums).
+ */
+#define PROCNUM_BASE 11
+
+/*
+ * Sizing the insert buffer - we use 10x the number of values specified
+ * in the reloption, but we cap it to 8192 not to get too large. When
+ * the buffer gets full, we reduce the number of values by half.
+ */
+#define MINMAX_BUFFER_FACTOR 10
+#define MINMAX_BUFFER_MIN 256
+#define MINMAX_BUFFER_MAX 8192
+#define MINMAX_BUFFER_LOAD_FACTOR 0.5
+
+typedef struct MinmaxMultiOpaque
+{
+ FmgrInfo extra_procinfos[MINMAX_MAX_PROCNUMS];
+ bool extra_proc_missing[MINMAX_MAX_PROCNUMS];
+ Oid cached_subtype;
+ FmgrInfo strategy_procinfos[BTMaxStrategyNumber];
+} MinmaxMultiOpaque;
+
+/*
+ * Storage type for BRIN's minmax reloptions
+ */
+typedef struct MinMaxOptions
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ int valuesPerRange; /* number of values per range */
+} MinMaxOptions;
+
+#define MINMAX_DEFAULT_VALUES_PER_PAGE 32
+
+#define MinMaxGetValuesPerRange(opts) \
+ ((opts) && (((MinMaxOptions *) (opts))->valuesPerRange != 0) ? \
+ ((MinMaxOptions *) (opts))->valuesPerRange : \
+ MINMAX_DEFAULT_VALUES_PER_PAGE)
+
+#define SAMESIGN(a,b) (((a) < 0) == ((b) < 0))
+
+/*
+ * The summary of multi-minmax indexes has two representations - Ranges for
+ * convenient processing, and SerializedRanges for storage in bytea value.
+ *
+ * The Ranges struct stores the boundary values in a single array, but we
+ * treat regular and single-point ranges differently to save space. For
+ * regular ranges (with different boundary values) we have to store both
+ * values, while for "single-point ranges" we only need to save one value.
+ *
+ * The 'values' array stores boundary values for regular ranges first (there
+ * are 2*nranges values to store), and then the nvalues boundary values for
+ * single-point ranges. That is, we have (2*nranges + nvalues) boundary
+ * values in the array.
+ *
+ * +---------------------------------+-------------------------------+
+ * | ranges (sorted pairs of values) | sorted values (single points) |
+ * +---------------------------------+-------------------------------+
+ *
+ * This allows us to quickly add new values, and store outliers without
+ * making the other ranges very wide.
+ *
+ * We never store more than maxvalues values (as set by values_per_range
+ * reloption). If needed we merge some of the ranges.
+ *
+ * To minimize palloc overhead, we always allocate the full array with
+ * space for maxvalues elements. This should be fine as long as the
+ * maxvalues is reasonably small (64 seems fine), which is the case
+ * thanks to values_per_range reloption being limited to 256.
+ */
+typedef struct Ranges
+{
+ /* Cache information that we need quite often. */
+ Oid typid;
+ Oid colloid;
+ AttrNumber attno;
+ FmgrInfo *cmp;
+
+ /* (2*nranges + nvalues) <= maxvalues */
+ int nranges; /* number of ranges in the array (stored) */
+ int nsorted; /* number of sorted values (ranges + points) */
+ int nvalues; /* number of values in the data array (all) */
+ int maxvalues; /* maximum number of values (reloption) */
+
+ /*
+ * We simply add the values into a large buffer, without any expensive
+ * steps (sorting, deduplication, ...). The buffer is a multiple of the
+ * target number of values, so the compaction happen less often,
+ * amortizing the costs. We keep the actual target and compact to the
+ * requested number of values at the very end, before serializing to
+ * on-disk representation.
+ */
+ /* requested number of values */
+ int target_maxvalues;
+
+ /* values stored for this range - either raw values, or ranges */
+ Datum values[FLEXIBLE_ARRAY_MEMBER];
+} Ranges;
+
+/*
+ * On-disk the summary is stored as a bytea value, with a simple header
+ * with basic metadata, followed by the boundary values. It has a varlena
+ * header, so can be treated as varlena directly.
+ *
+ * See range_serialize/range_deserialize for serialization details.
+ */
+typedef struct SerializedRanges
+{
+ /* varlena header (do not touch directly!) */
+ int32 vl_len_;
+
+ /* type of values stored in the data array */
+ Oid typid;
+
+ /* (2*nranges + nvalues) <= maxvalues */
+ int nranges; /* number of ranges in the array (stored) */
+ int nvalues; /* number of values in the data array (all) */
+ int maxvalues; /* maximum number of values (reloption) */
+
+ /* contains the actual data */
+ char data[FLEXIBLE_ARRAY_MEMBER];
+} SerializedRanges;
+
+static SerializedRanges *range_serialize(Ranges *range);
+
+static Ranges *range_deserialize(int maxvalues, SerializedRanges *range);
+
+
+/*
+ * Used to represent ranges expanded to make merging and combining easier.
+ *
+ * Each expanded range is essentially an interval, represented by min/max
+ * values, along with a flag whether it's a collapsed range (in which case
+ * the min and max values are equal). We have the flag to handle by-ref
+ * data types - we can't simply compare the datums, and this saves some
+ * calls to the type-specific comparator function.
+ */
+typedef struct ExpandedRange
+{
+ Datum minval; /* lower boundary */
+ Datum maxval; /* upper boundary */
+ bool collapsed; /* true if minval==maxval */
+} ExpandedRange;
+
+/*
+ * Represents a distance between two ranges (identified by index into
+ * an array of extended ranges).
+ */
+typedef struct DistanceValue
+{
+ int index;
+ double value;
+} DistanceValue;
+
+
+/* Cache for support and strategy procesures. */
+
+static FmgrInfo *minmax_multi_get_procinfo(BrinDesc *bdesc, uint16 attno,
+ uint16 procnum);
+
+static FmgrInfo *minmax_multi_get_strategy_procinfo(BrinDesc *bdesc,
+ uint16 attno, Oid subtype,
+ uint16 strategynum);
+
+typedef struct compare_context
+{
+ FmgrInfo *cmpFn;
+ Oid colloid;
+} compare_context;
+
+static int compare_values(const void *a, const void *b, void *arg);
+
+/* FIXME we have this in extended_stats_internal.h, which seems silly to
+ * include here */
+void *bsearch_arg(const void *key, const void *base,
+ size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg);
+
+
+#ifdef USE_ASSERT_CHECKING
+/*
+ * Check that the order of the array values is correct, using the cmp
+ * function (which should be BTLessStrategyNumber).
+ */
+static void
+AssertArrayOrder(FmgrInfo *cmp, Oid colloid, Datum *values, int nvalues)
+{
+ int i;
+ Datum lt;
+
+ for (i = 0; i < (nvalues - 1); i++)
+ {
+ lt = FunctionCall2Coll(cmp, colloid, values[i], values[i + 1]);
+ Assert(DatumGetBool(lt));
+ }
+}
+#endif
+
+/*
+ * Comprehensive check of the Ranges structure.
+ */
+static void
+AssertCheckRanges(Ranges *ranges, FmgrInfo *cmpFn, Oid colloid)
+{
+#ifdef USE_ASSERT_CHECKING
+ int i;
+
+ /* some basic sanity checks */
+ Assert(ranges->nranges >= 0);
+ Assert(ranges->nsorted >= 0);
+ Assert(ranges->nvalues >= ranges->nsorted);
+ Assert(ranges->maxvalues >= 2 * ranges->nranges + ranges->nvalues);
+ Assert(ranges->typid != InvalidOid);
+
+ /*
+ * First the ranges - there are 2*nranges boundary values, and the values
+ * have to be strictly ordered (equal values would mean the range is
+ * collapsed, and should be stored as a point). This also guarantees that
+ * the ranges do not overlap.
+ */
+ AssertArrayOrder(cmpFn, colloid, ranges->values, 2 * ranges->nranges);
+
+ /* then the single-point ranges (with nvalues boundar values ) */
+ AssertArrayOrder(cmpFn, colloid, &ranges->values[2 * ranges->nranges],
+ ranges->nsorted);
+
+ /*
+ * Check that none of the values are not covered by ranges (both sorted
+ * and unsorted)
+ */
+ for (i = 0; i < ranges->nvalues; i++)
+ {
+ Datum compar;
+ int start,
+ end;
+ Datum minvalue,
+ maxvalue;
+
+ Datum value = ranges->values[2 * ranges->nranges + i];
+
+ if (ranges->nranges == 0)
+ break;
+
+ minvalue = ranges->values[0];
+ maxvalue = ranges->values[2 * ranges->nranges - 1];
+
+ /*
+ * Is the value smaller than the minval? If yes, we'll recurse to the
+ * left side of range array.
+ */
+ compar = FunctionCall2Coll(cmpFn, colloid, value, minvalue);
+
+ /* smaller than the smallest value in the first range */
+ if (DatumGetBool(compar))
+ continue;
+
+ /*
+ * Is the value greater than the minval? If yes, we'll recurse to the
+ * right side of range array.
+ */
+ compar = FunctionCall2Coll(cmpFn, colloid, maxvalue, value);
+
+ /* larger than the largest value in the last range */
+ if (DatumGetBool(compar))
+ continue;
+
+ start = 0; /* first range */
+ end = ranges->nranges - 1; /* last range */
+ while (true)
+ {
+ int midpoint = (start + end) / 2;
+
+ /* this means we ran out of ranges in the last step */
+ if (start > end)
+ break;
+
+ /* copy the min/max values from the ranges */
+ minvalue = ranges->values[2 * midpoint];
+ maxvalue = ranges->values[2 * midpoint + 1];
+
+ /*
+ * Is the value smaller than the minval? If yes, we'll recurse to
+ * the left side of range array.
+ */
+ compar = FunctionCall2Coll(cmpFn, colloid, value, minvalue);
+
+ /* smaller than the smallest value in this range */
+ if (DatumGetBool(compar))
+ {
+ end = (midpoint - 1);
+ continue;
+ }
+
+ /*
+ * Is the value greater than the minval? If yes, we'll recurse to
+ * the right side of range array.
+ */
+ compar = FunctionCall2Coll(cmpFn, colloid, maxvalue, value);
+
+ /* larger than the largest value in this range */
+ if (DatumGetBool(compar))
+ {
+ start = (midpoint + 1);
+ continue;
+ }
+
+ /* hey, we found a matching range */
+ Assert(false);
+ }
+ }
+
+ /* and values in the unsorted part must not be in sorted part */
+ for (i = ranges->nsorted; i < ranges->nvalues; i++)
+ {
+ compare_context cxt;
+ Datum value = ranges->values[2 * ranges->nranges + i];
+
+ if (ranges->nsorted == 0)
+ break;
+
+ cxt.colloid = ranges->colloid;
+ cxt.cmpFn = ranges->cmp;
+
+ Assert(bsearch_arg(&value, &ranges->values[2 * ranges->nranges],
+ ranges->nsorted, sizeof(Datum),
+ compare_values, (void *) &cxt) == NULL);
+ }
+#endif
+}
+
+/*
+ * Check that the expanded ranges (built when reducing the number of ranges
+ * by combining some of them) are correctly sorted and do not overlap.
+ */
+static void
+AssertValidExpandedRanges(BrinDesc *bdesc, Oid colloid, AttrNumber attno,
+ Form_pg_attribute attr, ExpandedRange *ranges,
+ int nranges)
+{
+#ifdef USE_ASSERT_CHECKING
+ int i;
+ FmgrInfo *eq;
+ FmgrInfo *lt;
+
+ eq = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTEqualStrategyNumber);
+
+ lt = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTLessStrategyNumber);
+
+ /*
+ * Each range independently should be valid, i.e. that for the boundary
+ * values (lower <= upper).
+ */
+ for (i = 0; i < nranges; i++)
+ {
+ Datum r;
+ Datum minval = ranges[i].minval;
+ Datum maxval = ranges[i].maxval;
+
+ if (ranges[i].collapsed) /* collapsed: minval == maxval */
+ r = FunctionCall2Coll(eq, colloid, minval, maxval);
+ else /* non-collapsed: minval < maxval */
+ r = FunctionCall2Coll(lt, colloid, minval, maxval);
+
+ Assert(DatumGetBool(r));
+ }
+
+ /*
+ * And the ranges should be ordered and must nor overlap, i.e. upper <
+ * lower for boundaries of consecutive ranges.
+ */
+ for (i = 0; i < nranges - 1; i++)
+ {
+ Datum r;
+ Datum maxval = ranges[i].maxval;
+ Datum minval = ranges[i + 1].minval;
+
+ r = FunctionCall2Coll(lt, colloid, maxval, minval);
+
+ Assert(DatumGetBool(r));
+ }
+#endif
+}
+
+
+/*
+ * minmax_multi_init
+ * Initialize the deserialized range list, allocate all the memory.
+ *
+ * This is only in-memory representation of the ranges, so we allocate
+ * everything enough space for the maximum number of values (so as not
+ * to have to do repallocs as the ranges grow).
+ */
+static Ranges *
+minmax_multi_init(int maxvalues)
+{
+ Size len;
+ Ranges *ranges;
+
+ Assert(maxvalues > 0);
+
+ len = offsetof(Ranges, values); /* fixed header */
+ len += maxvalues * sizeof(Datum); /* Datum values */
+
+ ranges = (Ranges *) palloc0(len);
+
+ ranges->maxvalues = maxvalues;
+
+ return ranges;
+}
+
+
+/*
+ * range_deduplicate_values
+ * Deduplicate the part with values in the simple points.
+ *
+ * This is meant to be a cheaper way of reducing the size of the ranges. It
+ * does not touch the ranges, and only sorts the other values - it does not
+ * call the distance functions, which may be quite expensive, etc.
+ */
+static void
+range_deduplicate_values(Ranges *range)
+{
+ int i,
+ n;
+ int start;
+ compare_context cxt;
+
+ /*
+ * If there are no unsorted values, we're done (this probably can't
+ * happen, as we're adding values to unsorted part).
+ */
+ if (range->nsorted == range->nvalues)
+ return;
+
+ /* sort the values */
+ cxt.colloid = range->colloid;
+ cxt.cmpFn = range->cmp;
+
+ /* how many values to sort? */
+ start = 2 * range->nranges;
+
+ qsort_arg(&range->values[start],
+ range->nvalues, sizeof(Datum),
+ compare_values, (void *) &cxt);
+
+ n = 1;
+ for (i = 1; i < range->nvalues; i++)
+ {
+ /* same as preceding value, so store it */
+ if (compare_values(&range->values[start + i - 1],
+ &range->values[start + i],
+ (void *) &cxt) == 0)
+ continue;
+
+ range->values[start + n] = range->values[start + i];
+
+ n++;
+ }
+
+ /* now all the values are sorted */
+ range->nvalues = n;
+ range->nsorted = n;
+
+ AssertCheckRanges(range, range->cmp, range->colloid);
+}
+
+
+/*
+ * range_serialize
+ * Serialize the in-memory representation into a compact varlena value.
+ *
+ * Simply copy the header and then also the individual values, as stored
+ * in the in-memory value array.
+ */
+static SerializedRanges *
+range_serialize(Ranges *range)
+{
+ Size len;
+ int nvalues;
+ SerializedRanges *serialized;
+ Oid typid;
+ int typlen;
+ bool typbyval;
+
+ int i;
+ char *ptr;
+
+ /* simple sanity checks */
+ Assert(range->nranges >= 0);
+ Assert(range->nsorted >= 0);
+ Assert(range->nvalues >= 0);
+ Assert(range->maxvalues > 0);
+ Assert(range->target_maxvalues > 0);
+
+ /* at this point the range should be compacted to the target size */
+ Assert(2 * range->nranges + range->nvalues <= range->target_maxvalues);
+
+ Assert(range->target_maxvalues <= range->maxvalues);
+
+ /* range boundaries are always sorted */
+ Assert(range->nvalues >= range->nsorted);
+
+ /* deduplicate values, if there's unsorted part */
+ range_deduplicate_values(range);
+
+ /* see how many Datum values we actually have */
+ nvalues = 2 * range->nranges + range->nvalues;
+
+ typid = range->typid;
+ typbyval = get_typbyval(typid);
+ typlen = get_typlen(typid);
+
+ /* header is always needed */
+ len = offsetof(SerializedRanges, data);
+
+ /*
+ * The space needed depends on data type - for fixed-length data types
+ * (by-value and some by-reference) it's pretty simple, just multiply
+ * (attlen * nvalues) and we're done. For variable-length by-reference
+ * types we need to actually walk all the values and sum the lengths.
+ */
+ if (typlen == -1) /* varlena */
+ {
+ int i;
+
+ for (i = 0; i < nvalues; i++)
+ {
+ len += VARSIZE_ANY(range->values[i]);
+ }
+ }
+ else if (typlen == -2) /* cstring */
+ {
+ int i;
+
+ for (i = 0; i < nvalues; i++)
+ {
+ /* don't forget to include the null terminator ;-) */
+ len += strlen(DatumGetPointer(range->values[i])) + 1;
+ }
+ }
+ else /* fixed-length types (even by-reference) */
+ {
+ Assert(typlen > 0);
+ len += nvalues * typlen;
+ }
+
+ /*
+ * Allocate the serialized object, copy the basic information. The
+ * serialized object is a varlena, so update the header.
+ */
+ serialized = (SerializedRanges *) palloc0(len);
+ SET_VARSIZE(serialized, len);
+
+ serialized->typid = typid;
+ serialized->nranges = range->nranges;
+ serialized->nvalues = range->nvalues;
+ serialized->maxvalues = range->target_maxvalues;
+
+ /*
+ * And now copy also the boundary values (like the length calculation this
+ * depends on the particular data type).
+ */
+ ptr = serialized->data; /* start of the serialized data */
+
+ for (i = 0; i < nvalues; i++)
+ {
+ if (typbyval) /* simple by-value data types */
+ {
+ memcpy(ptr, &range->values[i], typlen);
+ ptr += typlen;
+ }
+ else if (typlen > 0) /* fixed-length by-ref types */
+ {
+ memcpy(ptr, DatumGetPointer(range->values[i]), typlen);
+ ptr += typlen;
+ }
+ else if (typlen == -1) /* varlena */
+ {
+ int tmp = VARSIZE_ANY(DatumGetPointer(range->values[i]));
+
+ memcpy(ptr, DatumGetPointer(range->values[i]), tmp);
+ ptr += tmp;
+ }
+ else if (typlen == -2) /* cstring */
+ {
+ int tmp = strlen(DatumGetPointer(range->values[i])) + 1;
+
+ memcpy(ptr, DatumGetPointer(range->values[i]), tmp);
+ ptr += tmp;
+ }
+
+ /* make sure we haven't overflown the buffer end */
+ Assert(ptr <= ((char *) serialized + len));
+ }
+
+ /* exact size */
+ Assert(ptr == ((char *) serialized + len));
+
+ return serialized;
+}
+
+/*
+ * range_deserialize
+ * Serialize the in-memory representation into a compact varlena value.
+ *
+ * Simply copy the header and then also the individual values, as stored
+ * in the in-memory value array.
+ */
+static Ranges *
+range_deserialize(int maxvalues, SerializedRanges *serialized)
+{
+ int i,
+ nvalues;
+ char *ptr;
+ bool typbyval;
+ int typlen;
+
+ Ranges *range;
+
+ Assert(serialized->nranges >= 0);
+ Assert(serialized->nvalues >= 0);
+ Assert(serialized->maxvalues > 0);
+
+ nvalues = 2 * serialized->nranges + serialized->nvalues;
+
+ Assert(nvalues <= serialized->maxvalues);
+ Assert(serialized->maxvalues <= maxvalues);
+
+ range = minmax_multi_init(maxvalues);
+
+ /* copy the header info */
+ range->nranges = serialized->nranges;
+ range->nvalues = serialized->nvalues;
+ range->nsorted = serialized->nvalues;
+ range->maxvalues = maxvalues;
+ range->target_maxvalues = serialized->maxvalues;
+
+ range->typid = serialized->typid;
+
+ typbyval = get_typbyval(serialized->typid);
+ typlen = get_typlen(serialized->typid);
+
+ /*
+ * And now deconstruct the values into Datum array. We don't need to copy
+ * the values and will instead just point the values to the serialized
+ * varlena value (assuming it will be kept around).
+ */
+ ptr = serialized->data;
+
+ for (i = 0; i < nvalues; i++)
+ {
+ if (typbyval) /* simple by-value data types */
+ {
+ memcpy(&range->values[i], ptr, typlen);
+ ptr += typlen;
+ }
+ else if (typlen > 0) /* fixed-length by-ref types */
+ {
+ /* no copy, just set the value to the pointer */
+ range->values[i] = PointerGetDatum(ptr);
+ ptr += typlen;
+ }
+ else if (typlen == -1) /* varlena */
+ {
+ range->values[i] = PointerGetDatum(ptr);
+ ptr += VARSIZE_ANY(DatumGetPointer(range->values[i]));
+ }
+ else if (typlen == -2) /* cstring */
+ {
+ range->values[i] = PointerGetDatum(ptr);
+ ptr += strlen(DatumGetPointer(range->values[i])) + 1;
+ }
+
+ /* make sure we haven't overflown the buffer end */
+ Assert(ptr <= ((char *) serialized + VARSIZE_ANY(serialized)));
+ }
+
+ /* should have consumed the whole input value exactly */
+ Assert(ptr == ((char *) serialized + VARSIZE_ANY(serialized)));
+
+ /* return the deserialized value */
+ return range;
+}
+
+/*
+ * compare_expanded_ranges
+ * Compare the expanded ranges - first by minimum, then by maximum.
+ *
+ * We do guarantee that ranges in a single Range object do not overlap,
+ * so it may seem strange that we don't order just by minimum. But when
+ * merging two Ranges (which happens in the union function), the ranges
+ * may in fact overlap. So we do compare both.
+ */
+static int
+compare_expanded_ranges(const void *a, const void *b, void *arg)
+{
+ ExpandedRange *ra = (ExpandedRange *) a;
+ ExpandedRange *rb = (ExpandedRange *) b;
+ Datum r;
+
+ compare_context *cxt = (compare_context *) arg;
+
+ /* first compare minvals */
+ r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, ra->minval, rb->minval);
+
+ if (DatumGetBool(r))
+ return -1;
+
+ r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, rb->minval, ra->minval);
+
+ if (DatumGetBool(r))
+ return 1;
+
+ /* then compare maxvals */
+ r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, ra->maxval, rb->maxval);
+
+ if (DatumGetBool(r))
+ return -1;
+
+ r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, rb->maxval, ra->maxval);
+
+ if (DatumGetBool(r))
+ return 1;
+
+ return 0;
+}
+
+/*
+ * compare_values
+ * Compare the values.
+ */
+static int
+compare_values(const void *a, const void *b, void *arg)
+{
+ Datum *da = (Datum *) a;
+ Datum *db = (Datum *) b;
+ Datum r;
+
+ compare_context *cxt = (compare_context *) arg;
+
+ r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, *da, *db);
+
+ if (DatumGetBool(r))
+ return -1;
+
+ r = FunctionCall2Coll(cxt->cmpFn, cxt->colloid, *db, *da);
+
+ if (DatumGetBool(r))
+ return 1;
+
+ return 0;
+}
+
+/*
+ * Check if the new value matches one of the existing ranges.
+ */
+static bool
+has_matching_range(BrinDesc *bdesc, Oid colloid, Ranges *ranges,
+ Datum newval, AttrNumber attno, Oid typid)
+{
+ Datum compar;
+
+ Datum minvalue = ranges->values[0];
+ Datum maxvalue = ranges->values[2 * ranges->nranges - 1];
+
+ FmgrInfo *cmpLessFn;
+ FmgrInfo *cmpGreaterFn;
+
+ /* binary search on ranges */
+ int start,
+ end;
+
+ if (ranges->nranges == 0)
+ return false;
+
+ /*
+ * Otherwise, need to compare the new value with boundaries of all the
+ * ranges. First check if it's less than the absolute minimum, which is
+ * the first value in the array.
+ */
+ cmpLessFn = minmax_multi_get_strategy_procinfo(bdesc, attno, typid,
+ BTLessStrategyNumber);
+ compar = FunctionCall2Coll(cmpLessFn, colloid, newval, minvalue);
+
+ /* smaller than the smallest value in the range list */
+ if (DatumGetBool(compar))
+ return false;
+
+ /*
+ * And now compare it to the existing maximum (last value in the data
+ * array). But only if we haven't already ruled out a possible match in
+ * the minvalue check.
+ */
+ cmpGreaterFn = minmax_multi_get_strategy_procinfo(bdesc, attno, typid,
+ BTGreaterStrategyNumber);
+ compar = FunctionCall2Coll(cmpGreaterFn, colloid, newval, maxvalue);
+
+ if (DatumGetBool(compar))
+ return false;
+
+ /*
+ * So we know it's in the general min/max, the question is whether it
+ * falls in one of the ranges or gaps. We'll do a binary search on the on
+ * the individual ranges - for each range we check equality (value falls
+ * into the range), and then check ranges either above or below the
+ * current range.
+ */
+ start = 0; /* first range */
+ end = (ranges->nranges - 1); /* last range */
+ while (true)
+ {
+ int midpoint = (start + end) / 2;
+
+ /* this means we ran out of ranges in the last step */
+ if (start > end)
+ return false;
+
+ /* copy the min/max values from the ranges */
+ minvalue = ranges->values[2 * midpoint];
+ maxvalue = ranges->values[2 * midpoint + 1];
+
+ /*
+ * Is the value smaller than the minval? If yes, we'll recurse to the
+ * left side of range array.
+ */
+ compar = FunctionCall2Coll(cmpLessFn, colloid, newval, minvalue);
+
+ /* smaller than the smallest value in this range */
+ if (DatumGetBool(compar))
+ {
+ end = (midpoint - 1);
+ continue;
+ }
+
+ /*
+ * Is the value greater than the minval? If yes, we'll recurse to the
+ * right side of range array.
+ */
+ compar = FunctionCall2Coll(cmpGreaterFn, colloid, newval, maxvalue);
+
+ /* larger than the largest value in this range */
+ if (DatumGetBool(compar))
+ {
+ start = (midpoint + 1);
+ continue;
+ }
+
+ /* hey, we found a matching range */
+ return true;
+ }
+
+ return false;
+}
+
+
+/*
+ * range_contains_value
+ * See if the new value is already contained in the range list.
+ *
+ * We first inspect the list of intervals. We use a small trick - we check
+ * the value against min/max of the whole range (min of the first interval,
+ * max of the last one) first, and only inspect the individual intervals if
+ * this passes.
+ *
+ * If the value matches none of the intervals, we check the exact values.
+ * We simply loop through them and invoke equality operator on them.
+ *
+ * XXX We only inspect the sorted parts, which means we may add duplicate
+ * values. It may produce some false negatives when adding the values, but
+ * only after we already added some values (otherwise there is no unsorted
+ * part). And when querying the index, there should be no unsorted values,
+ * because the values are sorted and deduplicated during serialization.
+ */
+static bool
+range_contains_value(BrinDesc *bdesc, Oid colloid,
+ AttrNumber attno, Form_pg_attribute attr,
+ Ranges *ranges, Datum newval)
+{
+ int i;
+ FmgrInfo *cmpEqualFn;
+ Oid typid = attr->atttypid;
+
+ /*
+ * First inspect the ranges, if there are any. We first check the whole
+ * range, and only when there's still a chance of getting a match we
+ * inspect the individual ranges.
+ */
+ if (has_matching_range(bdesc, colloid, ranges, newval, attno, typid))
+ return true;
+
+ cmpEqualFn = minmax_multi_get_strategy_procinfo(bdesc, attno, typid,
+ BTEqualStrategyNumber);
+
+ /*
+ * There is no matching range, so let's inspect the sorted values.
+ *
+ * XXX We do a sequential search for small number of values, and binary
+ * search once we have more than 16 values. This threshold is somewhat
+ * arbitrary, as it depends on how expensive the comparison function is.
+ * So maybe we should just do the binary search all the time.
+ *
+ * XXX If we use the threshold here, maybe we should do the same thing in
+ * has_matching_range?
+ */
+ if (ranges->nsorted >= 16)
+ {
+ compare_context cxt;
+
+ cxt.colloid = ranges->colloid;
+ cxt.cmpFn = ranges->cmp;
+
+ if (bsearch_arg(&newval, &ranges->values[2 * ranges->nranges],
+ ranges->nsorted, sizeof(Datum),
+ compare_values, (void *) &cxt) != NULL)
+ return true;
+ }
+ else
+ {
+ for (i = 2 * ranges->nranges; i < 2 * ranges->nranges + ranges->nsorted; i++)
+ {
+ Datum compar;
+
+ compar = FunctionCall2Coll(cmpEqualFn, colloid, newval, ranges->values[i]);
+
+ /* found an exact match */
+ if (DatumGetBool(compar))
+ return true;
+ }
+ }
+
+ /* the value is not covered by this BRIN tuple */
+ return false;
+}
+
+/*
+ * Expand ranges from Ranges into ExpandedRange array. This expects the
+ * eranges to be pre-allocated and with the correct size - there needs to be
+ * (nranges + nvalues) elements.
+ *
+ * The order of expanded ranges is arbitrary. We do expand the ranges first,
+ * and this part is sorted. But then we expand the values, and this part may
+ * be unsorted.
+ */
+static void
+fill_expanded_ranges(ExpandedRange *eranges, int neranges, Ranges *ranges)
+{
+ int idx;
+ int i;
+
+ /* Check that the output array has the right size. */
+ Assert(neranges == (2 * ranges->nranges + ranges->nvalues));
+
+ idx = 0;
+ for (i = 0; i < ranges->nranges; i++)
+ {
+ eranges[idx].minval = ranges->values[2 * i];
+ eranges[idx].maxval = ranges->values[2 * i + 1];
+ eranges[idx].collapsed = false;
+ idx++;
+
+ Assert(idx <= neranges);
+ }
+
+ for (i = 0; i < ranges->nvalues; i++)
+ {
+ eranges[idx].minval = ranges->values[2 * ranges->nranges + i];
+ eranges[idx].maxval = ranges->values[2 * ranges->nranges + i];
+ eranges[idx].collapsed = true;
+ idx++;
+
+ Assert(idx <= neranges);
+ }
+
+ /* Did we produce the expected number of elements? */
+ Assert(idx == neranges);
+
+ return;
+}
+
+/*
+ * Sort and deduplicate expanded ranges.
+ *
+ * The ranges may be deduplicated - we're simply appending values, without
+ * checking for duplicates etc. So maybe the deduplication will reduce the
+ * number of ranges enough, and we won't have to compute the distances etc.
+ *
+ * Returns the number of expanded ranges.
+ *
+ * XXX We do perform qsort on all the values, but we could also leverage
+ * the fact that some of the input data is already sorted (all the ranges
+ * and possibly some of the points) and do merge sort.
+ */
+static int
+sort_expanded_ranges(FmgrInfo *cmp, Oid colloid,
+ ExpandedRange *eranges, int neranges)
+{
+ int n;
+ int i;
+ compare_context cxt;
+
+ Assert(neranges > 0);
+
+ /* sort the values */
+ cxt.colloid = colloid;
+ cxt.cmpFn = cmp;
+
+ qsort_arg(eranges, neranges, sizeof(ExpandedRange),
+ compare_expanded_ranges, (void *) &cxt);
+
+ /*
+ * Deduplicate the ranges - simply compare each range to the preceding
+ * one, and skip the duplicate ones.
+ */
+ n = 1;
+ for (i = 1; i < neranges; i++)
+ {
+ /* if the current range is equal to the preceding one, do nothing */
+ if (!compare_expanded_ranges(&eranges[i - 1], &eranges[i], (void *) &cxt))
+ continue;
+
+ /* otherwise copy it to n-th place (if not already there) */
+ if (i != n)
+ memcpy(&eranges[n], &eranges[i], sizeof(ExpandedRange));
+
+ n++;
+ }
+
+ Assert((n > 0) && (n <= neranges));
+
+ return n;
+}
+
+/*
+ * When combining multiple Range values (in union function), some of the
+ * ranges may overlap. We simply merge the overlapping ranges to fix that.
+ *
+ * XXX This assumes the expanded ranges were previously sorted (by minval
+ * and then maxval). We leverage this when detecting overlap.
+ */
+static int
+merge_overlapping_ranges(FmgrInfo *cmp, Oid colloid,
+ ExpandedRange *eranges, int neranges)
+{
+ int idx;
+
+ /* Merge ranges (idx) and (idx+1) if they overlap. */
+ idx = 0;
+ while (idx < (neranges - 1))
+ {
+ Datum r;
+
+ /*
+ * comparing [?,maxval] vs. [minval,?] - the ranges overlap if (minval
+ * < maxval)
+ */
+ r = FunctionCall2Coll(cmp, colloid,
+ eranges[idx].maxval,
+ eranges[idx + 1].minval);
+
+ /*
+ * Nope, maxval < minval, so no overlap. And we know the ranges are
+ * ordered, so there are no more overlaps, because all the remaining
+ * ranges have greater or equal minval.
+ */
+ if (DatumGetBool(r))
+ {
+ /* proceed to the next range */
+ idx += 1;
+ continue;
+ }
+
+ /*
+ * So ranges 'idx' and 'idx+1' do overlap, but we don't know if
+ * 'idx+1' is contained in 'idx', or if they only overlap only
+ * partially. So compare the upper bounds and keep the larger one.
+ */
+ r = FunctionCall2Coll(cmp, colloid,
+ eranges[idx].maxval,
+ eranges[idx + 1].maxval);
+
+ if (DatumGetBool(r))
+ eranges[idx].maxval = eranges[idx + 1].maxval;
+
+ /*
+ * The range certainly is no longer collapsed (irrespectedly of the
+ * previous state).
+ */
+ eranges[idx].collapsed = false;
+
+ /*
+ * Now get rid of the (idx+1) range entirely by shifting the remaining
+ * ranges by 1. There are ncranges elements, and we need to move
+ * elements from (idx+2). That means the number of elements to move is
+ * [ncranges - (idx+2)].
+ */
+ memmove(&eranges[idx + 1], &eranges[idx + 2],
+ (neranges - (idx + 2)) * sizeof(ExpandedRange));
+
+ /*
+ * Decrease the number of ranges, and repeat (with the same range, as
+ * it might overlap with additional ranges thanks to the merge).
+ */
+ neranges--;
+ }
+
+ return neranges;
+}
+
+/*
+ * Simple comparator for distance values, comparing the double value.
+ * This is intentionally sorting the distances in descending order, i.e.
+ * the longer gaps will be at the front.
+ */
+static int
+compare_distances(const void *a, const void *b)
+{
+ DistanceValue *da = (DistanceValue *) a;
+ DistanceValue *db = (DistanceValue *) b;
+
+ if (da->value < db->value)
+ return 1;
+ else if (da->value > db->value)
+ return -1;
+
+ return 0;
+}
+
+/*
+ * Given an array of expanded ranges, compute distance of the gaps betwen
+ * the ranges - for ncranges there are (ncranges-1) gaps.
+ *
+ * We simply call the "distance" function to compute the (max-min) for pairs
+ * of consecutive ranges. The function may be fairly expensive, so we do that
+ * just once (and then use it to pick as many ranges to merge as possible).
+ *
+ * See reduce_expanded_ranges for details.
+ */
+static DistanceValue *
+build_distances(FmgrInfo *distanceFn, Oid colloid,
+ ExpandedRange *eranges, int neranges)
+{
+ int i;
+ int ndistances;
+ DistanceValue *distances;
+
+ Assert(neranges >= 2);
+
+ ndistances = (neranges - 1);
+ distances = (DistanceValue *) palloc0(sizeof(DistanceValue) * ndistances);
+
+ /*
+ * Walk though the ranges once and compute distance between the ranges so
+ * that we can sort them once.
+ */
+ for (i = 0; i < ndistances; i++)
+ {
+ Datum a1,
+ a2,
+ r;
+
+ a1 = eranges[i].maxval;
+ a2 = eranges[i + 1].minval;
+
+ /* compute length of the gap (between max/min) */
+ r = FunctionCall2Coll(distanceFn, colloid, a1, a2);
+
+ /* remember the index of the gap the distance is for */
+ distances[i].index = i;
+ distances[i].value = DatumGetFloat8(r);
+ }
+
+ /*
+ * Sort the distances in descending order, so that the longest gaps are at
+ * the front.
+ */
+ pg_qsort(distances, ndistances, sizeof(DistanceValue), compare_distances);
+
+ return distances;
+}
+
+/*
+ * Builds expanded ranges for the existing ranges (and single-point ranges),
+ * and also the new value (which did not fit into the array). This expanded
+ * representation makes the processing a bit easier, as it allows handling
+ * ranges and points the same way.
+ *
+ * We sort and deduplicate the expanded ranges - this is necessary, because
+ * the points may be unsorted. And moreover the two parts (ranges and
+ * points) are sorted on their own.
+ */
+static ExpandedRange *
+build_expanded_ranges(FmgrInfo *cmp, Oid colloid, Ranges *ranges,
+ int *nranges)
+{
+ int neranges;
+ ExpandedRange *eranges;
+
+ /* both ranges and points are expanded into a separate element */
+ neranges = ranges->nranges + ranges->nvalues;
+
+ eranges = (ExpandedRange *) palloc0(neranges * sizeof(ExpandedRange));
+
+ /* fill the expanded ranges */
+ fill_expanded_ranges(eranges, neranges, ranges);
+
+ /* sort and deduplicate the expanded ranges */
+ neranges = sort_expanded_ranges(cmp, colloid, eranges, neranges);
+
+ /* remember how many cranges we built */
+ *nranges = neranges;
+
+ return eranges;
+}
+
+#ifdef USE_ASSERT_CHECKING
+/*
+ * Counts bondary values needed to store the ranges. Each single-point
+ * range is stored using a single value, each regular range needs two.
+ */
+static int
+count_values(ExpandedRange *cranges, int ncranges)
+{
+ int i;
+ int count;
+
+ count = 0;
+ for (i = 0; i < ncranges; i++)
+ {
+ if (cranges[i].collapsed)
+ count += 1;
+ else
+ count += 2;
+ }
+
+ return count;
+}
+#endif
+
+/*
+ * reduce_expanded_ranges
+ * reduce the ranges until the number of values is low enough
+ *
+ * Combines ranges until the number of boundary values drops below the
+ * threshold specified by max_values. This happens by merging enough
+ * ranges by distance between them.
+ *
+ * Returns the number of result ranges.
+ *
+ * We simply use the global min/max and then add boundaries for enough
+ * largest gaps. Each gap adds 2 values, so we simply use (target/2-1)
+ * distances. Then we simply sort all the values - each two values are
+ * a boundary of a range (possibly collapsed).
+ *
+ * XXX Some of the ranges may be collapsed (i.e. the min/max values are
+ * equal), but we ignore that for now. We could repeat the process,
+ * adding a couple more gaps recursively.
+ *
+ * XXX The ranges to merge are selected solely using the distance. But
+ * that may not be the best strategy, for example when multiple gaps
+ * are of equal (or very similar) length.
+ *
+ * Consider for example points 1, 2, 3, .., 64, which have gaps of the
+ * same length 1 of course. In that case we tend to pick the first
+ * gap of that length, which leads to this:
+ *
+ * step 1: [1, 2], 3, 4, 5, .., 64
+ * step 2: [1, 3], 4, 5, .., 64
+ * step 3: [1, 4], 5, .., 64
+ * ...
+ *
+ * So in the end we'll have one "large" range and multiple small points.
+ * That may be fine, but it seems a bit strange and non-optimal. Maybe
+ * we should consider other things when picking ranges to merge - e.g.
+ * length of the ranges? Or perhaps randomize the choice of ranges, with
+ * probability inversely proportional to the distance (the gap lengths
+ * may be very close, but not exactly the same).
+ *
+ * XXX Or maybe we could just handle this by using random value as a
+ * tie-break, or by adding random noise to the actual distance.
+ */
+static int
+reduce_expanded_ranges(ExpandedRange *eranges, int neranges,
+ DistanceValue *distances, int max_values,
+ FmgrInfo *cmp, Oid colloid)
+{
+ int i;
+ int nvalues;
+ Datum *values;
+
+ compare_context cxt;
+
+ /* total number of gaps between ranges */
+ int ndistances = (neranges - 1);
+
+ /* number of gaps to keep */
+ int keep = (max_values / 2 - 1);
+
+ /*
+ * Maybe we have sufficiently low number of ranges already?
+ *
+ * XXX This should happen before we actually do the expensive stuff like
+ * sorting, so maybe this should be just an assert.
+ */
+ if (keep >= ndistances)
+ return neranges;
+
+ /* sort the values */
+ cxt.colloid = colloid;
+ cxt.cmpFn = cmp;
+
+ /* allocate space for the boundary values */
+ nvalues = 0;
+ values = (Datum *) palloc(sizeof(Datum) * max_values);
+
+ /* add the global min/max values, from the first/last range */
+ values[nvalues++] = eranges[0].minval;
+ values[nvalues++] = eranges[neranges - 1].maxval;
+
+ /* add boundary values for enough gaps */
+ for (i = 0; i < keep; i++)
+ {
+ /* index of the gap between (index) and (index+1) ranges */
+ int index = distances[i].index;
+
+ Assert((index >= 0) && ((index + 1) < neranges));
+
+ /* add max from the preceding range, minval from the next one */
+ values[nvalues++] = eranges[index].maxval;
+ values[nvalues++] = eranges[index + 1].minval;
+
+ Assert(nvalues <= max_values);
+ }
+
+ /* We should have even number of range values. */
+ Assert(nvalues % 2 == 0);
+
+ /*
+ * Sort the values using the comparator function, and form ranges from the
+ * sorted result.
+ */
+ qsort_arg(values, nvalues, sizeof(Datum),
+ compare_values, (void *) &cxt);
+
+ /* We have nvalues boundary values, which means nvalues/2 ranges. */
+ for (i = 0; i < (nvalues / 2); i++)
+ {
+ eranges[i].minval = values[2 * i];
+ eranges[i].maxval = values[2 * i + 1];
+
+ /* if the boundary values are the same, it's a collapsed range */
+ eranges[i].collapsed = (compare_values(&values[2 * i],
+ &values[2 * i + 1],
+ &cxt) == 0);
+ }
+
+ return (nvalues / 2);
+}
+
+/*
+ * Store the boundary values from ExpandedRanges back into Range (using
+ * only the minimal number of values needed).
+ */
+static void
+store_expanded_ranges(Ranges *ranges, ExpandedRange *eranges, int neranges)
+{
+ int i;
+ int idx = 0;
+
+ /* first copy in the regular ranges */
+ ranges->nranges = 0;
+ for (i = 0; i < neranges; i++)
+ {
+ if (!eranges[i].collapsed)
+ {
+ ranges->values[idx++] = eranges[i].minval;
+ ranges->values[idx++] = eranges[i].maxval;
+ ranges->nranges++;
+ }
+ }
+
+ /* now copy in the collapsed ones */
+ ranges->nvalues = 0;
+ for (i = 0; i < neranges; i++)
+ {
+ if (eranges[i].collapsed)
+ {
+ ranges->values[idx++] = eranges[i].minval;
+ ranges->nvalues++;
+ }
+ }
+
+ /* all the values are sorted */
+ ranges->nsorted = ranges->nvalues;
+
+ Assert(count_values(eranges, neranges) == 2 * ranges->nranges + ranges->nvalues);
+ Assert(2 * ranges->nranges + ranges->nvalues <= ranges->maxvalues);
+}
+
+
+/*
+ * Consider freeing space in the ranges.
+ *
+ * Returns true if the value was actually modified.
+ */
+static bool
+ensure_free_space_in_buffer(BrinDesc *bdesc, Oid colloid,
+ AttrNumber attno, Form_pg_attribute attr,
+ Ranges *range)
+{
+ MemoryContext ctx;
+ MemoryContext oldctx;
+
+ FmgrInfo *cmpFn,
+ *distanceFn;
+
+ /* expanded ranges */
+ ExpandedRange *eranges;
+ int neranges;
+ DistanceValue *distances;
+
+ /*
+ * If there is free space in the buffer, we're done without having to
+ * modify anything.
+ */
+ if (2 * range->nranges + range->nvalues < range->maxvalues)
+ return false;
+
+ /* we'll certainly need the comparator, so just look it up now */
+ cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTLessStrategyNumber);
+
+ /* deduplicate values, if there's unsorted part */
+ range_deduplicate_values(range);
+
+ /* did we reduce enough free space by just the deduplication? */
+ if (2 * range->nranges + range->nvalues <= range->maxvalues * MINMAX_BUFFER_LOAD_FACTOR)
+ return true;
+
+ /*
+ * We need to combine some of the existing ranges, to reduce the number of
+ * values we have to store.
+ *
+ * The distanceFn calls (which may internally call e.g. numeric_le) may
+ * allocate quite a bit of memory, and we must not leak it (we might have
+ * to do this repeatedly, even for a single BRIN page range). Otherwise
+ * we'd have problems e.g. when building new indexes. So we use a memory
+ * context and make sure we free the memory at the end (so if we call the
+ * distance function many times, it might be an issue, but meh).
+ */
+ ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "minmax-multi context",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldctx = MemoryContextSwitchTo(ctx);
+
+ /* build the expanded ranges */
+ eranges = build_expanded_ranges(cmpFn, colloid, range, &neranges);
+
+ /* and we'll also need the 'distance' procedure */
+ distanceFn = minmax_multi_get_procinfo(bdesc, attno, PROCNUM_DISTANCE);
+
+ /* build array of gap distances and sort them in ascending order */
+ distances = build_distances(distanceFn, colloid, eranges, neranges);
+
+ /*
+ * Combine ranges until we release at least 25% of the space. This
+ * threshold is somewhat arbitrary, perhaps needs tuning. We must not use
+ * too low or high value.
+ */
+ neranges = reduce_expanded_ranges(eranges, neranges, distances,
+ range->maxvalues * MINMAX_BUFFER_LOAD_FACTOR,
+ cmpFn, colloid);
+
+ /* Make sure we've sufficiently reduced the number of ranges. */
+ Assert(count_values(eranges, neranges) <= range->maxvalues * MINMAX_BUFFER_LOAD_FACTOR);
+
+ /* decompose the expanded ranges into regular ranges and single values */
+ store_expanded_ranges(range, eranges, neranges);
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextDelete(ctx);
+
+ /* Did we break the ranges somehow? */
+ AssertCheckRanges(range, cmpFn, colloid);
+
+ return true;
+}
+
+/*
+ * range_add_value
+ * Add the new value to the multi-minmax range.
+ */
+static bool
+range_add_value(BrinDesc *bdesc, Oid colloid,
+ AttrNumber attno, Form_pg_attribute attr,
+ Ranges *ranges, Datum newval)
+{
+ FmgrInfo *cmpFn;
+ bool modified = false;
+
+ /* we'll certainly need the comparator, so just look it up now */
+ cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTLessStrategyNumber);
+
+ /* comprehensive checks of the input ranges */
+ AssertCheckRanges(ranges, cmpFn, colloid);
+
+ /*
+ * Make sure there's enough free space in the buffer. We only trigger this
+ * when the buffer is full, which means it had to be modified as we size
+ * it to be larger than what is stored on disk.
+ *
+ * XXX This needs to happen before we check if the value is contained in
+ * the range, because the value might be in the unsorted part, and we
+ * don't check that in range_contains_value. The deduplication would then
+ * move it to the sorted part, and we'd add the value too, which violates
+ * the rule that we never have duplicates with the ranges or sorted
+ * values.
+ *
+ * XXX At the moment this only does the deduplication.
+ *
+ * XXX We might also deduplicate and recheck if the value is contained,
+ * but that seems like an overkill. We'd need to deduplicate anyway, so
+ * why not do it now.
+ */
+ modified = ensure_free_space_in_buffer(bdesc, colloid,
+ attno, attr, ranges);
+
+ /*
+ * Bail out if the value already is covered by the range.
+ *
+ * We could also add values until we hit values_per_range, and then do the
+ * deduplication in a batch, hoping for better efficiency. But that would
+ * mean we actually modify the range every time, which means having to
+ * serialize the value, which does palloc, walks the values, copies them,
+ * etc. Not exactly cheap.
+ *
+ * So instead we do the check, which should be fairly cheap - assuming the
+ * comparator function is not very expensive.
+ *
+ * This also implies means the values array can't contain duplicities.
+ */
+ if (range_contains_value(bdesc, colloid, attno, attr, ranges, newval))
+ return modified;
+
+ /* Make a copy of the value, if needed. */
+ newval = datumCopy(newval, attr->attbyval, attr->attlen);
+
+ /*
+ * If there's space in the values array, copy it in and we're done.
+ *
+ * We do want to keep the values sorted (to speed up searches), so we do a
+ * simple insertion sort. We could do something more elaborate, e.g. by
+ * sorting the values only now and then, but for small counts (e.g. when
+ * maxvalues is 64) this should be fine.
+ */
+ ranges->values[2 * ranges->nranges + ranges->nvalues] = newval;
+ ranges->nvalues++;
+
+ /*
+ * Check we haven't broken the ordering of boundary values (checks both
+ * parts, but that doesn't hurt).
+ */
+ AssertCheckRanges(ranges, cmpFn, colloid);
+
+ /* Also check the range contains the value we just added. */
+ /* FIXME Assert(ranges, cmpFn, colloid); */
+
+ /* yep, we've modified the range */
+ return true;
+}
+
+/*
+ * Generate range representation of data collected during "batch mode".
+ * This is similar to reduce_expanded_ranges, except that we can't assume
+ * the values are sorted and there may be duplicate values.
+ */
+static void
+compactify_ranges(BrinDesc *bdesc, Ranges *ranges, int max_values)
+{
+ FmgrInfo *cmpFn,
+ *distanceFn;
+
+ /* expanded ranges */
+ ExpandedRange *eranges;
+ int neranges;
+ DistanceValue *distances;
+
+ MemoryContext ctx;
+ MemoryContext oldctx;
+
+ /* we'll certainly need the comparator, so just look it up now */
+ cmpFn = minmax_multi_get_strategy_procinfo(bdesc, ranges->attno, ranges->typid,
+ BTLessStrategyNumber);
+
+ /* and we'll also need the 'distance' procedure */
+ distanceFn = minmax_multi_get_procinfo(bdesc, ranges->attno, PROCNUM_DISTANCE);
+
+ /*
+ * The distanceFn calls (which may internally call e.g. numeric_le) may
+ * allocate quite a bit of memory, and we must not leak it. Otherwise we'd
+ * have problems e.g. when building indexes. So we create a local memory
+ * context and make sure we free the memory before leaving this function
+ * (not after every call).
+ */
+ ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "minmax-multi context",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldctx = MemoryContextSwitchTo(ctx);
+
+ /* build the expanded ranges */
+ eranges = build_expanded_ranges(cmpFn, ranges->colloid, ranges, &neranges);
+
+ /* FIXME this should do neranges >= max_values */
+ if (neranges > 1)
+ {
+ /* build array of gap distances and sort them in ascending order */
+ distances = build_distances(distanceFn, ranges->colloid,
+ eranges, neranges);
+
+ /*
+ * Combine ranges until we get below max_values. We don't use any
+ * scale factor, because this is used during serialization, and we
+ * don't expect more tuples to be inserted anytime soon.
+ */
+ neranges = reduce_expanded_ranges(eranges, neranges, distances,
+ max_values, cmpFn, ranges->colloid);
+
+ Assert(count_values(eranges, neranges) <= max_values);
+ }
+
+ /* decompose the expanded ranges into regular ranges and single values */
+ store_expanded_ranges(ranges, eranges, neranges);
+
+ /* check all the range invariants */
+ AssertCheckRanges(ranges, cmpFn, ranges->colloid);
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextDelete(ctx);
+}
+
+Datum
+brin_minmax_multi_opcinfo(PG_FUNCTION_ARGS)
+{
+ BrinOpcInfo *result;
+
+ /*
+ * opaque->strategy_procinfos is initialized lazily; here it is set to
+ * all-uninitialized by palloc0 which sets fn_oid to InvalidOid.
+ */
+
+ result = palloc0(MAXALIGN(SizeofBrinOpcInfo(1)) +
+ sizeof(MinmaxMultiOpaque));
+ result->oi_nstored = 1;
+ result->oi_regular_nulls = true;
+ result->oi_opaque = (MinmaxMultiOpaque *)
+ MAXALIGN((char *) result + SizeofBrinOpcInfo(1));
+ result->oi_typcache[0] = lookup_type_cache(PG_BRIN_MINMAX_MULTI_SUMMARYOID, 0);
+
+ PG_RETURN_POINTER(result);
+}
+
+/*
+ * Compute distance between two float4 values (plain subtraction).
+ */
+Datum
+brin_minmax_multi_distance_float4(PG_FUNCTION_ARGS)
+{
+ float a1 = PG_GETARG_FLOAT4(0);
+ float a2 = PG_GETARG_FLOAT4(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(a1 <= a2);
+
+ PG_RETURN_FLOAT8((double) a2 - (double) a1);
+}
+
+/*
+ * Compute distance between two float8 values (plain subtraction).
+ */
+Datum
+brin_minmax_multi_distance_float8(PG_FUNCTION_ARGS)
+{
+ double a1 = PG_GETARG_FLOAT8(0);
+ double a2 = PG_GETARG_FLOAT8(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(a1 <= a2);
+
+ PG_RETURN_FLOAT8(a2 - a1);
+}
+
+/*
+ * Compute distance between two int2 values (plain subtraction).
+ */
+Datum
+brin_minmax_multi_distance_int2(PG_FUNCTION_ARGS)
+{
+ int16 a1 = PG_GETARG_INT16(0);
+ int16 a2 = PG_GETARG_INT16(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(a1 <= a2);
+
+ PG_RETURN_FLOAT8((double) a2 - (double) a1);
+}
+
+/*
+ * Compute distance between two int4 values (plain subtraction).
+ */
+Datum
+brin_minmax_multi_distance_int4(PG_FUNCTION_ARGS)
+{
+ int32 a1 = PG_GETARG_INT32(0);
+ int32 a2 = PG_GETARG_INT32(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(a1 <= a2);
+
+ PG_RETURN_FLOAT8((double) a2 - (double) a1);
+}
+
+/*
+ * Compute distance between two int8 values (plain subtraction).
+ */
+Datum
+brin_minmax_multi_distance_int8(PG_FUNCTION_ARGS)
+{
+ int64 a1 = PG_GETARG_INT64(0);
+ int64 a2 = PG_GETARG_INT64(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(a1 <= a2);
+
+ PG_RETURN_FLOAT8((double) a2 - (double) a1);
+}
+
+/*
+ * Compute distance between two tid values (by mapping them to float8
+ * and then subtracting them).
+ */
+Datum
+brin_minmax_multi_distance_tid(PG_FUNCTION_ARGS)
+{
+ double da1,
+ da2;
+
+ ItemPointer pa1 = (ItemPointer) PG_GETARG_DATUM(0);
+ ItemPointer pa2 = (ItemPointer) PG_GETARG_DATUM(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(ItemPointerCompare(pa1, pa2) <= 0);
+
+ /*
+ * We use the no-check variants here, because user-supplied values may
+ * have (ip_posid == 0). See ItemPointerCompare.
+ */
+ da1 = ItemPointerGetBlockNumberNoCheck(pa1) * MaxHeapTuplesPerPage +
+ ItemPointerGetOffsetNumberNoCheck(pa1);
+
+ da2 = ItemPointerGetBlockNumberNoCheck(pa2) * MaxHeapTuplesPerPage +
+ ItemPointerGetOffsetNumberNoCheck(pa2);
+
+ PG_RETURN_FLOAT8(da2 - da1);
+}
+
+/*
+ * Comutes distance between two numeric values (plain subtraction).
+ */
+Datum
+brin_minmax_multi_distance_numeric(PG_FUNCTION_ARGS)
+{
+ Datum d;
+ Datum a1 = PG_GETARG_DATUM(0);
+ Datum a2 = PG_GETARG_DATUM(1);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(DatumGetBool(DirectFunctionCall2(numeric_le, a1, a2)));
+
+ d = DirectFunctionCall2(numeric_sub, a2, a1); /* a2 - a1 */
+
+ PG_RETURN_FLOAT8(DirectFunctionCall1(numeric_float8, d));
+}
+
+/*
+ * Computes approximate distance between two UUID values.
+ *
+ * XXX We do not need a perfectly accurate value, so we approximate the
+ * deltas (which would have to be 128-bit integers) with a 64-bit float.
+ * The small inaccuracies do not matter in practice, in the worst case
+ * we'll decide to merge ranges that are not the closest ones.
+ */
+Datum
+brin_minmax_multi_distance_uuid(PG_FUNCTION_ARGS)
+{
+ int i;
+ float8 delta = 0;
+
+ Datum a1 = PG_GETARG_DATUM(0);
+ Datum a2 = PG_GETARG_DATUM(1);
+
+ pg_uuid_t *u1 = DatumGetUUIDP(a1);
+ pg_uuid_t *u2 = DatumGetUUIDP(a2);
+
+ /*
+ * We know the values are range boundaries, but the range may be collapsed
+ * (i.e. single points), with equal values.
+ */
+ Assert(DatumGetBool(DirectFunctionCall2(uuid_le, a1, a2)));
+
+ /* compute approximate delta as a double precision value */
+ for (i = UUID_LEN - 1; i >= 0; i--)
+ {
+ delta += (int) u2->data[i] - (int) u1->data[i];
+ delta /= 256;
+ }
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Compute approximate distance between two dates.
+ */
+Datum
+brin_minmax_multi_distance_date(PG_FUNCTION_ARGS)
+{
+ DateADT dateVal1 = PG_GETARG_DATEADT(0);
+ DateADT dateVal2 = PG_GETARG_DATEADT(1);
+
+ if (DATE_NOT_FINITE(dateVal1) || DATE_NOT_FINITE(dateVal2))
+ PG_RETURN_FLOAT8(0);
+
+ PG_RETURN_FLOAT8(dateVal1 - dateVal2);
+}
+
+/*
+ * Computes approximate distance between two time (without tz) values.
+ *
+ * TimeADT is just an int64, so we simply subtract the values directly.
+ */
+Datum
+brin_minmax_multi_distance_time(PG_FUNCTION_ARGS)
+{
+ float8 delta = 0;
+
+ TimeADT ta = PG_GETARG_TIMEADT(0);
+ TimeADT tb = PG_GETARG_TIMEADT(1);
+
+ delta = (tb - ta);
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Computes approximate distance between two timetz values.
+ *
+ * Simply subtracts the TimeADT (int64) values embedded in TimeTzADT.
+ */
+Datum
+brin_minmax_multi_distance_timetz(PG_FUNCTION_ARGS)
+{
+ float8 delta = 0;
+
+ TimeTzADT *ta = PG_GETARG_TIMETZADT_P(0);
+ TimeTzADT *tb = PG_GETARG_TIMETZADT_P(1);
+
+ delta = tb->time - ta->time;
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+Datum
+brin_minmax_multi_distance_timestamp(PG_FUNCTION_ARGS)
+{
+ float8 delta = 0;
+
+ Timestamp dt1 = PG_GETARG_TIMESTAMP(0);
+ Timestamp dt2 = PG_GETARG_TIMESTAMP(1);
+
+ if (TIMESTAMP_NOT_FINITE(dt1) || TIMESTAMP_NOT_FINITE(dt2))
+ PG_RETURN_FLOAT8(0);
+
+ delta = dt2 - dt1;
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Computes distance between two interval values.
+ */
+Datum
+brin_minmax_multi_distance_interval(PG_FUNCTION_ARGS)
+{
+ float8 delta = 0;
+
+ Interval *ia = PG_GETARG_INTERVAL_P(0);
+ Interval *ib = PG_GETARG_INTERVAL_P(1);
+ Interval *result;
+
+ result = (Interval *) palloc(sizeof(Interval));
+
+ result->month = ib->month - ia->month;
+ /* overflow check copied from int4mi */
+ if (!SAMESIGN(ib->month, ia->month) &&
+ !SAMESIGN(result->month, ib->month))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATETIME_VALUE_OUT_OF_RANGE),
+ errmsg("interval out of range")));
+
+ result->day = ib->day - ia->day;
+ if (!SAMESIGN(ib->day, ia->day) &&
+ !SAMESIGN(result->day, ib->day))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATETIME_VALUE_OUT_OF_RANGE),
+ errmsg("interval out of range")));
+
+ result->time = ib->time - ia->time;
+ if (!SAMESIGN(ib->time, ia->time) &&
+ !SAMESIGN(result->time, ib->time))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATETIME_VALUE_OUT_OF_RANGE),
+ errmsg("interval out of range")));
+
+ /*
+ * We assume months have 31 days - we don't need to be precise, in the
+ * worst case we'll build somewhat less efficient ranges.
+ */
+ delta = (float8) (result->month * 31 + result->day);
+
+ /* convert to microseconds (just like the time part) */
+ delta = 24L * 3600L * delta;
+
+ /* and add the time part */
+ delta += result->time / (float8) 1000000.0;
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Compute distance between two pg_lsn values.
+ *
+ * LSN is just an int64 encoding position in the stream, so just subtract
+ * those int64 values directly.
+ */
+Datum
+brin_minmax_multi_distance_pg_lsn(PG_FUNCTION_ARGS)
+{
+ float8 delta = 0;
+
+ XLogRecPtr lsna = PG_GETARG_LSN(0);
+ XLogRecPtr lsnb = PG_GETARG_LSN(1);
+
+ delta = (lsnb - lsna);
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Compute distance between two macaddr values.
+ *
+ * mac addresses are treated as 6 unsigned chars, so do the same thing we
+ * already do for UUID values.
+ */
+Datum
+brin_minmax_multi_distance_macaddr(PG_FUNCTION_ARGS)
+{
+ float8 delta;
+
+ macaddr *a = PG_GETARG_MACADDR_P(0);
+ macaddr *b = PG_GETARG_MACADDR_P(1);
+
+ delta = ((float8) b->f - (float8) a->f);
+ delta /= 256;
+
+ delta += ((float8) b->e - (float8) a->e);
+ delta /= 256;
+
+ delta += ((float8) b->d - (float8) a->d);
+ delta /= 256;
+
+ delta += ((float8) b->c - (float8) a->c);
+ delta /= 256;
+
+ delta += ((float8) b->b - (float8) a->b);
+ delta /= 256;
+
+ delta += ((float8) b->a - (float8) a->a);
+ delta /= 256;
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Compute distance between two macaddr8 values.
+ *
+ * macaddr8 addresses are 8 unsigned chars, so do the same thing we
+ * already do for UUID values.
+ */
+Datum
+brin_minmax_multi_distance_macaddr8(PG_FUNCTION_ARGS)
+{
+ float8 delta;
+
+ macaddr8 *a = PG_GETARG_MACADDR8_P(0);
+ macaddr8 *b = PG_GETARG_MACADDR8_P(1);
+
+ delta = ((float8) b->h - (float8) a->h);
+ delta /= 256;
+
+ delta += ((float8) b->g - (float8) a->g);
+ delta /= 256;
+
+ delta += ((float8) b->f - (float8) a->f);
+ delta /= 256;
+
+ delta += ((float8) b->e - (float8) a->e);
+ delta /= 256;
+
+ delta += ((float8) b->d - (float8) a->d);
+ delta /= 256;
+
+ delta += ((float8) b->c - (float8) a->c);
+ delta /= 256;
+
+ delta += ((float8) b->b - (float8) a->b);
+ delta /= 256;
+
+ delta += ((float8) b->a - (float8) a->a);
+ delta /= 256;
+
+ Assert(delta >= 0);
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+/*
+ * Compute distance between two inet values.
+ *
+ * The distance is defined as difference between 32-bit/128-bit values,
+ * depending on the IP version. The distance is computed by subtracting
+ * the bytes and normalizing it to [0,1] range for each IP family.
+ * Addresses from difference families are consider to be in maximum
+ * distance, which is 1.0.
+ *
+ * XXX Does this need to consider the mask (bits)? For now it's ignored.
+ */
+Datum
+brin_minmax_multi_distance_inet(PG_FUNCTION_ARGS)
+{
+ float8 delta;
+ int i;
+ int len;
+ unsigned char *addra,
+ *addrb;
+
+ inet *ipa = PG_GETARG_INET_PP(0);
+ inet *ipb = PG_GETARG_INET_PP(1);
+
+ /*
+ * If the addresses are from different families, consider them to be in
+ * maximal possible distance (which is 1.0).
+ */
+ if (ip_family(ipa) != ip_family(ipb))
+ PG_RETURN_FLOAT8(1.0);
+
+ /* ipv4 or ipv6 */
+ if (ip_family(ipa) == PGSQL_AF_INET)
+ len = 4;
+ else
+ len = 16; /* NS_IN6ADDRSZ */
+
+ addra = ip_addr(ipa);
+ addrb = ip_addr(ipb);
+
+ delta = 0;
+ for (i = len - 1; i >= 0; i--)
+ {
+ delta += (float8) addrb[i] - (float8) addra[i];
+ delta /= 256;
+ }
+
+ Assert((delta >= 0) && (delta <= 1));
+
+ PG_RETURN_FLOAT8(delta);
+}
+
+static void
+brin_minmax_multi_serialize(BrinDesc *bdesc, Datum src, Datum *dst)
+{
+ Ranges *ranges = (Ranges *) DatumGetPointer(src);
+ SerializedRanges *s;
+
+ /*
+ * In batch mode, we need to compress the accumulated values to the
+ * actually requested number of values/ranges.
+ */
+ compactify_ranges(bdesc, ranges, ranges->target_maxvalues);
+
+ s = range_serialize(ranges);
+ dst[0] = PointerGetDatum(s);
+}
+
+static int
+brin_minmax_multi_get_values(BrinDesc *bdesc, MinMaxOptions *opts)
+{
+ return MinMaxGetValuesPerRange(opts);
+}
+
+/*
+ * Examine the given index tuple (which contains partial status of a certain
+ * page range) by comparing it to the given value that comes from another heap
+ * tuple. If the new value is outside the min/max range specified by the
+ * existing tuple values, update the index tuple and return true. Otherwise,
+ * return false and do not modify in this case.
+ */
+Datum
+brin_minmax_multi_add_value(PG_FUNCTION_ARGS)
+{
+ BrinDesc *bdesc = (BrinDesc *) PG_GETARG_POINTER(0);
+ BrinValues *column = (BrinValues *) PG_GETARG_POINTER(1);
+ Datum newval = PG_GETARG_DATUM(2);
+ bool isnull PG_USED_FOR_ASSERTS_ONLY = PG_GETARG_DATUM(3);
+ MinMaxOptions *opts = (MinMaxOptions *) PG_GET_OPCLASS_OPTIONS();
+ Oid colloid = PG_GET_COLLATION();
+ bool modified = false;
+ Form_pg_attribute attr;
+ AttrNumber attno;
+ Ranges *ranges;
+ SerializedRanges *serialized = NULL;
+
+ Assert(!isnull);
+
+ attno = column->bv_attno;
+ attr = TupleDescAttr(bdesc->bd_tupdesc, attno - 1);
+
+ /* use the already deserialized value, is possible */
+ ranges = (Ranges *) DatumGetPointer(column->bv_mem_value);
+
+ /*
+ * If this is the first non-null value, we need to initialize the range
+ * list. Otherwise just extract the existing range list from BrinValues.
+ *
+ * When starting with an empty range, we assume this is a batch mode, i.e.
+ * we size the buffer for the maximum possible number of items in the
+ * range (based on range size and max number of items on a page).
+ *
+ * XXX This may require quite a bit of memory, so maybe we should use some
+ * value in between. OTOH most tables will have much wider rows, so the
+ * number of rows per page is much lower.
+ *
+ * XXX Maybe we should do this (using larger buffer) even when there
+ * already is a summary?
+ */
+ if (column->bv_allnulls)
+ {
+ MemoryContext oldctx;
+
+ int target_maxvalues;
+ int maxvalues;
+ BlockNumber pagesPerRange = BrinGetPagesPerRange(bdesc->bd_index);
+
+ /* what was specified as a reloption? */
+ target_maxvalues = brin_minmax_multi_get_values(bdesc, opts);
+
+ /*
+ * Determine the insert buffer size - we use 10x the target, capped to
+ * the maximum number of values in the heap range. This is more than
+ * enough, considering the actual number of rows per page is likely
+ * much lower, but meh.
+ */
+ maxvalues = Min(target_maxvalues * MINMAX_BUFFER_FACTOR,
+ MaxHeapTuplesPerPage * pagesPerRange);
+
+ /* but always at least the original value */
+ maxvalues = Max(maxvalues, target_maxvalues);
+
+ /* always cap by MIN/MAX */
+ maxvalues = Max(maxvalues, MINMAX_BUFFER_MIN);
+ maxvalues = Min(maxvalues, MINMAX_BUFFER_MAX);
+
+ oldctx = MemoryContextSwitchTo(column->bv_context);
+ ranges = minmax_multi_init(maxvalues);
+ ranges->attno = attno;
+ ranges->colloid = colloid;
+ ranges->typid = attr->atttypid;
+ ranges->target_maxvalues = target_maxvalues;
+
+ /* we'll certainly need the comparator, so just look it up now */
+ ranges->cmp = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTLessStrategyNumber);
+
+ MemoryContextSwitchTo(oldctx);
+
+ column->bv_allnulls = false;
+ modified = true;
+
+ column->bv_mem_value = PointerGetDatum(ranges);
+ column->bv_serialize = brin_minmax_multi_serialize;
+ }
+ else if (!ranges)
+ {
+ MemoryContext oldctx;
+
+ int maxvalues;
+ BlockNumber pagesPerRange = BrinGetPagesPerRange(bdesc->bd_index);
+
+ oldctx = MemoryContextSwitchTo(column->bv_context);
+
+ serialized = (SerializedRanges *) PG_DETOAST_DATUM(column->bv_values[0]);
+
+ /*
+ * Determine the insert buffer size - we use 10x the target, capped to
+ * the maximum number of values in the heap range. This is more than
+ * enough, considering the actual number of rows per page is likely
+ * much lower, but meh.
+ */
+ maxvalues = Min(serialized->maxvalues * MINMAX_BUFFER_FACTOR,
+ MaxHeapTuplesPerPage * pagesPerRange);
+
+ /* but always at least the original value */
+ maxvalues = Max(maxvalues, serialized->maxvalues);
+
+ /* always cap by MIN/MAX */
+ maxvalues = Max(maxvalues, MINMAX_BUFFER_MIN);
+ maxvalues = Min(maxvalues, MINMAX_BUFFER_MAX);
+
+ ranges = range_deserialize(maxvalues, serialized);
+
+ ranges->attno = attno;
+ ranges->colloid = colloid;
+ ranges->typid = attr->atttypid;
+
+ /* we'll certainly need the comparator, so just look it up now */
+ ranges->cmp = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTLessStrategyNumber);
+
+ column->bv_mem_value = PointerGetDatum(ranges);
+ column->bv_serialize = brin_minmax_multi_serialize;
+
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ /*
+ * Try to add the new value to the range. We need to update the modified
+ * flag, so that we serialize the updated summary later.
+ */
+ modified |= range_add_value(bdesc, colloid, attno, attr, ranges, newval);
+
+
+ PG_RETURN_BOOL(modified);
+}
+
+/*
+ * Given an index tuple corresponding to a certain page range and a scan key,
+ * return whether the scan key is consistent with the index tuple's min/max
+ * values. Return true if so, false otherwise.
+ */
+Datum
+brin_minmax_multi_consistent(PG_FUNCTION_ARGS)
+{
+ BrinDesc *bdesc = (BrinDesc *) PG_GETARG_POINTER(0);
+ BrinValues *column = (BrinValues *) PG_GETARG_POINTER(1);
+ ScanKey *keys = (ScanKey *) PG_GETARG_POINTER(2);
+ int nkeys = PG_GETARG_INT32(3);
+
+ /* MinMaxOptions *opts = (MinMaxOptions *) PG_GET_OPCLASS_OPTIONS(); */
+ Oid colloid = PG_GET_COLLATION(),
+ subtype;
+ AttrNumber attno;
+ Datum value;
+ FmgrInfo *finfo;
+ SerializedRanges *serialized;
+ Ranges *ranges;
+ int keyno;
+ int rangeno;
+ int i;
+
+ attno = column->bv_attno;
+
+ serialized = (SerializedRanges *) PG_DETOAST_DATUM(column->bv_values[0]);
+ ranges = range_deserialize(serialized->maxvalues, serialized);
+
+ /* inspect the ranges, and for each one evaluate the scan keys */
+ for (rangeno = 0; rangeno < ranges->nranges; rangeno++)
+ {
+ Datum minval = ranges->values[2 * rangeno];
+ Datum maxval = ranges->values[2 * rangeno + 1];
+
+ /* assume the range is matching, and we'll try to prove otherwise */
+ bool matching = true;
+
+ for (keyno = 0; keyno < nkeys; keyno++)
+ {
+ Datum matches;
+ ScanKey key = keys[keyno];
+
+ /* NULL keys are handled and filtered-out in bringetbitmap */
+ Assert(!(key->sk_flags & SK_ISNULL));
+
+ attno = key->sk_attno;
+ subtype = key->sk_subtype;
+ value = key->sk_argument;
+ switch (key->sk_strategy)
+ {
+ case BTLessStrategyNumber:
+ case BTLessEqualStrategyNumber:
+ finfo = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype,
+ key->sk_strategy);
+ /* first value from the array */
+ matches = FunctionCall2Coll(finfo, colloid, minval, value);
+ break;
+
+ case BTEqualStrategyNumber:
+ {
+ Datum compar;
+ FmgrInfo *cmpFn;
+
+ /* by default this range does not match */
+ matches = false;
+
+ /*
+ * Otherwise, need to compare the new value with
+ * boundaries of all the ranges. First check if it's
+ * less than the absolute minimum, which is the first
+ * value in the array.
+ */
+ cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype,
+ BTLessStrategyNumber);
+ compar = FunctionCall2Coll(cmpFn, colloid, value, minval);
+
+ /* smaller than the smallest value in this range */
+ if (DatumGetBool(compar))
+ break;
+
+ cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype,
+ BTGreaterStrategyNumber);
+ compar = FunctionCall2Coll(cmpFn, colloid, value, maxval);
+
+ /* larger than the largest value in this range */
+ if (DatumGetBool(compar))
+ break;
+
+ /*
+ * haven't managed to eliminate this range, so
+ * consider it matching
+ */
+ matches = true;
+
+ break;
+ }
+ case BTGreaterEqualStrategyNumber:
+ case BTGreaterStrategyNumber:
+ finfo = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype,
+ key->sk_strategy);
+ /* last value from the array */
+ matches = FunctionCall2Coll(finfo, colloid, maxval, value);
+ break;
+
+ default:
+ /* shouldn't happen */
+ elog(ERROR, "invalid strategy number %d", key->sk_strategy);
+ matches = 0;
+ break;
+ }
+
+ /* the range has to match all the scan keys */
+ matching &= DatumGetBool(matches);
+
+ /* once we find a non-matching key, we're done */
+ if (!matching)
+ break;
+ }
+
+ /*
+ * have we found a range matching all scan keys? if yes, we're done
+ */
+ if (matching)
+ PG_RETURN_DATUM(BoolGetDatum(true));
+ }
+
+ /* and now inspect the values */
+ for (i = 0; i < ranges->nvalues; i++)
+ {
+ Datum val = ranges->values[2 * ranges->nranges + i];
+
+ /* assume the range is matching, and we'll try to prove otherwise */
+ bool matching = true;
+
+ for (keyno = 0; keyno < nkeys; keyno++)
+ {
+ Datum matches;
+ ScanKey key = keys[keyno];
+
+ /* we've already dealt with NULL keys at the beginning */
+ if (key->sk_flags & SK_ISNULL)
+ continue;
+
+ attno = key->sk_attno;
+ subtype = key->sk_subtype;
+ value = key->sk_argument;
+ switch (key->sk_strategy)
+ {
+ case BTLessStrategyNumber:
+ case BTLessEqualStrategyNumber:
+ case BTEqualStrategyNumber:
+ case BTGreaterEqualStrategyNumber:
+ case BTGreaterStrategyNumber:
+
+ finfo = minmax_multi_get_strategy_procinfo(bdesc, attno, subtype,
+ key->sk_strategy);
+ matches = FunctionCall2Coll(finfo, colloid, val, value);
+ break;
+
+ default:
+ /* shouldn't happen */
+ elog(ERROR, "invalid strategy number %d", key->sk_strategy);
+ matches = 0;
+ break;
+ }
+
+ /* the range has to match all the scan keys */
+ matching &= DatumGetBool(matches);
+
+ /* once we find a non-matching key, we're done */
+ if (!matching)
+ break;
+ }
+
+ /*
+ * have we found a range matching all scan keys? if yes, we're done
+ */
+ if (matching)
+ PG_RETURN_DATUM(BoolGetDatum(true));
+ }
+
+ PG_RETURN_DATUM(BoolGetDatum(false));
+}
+
+/*
+ * Given two BrinValues, update the first of them as a union of the summary
+ * values contained in both. The second one is untouched.
+ */
+Datum
+brin_minmax_multi_union(PG_FUNCTION_ARGS)
+{
+ BrinDesc *bdesc = (BrinDesc *) PG_GETARG_POINTER(0);
+ BrinValues *col_a = (BrinValues *) PG_GETARG_POINTER(1);
+ BrinValues *col_b = (BrinValues *) PG_GETARG_POINTER(2);
+
+ /* MinMaxOptions *opts = (MinMaxOptions *) PG_GET_OPCLASS_OPTIONS(); */
+ Oid colloid = PG_GET_COLLATION();
+ SerializedRanges *serialized_a;
+ SerializedRanges *serialized_b;
+ Ranges *ranges_a;
+ Ranges *ranges_b;
+ AttrNumber attno;
+ Form_pg_attribute attr;
+ ExpandedRange *eranges;
+ int neranges;
+ FmgrInfo *cmpFn,
+ *distanceFn;
+ DistanceValue *distances;
+ MemoryContext ctx;
+ MemoryContext oldctx;
+
+ Assert(col_a->bv_attno == col_b->bv_attno);
+ Assert(!col_a->bv_allnulls && !col_b->bv_allnulls);
+
+ attno = col_a->bv_attno;
+ attr = TupleDescAttr(bdesc->bd_tupdesc, attno - 1);
+
+ serialized_a = (SerializedRanges *) PG_DETOAST_DATUM(col_a->bv_values[0]);
+ serialized_b = (SerializedRanges *) PG_DETOAST_DATUM(col_b->bv_values[0]);
+
+ ranges_a = range_deserialize(serialized_a->maxvalues, serialized_a);
+ ranges_b = range_deserialize(serialized_b->maxvalues, serialized_b);
+
+ /* make sure neither of the ranges is NULL */
+ Assert(ranges_a && ranges_b);
+
+ neranges = (ranges_a->nranges + ranges_a->nvalues) +
+ (ranges_b->nranges + ranges_b->nvalues);
+
+ /*
+ * The distanceFn calls (which may internally call e.g. numeric_le) may
+ * allocate quite a bit of memory, and we must not leak it. Otherwise we'd
+ * have problems e.g. when building indexes. So we create a local memory
+ * context and make sure we free the memory before leaving this function
+ * (not after every call).
+ */
+ ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "minmax-multi context",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldctx = MemoryContextSwitchTo(ctx);
+
+ /* allocate and fill */
+ eranges = (ExpandedRange *) palloc0(neranges * sizeof(ExpandedRange));
+
+ /* fill the expanded ranges with entries for the first range */
+ fill_expanded_ranges(eranges, ranges_a->nranges + ranges_a->nvalues,
+ ranges_a);
+
+ /* and now add combine ranges for the second range */
+ fill_expanded_ranges(&eranges[ranges_a->nranges + ranges_a->nvalues],
+ ranges_b->nranges + ranges_b->nvalues,
+ ranges_b);
+
+ cmpFn = minmax_multi_get_strategy_procinfo(bdesc, attno, attr->atttypid,
+ BTLessStrategyNumber);
+
+ /* sort the expanded ranges */
+ sort_expanded_ranges(cmpFn, colloid, eranges, neranges);
+
+ /*
+ * We've loaded two different lists of expanded ranges, so some of them
+ * may be overlapping. So walk through them and merge them.
+ */
+ neranges = merge_overlapping_ranges(cmpFn, colloid, eranges, neranges);
+
+ /* check that the combine ranges are correct (no overlaps, ordering) */
+ AssertValidExpandedRanges(bdesc, colloid, attno, attr, eranges, neranges);
+
+ /*
+ * If needed, reduce some of the ranges.
+ *
+ * XXX This may be fairly expensive, so maybe we should do it only when
+ * it's actually needed (when we have too many ranges).
+ */
+
+ /* build array of gap distances and sort them in ascending order */
+ distanceFn = minmax_multi_get_procinfo(bdesc, attno, PROCNUM_DISTANCE);
+ distances = build_distances(distanceFn, colloid, eranges, neranges);
+
+ /*
+ * See how many values would be needed to store the current ranges, and if
+ * needed combine as many off them to get below the threshold. The
+ * collapsed ranges will be stored as a single value.
+ *
+ * XXX This does not apply the load factor, as we don't expect to add more
+ * values to the range, so we prefer to keep as many ranges as possible.
+ *
+ * XXX Can the maxvalues be different in the two ranges? Perhaps we should
+ * use maximum of those?
+ */
+ neranges = reduce_expanded_ranges(eranges, neranges, distances,
+ ranges_a->maxvalues,
+ cmpFn, colloid);
+
+ /* update the first range summary */
+ store_expanded_ranges(ranges_a, eranges, neranges);
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextDelete(ctx);
+
+ /* cleanup and update the serialized value */
+ pfree(serialized_a);
+ col_a->bv_values[0] = PointerGetDatum(range_serialize(ranges_a));
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Cache and return minmax multi opclass support procedure
+ *
+ * Return the procedure corresponding to the given function support number
+ * or null if it is not exists.
+ */
+static FmgrInfo *
+minmax_multi_get_procinfo(BrinDesc *bdesc, uint16 attno, uint16 procnum)
+{
+ MinmaxMultiOpaque *opaque;
+ uint16 basenum = procnum - PROCNUM_BASE;
+
+ /*
+ * We cache these in the opaque struct, to avoid repetitive syscache
+ * lookups.
+ */
+ opaque = (MinmaxMultiOpaque *) bdesc->bd_info[attno - 1]->oi_opaque;
+
+ /*
+ * If we already searched for this proc and didn't find it, don't bother
+ * searching again.
+ */
+ if (opaque->extra_proc_missing[basenum])
+ return NULL;
+
+ if (opaque->extra_procinfos[basenum].fn_oid == InvalidOid)
+ {
+ if (RegProcedureIsValid(index_getprocid(bdesc->bd_index, attno,
+ procnum)))
+ {
+ fmgr_info_copy(&opaque->extra_procinfos[basenum],
+ index_getprocinfo(bdesc->bd_index, attno, procnum),
+ bdesc->bd_context);
+ }
+ else
+ {
+ opaque->extra_proc_missing[basenum] = true;
+ return NULL;
+ }
+ }
+
+ return &opaque->extra_procinfos[basenum];
+}
+
+/*
+ * Cache and return the procedure for the given strategy.
+ *
+ * Note: this function mirrors minmax_multi_get_strategy_procinfo; see notes
+ * there. If changes are made here, see that function too.
+ */
+static FmgrInfo *
+minmax_multi_get_strategy_procinfo(BrinDesc *bdesc, uint16 attno, Oid subtype,
+ uint16 strategynum)
+{
+ MinmaxMultiOpaque *opaque;
+
+ Assert(strategynum >= 1 &&
+ strategynum <= BTMaxStrategyNumber);
+
+ opaque = (MinmaxMultiOpaque *) bdesc->bd_info[attno - 1]->oi_opaque;
+
+ /*
+ * We cache the procedures for the previous subtype in the opaque struct,
+ * to avoid repetitive syscache lookups. If the subtype changed,
+ * invalidate all the cached entries.
+ */
+ if (opaque->cached_subtype != subtype)
+ {
+ uint16 i;
+
+ for (i = 1; i <= BTMaxStrategyNumber; i++)
+ opaque->strategy_procinfos[i - 1].fn_oid = InvalidOid;
+ opaque->cached_subtype = subtype;
+ }
+
+ if (opaque->strategy_procinfos[strategynum - 1].fn_oid == InvalidOid)
+ {
+ Form_pg_attribute attr;
+ HeapTuple tuple;
+ Oid opfamily,
+ oprid;
+ bool isNull;
+
+ opfamily = bdesc->bd_index->rd_opfamily[attno - 1];
+ attr = TupleDescAttr(bdesc->bd_tupdesc, attno - 1);
+ tuple = SearchSysCache4(AMOPSTRATEGY, ObjectIdGetDatum(opfamily),
+ ObjectIdGetDatum(attr->atttypid),
+ ObjectIdGetDatum(subtype),
+ Int16GetDatum(strategynum));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "missing operator %d(%u,%u) in opfamily %u",
+ strategynum, attr->atttypid, subtype, opfamily);
+
+ oprid = DatumGetObjectId(SysCacheGetAttr(AMOPSTRATEGY, tuple,
+ Anum_pg_amop_amopopr, &isNull));
+ ReleaseSysCache(tuple);
+ Assert(!isNull && RegProcedureIsValid(oprid));
+
+ fmgr_info_cxt(get_opcode(oprid),
+ &opaque->strategy_procinfos[strategynum - 1],
+ bdesc->bd_context);
+ }
+
+ return &opaque->strategy_procinfos[strategynum - 1];
+}
+
+Datum
+brin_minmax_multi_options(PG_FUNCTION_ARGS)
+{
+ local_relopts *relopts = (local_relopts *) PG_GETARG_POINTER(0);
+
+ init_local_reloptions(relopts, sizeof(MinMaxOptions));
+
+ add_local_int_reloption(relopts, "values_per_range", "desc",
+ 32, 8, 256, offsetof(MinMaxOptions, valuesPerRange));
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * brin_minmax_multi_summary_in
+ * - input routine for type brin_minmax_multi_summary.
+ *
+ * brin_minmax_multi_summary is only used internally to represent summaries
+ * in BRIN minmax-multi indexes, so it has no operations of its own, and we
+ * disallow input too.
+ */
+Datum
+brin_minmax_multi_summary_in(PG_FUNCTION_ARGS)
+{
+ /*
+ * brin_minmax_multi_summary stores the data in binary form and parsing
+ * text input is not needed, so disallow this.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type %s", "brin_minmax_multi_summary")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+
+/*
+ * brin_minmax_multi_summary_out
+ * - output routine for type brin_minmax_multi_summary.
+ *
+ * BRIN minmax-multi summaries are serialized into a bytea value, but we
+ * want to output something nicer humans can understand.
+ */
+Datum
+brin_minmax_multi_summary_out(PG_FUNCTION_ARGS)
+{
+ int i;
+ int idx;
+ SerializedRanges *ranges;
+ Ranges *ranges_deserialized;
+ StringInfoData str;
+ bool isvarlena;
+ Oid outfunc;
+ FmgrInfo fmgrinfo;
+ ArrayBuildState *astate_values = NULL;
+
+ initStringInfo(&str);
+ appendStringInfoChar(&str, '{');
+
+ /*
+ * Detoast to get value with full 4B header (can't be stored in a toast
+ * table, but can use 1B header).
+ */
+ ranges = (SerializedRanges *) PG_DETOAST_DATUM(PG_GETARG_BYTEA_PP(0));
+
+ /* lookup output func for the type */
+ getTypeOutputInfo(ranges->typid, &outfunc, &isvarlena);
+ fmgr_info(outfunc, &fmgrinfo);
+
+ /* deserialize the range info easy-to-process pieces */
+ ranges_deserialized = range_deserialize(ranges->maxvalues, ranges);
+
+ appendStringInfo(&str, "nranges: %u nvalues: %u maxvalues: %u",
+ ranges_deserialized->nranges,
+ ranges_deserialized->nvalues,
+ ranges_deserialized->maxvalues);
+
+ /* serialize ranges */
+ idx = 0;
+ for (i = 0; i < ranges_deserialized->nranges; i++)
+ {
+ Datum a,
+ b;
+ text *c;
+ StringInfoData str;
+
+ initStringInfo(&str);
+
+ a = FunctionCall1(&fmgrinfo, ranges_deserialized->values[idx++]);
+ b = FunctionCall1(&fmgrinfo, ranges_deserialized->values[idx++]);
+
+ appendStringInfo(&str, "%s ... %s",
+ DatumGetPointer(a),
+ DatumGetPointer(b));
+
+ c = cstring_to_text(str.data);
+
+ astate_values = accumArrayResult(astate_values,
+ PointerGetDatum(c),
+ false,
+ TEXTOID,
+ CurrentMemoryContext);
+ }
+
+ if (ranges_deserialized->nranges > 0)
+ {
+ Oid typoutput;
+ bool typIsVarlena;
+ Datum val;
+ char *extval;
+
+ getTypeOutputInfo(ANYARRAYOID, &typoutput, &typIsVarlena);
+
+ val = PointerGetDatum(makeArrayResult(astate_values, CurrentMemoryContext));
+
+ extval = OidOutputFunctionCall(typoutput, val);
+
+ appendStringInfo(&str, " ranges: %s", extval);
+ }
+
+ /* serialize individual values */
+ astate_values = NULL;
+
+ for (i = 0; i < ranges_deserialized->nvalues; i++)
+ {
+ Datum a;
+ text *b;
+ StringInfoData str;
+
+ initStringInfo(&str);
+
+ a = FunctionCall1(&fmgrinfo, ranges_deserialized->values[idx++]);
+
+ appendStringInfo(&str, "%s", DatumGetPointer(a));
+
+ b = cstring_to_text(str.data);
+
+ astate_values = accumArrayResult(astate_values,
+ PointerGetDatum(b),
+ false,
+ TEXTOID,
+ CurrentMemoryContext);
+ }
+
+ if (ranges_deserialized->nvalues > 0)
+ {
+ Oid typoutput;
+ bool typIsVarlena;
+ Datum val;
+ char *extval;
+
+ getTypeOutputInfo(ANYARRAYOID, &typoutput, &typIsVarlena);
+
+ val = PointerGetDatum(makeArrayResult(astate_values, CurrentMemoryContext));
+
+ extval = OidOutputFunctionCall(typoutput, val);
+
+ appendStringInfo(&str, " values: %s", extval);
+ }
+
+
+ appendStringInfoChar(&str, '}');
+
+ PG_RETURN_CSTRING(str.data);
+}
+
+/*
+ * brin_minmax_multi_summary_recv
+ * - binary input routine for type brin_minmax_multi_summary.
+ */
+Datum
+brin_minmax_multi_summary_recv(PG_FUNCTION_ARGS)
+{
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type %s", "brin_minmax_multi_summary")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+/*
+ * brin_minmax_multi_summary_send
+ * - binary output routine for type brin_minmax_multi_summary.
+ *
+ * BRIN minmax-multi summaries are serialized in a bytea value (although
+ * the type is named differently), so let's just send that.
+ */
+Datum
+brin_minmax_multi_summary_send(PG_FUNCTION_ARGS)
+{
+ return byteasend(fcinfo);
+}
diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c
index a7eb1c9473..69ce6dfca4 100644
--- a/src/backend/access/brin/brin_tuple.c
+++ b/src/backend/access/brin/brin_tuple.c
@@ -159,6 +159,14 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
if (tuple->bt_columns[keyno].bv_hasnulls)
anynulls = true;
+ /* if needed, serialize the values before forming the on-disk tuple */
+ if (tuple->bt_columns[keyno].bv_serialize)
+ {
+ tuple->bt_columns[keyno].bv_serialize(brdesc,
+ tuple->bt_columns[keyno].bv_mem_value,
+ tuple->bt_columns[keyno].bv_values);
+ }
+
/*
* Now obtain the values of each stored datum. Note that some values
* might be toasted, and we cannot rely on the original heap values
@@ -169,15 +177,15 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
datumno < brdesc->bd_info[keyno]->oi_nstored;
datumno++)
{
- Datum value = tuple->bt_columns[keyno].bv_values[datumno];
+ Datum value = tuple->bt_columns[keyno].bv_values[datumno];
#ifdef TOAST_INDEX_HACK
/* We must look at the stored type, not at the index descriptor. */
- TypeCacheEntry *atttype = brdesc->bd_info[keyno]->oi_typcache[datumno];
+ TypeCacheEntry *atttype = brdesc->bd_info[keyno]->oi_typcache[datumno];
/* Do we need to free the value at the end? */
- bool free_value = false;
+ bool free_value = false;
/* For non-varlena types we don't need to do anything special */
if (atttype->typlen != -1)
@@ -193,9 +201,9 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
* If value is stored EXTERNAL, must fetch it so we are not
* depending on outside storage.
*
- * XXX Is this actually true? Could it be that the summary is
- * NULL even for range with non-NULL data? E.g. degenerate bloom
- * filter may be thrown away, etc.
+ * XXX Is this actually true? Could it be that the summary is NULL
+ * even for range with non-NULL data? E.g. degenerate bloom filter
+ * may be thrown away, etc.
*/
if (VARATT_IS_EXTERNAL(DatumGetPointer(value)))
{
@@ -205,8 +213,8 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
}
/*
- * If value is above size target, and is of a compressible datatype,
- * try to compress it in-line.
+ * If value is above size target, and is of a compressible
+ * datatype, try to compress it in-line.
*/
if (!VARATT_IS_EXTENDED(DatumGetPointer(value)) &&
VARSIZE(DatumGetPointer(value)) > TOAST_INDEX_TARGET &&
@@ -495,6 +503,11 @@ brin_memtuple_initialize(BrinMemTuple *dtuple, BrinDesc *brdesc)
dtuple->bt_columns[i].bv_allnulls = true;
dtuple->bt_columns[i].bv_hasnulls = false;
dtuple->bt_columns[i].bv_values = (Datum *) currdatum;
+
+ dtuple->bt_columns[i].bv_mem_value = PointerGetDatum(NULL);
+ dtuple->bt_columns[i].bv_serialize = NULL;
+ dtuple->bt_columns[i].bv_context = dtuple->bt_context;
+
currdatum += sizeof(Datum) * brdesc->bd_info[i]->oi_nstored;
}
@@ -574,6 +587,10 @@ brin_deform_tuple(BrinDesc *brdesc, BrinTuple *tuple, BrinMemTuple *dMemtuple)
dtup->bt_columns[keyno].bv_hasnulls = hasnulls[keyno];
dtup->bt_columns[keyno].bv_allnulls = false;
+
+ dtup->bt_columns[keyno].bv_mem_value = PointerGetDatum(NULL);
+ dtup->bt_columns[keyno].bv_serialize = NULL;
+ dtup->bt_columns[keyno].bv_context = dtup->bt_context;
}
MemoryContextSwitchTo(oldcxt);
diff --git a/src/include/access/brin.h b/src/include/access/brin.h
index 0e52d75457..9cd5fa9f62 100644
--- a/src/include/access/brin.h
+++ b/src/include/access/brin.h
@@ -24,6 +24,7 @@ typedef struct BrinOptions
bool autosummarize;
double nDistinctPerRange; /* number of distinct values per range */
double falsePositiveRate; /* false positive for bloom filter */
+ int valuesPerRange; /* number of values per range */
} BrinOptions;
diff --git a/src/include/access/brin_internal.h b/src/include/access/brin_internal.h
index 8cc4e532e6..fdaff42722 100644
--- a/src/include/access/brin_internal.h
+++ b/src/include/access/brin_internal.h
@@ -62,6 +62,9 @@ typedef struct BrinDesc
double bd_nDistinctPerRange;
double bd_falsePositiveRange;
+ /* parameters for multi-minmax indexes */
+ int bd_valuesPerRange;
+
/* per-column info; bd_tupdesc->natts entries long */
BrinOpcInfo *bd_info[FLEXIBLE_ARRAY_MEMBER];
} BrinDesc;
diff --git a/src/include/access/brin_tuple.h b/src/include/access/brin_tuple.h
index 5715bd58df..87de94f397 100644
--- a/src/include/access/brin_tuple.h
+++ b/src/include/access/brin_tuple.h
@@ -14,6 +14,11 @@
#include "access/brin_internal.h"
#include "access/tupdesc.h"
+/*
+ * The BRIN opclasses may register serialization callback, in case the on-disk
+ * and in-memory representations differ (e.g. for performance reasons).
+ */
+typedef void (*brin_serialize_callback_type) (BrinDesc *bdesc, Datum src, Datum *dst);
/*
* A BRIN index stores one index tuple per page range. Each index tuple
@@ -27,6 +32,9 @@ typedef struct BrinValues
bool bv_hasnulls; /* are there any nulls in the page range? */
bool bv_allnulls; /* are all values nulls in the page range? */
Datum *bv_values; /* current accumulated values */
+ Datum bv_mem_value; /* expanded accumulated values */
+ MemoryContext bv_context;
+ brin_serialize_callback_type bv_serialize;
} BrinValues;
/*
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 82e874130d..654584a03f 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -186,7 +186,7 @@ FullTransactionIdAdvance(FullTransactionId *dest)
* ----------
*/
#define FirstGenbkiObjectId 10000
-#define FirstBootstrapObjectId 12000
+#define FirstBootstrapObjectId 13000
#define FirstNormalObjectId 16384
/*
diff --git a/src/include/catalog/pg_amop.dat b/src/include/catalog/pg_amop.dat
index 04d678f96a..8135854163 100644
--- a/src/include/catalog/pg_amop.dat
+++ b/src/include/catalog/pg_amop.dat
@@ -2009,6 +2009,152 @@
amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int4,int8)',
amopmethod => 'brin' },
+# minmax multi integer
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int8', amopstrategy => '1', amopopr => '<(int8,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int8', amopstrategy => '2', amopopr => '<=(int8,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int8', amopstrategy => '3', amopopr => '=(int8,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int8', amopstrategy => '4', amopopr => '>=(int8,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int8,int8)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int2', amopstrategy => '1', amopopr => '<(int8,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int2', amopstrategy => '2', amopopr => '<=(int8,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int2', amopstrategy => '3', amopopr => '=(int8,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int2', amopstrategy => '4', amopopr => '>=(int8,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int2', amopstrategy => '5', amopopr => '>(int8,int2)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int4', amopstrategy => '1', amopopr => '<(int8,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int4', amopstrategy => '2', amopopr => '<=(int8,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int4', amopstrategy => '3', amopopr => '=(int8,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int4', amopstrategy => '4', amopopr => '>=(int8,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int8',
+ amoprighttype => 'int4', amopstrategy => '5', amopopr => '>(int8,int4)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int2', amopstrategy => '1', amopopr => '<(int2,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int2', amopstrategy => '2', amopopr => '<=(int2,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int2', amopstrategy => '3', amopopr => '=(int2,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int2', amopstrategy => '4', amopopr => '>=(int2,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int2', amopstrategy => '5', amopopr => '>(int2,int2)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int8', amopstrategy => '1', amopopr => '<(int2,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int8', amopstrategy => '2', amopopr => '<=(int2,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int8', amopstrategy => '3', amopopr => '=(int2,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int8', amopstrategy => '4', amopopr => '>=(int2,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int2,int8)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int4', amopstrategy => '1', amopopr => '<(int2,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int4', amopstrategy => '2', amopopr => '<=(int2,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int4', amopstrategy => '3', amopopr => '=(int2,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int4', amopstrategy => '4', amopopr => '>=(int2,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int2',
+ amoprighttype => 'int4', amopstrategy => '5', amopopr => '>(int2,int4)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int4', amopstrategy => '1', amopopr => '<(int4,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int4', amopstrategy => '2', amopopr => '<=(int4,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int4', amopstrategy => '3', amopopr => '=(int4,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int4', amopstrategy => '4', amopopr => '>=(int4,int4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int4', amopstrategy => '5', amopopr => '>(int4,int4)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int2', amopstrategy => '1', amopopr => '<(int4,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int2', amopstrategy => '2', amopopr => '<=(int4,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int2', amopstrategy => '3', amopopr => '=(int4,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int2', amopstrategy => '4', amopopr => '>=(int4,int2)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int2', amopstrategy => '5', amopopr => '>(int4,int2)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int8', amopstrategy => '1', amopopr => '<(int4,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int8', amopstrategy => '2', amopopr => '<=(int4,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int8', amopstrategy => '3', amopopr => '=(int4,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int8', amopstrategy => '4', amopopr => '>=(int4,int8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/integer_minmax_multi_ops', amoplefttype => 'int4',
+ amoprighttype => 'int8', amopstrategy => '5', amopopr => '>(int4,int8)',
+ amopmethod => 'brin' },
+
# bloom integer
{ amopfamily => 'brin/integer_bloom_ops', amoplefttype => 'int8',
@@ -2062,6 +2208,23 @@
amoprighttype => 'oid', amopstrategy => '5', amopopr => '>(oid,oid)',
amopmethod => 'brin' },
+# minmax multi oid
+{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid',
+ amoprighttype => 'oid', amopstrategy => '1', amopopr => '<(oid,oid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid',
+ amoprighttype => 'oid', amopstrategy => '2', amopopr => '<=(oid,oid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid',
+ amoprighttype => 'oid', amopstrategy => '3', amopopr => '=(oid,oid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid',
+ amoprighttype => 'oid', amopstrategy => '4', amopopr => '>=(oid,oid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/oid_minmax_multi_ops', amoplefttype => 'oid',
+ amoprighttype => 'oid', amopstrategy => '5', amopopr => '>(oid,oid)',
+ amopmethod => 'brin' },
+
# bloom oid
{ amopfamily => 'brin/oid_bloom_ops', amoplefttype => 'oid',
amoprighttype => 'oid', amopstrategy => '1', amopopr => '=(oid,oid)',
@@ -2088,6 +2251,22 @@
{ amopfamily => 'brin/tid_bloom_ops', amoplefttype => 'tid',
amoprighttype => 'tid', amopstrategy => '1', amopopr => '=(tid,tid)',
amopmethod => 'brin' },
+# minmax multi tid
+{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid',
+ amoprighttype => 'tid', amopstrategy => '1', amopopr => '<(tid,tid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid',
+ amoprighttype => 'tid', amopstrategy => '2', amopopr => '<=(tid,tid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid',
+ amoprighttype => 'tid', amopstrategy => '3', amopopr => '=(tid,tid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid',
+ amoprighttype => 'tid', amopstrategy => '4', amopopr => '>=(tid,tid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/tid_minmax_multi_ops', amoplefttype => 'tid',
+ amoprighttype => 'tid', amopstrategy => '5', amopopr => '>(tid,tid)',
+ amopmethod => 'brin' },
# minmax float (float4, float8)
@@ -2155,6 +2334,72 @@
amoprighttype => 'float8', amopstrategy => '5', amopopr => '>(float8,float8)',
amopmethod => 'brin' },
+# minmax multi float (float4, float8)
+
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float4', amopstrategy => '1', amopopr => '<(float4,float4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float4', amopstrategy => '2',
+ amopopr => '<=(float4,float4)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float4', amopstrategy => '3', amopopr => '=(float4,float4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float4', amopstrategy => '4',
+ amopopr => '>=(float4,float4)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float4', amopstrategy => '5', amopopr => '>(float4,float4)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float8', amopstrategy => '1', amopopr => '<(float4,float8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float8', amopstrategy => '2',
+ amopopr => '<=(float4,float8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float8', amopstrategy => '3', amopopr => '=(float4,float8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float8', amopstrategy => '4',
+ amopopr => '>=(float4,float8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float4',
+ amoprighttype => 'float8', amopstrategy => '5', amopopr => '>(float4,float8)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float4', amopstrategy => '1', amopopr => '<(float8,float4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float4', amopstrategy => '2',
+ amopopr => '<=(float8,float4)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float4', amopstrategy => '3', amopopr => '=(float8,float4)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float4', amopstrategy => '4',
+ amopopr => '>=(float8,float4)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float4', amopstrategy => '5', amopopr => '>(float8,float4)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float8', amopstrategy => '1', amopopr => '<(float8,float8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float8', amopstrategy => '2',
+ amopopr => '<=(float8,float8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float8', amopstrategy => '3', amopopr => '=(float8,float8)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float8', amopstrategy => '4',
+ amopopr => '>=(float8,float8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/float_minmax_multi_ops', amoplefttype => 'float8',
+ amoprighttype => 'float8', amopstrategy => '5', amopopr => '>(float8,float8)',
+ amopmethod => 'brin' },
+
# bloom float
{ amopfamily => 'brin/float_bloom_ops', amoplefttype => 'float4',
amoprighttype => 'float4', amopstrategy => '1', amopopr => '=(float4,float4)',
@@ -2180,6 +2425,23 @@
amoprighttype => 'macaddr', amopstrategy => '5',
amopopr => '>(macaddr,macaddr)', amopmethod => 'brin' },
+# minmax multi macaddr
+{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr',
+ amoprighttype => 'macaddr', amopstrategy => '1',
+ amopopr => '<(macaddr,macaddr)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr',
+ amoprighttype => 'macaddr', amopstrategy => '2',
+ amopopr => '<=(macaddr,macaddr)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr',
+ amoprighttype => 'macaddr', amopstrategy => '3',
+ amopopr => '=(macaddr,macaddr)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr',
+ amoprighttype => 'macaddr', amopstrategy => '4',
+ amopopr => '>=(macaddr,macaddr)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr_minmax_multi_ops', amoplefttype => 'macaddr',
+ amoprighttype => 'macaddr', amopstrategy => '5',
+ amopopr => '>(macaddr,macaddr)', amopmethod => 'brin' },
+
# bloom macaddr
{ amopfamily => 'brin/macaddr_bloom_ops', amoplefttype => 'macaddr',
amoprighttype => 'macaddr', amopstrategy => '1',
@@ -2202,6 +2464,23 @@
amoprighttype => 'macaddr8', amopstrategy => '5',
amopopr => '>(macaddr8,macaddr8)', amopmethod => 'brin' },
+# minmax multi macaddr8
+{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8',
+ amoprighttype => 'macaddr8', amopstrategy => '1',
+ amopopr => '<(macaddr8,macaddr8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8',
+ amoprighttype => 'macaddr8', amopstrategy => '2',
+ amopopr => '<=(macaddr8,macaddr8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8',
+ amoprighttype => 'macaddr8', amopstrategy => '3',
+ amopopr => '=(macaddr8,macaddr8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8',
+ amoprighttype => 'macaddr8', amopstrategy => '4',
+ amopopr => '>=(macaddr8,macaddr8)', amopmethod => 'brin' },
+{ amopfamily => 'brin/macaddr8_minmax_multi_ops', amoplefttype => 'macaddr8',
+ amoprighttype => 'macaddr8', amopstrategy => '5',
+ amopopr => '>(macaddr8,macaddr8)', amopmethod => 'brin' },
+
# bloom macaddr8
{ amopfamily => 'brin/macaddr8_bloom_ops', amoplefttype => 'macaddr8',
amoprighttype => 'macaddr8', amopstrategy => '1',
@@ -2224,6 +2503,23 @@
amoprighttype => 'inet', amopstrategy => '5', amopopr => '>(inet,inet)',
amopmethod => 'brin' },
+# minmax multi inet
+{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet',
+ amoprighttype => 'inet', amopstrategy => '1', amopopr => '<(inet,inet)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet',
+ amoprighttype => 'inet', amopstrategy => '2', amopopr => '<=(inet,inet)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet',
+ amoprighttype => 'inet', amopstrategy => '3', amopopr => '=(inet,inet)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet',
+ amoprighttype => 'inet', amopstrategy => '4', amopopr => '>=(inet,inet)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/network_minmax_multi_ops', amoplefttype => 'inet',
+ amoprighttype => 'inet', amopstrategy => '5', amopopr => '>(inet,inet)',
+ amopmethod => 'brin' },
+
# bloom inet
{ amopfamily => 'brin/network_bloom_ops', amoplefttype => 'inet',
amoprighttype => 'inet', amopstrategy => '1', amopopr => '=(inet,inet)',
@@ -2288,6 +2584,23 @@
amoprighttype => 'time', amopstrategy => '5', amopopr => '>(time,time)',
amopmethod => 'brin' },
+# minmax multi time without time zone
+{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time',
+ amoprighttype => 'time', amopstrategy => '1', amopopr => '<(time,time)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time',
+ amoprighttype => 'time', amopstrategy => '2', amopopr => '<=(time,time)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time',
+ amoprighttype => 'time', amopstrategy => '3', amopopr => '=(time,time)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time',
+ amoprighttype => 'time', amopstrategy => '4', amopopr => '>=(time,time)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/time_minmax_multi_ops', amoplefttype => 'time',
+ amoprighttype => 'time', amopstrategy => '5', amopopr => '>(time,time)',
+ amopmethod => 'brin' },
+
# bloom time without time zone
{ amopfamily => 'brin/time_bloom_ops', amoplefttype => 'time',
amoprighttype => 'time', amopstrategy => '1', amopopr => '=(time,time)',
@@ -2439,6 +2752,152 @@
amoprighttype => 'timestamptz', amopstrategy => '5',
amopopr => '>(timestamptz,timestamptz)', amopmethod => 'brin' },
+# minmax multi datetime (date, timestamp, timestamptz)
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamp', amopstrategy => '1',
+ amopopr => '<(timestamp,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamp', amopstrategy => '2',
+ amopopr => '<=(timestamp,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamp', amopstrategy => '3',
+ amopopr => '=(timestamp,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamp', amopstrategy => '4',
+ amopopr => '>=(timestamp,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamp', amopstrategy => '5',
+ amopopr => '>(timestamp,timestamp)', amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'date', amopstrategy => '1', amopopr => '<(timestamp,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'date', amopstrategy => '2', amopopr => '<=(timestamp,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'date', amopstrategy => '3', amopopr => '=(timestamp,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'date', amopstrategy => '4', amopopr => '>=(timestamp,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'date', amopstrategy => '5', amopopr => '>(timestamp,date)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamptz', amopstrategy => '1',
+ amopopr => '<(timestamp,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamptz', amopstrategy => '2',
+ amopopr => '<=(timestamp,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamptz', amopstrategy => '3',
+ amopopr => '=(timestamp,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamptz', amopstrategy => '4',
+ amopopr => '>=(timestamp,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamp',
+ amoprighttype => 'timestamptz', amopstrategy => '5',
+ amopopr => '>(timestamp,timestamptz)', amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'date', amopstrategy => '1', amopopr => '<(date,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'date', amopstrategy => '2', amopopr => '<=(date,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'date', amopstrategy => '3', amopopr => '=(date,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'date', amopstrategy => '4', amopopr => '>=(date,date)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'date', amopstrategy => '5', amopopr => '>(date,date)',
+ amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamp', amopstrategy => '1',
+ amopopr => '<(date,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamp', amopstrategy => '2',
+ amopopr => '<=(date,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamp', amopstrategy => '3',
+ amopopr => '=(date,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamp', amopstrategy => '4',
+ amopopr => '>=(date,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamp', amopstrategy => '5',
+ amopopr => '>(date,timestamp)', amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamptz', amopstrategy => '1',
+ amopopr => '<(date,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamptz', amopstrategy => '2',
+ amopopr => '<=(date,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamptz', amopstrategy => '3',
+ amopopr => '=(date,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamptz', amopstrategy => '4',
+ amopopr => '>=(date,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'date',
+ amoprighttype => 'timestamptz', amopstrategy => '5',
+ amopopr => '>(date,timestamptz)', amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'date', amopstrategy => '1',
+ amopopr => '<(timestamptz,date)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'date', amopstrategy => '2',
+ amopopr => '<=(timestamptz,date)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'date', amopstrategy => '3',
+ amopopr => '=(timestamptz,date)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'date', amopstrategy => '4',
+ amopopr => '>=(timestamptz,date)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'date', amopstrategy => '5',
+ amopopr => '>(timestamptz,date)', amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamp', amopstrategy => '1',
+ amopopr => '<(timestamptz,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamp', amopstrategy => '2',
+ amopopr => '<=(timestamptz,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamp', amopstrategy => '3',
+ amopopr => '=(timestamptz,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamp', amopstrategy => '4',
+ amopopr => '>=(timestamptz,timestamp)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamp', amopstrategy => '5',
+ amopopr => '>(timestamptz,timestamp)', amopmethod => 'brin' },
+
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamptz', amopstrategy => '1',
+ amopopr => '<(timestamptz,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamptz', amopstrategy => '2',
+ amopopr => '<=(timestamptz,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamptz', amopstrategy => '3',
+ amopopr => '=(timestamptz,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamptz', amopstrategy => '4',
+ amopopr => '>=(timestamptz,timestamptz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/datetime_minmax_multi_ops', amoplefttype => 'timestamptz',
+ amoprighttype => 'timestamptz', amopstrategy => '5',
+ amopopr => '>(timestamptz,timestamptz)', amopmethod => 'brin' },
+
# bloom datetime (date, timestamp, timestamptz)
{ amopfamily => 'brin/datetime_bloom_ops', amoplefttype => 'timestamp',
@@ -2470,6 +2929,23 @@
amoprighttype => 'interval', amopstrategy => '5',
amopopr => '>(interval,interval)', amopmethod => 'brin' },
+# minmax multi interval
+{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval',
+ amoprighttype => 'interval', amopstrategy => '1',
+ amopopr => '<(interval,interval)', amopmethod => 'brin' },
+{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval',
+ amoprighttype => 'interval', amopstrategy => '2',
+ amopopr => '<=(interval,interval)', amopmethod => 'brin' },
+{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval',
+ amoprighttype => 'interval', amopstrategy => '3',
+ amopopr => '=(interval,interval)', amopmethod => 'brin' },
+{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval',
+ amoprighttype => 'interval', amopstrategy => '4',
+ amopopr => '>=(interval,interval)', amopmethod => 'brin' },
+{ amopfamily => 'brin/interval_minmax_multi_ops', amoplefttype => 'interval',
+ amoprighttype => 'interval', amopstrategy => '5',
+ amopopr => '>(interval,interval)', amopmethod => 'brin' },
+
# bloom interval
{ amopfamily => 'brin/interval_bloom_ops', amoplefttype => 'interval',
amoprighttype => 'interval', amopstrategy => '1',
@@ -2492,6 +2968,23 @@
amoprighttype => 'timetz', amopstrategy => '5', amopopr => '>(timetz,timetz)',
amopmethod => 'brin' },
+# minmax multi time with time zone
+{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz',
+ amoprighttype => 'timetz', amopstrategy => '1', amopopr => '<(timetz,timetz)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz',
+ amoprighttype => 'timetz', amopstrategy => '2',
+ amopopr => '<=(timetz,timetz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz',
+ amoprighttype => 'timetz', amopstrategy => '3', amopopr => '=(timetz,timetz)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz',
+ amoprighttype => 'timetz', amopstrategy => '4',
+ amopopr => '>=(timetz,timetz)', amopmethod => 'brin' },
+{ amopfamily => 'brin/timetz_minmax_multi_ops', amoplefttype => 'timetz',
+ amoprighttype => 'timetz', amopstrategy => '5', amopopr => '>(timetz,timetz)',
+ amopmethod => 'brin' },
+
# bloom time with time zone
{ amopfamily => 'brin/timetz_bloom_ops', amoplefttype => 'timetz',
amoprighttype => 'timetz', amopstrategy => '1', amopopr => '=(timetz,timetz)',
@@ -2548,6 +3041,23 @@
amoprighttype => 'numeric', amopstrategy => '5',
amopopr => '>(numeric,numeric)', amopmethod => 'brin' },
+# minmax multi numeric
+{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric',
+ amoprighttype => 'numeric', amopstrategy => '1',
+ amopopr => '<(numeric,numeric)', amopmethod => 'brin' },
+{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric',
+ amoprighttype => 'numeric', amopstrategy => '2',
+ amopopr => '<=(numeric,numeric)', amopmethod => 'brin' },
+{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric',
+ amoprighttype => 'numeric', amopstrategy => '3',
+ amopopr => '=(numeric,numeric)', amopmethod => 'brin' },
+{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric',
+ amoprighttype => 'numeric', amopstrategy => '4',
+ amopopr => '>=(numeric,numeric)', amopmethod => 'brin' },
+{ amopfamily => 'brin/numeric_minmax_multi_ops', amoplefttype => 'numeric',
+ amoprighttype => 'numeric', amopstrategy => '5',
+ amopopr => '>(numeric,numeric)', amopmethod => 'brin' },
+
# bloom numeric
{ amopfamily => 'brin/numeric_bloom_ops', amoplefttype => 'numeric',
amoprighttype => 'numeric', amopstrategy => '1',
@@ -2570,6 +3080,23 @@
amoprighttype => 'uuid', amopstrategy => '5', amopopr => '>(uuid,uuid)',
amopmethod => 'brin' },
+# minmax multi uuid
+{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid',
+ amoprighttype => 'uuid', amopstrategy => '1', amopopr => '<(uuid,uuid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid',
+ amoprighttype => 'uuid', amopstrategy => '2', amopopr => '<=(uuid,uuid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid',
+ amoprighttype => 'uuid', amopstrategy => '3', amopopr => '=(uuid,uuid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid',
+ amoprighttype => 'uuid', amopstrategy => '4', amopopr => '>=(uuid,uuid)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/uuid_minmax_multi_ops', amoplefttype => 'uuid',
+ amoprighttype => 'uuid', amopstrategy => '5', amopopr => '>(uuid,uuid)',
+ amopmethod => 'brin' },
+
# bloom uuid
{ amopfamily => 'brin/uuid_bloom_ops', amoplefttype => 'uuid',
amoprighttype => 'uuid', amopstrategy => '1', amopopr => '=(uuid,uuid)',
@@ -2636,6 +3163,23 @@
amoprighttype => 'pg_lsn', amopstrategy => '5', amopopr => '>(pg_lsn,pg_lsn)',
amopmethod => 'brin' },
+# minmax multi pg_lsn
+{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn',
+ amoprighttype => 'pg_lsn', amopstrategy => '1', amopopr => '<(pg_lsn,pg_lsn)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn',
+ amoprighttype => 'pg_lsn', amopstrategy => '2',
+ amopopr => '<=(pg_lsn,pg_lsn)', amopmethod => 'brin' },
+{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn',
+ amoprighttype => 'pg_lsn', amopstrategy => '3', amopopr => '=(pg_lsn,pg_lsn)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn',
+ amoprighttype => 'pg_lsn', amopstrategy => '4',
+ amopopr => '>=(pg_lsn,pg_lsn)', amopmethod => 'brin' },
+{ amopfamily => 'brin/pg_lsn_minmax_multi_ops', amoplefttype => 'pg_lsn',
+ amoprighttype => 'pg_lsn', amopstrategy => '5', amopopr => '>(pg_lsn,pg_lsn)',
+ amopmethod => 'brin' },
+
# bloom pg_lsn
{ amopfamily => 'brin/pg_lsn_bloom_ops', amoplefttype => 'pg_lsn',
amoprighttype => 'pg_lsn', amopstrategy => '1', amopopr => '=(pg_lsn,pg_lsn)',
diff --git a/src/include/catalog/pg_amproc.dat b/src/include/catalog/pg_amproc.dat
index 6709c8dfea..51403716b1 100644
--- a/src/include/catalog/pg_amproc.dat
+++ b/src/include/catalog/pg_amproc.dat
@@ -986,6 +986,152 @@
{ amprocfamily => 'brin/integer_minmax_ops', amproclefttype => 'int4',
amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi integer: int2, int4, int8
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int8', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int8', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int8', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int8', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int2', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int2', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int2', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int2', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int4', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int4', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int4', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int4', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int8',
+ amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int2', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int2', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int2', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int2', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int2' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int8', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int8', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int8', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int8', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int4', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int4', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int4', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int4', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int2',
+ amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int4', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int4', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int4', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int4', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int4', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int4', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int8', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int8', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int8', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int8', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int8', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int8', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int8' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int2', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int2', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int2', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int2', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int2', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/integer_minmax_multi_ops', amproclefttype => 'int4',
+ amprocrighttype => 'int2', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' },
+
# bloom integer: int2, int4, int8
{ amprocfamily => 'brin/integer_bloom_ops', amproclefttype => 'int8',
amprocrighttype => 'int8', amprocnum => '1',
@@ -1081,6 +1227,23 @@
{ amprocfamily => 'brin/oid_minmax_ops', amproclefttype => 'oid',
amprocrighttype => 'oid', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi oid
+{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid',
+ amprocrighttype => 'oid', amprocnum => '1', amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid',
+ amprocrighttype => 'oid', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid',
+ amprocrighttype => 'oid', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid',
+ amprocrighttype => 'oid', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid',
+ amprocrighttype => 'oid', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/oid_minmax_multi_ops', amproclefttype => 'oid',
+ amprocrighttype => 'oid', amprocnum => '11', amproc => 'brin_minmax_multi_distance_int4' },
+
# bloom oid
{ amprocfamily => 'brin/oid_bloom_ops', amproclefttype => 'oid',
amprocrighttype => 'oid', amprocnum => '1', amproc => 'brin_bloom_opcinfo' },
@@ -1127,6 +1290,23 @@
{ amprocfamily => 'brin/tid_bloom_ops', amproclefttype => 'tid',
amprocrighttype => 'tid', amprocnum => '11', amproc => 'hashtid' },
+# minmax multi tid
+{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid',
+ amprocrighttype => 'tid', amprocnum => '1', amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid',
+ amprocrighttype => 'tid', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid',
+ amprocrighttype => 'tid', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid',
+ amprocrighttype => 'tid', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid',
+ amprocrighttype => 'tid', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/tid_minmax_multi_ops', amproclefttype => 'tid',
+ amprocrighttype => 'tid', amprocnum => '11', amproc => 'brin_minmax_multi_distance_tid' },
+
# minmax float
{ amprocfamily => 'brin/float_minmax_ops', amproclefttype => 'float4',
amprocrighttype => 'float4', amprocnum => '1',
@@ -1177,6 +1357,80 @@
amprocrighttype => 'float4', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi float
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float4', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float4', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float4', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float4', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float4', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float4', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_float4' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float8', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float8', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float8', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float8', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float8', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float4',
+ amprocrighttype => 'float8', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_float8' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float8', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float8', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float8', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float8', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float8', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float8', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_float8' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float4', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float4', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float4', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float4', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float4', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/float_minmax_multi_ops', amproclefttype => 'float8',
+ amprocrighttype => 'float4', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_float8' },
+
# bloom float
{ amprocfamily => 'brin/float_bloom_ops', amproclefttype => 'float4',
amprocrighttype => 'float4', amprocnum => '1',
@@ -1230,6 +1484,26 @@
amprocrighttype => 'macaddr', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi macaddr
+{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr',
+ amprocrighttype => 'macaddr', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr',
+ amprocrighttype => 'macaddr', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr',
+ amprocrighttype => 'macaddr', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr',
+ amprocrighttype => 'macaddr', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr',
+ amprocrighttype => 'macaddr', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/macaddr_minmax_multi_ops', amproclefttype => 'macaddr',
+ amprocrighttype => 'macaddr', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_macaddr' },
+
# bloom macaddr
{ amprocfamily => 'brin/macaddr_bloom_ops', amproclefttype => 'macaddr',
amprocrighttype => 'macaddr', amprocnum => '1',
@@ -1264,6 +1538,26 @@
amprocrighttype => 'macaddr8', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi macaddr8
+{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8',
+ amprocrighttype => 'macaddr8', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8',
+ amprocrighttype => 'macaddr8', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8',
+ amprocrighttype => 'macaddr8', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8',
+ amprocrighttype => 'macaddr8', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8',
+ amprocrighttype => 'macaddr8', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/macaddr8_minmax_multi_ops', amproclefttype => 'macaddr8',
+ amprocrighttype => 'macaddr8', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_macaddr8' },
+
# bloom macaddr8
{ amprocfamily => 'brin/macaddr8_bloom_ops', amproclefttype => 'macaddr8',
amprocrighttype => 'macaddr8', amprocnum => '1',
@@ -1297,6 +1591,26 @@
{ amprocfamily => 'brin/network_minmax_ops', amproclefttype => 'inet',
amprocrighttype => 'inet', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi inet
+{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet',
+ amprocrighttype => 'inet', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet',
+ amprocrighttype => 'inet', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet',
+ amprocrighttype => 'inet', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet',
+ amprocrighttype => 'inet', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet',
+ amprocrighttype => 'inet', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/network_minmax_multi_ops', amproclefttype => 'inet',
+ amprocrighttype => 'inet', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_inet' },
+
# bloom inet
{ amprocfamily => 'brin/network_bloom_ops', amproclefttype => 'inet',
amprocrighttype => 'inet', amprocnum => '1',
@@ -1382,6 +1696,25 @@
{ amprocfamily => 'brin/time_minmax_ops', amproclefttype => 'time',
amprocrighttype => 'time', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi time without time zone
+{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time',
+ amprocrighttype => 'time', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time',
+ amprocrighttype => 'time', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time',
+ amprocrighttype => 'time', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time',
+ amprocrighttype => 'time', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time',
+ amprocrighttype => 'time', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/time_minmax_multi_ops', amproclefttype => 'time',
+ amprocrighttype => 'time', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_time' },
+
# bloom time without time zone
{ amprocfamily => 'brin/time_bloom_ops', amproclefttype => 'time',
amprocrighttype => 'time', amprocnum => '1',
@@ -1507,6 +1840,170 @@
amprocrighttype => 'timestamptz', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi datetime (date, timestamp, timestamptz)
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamp', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamp', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamp', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamp', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamp', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamp', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_time' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamptz', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamptz', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamptz', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamptz', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamptz', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'timestamptz', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_time' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'date', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'date', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'date', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'date', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'date', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamp',
+ amprocrighttype => 'date', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_time' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamptz', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamptz', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamptz', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamptz', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamptz', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamptz', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_time' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamp', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamp', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamp', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamp', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamp', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'timestamp', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_timestamp' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'date', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'date', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'date', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'date', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'date', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'timestamptz',
+ amprocrighttype => 'date', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_timestamp' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'date', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'date', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'date', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'date', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'date', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'date', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_date' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamp', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamp', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamp', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamp', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamp', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamp', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_float8' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamptz', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamptz', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamptz', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamptz', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamptz', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/datetime_minmax_multi_ops', amproclefttype => 'date',
+ amprocrighttype => 'timestamptz', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_float8' },
+
# bloom datetime (date, timestamp, timestamptz)
{ amprocfamily => 'brin/datetime_bloom_ops', amproclefttype => 'timestamp',
amprocrighttype => 'timestamp', amprocnum => '1',
@@ -1577,6 +2074,26 @@
amprocrighttype => 'interval', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi interval
+{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval',
+ amprocrighttype => 'interval', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval',
+ amprocrighttype => 'interval', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval',
+ amprocrighttype => 'interval', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval',
+ amprocrighttype => 'interval', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval',
+ amprocrighttype => 'interval', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/interval_minmax_multi_ops', amproclefttype => 'interval',
+ amprocrighttype => 'interval', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_interval' },
+
# bloom interval
{ amprocfamily => 'brin/interval_bloom_ops', amproclefttype => 'interval',
amprocrighttype => 'interval', amprocnum => '1',
@@ -1611,6 +2128,26 @@
amprocrighttype => 'timetz', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi time with time zone
+{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz',
+ amprocrighttype => 'timetz', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz',
+ amprocrighttype => 'timetz', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz',
+ amprocrighttype => 'timetz', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz',
+ amprocrighttype => 'timetz', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz',
+ amprocrighttype => 'timetz', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/timetz_minmax_multi_ops', amproclefttype => 'timetz',
+ amprocrighttype => 'timetz', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_timetz' },
+
# bloom time with time zone
{ amprocfamily => 'brin/timetz_bloom_ops', amproclefttype => 'timetz',
amprocrighttype => 'timetz', amprocnum => '1',
@@ -1671,6 +2208,26 @@
amprocrighttype => 'numeric', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi numeric
+{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric',
+ amprocrighttype => 'numeric', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric',
+ amprocrighttype => 'numeric', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric',
+ amprocrighttype => 'numeric', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric',
+ amprocrighttype => 'numeric', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric',
+ amprocrighttype => 'numeric', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/numeric_minmax_multi_ops', amproclefttype => 'numeric',
+ amprocrighttype => 'numeric', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_numeric' },
+
# bloom numeric
{ amprocfamily => 'brin/numeric_bloom_ops', amproclefttype => 'numeric',
amprocrighttype => 'numeric', amprocnum => '1',
@@ -1702,7 +2259,28 @@
amprocrighttype => 'uuid', amprocnum => '3',
amproc => 'brin_minmax_consistent' },
{ amprocfamily => 'brin/uuid_minmax_ops', amproclefttype => 'uuid',
- amprocrighttype => 'uuid', amprocnum => '4', amproc => 'brin_minmax_union' },
+ amprocrighttype => 'uuid', amprocnum => '4',
+ amproc => 'brin_minmax_union' },
+
+# minmax multi uuid
+{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid',
+ amprocrighttype => 'uuid', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid',
+ amprocrighttype => 'uuid', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid',
+ amprocrighttype => 'uuid', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid',
+ amprocrighttype => 'uuid', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid',
+ amprocrighttype => 'uuid', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/uuid_minmax_multi_ops', amproclefttype => 'uuid',
+ amprocrighttype => 'uuid', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_uuid' },
# bloom uuid
{ amprocfamily => 'brin/uuid_bloom_ops', amproclefttype => 'uuid',
@@ -1759,6 +2337,26 @@
amprocrighttype => 'pg_lsn', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi pg_lsn
+{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn',
+ amprocrighttype => 'pg_lsn', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn',
+ amprocrighttype => 'pg_lsn', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn',
+ amprocrighttype => 'pg_lsn', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn',
+ amprocrighttype => 'pg_lsn', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn',
+ amprocrighttype => 'pg_lsn', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+{ amprocfamily => 'brin/pg_lsn_minmax_multi_ops', amproclefttype => 'pg_lsn',
+ amprocrighttype => 'pg_lsn', amprocnum => '11',
+ amproc => 'brin_minmax_multi_distance_pg_lsn' },
+
# bloom pg_lsn
{ amprocfamily => 'brin/pg_lsn_bloom_ops', amproclefttype => 'pg_lsn',
amprocrighttype => 'pg_lsn', amprocnum => '1',
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index 6a5bb58baf..da25befefe 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -284,18 +284,27 @@
{ opcmethod => 'brin', opcname => 'int8_minmax_ops',
opcfamily => 'brin/integer_minmax_ops', opcintype => 'int8',
opckeytype => 'int8' },
+{ opcmethod => 'brin', opcname => 'int8_minmax_multi_ops',
+ opcfamily => 'brin/integer_minmax_multi_ops', opcintype => 'int8',
+ opckeytype => 'int8', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'int8_bloom_ops',
opcfamily => 'brin/integer_bloom_ops', opcintype => 'int8',
opckeytype => 'int8', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'int2_minmax_ops',
opcfamily => 'brin/integer_minmax_ops', opcintype => 'int2',
opckeytype => 'int2' },
+{ opcmethod => 'brin', opcname => 'int2_minmax_multi_ops',
+ opcfamily => 'brin/integer_minmax_multi_ops', opcintype => 'int2',
+ opckeytype => 'int2', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'int2_bloom_ops',
opcfamily => 'brin/integer_bloom_ops', opcintype => 'int2',
opckeytype => 'int2', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'int4_minmax_ops',
opcfamily => 'brin/integer_minmax_ops', opcintype => 'int4',
opckeytype => 'int4' },
+{ opcmethod => 'brin', opcname => 'int4_minmax_multi_ops',
+ opcfamily => 'brin/integer_minmax_multi_ops', opcintype => 'int4',
+ opckeytype => 'int4', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'int4_bloom_ops',
opcfamily => 'brin/integer_bloom_ops', opcintype => 'int4',
opckeytype => 'int4', opcdefault => 'f' },
@@ -307,6 +316,9 @@
opckeytype => 'text', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'oid_minmax_ops',
opcfamily => 'brin/oid_minmax_ops', opcintype => 'oid', opckeytype => 'oid' },
+{ opcmethod => 'brin', opcname => 'oid_minmax_multi_ops',
+ opcfamily => 'brin/oid_minmax_multi_ops', opcintype => 'oid',
+ opckeytype => 'oid', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'oid_bloom_ops',
opcfamily => 'brin/oid_bloom_ops', opcintype => 'oid',
opckeytype => 'oid', opcdefault => 'f' },
@@ -315,33 +327,51 @@
{ opcmethod => 'brin', opcname => 'tid_bloom_ops',
opcfamily => 'brin/tid_bloom_ops', opcintype => 'tid', opckeytype => 'tid',
opcdefault => 'f'},
+{ opcmethod => 'brin', opcname => 'tid_minmax_multi_ops',
+ opcfamily => 'brin/tid_minmax_multi_ops', opcintype => 'tid',
+ opckeytype => 'tid', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'float4_minmax_ops',
opcfamily => 'brin/float_minmax_ops', opcintype => 'float4',
opckeytype => 'float4' },
+{ opcmethod => 'brin', opcname => 'float4_minmax_multi_ops',
+ opcfamily => 'brin/float_minmax_multi_ops', opcintype => 'float4',
+ opckeytype => 'float4', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'float4_bloom_ops',
opcfamily => 'brin/float_bloom_ops', opcintype => 'float4',
opckeytype => 'float4', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'float8_minmax_ops',
opcfamily => 'brin/float_minmax_ops', opcintype => 'float8',
opckeytype => 'float8' },
+{ opcmethod => 'brin', opcname => 'float8_minmax_multi_ops',
+ opcfamily => 'brin/float_minmax_multi_ops', opcintype => 'float8',
+ opckeytype => 'float8', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'float8_bloom_ops',
opcfamily => 'brin/float_bloom_ops', opcintype => 'float8',
opckeytype => 'float8', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'macaddr_minmax_ops',
opcfamily => 'brin/macaddr_minmax_ops', opcintype => 'macaddr',
opckeytype => 'macaddr' },
+{ opcmethod => 'brin', opcname => 'macaddr_minmax_multi_ops',
+ opcfamily => 'brin/macaddr_minmax_multi_ops', opcintype => 'macaddr',
+ opckeytype => 'macaddr', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'macaddr_bloom_ops',
opcfamily => 'brin/macaddr_bloom_ops', opcintype => 'macaddr',
opckeytype => 'macaddr', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'macaddr8_minmax_ops',
opcfamily => 'brin/macaddr8_minmax_ops', opcintype => 'macaddr8',
opckeytype => 'macaddr8' },
+{ opcmethod => 'brin', opcname => 'macaddr8_minmax_multi_ops',
+ opcfamily => 'brin/macaddr8_minmax_multi_ops', opcintype => 'macaddr8',
+ opckeytype => 'macaddr8', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'macaddr8_bloom_ops',
opcfamily => 'brin/macaddr8_bloom_ops', opcintype => 'macaddr8',
opckeytype => 'macaddr8', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'inet_minmax_ops',
opcfamily => 'brin/network_minmax_ops', opcintype => 'inet',
opcdefault => 'f', opckeytype => 'inet' },
+{ opcmethod => 'brin', opcname => 'inet_minmax_multi_ops',
+ opcfamily => 'brin/network_minmax_multi_ops', opcintype => 'inet',
+ opcdefault => 'f', opckeytype => 'inet', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'inet_bloom_ops',
opcfamily => 'brin/network_bloom_ops', opcintype => 'inet',
opcdefault => 'f', opckeytype => 'inet', opcdefault => 'f' },
@@ -357,36 +387,54 @@
{ opcmethod => 'brin', opcname => 'time_minmax_ops',
opcfamily => 'brin/time_minmax_ops', opcintype => 'time',
opckeytype => 'time' },
+{ opcmethod => 'brin', opcname => 'time_minmax_multi_ops',
+ opcfamily => 'brin/time_minmax_multi_ops', opcintype => 'time',
+ opckeytype => 'time', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'time_bloom_ops',
opcfamily => 'brin/time_bloom_ops', opcintype => 'time',
opckeytype => 'time', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'date_minmax_ops',
opcfamily => 'brin/datetime_minmax_ops', opcintype => 'date',
opckeytype => 'date' },
+{ opcmethod => 'brin', opcname => 'date_minmax_multi_ops',
+ opcfamily => 'brin/datetime_minmax_multi_ops', opcintype => 'date',
+ opckeytype => 'date', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'date_bloom_ops',
opcfamily => 'brin/datetime_bloom_ops', opcintype => 'date',
opckeytype => 'date', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'timestamp_minmax_ops',
opcfamily => 'brin/datetime_minmax_ops', opcintype => 'timestamp',
opckeytype => 'timestamp' },
+{ opcmethod => 'brin', opcname => 'timestamp_minmax_multi_ops',
+ opcfamily => 'brin/datetime_minmax_multi_ops', opcintype => 'timestamp',
+ opckeytype => 'timestamp', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'timestamp_bloom_ops',
opcfamily => 'brin/datetime_bloom_ops', opcintype => 'timestamp',
opckeytype => 'timestamp', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'timestamptz_minmax_ops',
opcfamily => 'brin/datetime_minmax_ops', opcintype => 'timestamptz',
opckeytype => 'timestamptz' },
+{ opcmethod => 'brin', opcname => 'timestamptz_minmax_multi_ops',
+ opcfamily => 'brin/datetime_minmax_multi_ops', opcintype => 'timestamptz',
+ opckeytype => 'timestamptz', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'timestamptz_bloom_ops',
opcfamily => 'brin/datetime_bloom_ops', opcintype => 'timestamptz',
opckeytype => 'timestamptz', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'interval_minmax_ops',
opcfamily => 'brin/interval_minmax_ops', opcintype => 'interval',
opckeytype => 'interval' },
+{ opcmethod => 'brin', opcname => 'interval_minmax_multi_ops',
+ opcfamily => 'brin/interval_minmax_multi_ops', opcintype => 'interval',
+ opckeytype => 'interval', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'interval_bloom_ops',
opcfamily => 'brin/interval_bloom_ops', opcintype => 'interval',
opckeytype => 'interval', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'timetz_minmax_ops',
opcfamily => 'brin/timetz_minmax_ops', opcintype => 'timetz',
opckeytype => 'timetz' },
+{ opcmethod => 'brin', opcname => 'timetz_minmax_multi_ops',
+ opcfamily => 'brin/timetz_minmax_multi_ops', opcintype => 'timetz',
+ opckeytype => 'timetz', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'timetz_bloom_ops',
opcfamily => 'brin/timetz_bloom_ops', opcintype => 'timetz',
opckeytype => 'timetz', opcdefault => 'f' },
@@ -398,6 +446,9 @@
{ opcmethod => 'brin', opcname => 'numeric_minmax_ops',
opcfamily => 'brin/numeric_minmax_ops', opcintype => 'numeric',
opckeytype => 'numeric' },
+{ opcmethod => 'brin', opcname => 'numeric_minmax_multi_ops',
+ opcfamily => 'brin/numeric_minmax_multi_ops', opcintype => 'numeric',
+ opckeytype => 'numeric', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'numeric_bloom_ops',
opcfamily => 'brin/numeric_bloom_ops', opcintype => 'numeric',
opckeytype => 'numeric', opcdefault => 'f' },
@@ -407,6 +458,9 @@
{ opcmethod => 'brin', opcname => 'uuid_minmax_ops',
opcfamily => 'brin/uuid_minmax_ops', opcintype => 'uuid',
opckeytype => 'uuid' },
+{ opcmethod => 'brin', opcname => 'uuid_minmax_multi_ops',
+ opcfamily => 'brin/uuid_minmax_multi_ops', opcintype => 'uuid',
+ opckeytype => 'uuid', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'uuid_bloom_ops',
opcfamily => 'brin/uuid_bloom_ops', opcintype => 'uuid',
opckeytype => 'uuid', opcdefault => 'f' },
@@ -416,6 +470,9 @@
{ opcmethod => 'brin', opcname => 'pg_lsn_minmax_ops',
opcfamily => 'brin/pg_lsn_minmax_ops', opcintype => 'pg_lsn',
opckeytype => 'pg_lsn' },
+{ opcmethod => 'brin', opcname => 'pg_lsn_minmax_multi_ops',
+ opcfamily => 'brin/pg_lsn_minmax_multi_ops', opcintype => 'pg_lsn',
+ opckeytype => 'pg_lsn', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'pg_lsn_bloom_ops',
opcfamily => 'brin/pg_lsn_bloom_ops', opcintype => 'pg_lsn',
opckeytype => 'pg_lsn', opcdefault => 'f' },
diff --git a/src/include/catalog/pg_opfamily.dat b/src/include/catalog/pg_opfamily.dat
index dea9adaf98..ba9231ac8c 100644
--- a/src/include/catalog/pg_opfamily.dat
+++ b/src/include/catalog/pg_opfamily.dat
@@ -182,10 +182,14 @@
opfmethod => 'gin', opfname => 'jsonb_path_ops' },
{ oid => '4054',
opfmethod => 'brin', opfname => 'integer_minmax_ops' },
+{ oid => '9926',
+ opfmethod => 'brin', opfname => 'integer_minmax_multi_ops' },
{ oid => '9901',
opfmethod => 'brin', opfname => 'integer_bloom_ops' },
{ oid => '4055',
opfmethod => 'brin', opfname => 'numeric_minmax_ops' },
+{ oid => '9927',
+ opfmethod => 'brin', opfname => 'numeric_minmax_multi_ops' },
{ oid => '4056',
opfmethod => 'brin', opfname => 'text_minmax_ops' },
{ oid => '9902',
@@ -194,10 +198,14 @@
opfmethod => 'brin', opfname => 'numeric_bloom_ops' },
{ oid => '4058',
opfmethod => 'brin', opfname => 'timetz_minmax_ops' },
+{ oid => '9928',
+ opfmethod => 'brin', opfname => 'timetz_minmax_multi_ops' },
{ oid => '9904',
opfmethod => 'brin', opfname => 'timetz_bloom_ops' },
{ oid => '4059',
opfmethod => 'brin', opfname => 'datetime_minmax_ops' },
+{ oid => '9929',
+ opfmethod => 'brin', opfname => 'datetime_minmax_multi_ops' },
{ oid => '9905',
opfmethod => 'brin', opfname => 'datetime_bloom_ops' },
{ oid => '4062',
@@ -214,26 +222,38 @@
opfmethod => 'brin', opfname => 'name_bloom_ops' },
{ oid => '4068',
opfmethod => 'brin', opfname => 'oid_minmax_ops' },
+{ oid => '9930',
+ opfmethod => 'brin', opfname => 'oid_minmax_multi_ops' },
{ oid => '9909',
opfmethod => 'brin', opfname => 'oid_bloom_ops' },
{ oid => '4069',
opfmethod => 'brin', opfname => 'tid_minmax_ops' },
{ oid => '9910',
opfmethod => 'brin', opfname => 'tid_bloom_ops' },
+{ oid => '9931',
+ opfmethod => 'brin', opfname => 'tid_minmax_multi_ops' },
{ oid => '4070',
opfmethod => 'brin', opfname => 'float_minmax_ops' },
+{ oid => '9932',
+ opfmethod => 'brin', opfname => 'float_minmax_multi_ops' },
{ oid => '9911',
opfmethod => 'brin', opfname => 'float_bloom_ops' },
{ oid => '4074',
opfmethod => 'brin', opfname => 'macaddr_minmax_ops' },
+{ oid => '9933',
+ opfmethod => 'brin', opfname => 'macaddr_minmax_multi_ops' },
{ oid => '9912',
opfmethod => 'brin', opfname => 'macaddr_bloom_ops' },
{ oid => '4109',
opfmethod => 'brin', opfname => 'macaddr8_minmax_ops' },
+{ oid => '9934',
+ opfmethod => 'brin', opfname => 'macaddr8_minmax_multi_ops' },
{ oid => '9913',
opfmethod => 'brin', opfname => 'macaddr8_bloom_ops' },
{ oid => '4075',
opfmethod => 'brin', opfname => 'network_minmax_ops' },
+{ oid => '9935',
+ opfmethod => 'brin', opfname => 'network_minmax_multi_ops' },
{ oid => '4102',
opfmethod => 'brin', opfname => 'network_inclusion_ops' },
{ oid => '9914',
@@ -244,10 +264,14 @@
opfmethod => 'brin', opfname => 'bpchar_bloom_ops' },
{ oid => '4077',
opfmethod => 'brin', opfname => 'time_minmax_ops' },
+{ oid => '9936',
+ opfmethod => 'brin', opfname => 'time_minmax_multi_ops' },
{ oid => '9916',
opfmethod => 'brin', opfname => 'time_bloom_ops' },
{ oid => '4078',
opfmethod => 'brin', opfname => 'interval_minmax_ops' },
+{ oid => '9937',
+ opfmethod => 'brin', opfname => 'interval_minmax_multi_ops' },
{ oid => '9917',
opfmethod => 'brin', opfname => 'interval_bloom_ops' },
{ oid => '4079',
@@ -256,12 +280,16 @@
opfmethod => 'brin', opfname => 'varbit_minmax_ops' },
{ oid => '4081',
opfmethod => 'brin', opfname => 'uuid_minmax_ops' },
+{ oid => '9938',
+ opfmethod => 'brin', opfname => 'uuid_minmax_multi_ops' },
{ oid => '9918',
opfmethod => 'brin', opfname => 'uuid_bloom_ops' },
{ oid => '4103',
opfmethod => 'brin', opfname => 'range_inclusion_ops' },
{ oid => '4082',
opfmethod => 'brin', opfname => 'pg_lsn_minmax_ops' },
+{ oid => '9939',
+ opfmethod => 'brin', opfname => 'pg_lsn_minmax_multi_ops' },
{ oid => '9919',
opfmethod => 'brin', opfname => 'pg_lsn_bloom_ops' },
{ oid => '4104',
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3b92bb66e5..728763b97e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8197,6 +8197,77 @@
proname => 'brin_minmax_union', prorettype => 'bool',
proargtypes => 'internal internal internal', prosrc => 'brin_minmax_union' },
+# BRIN minmax multi
+{ oid => '9940', descr => 'BRIN multi minmax support',
+ proname => 'brin_minmax_multi_opcinfo', prorettype => 'internal',
+ proargtypes => 'internal', prosrc => 'brin_minmax_multi_opcinfo' },
+{ oid => '9941', descr => 'BRIN multi minmax support',
+ proname => 'brin_minmax_multi_add_value', prorettype => 'bool',
+ proargtypes => 'internal internal internal internal',
+ prosrc => 'brin_minmax_multi_add_value' },
+{ oid => '9942', descr => 'BRIN multi minmax support',
+ proname => 'brin_minmax_multi_consistent', prorettype => 'bool',
+ proargtypes => 'internal internal internal int4',
+ prosrc => 'brin_minmax_multi_consistent' },
+{ oid => '9943', descr => 'BRIN multi minmax support',
+ proname => 'brin_minmax_multi_union', prorettype => 'bool',
+ proargtypes => 'internal internal internal', prosrc => 'brin_minmax_multi_union' },
+{ oid => '9944', descr => 'BRIN multi minmax support',
+ proname => 'brin_minmax_multi_options', prorettype => 'void', proisstrict => 'f',
+ proargtypes => 'internal', prosrc => 'brin_minmax_multi_options' },
+
+{ oid => '9945', descr => 'BRIN multi minmax int2 distance',
+ proname => 'brin_minmax_multi_distance_int2', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_int2' },
+{ oid => '9946', descr => 'BRIN multi minmax int4 distance',
+ proname => 'brin_minmax_multi_distance_int4', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_int4' },
+{ oid => '9947', descr => 'BRIN multi minmax int8 distance',
+ proname => 'brin_minmax_multi_distance_int8', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_int8' },
+{ oid => '9948', descr => 'BRIN multi minmax float4 distance',
+ proname => 'brin_minmax_multi_distance_float4', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_float4' },
+{ oid => '9949', descr => 'BRIN multi minmax float8 distance',
+ proname => 'brin_minmax_multi_distance_float8', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_float8' },
+{ oid => '9950', descr => 'BRIN multi minmax numeric distance',
+ proname => 'brin_minmax_multi_distance_numeric', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_numeric' },
+{ oid => '9951', descr => 'BRIN multi minmax tid distance',
+ proname => 'brin_minmax_multi_distance_tid', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_tid' },
+{ oid => '9952', descr => 'BRIN multi minmax uuid distance',
+ proname => 'brin_minmax_multi_distance_uuid', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_uuid' },
+{ oid => '9953', descr => 'BRIN multi minmax date distance',
+ proname => 'brin_minmax_multi_distance_date', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_date' },
+{ oid => '9954', descr => 'BRIN multi minmax time distance',
+ proname => 'brin_minmax_multi_distance_time', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_time' },
+{ oid => '9955', descr => 'BRIN multi minmax interval distance',
+ proname => 'brin_minmax_multi_distance_interval', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_interval' },
+{ oid => '9956', descr => 'BRIN multi minmax timetz distance',
+ proname => 'brin_minmax_multi_distance_timetz', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_timetz' },
+{ oid => '9957', descr => 'BRIN multi minmax pg_lsn distance',
+ proname => 'brin_minmax_multi_distance_pg_lsn', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_pg_lsn' },
+{ oid => '9958', descr => 'BRIN multi minmax macaddr distance',
+ proname => 'brin_minmax_multi_distance_macaddr', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_macaddr' },
+{ oid => '9959', descr => 'BRIN multi minmax macaddr8 distance',
+ proname => 'brin_minmax_multi_distance_macaddr8', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_macaddr8' },
+{ oid => '9960', descr => 'BRIN multi minmax inet distance',
+ proname => 'brin_minmax_multi_distance_inet', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_inet' },
+{ oid => '9961', descr => 'BRIN multi minmax timestamp distance',
+ proname => 'brin_minmax_multi_distance_timestamp', prorettype => 'float8',
+ proargtypes => 'internal internal', prosrc => 'brin_minmax_multi_distance_timestamp' },
+
# BRIN inclusion
{ oid => '4105', descr => 'BRIN inclusion support',
proname => 'brin_inclusion_opcinfo', prorettype => 'internal',
@@ -11421,4 +11492,18 @@
proname => 'brin_bloom_summary_send', provolatile => 's', prorettype => 'bytea',
proargtypes => 'pg_brin_bloom_summary', prosrc => 'brin_bloom_summary_send' },
+{ oid => '9962', descr => 'I/O',
+ proname => 'brin_minmax_multi_summary_in', prorettype => 'pg_brin_minmax_multi_summary',
+ proargtypes => 'cstring', prosrc => 'brin_minmax_multi_summary_in' },
+{ oid => '9963', descr => 'I/O',
+ proname => 'brin_minmax_multi_summary_out', prorettype => 'cstring',
+ proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_out' },
+{ oid => '9964', descr => 'I/O',
+ proname => 'brin_minmax_multi_summary_recv', provolatile => 's',
+ prorettype => 'pg_brin_minmax_multi_summary', proargtypes => 'internal',
+ prosrc => 'brin_minmax_multi_summary_recv' },
+{ oid => '9965', descr => 'I/O',
+ proname => 'brin_minmax_multi_summary_send', provolatile => 's', prorettype => 'bytea',
+ proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_send' },
+
]
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 74e279cbf9..e809094490 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -685,4 +685,10 @@
typinput => 'brin_bloom_summary_in', typoutput => 'brin_bloom_summary_out',
typreceive => 'brin_bloom_summary_recv', typsend => 'brin_bloom_summary_send',
typalign => 'i', typstorage => 'x', typcollation => 'default' },
+{ oid => '9966',
+ descr => 'BRIN minmax-multi summary',
+ typname => 'pg_brin_minmax_multi_summary', typlen => '-1', typbyval => 'f', typcategory => 'S',
+ typinput => 'brin_minmax_multi_summary_in', typoutput => 'brin_minmax_multi_summary_out',
+ typreceive => 'brin_minmax_multi_summary_recv', typsend => 'brin_minmax_multi_summary_send',
+ typalign => 'i', typstorage => 'x', typcollation => 'default' },
]
diff --git a/src/test/regress/expected/brin_multi.out b/src/test/regress/expected/brin_multi.out
new file mode 100644
index 0000000000..e13cb59c7e
--- /dev/null
+++ b/src/test/regress/expected/brin_multi.out
@@ -0,0 +1,445 @@
+CREATE TABLE brintest_multi (
+ int8col bigint,
+ int2col smallint,
+ int4col integer,
+ oidcol oid,
+ tidcol tid,
+ float4col real,
+ float8col double precision,
+ macaddrcol macaddr,
+ inetcol inet,
+ cidrcol cidr,
+ datecol date,
+ timecol time without time zone,
+ timestampcol timestamp without time zone,
+ timestamptzcol timestamp with time zone,
+ intervalcol interval,
+ timetzcol time with time zone,
+ numericcol numeric,
+ uuidcol uuid,
+ lsncol pg_lsn
+) WITH (fillfactor=10);
+INSERT INTO brintest_multi SELECT
+ 142857 * tenthous,
+ thousand,
+ twothousand,
+ unique1::oid,
+ format('(%s,%s)', tenthous, twenty)::tid,
+ (four + 1.0)/(hundred+1),
+ odd::float8 / (tenthous + 1),
+ format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr,
+ inet '10.2.3.4/24' + tenthous,
+ cidr '10.2.3/24' + tenthous,
+ date '1995-08-15' + tenthous,
+ time '01:20:30' + thousand * interval '18.5 second',
+ timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours',
+ timestamptz '1972-10-10 03:00' + thousand * interval '1 hour',
+ justify_days(justify_hours(tenthous * interval '12 minutes')),
+ timetz '01:30:20+02' + hundred * interval '15 seconds',
+ tenthous::numeric(36,30) * fivethous * even / (hundred + 1),
+ format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid,
+ format('%s/%s%s', odd, even, tenthous)::pg_lsn
+FROM tenk1 ORDER BY unique2 LIMIT 100;
+-- throw in some NULL's and different values
+INSERT INTO brintest_multi (inetcol, cidrcol) SELECT
+ inet 'fe80::6e40:8ff:fea9:8c46' + tenthous,
+ cidr 'fe80::6e40:8ff:fea9:8c46' + tenthous
+FROM tenk1 ORDER BY thousand, tenthous LIMIT 25;
+-- test minmax-multi specific index options
+-- number of values must be >= 16
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops(values_per_range = 7)
+);
+ERROR: value 7 out of bounds for option "values_per_range"
+DETAIL: Valid values are between "8" and "256".
+-- number of values must be <= 256
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops(values_per_range = 257)
+);
+ERROR: value 257 out of bounds for option "values_per_range"
+DETAIL: Valid values are between "8" and "256".
+-- first create an index with a single page range, to force compaction
+-- due to exceeding the number of values per summary
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops,
+ int2col int2_minmax_multi_ops,
+ int4col int4_minmax_multi_ops,
+ oidcol oid_minmax_multi_ops,
+ tidcol tid_minmax_multi_ops,
+ float4col float4_minmax_multi_ops,
+ float8col float8_minmax_multi_ops,
+ macaddrcol macaddr_minmax_multi_ops,
+ inetcol inet_minmax_multi_ops,
+ cidrcol inet_minmax_multi_ops,
+ datecol date_minmax_multi_ops,
+ timecol time_minmax_multi_ops,
+ timestampcol timestamp_minmax_multi_ops,
+ timestamptzcol timestamptz_minmax_multi_ops,
+ intervalcol interval_minmax_multi_ops,
+ timetzcol timetz_minmax_multi_ops,
+ numericcol numeric_minmax_multi_ops,
+ uuidcol uuid_minmax_multi_ops,
+ lsncol pg_lsn_minmax_multi_ops
+);
+DROP INDEX brinidx_multi;
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops,
+ int2col int2_minmax_multi_ops,
+ int4col int4_minmax_multi_ops,
+ oidcol oid_minmax_multi_ops,
+ tidcol tid_minmax_multi_ops,
+ float4col float4_minmax_multi_ops,
+ float8col float8_minmax_multi_ops,
+ macaddrcol macaddr_minmax_multi_ops,
+ inetcol inet_minmax_multi_ops,
+ cidrcol inet_minmax_multi_ops,
+ datecol date_minmax_multi_ops,
+ timecol time_minmax_multi_ops,
+ timestampcol timestamp_minmax_multi_ops,
+ timestamptzcol timestamptz_minmax_multi_ops,
+ intervalcol interval_minmax_multi_ops,
+ timetzcol timetz_minmax_multi_ops,
+ numericcol numeric_minmax_multi_ops,
+ uuidcol uuid_minmax_multi_ops,
+ lsncol pg_lsn_minmax_multi_ops
+) with (pages_per_range = 1);
+CREATE TABLE brinopers_multi (colname name, typ text,
+ op text[], value text[], matches int[],
+ check (cardinality(op) = cardinality(value)),
+ check (cardinality(op) = cardinality(matches)));
+INSERT INTO brinopers_multi VALUES
+ ('int2col', 'int2',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 999, 999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int2col', 'int4',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 999, 1999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int2col', 'int8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 999, 1428427143}',
+ '{100, 100, 1, 100, 100}'),
+ ('int4col', 'int2',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 1999, 1999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int4col', 'int4',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 1999, 1999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int4col', 'int8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 1999, 1428427143}',
+ '{100, 100, 1, 100, 100}'),
+ ('int8col', 'int2',
+ '{>, >=}',
+ '{0, 0}',
+ '{100, 100}'),
+ ('int8col', 'int4',
+ '{>, >=}',
+ '{0, 0}',
+ '{100, 100}'),
+ ('int8col', 'int8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 1257141600, 1428427143, 1428427143}',
+ '{100, 100, 1, 100, 100}'),
+ ('oidcol', 'oid',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 8800, 9999, 9999}',
+ '{100, 100, 1, 100, 100}'),
+ ('tidcol', 'tid',
+ '{>, >=, =, <=, <}',
+ '{"(0,0)", "(0,0)", "(8800,0)", "(9999,19)", "(9999,19)"}',
+ '{100, 100, 1, 100, 100}'),
+ ('float4col', 'float4',
+ '{>, >=, =, <=, <}',
+ '{0.0103093, 0.0103093, 1, 1, 1}',
+ '{100, 100, 4, 100, 96}'),
+ ('float4col', 'float8',
+ '{>, >=, =, <=, <}',
+ '{0.0103093, 0.0103093, 1, 1, 1}',
+ '{100, 100, 4, 100, 96}'),
+ ('float8col', 'float4',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 0, 1.98, 1.98}',
+ '{99, 100, 1, 100, 100}'),
+ ('float8col', 'float8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 0, 1.98, 1.98}',
+ '{99, 100, 1, 100, 100}'),
+ ('macaddrcol', 'macaddr',
+ '{>, >=, =, <=, <}',
+ '{00:00:01:00:00:00, 00:00:01:00:00:00, 2c:00:2d:00:16:00, ff:fe:00:00:00:00, ff:fe:00:00:00:00}',
+ '{99, 100, 2, 100, 100}'),
+ ('inetcol', 'inet',
+ '{=, <, <=, >, >=}',
+ '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{1, 100, 100, 125, 125}'),
+ ('inetcol', 'cidr',
+ '{<, <=, >, >=}',
+ '{255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{100, 100, 125, 125}'),
+ ('cidrcol', 'inet',
+ '{=, <, <=, >, >=}',
+ '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{2, 100, 100, 125, 125}'),
+ ('cidrcol', 'cidr',
+ '{=, <, <=, >, >=}',
+ '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{2, 100, 100, 125, 125}'),
+ ('datecol', 'date',
+ '{>, >=, =, <=, <}',
+ '{1995-08-15, 1995-08-15, 2009-12-01, 2022-12-30, 2022-12-30}',
+ '{100, 100, 1, 100, 100}'),
+ ('timecol', 'time',
+ '{>, >=, =, <=, <}',
+ '{01:20:30, 01:20:30, 02:28:57, 06:28:31.5, 06:28:31.5}',
+ '{100, 100, 1, 100, 100}'),
+ ('timestampcol', 'timestamp',
+ '{>, >=, =, <=, <}',
+ '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}',
+ '{100, 100, 1, 100, 100}'),
+ ('timestampcol', 'timestamptz',
+ '{>, >=, =, <=, <}',
+ '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}',
+ '{100, 100, 1, 100, 100}'),
+ ('timestamptzcol', 'timestamptz',
+ '{>, >=, =, <=, <}',
+ '{1972-10-10 03:00:00-04, 1972-10-10 03:00:00-04, 1972-10-19 09:00:00-07, 1972-11-20 19:00:00-03, 1972-11-20 19:00:00-03}',
+ '{100, 100, 1, 100, 100}'),
+ ('intervalcol', 'interval',
+ '{>, >=, =, <=, <}',
+ '{00:00:00, 00:00:00, 1 mons 13 days 12:24, 2 mons 23 days 07:48:00, 1 year}',
+ '{100, 100, 1, 100, 100}'),
+ ('timetzcol', 'timetz',
+ '{>, >=, =, <=, <}',
+ '{01:30:20+02, 01:30:20+02, 01:35:50+02, 23:55:05+02, 23:55:05+02}',
+ '{99, 100, 2, 100, 100}'),
+ ('numericcol', 'numeric',
+ '{>, >=, =, <=, <}',
+ '{0.00, 0.01, 2268164.347826086956521739130434782609, 99470151.9, 99470151.9}',
+ '{100, 100, 1, 100, 100}'),
+ ('uuidcol', 'uuid',
+ '{>, >=, =, <=, <}',
+ '{00040004-0004-0004-0004-000400040004, 00040004-0004-0004-0004-000400040004, 52225222-5222-5222-5222-522252225222, 99989998-9998-9998-9998-999899989998, 99989998-9998-9998-9998-999899989998}',
+ '{100, 100, 1, 100, 100}'),
+ ('lsncol', 'pg_lsn',
+ '{>, >=, =, <=, <, IS, IS NOT}',
+ '{0/1200, 0/1200, 44/455222, 198/1999799, 198/1999799, NULL, NULL}',
+ '{100, 100, 1, 100, 100, 25, 100}');
+DO $x$
+DECLARE
+ r record;
+ r2 record;
+ cond text;
+ idx_ctids tid[];
+ ss_ctids tid[];
+ count int;
+ plan_ok bool;
+ plan_line text;
+BEGIN
+ FOR r IN SELECT colname, oper, typ, value[ordinality], matches[ordinality] FROM brinopers_multi, unnest(op) WITH ORDINALITY AS oper LOOP
+
+ -- prepare the condition
+ IF r.value IS NULL THEN
+ cond := format('%I %s %L', r.colname, r.oper, r.value);
+ ELSE
+ cond := format('%I %s %L::%s', r.colname, r.oper, r.value, r.typ);
+ END IF;
+
+ -- run the query using the brin index
+ SET enable_seqscan = 0;
+ SET enable_bitmapscan = 1;
+
+ plan_ok := false;
+ FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP
+ IF plan_line LIKE '%Bitmap Heap Scan on brintest_multi%' THEN
+ plan_ok := true;
+ END IF;
+ END LOOP;
+ IF NOT plan_ok THEN
+ RAISE WARNING 'did not get bitmap indexscan plan for %', r;
+ END IF;
+
+ EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond)
+ INTO idx_ctids;
+
+ -- run the query using a seqscan
+ SET enable_seqscan = 1;
+ SET enable_bitmapscan = 0;
+
+ plan_ok := false;
+ FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP
+ IF plan_line LIKE '%Seq Scan on brintest_multi%' THEN
+ plan_ok := true;
+ END IF;
+ END LOOP;
+ IF NOT plan_ok THEN
+ RAISE WARNING 'did not get seqscan plan for %', r;
+ END IF;
+
+ EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond)
+ INTO ss_ctids;
+
+ -- make sure both return the same results
+ count := array_length(idx_ctids, 1);
+
+ IF NOT (count = array_length(ss_ctids, 1) AND
+ idx_ctids @> ss_ctids AND
+ idx_ctids <@ ss_ctids) THEN
+ -- report the results of each scan to make the differences obvious
+ RAISE WARNING 'something not right in %: count %', r, count;
+ SET enable_seqscan = 1;
+ SET enable_bitmapscan = 0;
+ FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP
+ RAISE NOTICE 'seqscan: %', r2;
+ END LOOP;
+
+ SET enable_seqscan = 0;
+ SET enable_bitmapscan = 1;
+ FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP
+ RAISE NOTICE 'bitmapscan: %', r2;
+ END LOOP;
+ END IF;
+
+ -- make sure we found expected number of matches
+ IF count != r.matches THEN RAISE WARNING 'unexpected number of results % for %', count, r; END IF;
+ END LOOP;
+END;
+$x$;
+RESET enable_seqscan;
+RESET enable_bitmapscan;
+INSERT INTO brintest_multi SELECT
+ 142857 * tenthous,
+ thousand,
+ twothousand,
+ unique1::oid,
+ format('(%s,%s)', tenthous, twenty)::tid,
+ (four + 1.0)/(hundred+1),
+ odd::float8 / (tenthous + 1),
+ format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr,
+ inet '10.2.3.4' + tenthous,
+ cidr '10.2.3/24' + tenthous,
+ date '1995-08-15' + tenthous,
+ time '01:20:30' + thousand * interval '18.5 second',
+ timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours',
+ timestamptz '1972-10-10 03:00' + thousand * interval '1 hour',
+ justify_days(justify_hours(tenthous * interval '12 minutes')),
+ timetz '01:30:20' + hundred * interval '15 seconds',
+ tenthous::numeric(36,30) * fivethous * even / (hundred + 1),
+ format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid,
+ format('%s/%s%s', odd, even, tenthous)::pg_lsn
+FROM tenk1 ORDER BY unique2 LIMIT 5 OFFSET 5;
+SELECT brin_desummarize_range('brinidx_multi', 0);
+ brin_desummarize_range
+------------------------
+
+(1 row)
+
+VACUUM brintest_multi; -- force a summarization cycle in brinidx
+UPDATE brintest_multi SET int8col = int8col * int4col;
+-- Tests for brin_summarize_new_values
+SELECT brin_summarize_new_values('brintest_multi'); -- error, not an index
+ERROR: "brintest_multi" is not an index
+SELECT brin_summarize_new_values('tenk1_unique1'); -- error, not a BRIN index
+ERROR: "tenk1_unique1" is not a BRIN index
+SELECT brin_summarize_new_values('brinidx_multi'); -- ok, no change expected
+ brin_summarize_new_values
+---------------------------
+ 0
+(1 row)
+
+-- Tests for brin_desummarize_range
+SELECT brin_desummarize_range('brinidx_multi', -1); -- error, invalid range
+ERROR: block number out of range: -1
+SELECT brin_desummarize_range('brinidx_multi', 0);
+ brin_desummarize_range
+------------------------
+
+(1 row)
+
+SELECT brin_desummarize_range('brinidx_multi', 0);
+ brin_desummarize_range
+------------------------
+
+(1 row)
+
+SELECT brin_desummarize_range('brinidx_multi', 100000000);
+ brin_desummarize_range
+------------------------
+
+(1 row)
+
+-- Test brin_summarize_range
+CREATE TABLE brin_summarize_multi (
+ value int
+) WITH (fillfactor=10, autovacuum_enabled=false);
+CREATE INDEX brin_summarize_multi_idx ON brin_summarize_multi USING brin (value) WITH (pages_per_range=2);
+-- Fill a few pages
+DO $$
+DECLARE curtid tid;
+BEGIN
+ LOOP
+ INSERT INTO brin_summarize_multi VALUES (1) RETURNING ctid INTO curtid;
+ EXIT WHEN curtid > tid '(2, 0)';
+ END LOOP;
+END;
+$$;
+-- summarize one range
+SELECT brin_summarize_range('brin_summarize_multi_idx', 0);
+ brin_summarize_range
+----------------------
+ 0
+(1 row)
+
+-- nothing: already summarized
+SELECT brin_summarize_range('brin_summarize_multi_idx', 1);
+ brin_summarize_range
+----------------------
+ 0
+(1 row)
+
+-- summarize one range
+SELECT brin_summarize_range('brin_summarize_multi_idx', 2);
+ brin_summarize_range
+----------------------
+ 1
+(1 row)
+
+-- nothing: page doesn't exist in table
+SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967295);
+ brin_summarize_range
+----------------------
+ 0
+(1 row)
+
+-- invalid block number values
+SELECT brin_summarize_range('brin_summarize_multi_idx', -1);
+ERROR: block number out of range: -1
+SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967296);
+ERROR: block number out of range: 4294967296
+-- test brin cost estimates behave sanely based on correlation of values
+CREATE TABLE brin_test_multi (a INT, b INT);
+INSERT INTO brin_test_multi SELECT x/100,x%100 FROM generate_series(1,10000) x(x);
+CREATE INDEX brin_test_multi_a_idx ON brin_test_multi USING brin (a) WITH (pages_per_range = 2);
+CREATE INDEX brin_test_multi_b_idx ON brin_test_multi USING brin (b) WITH (pages_per_range = 2);
+VACUUM ANALYZE brin_test_multi;
+-- Ensure brin index is used when columns are perfectly correlated
+EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE a = 1;
+ QUERY PLAN
+--------------------------------------------------
+ Bitmap Heap Scan on brin_test_multi
+ Recheck Cond: (a = 1)
+ -> Bitmap Index Scan on brin_test_multi_a_idx
+ Index Cond: (a = 1)
+(4 rows)
+
+-- Ensure brin index is not used when values are not correlated
+EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE b = 1;
+ QUERY PLAN
+-----------------------------
+ Seq Scan on brin_test_multi
+ Filter: (b = 1)
+(2 rows)
+
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index a7a5b22d4f..76050af797 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -4990,12 +4990,13 @@ List of access methods
(0 rows)
\dAc brin pg*.oid*
- List of operator classes
- AM | Input type | Storage type | Operator class | Default?
-------+------------+--------------+----------------+----------
- brin | oid | | oid_bloom_ops | no
- brin | oid | | oid_minmax_ops | yes
-(2 rows)
+ List of operator classes
+ AM | Input type | Storage type | Operator class | Default?
+------+------------+--------------+----------------------+----------
+ brin | oid | | oid_bloom_ops | no
+ brin | oid | | oid_minmax_multi_ops | no
+ brin | oid | | oid_minmax_ops | yes
+(3 rows)
\dAf spgist
List of operator families
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index e568b9fea2..0541c12a25 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -67,14 +67,15 @@ WHERE p1.typtype not in ('p') AND p1.typname NOT LIKE E'\\_%'
WHERE p2.typname = ('_' || p1.typname)::name AND
p2.typelem = p1.oid and p1.typarray = p2.oid)
ORDER BY p1.oid;
- oid | typname
-------+-----------------------
+ oid | typname
+------+------------------------------
194 | pg_node_tree
3361 | pg_ndistinct
3402 | pg_dependencies
5017 | pg_mcv_list
9925 | pg_brin_bloom_summary
-(5 rows)
+ 9966 | pg_brin_minmax_multi_summary
+(6 rows)
-- Make sure typarray points to a "true" array type of our own base
SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype,
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index ecd0806718..d34fc7ef88 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -80,7 +80,7 @@ test: brin gin gist spgist privileges init_privs security_label collate matview
# ----------
# Additional BRIN tests
# ----------
-test: brin_bloom
+test: brin_bloom brin_multi
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index c0d7fa76f1..91834a984e 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -109,6 +109,7 @@ test: namespace
test: prepared_xacts
test: brin
test: brin_bloom
+test: brin_multi
test: gin
test: gist
test: spgist
diff --git a/src/test/regress/sql/brin_multi.sql b/src/test/regress/sql/brin_multi.sql
new file mode 100644
index 0000000000..6d61fb84c6
--- /dev/null
+++ b/src/test/regress/sql/brin_multi.sql
@@ -0,0 +1,397 @@
+CREATE TABLE brintest_multi (
+ int8col bigint,
+ int2col smallint,
+ int4col integer,
+ oidcol oid,
+ tidcol tid,
+ float4col real,
+ float8col double precision,
+ macaddrcol macaddr,
+ inetcol inet,
+ cidrcol cidr,
+ datecol date,
+ timecol time without time zone,
+ timestampcol timestamp without time zone,
+ timestamptzcol timestamp with time zone,
+ intervalcol interval,
+ timetzcol time with time zone,
+ numericcol numeric,
+ uuidcol uuid,
+ lsncol pg_lsn
+) WITH (fillfactor=10);
+
+INSERT INTO brintest_multi SELECT
+ 142857 * tenthous,
+ thousand,
+ twothousand,
+ unique1::oid,
+ format('(%s,%s)', tenthous, twenty)::tid,
+ (four + 1.0)/(hundred+1),
+ odd::float8 / (tenthous + 1),
+ format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr,
+ inet '10.2.3.4/24' + tenthous,
+ cidr '10.2.3/24' + tenthous,
+ date '1995-08-15' + tenthous,
+ time '01:20:30' + thousand * interval '18.5 second',
+ timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours',
+ timestamptz '1972-10-10 03:00' + thousand * interval '1 hour',
+ justify_days(justify_hours(tenthous * interval '12 minutes')),
+ timetz '01:30:20+02' + hundred * interval '15 seconds',
+ tenthous::numeric(36,30) * fivethous * even / (hundred + 1),
+ format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid,
+ format('%s/%s%s', odd, even, tenthous)::pg_lsn
+FROM tenk1 ORDER BY unique2 LIMIT 100;
+
+-- throw in some NULL's and different values
+INSERT INTO brintest_multi (inetcol, cidrcol) SELECT
+ inet 'fe80::6e40:8ff:fea9:8c46' + tenthous,
+ cidr 'fe80::6e40:8ff:fea9:8c46' + tenthous
+FROM tenk1 ORDER BY thousand, tenthous LIMIT 25;
+
+-- test minmax-multi specific index options
+-- number of values must be >= 16
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops(values_per_range = 7)
+);
+-- number of values must be <= 256
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops(values_per_range = 257)
+);
+
+-- first create an index with a single page range, to force compaction
+-- due to exceeding the number of values per summary
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops,
+ int2col int2_minmax_multi_ops,
+ int4col int4_minmax_multi_ops,
+ oidcol oid_minmax_multi_ops,
+ tidcol tid_minmax_multi_ops,
+ float4col float4_minmax_multi_ops,
+ float8col float8_minmax_multi_ops,
+ macaddrcol macaddr_minmax_multi_ops,
+ inetcol inet_minmax_multi_ops,
+ cidrcol inet_minmax_multi_ops,
+ datecol date_minmax_multi_ops,
+ timecol time_minmax_multi_ops,
+ timestampcol timestamp_minmax_multi_ops,
+ timestamptzcol timestamptz_minmax_multi_ops,
+ intervalcol interval_minmax_multi_ops,
+ timetzcol timetz_minmax_multi_ops,
+ numericcol numeric_minmax_multi_ops,
+ uuidcol uuid_minmax_multi_ops,
+ lsncol pg_lsn_minmax_multi_ops
+);
+
+DROP INDEX brinidx_multi;
+
+CREATE INDEX brinidx_multi ON brintest_multi USING brin (
+ int8col int8_minmax_multi_ops,
+ int2col int2_minmax_multi_ops,
+ int4col int4_minmax_multi_ops,
+ oidcol oid_minmax_multi_ops,
+ tidcol tid_minmax_multi_ops,
+ float4col float4_minmax_multi_ops,
+ float8col float8_minmax_multi_ops,
+ macaddrcol macaddr_minmax_multi_ops,
+ inetcol inet_minmax_multi_ops,
+ cidrcol inet_minmax_multi_ops,
+ datecol date_minmax_multi_ops,
+ timecol time_minmax_multi_ops,
+ timestampcol timestamp_minmax_multi_ops,
+ timestamptzcol timestamptz_minmax_multi_ops,
+ intervalcol interval_minmax_multi_ops,
+ timetzcol timetz_minmax_multi_ops,
+ numericcol numeric_minmax_multi_ops,
+ uuidcol uuid_minmax_multi_ops,
+ lsncol pg_lsn_minmax_multi_ops
+) with (pages_per_range = 1);
+
+CREATE TABLE brinopers_multi (colname name, typ text,
+ op text[], value text[], matches int[],
+ check (cardinality(op) = cardinality(value)),
+ check (cardinality(op) = cardinality(matches)));
+
+INSERT INTO brinopers_multi VALUES
+ ('int2col', 'int2',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 999, 999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int2col', 'int4',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 999, 1999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int2col', 'int8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 999, 1428427143}',
+ '{100, 100, 1, 100, 100}'),
+ ('int4col', 'int2',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 1999, 1999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int4col', 'int4',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 1999, 1999}',
+ '{100, 100, 1, 100, 100}'),
+ ('int4col', 'int8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 800, 1999, 1428427143}',
+ '{100, 100, 1, 100, 100}'),
+ ('int8col', 'int2',
+ '{>, >=}',
+ '{0, 0}',
+ '{100, 100}'),
+ ('int8col', 'int4',
+ '{>, >=}',
+ '{0, 0}',
+ '{100, 100}'),
+ ('int8col', 'int8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 1257141600, 1428427143, 1428427143}',
+ '{100, 100, 1, 100, 100}'),
+ ('oidcol', 'oid',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 8800, 9999, 9999}',
+ '{100, 100, 1, 100, 100}'),
+ ('tidcol', 'tid',
+ '{>, >=, =, <=, <}',
+ '{"(0,0)", "(0,0)", "(8800,0)", "(9999,19)", "(9999,19)"}',
+ '{100, 100, 1, 100, 100}'),
+ ('float4col', 'float4',
+ '{>, >=, =, <=, <}',
+ '{0.0103093, 0.0103093, 1, 1, 1}',
+ '{100, 100, 4, 100, 96}'),
+ ('float4col', 'float8',
+ '{>, >=, =, <=, <}',
+ '{0.0103093, 0.0103093, 1, 1, 1}',
+ '{100, 100, 4, 100, 96}'),
+ ('float8col', 'float4',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 0, 1.98, 1.98}',
+ '{99, 100, 1, 100, 100}'),
+ ('float8col', 'float8',
+ '{>, >=, =, <=, <}',
+ '{0, 0, 0, 1.98, 1.98}',
+ '{99, 100, 1, 100, 100}'),
+ ('macaddrcol', 'macaddr',
+ '{>, >=, =, <=, <}',
+ '{00:00:01:00:00:00, 00:00:01:00:00:00, 2c:00:2d:00:16:00, ff:fe:00:00:00:00, ff:fe:00:00:00:00}',
+ '{99, 100, 2, 100, 100}'),
+ ('inetcol', 'inet',
+ '{=, <, <=, >, >=}',
+ '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{1, 100, 100, 125, 125}'),
+ ('inetcol', 'cidr',
+ '{<, <=, >, >=}',
+ '{255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{100, 100, 125, 125}'),
+ ('cidrcol', 'inet',
+ '{=, <, <=, >, >=}',
+ '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{2, 100, 100, 125, 125}'),
+ ('cidrcol', 'cidr',
+ '{=, <, <=, >, >=}',
+ '{10.2.14/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}',
+ '{2, 100, 100, 125, 125}'),
+ ('datecol', 'date',
+ '{>, >=, =, <=, <}',
+ '{1995-08-15, 1995-08-15, 2009-12-01, 2022-12-30, 2022-12-30}',
+ '{100, 100, 1, 100, 100}'),
+ ('timecol', 'time',
+ '{>, >=, =, <=, <}',
+ '{01:20:30, 01:20:30, 02:28:57, 06:28:31.5, 06:28:31.5}',
+ '{100, 100, 1, 100, 100}'),
+ ('timestampcol', 'timestamp',
+ '{>, >=, =, <=, <}',
+ '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}',
+ '{100, 100, 1, 100, 100}'),
+ ('timestampcol', 'timestamptz',
+ '{>, >=, =, <=, <}',
+ '{1942-07-23 03:05:09, 1942-07-23 03:05:09, 1964-03-24 19:26:45, 1984-01-20 22:42:21, 1984-01-20 22:42:21}',
+ '{100, 100, 1, 100, 100}'),
+ ('timestamptzcol', 'timestamptz',
+ '{>, >=, =, <=, <}',
+ '{1972-10-10 03:00:00-04, 1972-10-10 03:00:00-04, 1972-10-19 09:00:00-07, 1972-11-20 19:00:00-03, 1972-11-20 19:00:00-03}',
+ '{100, 100, 1, 100, 100}'),
+ ('intervalcol', 'interval',
+ '{>, >=, =, <=, <}',
+ '{00:00:00, 00:00:00, 1 mons 13 days 12:24, 2 mons 23 days 07:48:00, 1 year}',
+ '{100, 100, 1, 100, 100}'),
+ ('timetzcol', 'timetz',
+ '{>, >=, =, <=, <}',
+ '{01:30:20+02, 01:30:20+02, 01:35:50+02, 23:55:05+02, 23:55:05+02}',
+ '{99, 100, 2, 100, 100}'),
+ ('numericcol', 'numeric',
+ '{>, >=, =, <=, <}',
+ '{0.00, 0.01, 2268164.347826086956521739130434782609, 99470151.9, 99470151.9}',
+ '{100, 100, 1, 100, 100}'),
+ ('uuidcol', 'uuid',
+ '{>, >=, =, <=, <}',
+ '{00040004-0004-0004-0004-000400040004, 00040004-0004-0004-0004-000400040004, 52225222-5222-5222-5222-522252225222, 99989998-9998-9998-9998-999899989998, 99989998-9998-9998-9998-999899989998}',
+ '{100, 100, 1, 100, 100}'),
+ ('lsncol', 'pg_lsn',
+ '{>, >=, =, <=, <, IS, IS NOT}',
+ '{0/1200, 0/1200, 44/455222, 198/1999799, 198/1999799, NULL, NULL}',
+ '{100, 100, 1, 100, 100, 25, 100}');
+
+DO $x$
+DECLARE
+ r record;
+ r2 record;
+ cond text;
+ idx_ctids tid[];
+ ss_ctids tid[];
+ count int;
+ plan_ok bool;
+ plan_line text;
+BEGIN
+ FOR r IN SELECT colname, oper, typ, value[ordinality], matches[ordinality] FROM brinopers_multi, unnest(op) WITH ORDINALITY AS oper LOOP
+
+ -- prepare the condition
+ IF r.value IS NULL THEN
+ cond := format('%I %s %L', r.colname, r.oper, r.value);
+ ELSE
+ cond := format('%I %s %L::%s', r.colname, r.oper, r.value, r.typ);
+ END IF;
+
+ -- run the query using the brin index
+ SET enable_seqscan = 0;
+ SET enable_bitmapscan = 1;
+
+ plan_ok := false;
+ FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP
+ IF plan_line LIKE '%Bitmap Heap Scan on brintest_multi%' THEN
+ plan_ok := true;
+ END IF;
+ END LOOP;
+ IF NOT plan_ok THEN
+ RAISE WARNING 'did not get bitmap indexscan plan for %', r;
+ END IF;
+
+ EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond)
+ INTO idx_ctids;
+
+ -- run the query using a seqscan
+ SET enable_seqscan = 1;
+ SET enable_bitmapscan = 0;
+
+ plan_ok := false;
+ FOR plan_line IN EXECUTE format($y$EXPLAIN SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond) LOOP
+ IF plan_line LIKE '%Seq Scan on brintest_multi%' THEN
+ plan_ok := true;
+ END IF;
+ END LOOP;
+ IF NOT plan_ok THEN
+ RAISE WARNING 'did not get seqscan plan for %', r;
+ END IF;
+
+ EXECUTE format($y$SELECT array_agg(ctid) FROM brintest_multi WHERE %s $y$, cond)
+ INTO ss_ctids;
+
+ -- make sure both return the same results
+ count := array_length(idx_ctids, 1);
+
+ IF NOT (count = array_length(ss_ctids, 1) AND
+ idx_ctids @> ss_ctids AND
+ idx_ctids <@ ss_ctids) THEN
+ -- report the results of each scan to make the differences obvious
+ RAISE WARNING 'something not right in %: count %', r, count;
+ SET enable_seqscan = 1;
+ SET enable_bitmapscan = 0;
+ FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP
+ RAISE NOTICE 'seqscan: %', r2;
+ END LOOP;
+
+ SET enable_seqscan = 0;
+ SET enable_bitmapscan = 1;
+ FOR r2 IN EXECUTE 'SELECT ' || r.colname || ' FROM brintest_multi WHERE ' || cond LOOP
+ RAISE NOTICE 'bitmapscan: %', r2;
+ END LOOP;
+ END IF;
+
+ -- make sure we found expected number of matches
+ IF count != r.matches THEN RAISE WARNING 'unexpected number of results % for %', count, r; END IF;
+ END LOOP;
+END;
+$x$;
+
+RESET enable_seqscan;
+RESET enable_bitmapscan;
+
+INSERT INTO brintest_multi SELECT
+ 142857 * tenthous,
+ thousand,
+ twothousand,
+ unique1::oid,
+ format('(%s,%s)', tenthous, twenty)::tid,
+ (four + 1.0)/(hundred+1),
+ odd::float8 / (tenthous + 1),
+ format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr,
+ inet '10.2.3.4' + tenthous,
+ cidr '10.2.3/24' + tenthous,
+ date '1995-08-15' + tenthous,
+ time '01:20:30' + thousand * interval '18.5 second',
+ timestamp '1942-07-23 03:05:09' + tenthous * interval '36.38 hours',
+ timestamptz '1972-10-10 03:00' + thousand * interval '1 hour',
+ justify_days(justify_hours(tenthous * interval '12 minutes')),
+ timetz '01:30:20' + hundred * interval '15 seconds',
+ tenthous::numeric(36,30) * fivethous * even / (hundred + 1),
+ format('%s%s-%s-%s-%s-%s%s%s', to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'), to_char(tenthous, 'FM0000'))::uuid,
+ format('%s/%s%s', odd, even, tenthous)::pg_lsn
+FROM tenk1 ORDER BY unique2 LIMIT 5 OFFSET 5;
+
+SELECT brin_desummarize_range('brinidx_multi', 0);
+VACUUM brintest_multi; -- force a summarization cycle in brinidx
+
+UPDATE brintest_multi SET int8col = int8col * int4col;
+
+-- Tests for brin_summarize_new_values
+SELECT brin_summarize_new_values('brintest_multi'); -- error, not an index
+SELECT brin_summarize_new_values('tenk1_unique1'); -- error, not a BRIN index
+SELECT brin_summarize_new_values('brinidx_multi'); -- ok, no change expected
+
+-- Tests for brin_desummarize_range
+SELECT brin_desummarize_range('brinidx_multi', -1); -- error, invalid range
+SELECT brin_desummarize_range('brinidx_multi', 0);
+SELECT brin_desummarize_range('brinidx_multi', 0);
+SELECT brin_desummarize_range('brinidx_multi', 100000000);
+
+-- Test brin_summarize_range
+CREATE TABLE brin_summarize_multi (
+ value int
+) WITH (fillfactor=10, autovacuum_enabled=false);
+CREATE INDEX brin_summarize_multi_idx ON brin_summarize_multi USING brin (value) WITH (pages_per_range=2);
+-- Fill a few pages
+DO $$
+DECLARE curtid tid;
+BEGIN
+ LOOP
+ INSERT INTO brin_summarize_multi VALUES (1) RETURNING ctid INTO curtid;
+ EXIT WHEN curtid > tid '(2, 0)';
+ END LOOP;
+END;
+$$;
+
+-- summarize one range
+SELECT brin_summarize_range('brin_summarize_multi_idx', 0);
+-- nothing: already summarized
+SELECT brin_summarize_range('brin_summarize_multi_idx', 1);
+-- summarize one range
+SELECT brin_summarize_range('brin_summarize_multi_idx', 2);
+-- nothing: page doesn't exist in table
+SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967295);
+-- invalid block number values
+SELECT brin_summarize_range('brin_summarize_multi_idx', -1);
+SELECT brin_summarize_range('brin_summarize_multi_idx', 4294967296);
+
+
+-- test brin cost estimates behave sanely based on correlation of values
+CREATE TABLE brin_test_multi (a INT, b INT);
+INSERT INTO brin_test_multi SELECT x/100,x%100 FROM generate_series(1,10000) x(x);
+CREATE INDEX brin_test_multi_a_idx ON brin_test_multi USING brin (a) WITH (pages_per_range = 2);
+CREATE INDEX brin_test_multi_b_idx ON brin_test_multi USING brin (b) WITH (pages_per_range = 2);
+VACUUM ANALYZE brin_test_multi;
+
+-- Ensure brin index is used when columns are perfectly correlated
+EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE a = 1;
+-- Ensure brin index is not used when values are not correlated
+EXPLAIN (COSTS OFF) SELECT * FROM brin_test_multi WHERE b = 1;
--
2.26.2
--------------6A37408A2C42B5B56C3B3D49
Content-Type: text/x-patch; charset=UTF-8;
name="0006-Ignore-correlation-for-new-BRIN-opclasses-20210303.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename*0="0006-Ignore-correlation-for-new-BRIN-opclasses-20210303.patc";
filename*1="h"
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
@ 2022-04-27 06:12 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: Michael Paquier @ 2022-04-27 06:12 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Pg Hackers <[email protected]>
On Wed, Apr 27, 2022 at 10:25:50AM +0530, Amit Kapila wrote:
> I feel we can explain a bit more about this in docs. We already have
> some explanation of how row filters are combined [1]. We can probably
> add a few examples for column lists.
I am not completely sure exactly what we should do here, but this
stuff needs to be at least discussed. I have added an open item.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../Ymje09ToDVKpW%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
@ 2022-04-29 05:05 ` Amit Kapila <[email protected]>
2022-05-01 21:42 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-04-29 05:05 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Pg Hackers <[email protected]>
On Thu, Apr 28, 2022 at 5:56 PM Peter Eisentraut
<[email protected]> wrote:
>
> On 27.04.22 12:33, Amit Kapila wrote:
> > Currently, when the subscription has multiple publications, we combine
> > the objects, and actions of those publications. It happens for
> > 'publish_via_partition_root', publication actions, tables, column
> > lists, or row filters. I think the whole design works on this idea
> > even the initial table sync. I think it might need a major change
> > (which I am not sure about at this stage) if we want to make the
> > initial sync also behave similar to what you are proposing.
>
> If one publication says "publish if insert" and another publication says
> "publish if update", then the combination of that is clearly "publish if
> insert or update". Similarly, if one publication says "WHERE (foo)" and
> one says "WHERE (bar)", then the combination is "WHERE (foo OR bar)".
>
> But if one publication says "publish columns a and b if condition-X" and
> another publication says "publish columns a and c if not-condition-X",
> then the combination is clearly *not* "publish columns a, b, c if true".
> That is not logical, in the literal sense of that word.
>
So, what should be the behavior in the below cases:
Case-1:
pub1: "publish columns a and b if condition-X"
pub2: "publish column c if condition-X"
Isn't it okay to combine these?
Case-2:
pub1: "publish columns a and b if condition-X"
pub2: "publish columns c if condition-Y"
Here Y is subset of condition X (say something like condition-X: "col1
> 5" and condition-Y: "col1 > 10").
What should we do in such a case?
I think if there are some cases where combining them is okay but in
other cases, it is not okay then it is better to prohibit 'not-okay'
cases if that is feasible.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-04-29 05:05 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-01 21:42 ` Tomas Vondra <[email protected]>
2022-05-02 20:34 ` Re: bogus: logical replication rows/cols combinations Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-01 21:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pg Hackers <[email protected]>
On 4/29/22 07:05, Amit Kapila wrote:
> On Thu, Apr 28, 2022 at 5:56 PM Peter Eisentraut
> <[email protected]> wrote:
>>
>> On 27.04.22 12:33, Amit Kapila wrote:
>>> Currently, when the subscription has multiple publications, we combine
>>> the objects, and actions of those publications. It happens for
>>> 'publish_via_partition_root', publication actions, tables, column
>>> lists, or row filters. I think the whole design works on this idea
>>> even the initial table sync. I think it might need a major change
>>> (which I am not sure about at this stage) if we want to make the
>>> initial sync also behave similar to what you are proposing.
>>
>> If one publication says "publish if insert" and another publication says
>> "publish if update", then the combination of that is clearly "publish if
>> insert or update". Similarly, if one publication says "WHERE (foo)" and
>> one says "WHERE (bar)", then the combination is "WHERE (foo OR bar)".
>>
>> But if one publication says "publish columns a and b if condition-X" and
>> another publication says "publish columns a and c if not-condition-X",
>> then the combination is clearly *not* "publish columns a, b, c if true".
>> That is not logical, in the literal sense of that word.
>>
>
> So, what should be the behavior in the below cases:
>
> Case-1:
> pub1: "publish columns a and b if condition-X"
> pub2: "publish column c if condition-X"
>
> Isn't it okay to combine these?
>
Yes, I think it's reasonable to combine those. So the whole publication
will have
WHERE (condition-X)
and the column list will be (a,b,c).
> Case-2:
> pub1: "publish columns a and b if condition-X"
> pub2: "publish columns c if condition-Y"
>
In this case the publication will have
WHERE (condition-X or condition-Y)
and there will be different column filters for different row sets:
if (condition-X and condition-Y)
=> (a,b,c)
else if (condition-X and NOT condition-Y)
=> (a,b)
else if (condition-Y and NOT condition-X)
=> (c)
I think this behavior is reasonable, and it's what the patch does.
> Here Y is subset of condition X (say something like condition-X:
> "col > 5" and condition-Y: "col1 > 10").>
> What should we do in such a case?
>
> I think if there are some cases where combining them is okay but in
> other cases, it is not okay then it is better to prohibit 'not-okay'
> cases if that is feasible.
>
Not sure I understand what's the (supposed) issue with this example.
We'll simply do this:
if (col1 > 5 and col1 > 10)
=> (a,b,c)
else if (col1 > 5 and col1 <= 10)
=> (a,b)
else if (col1 > 10 and col1 <= 5)
=> (c)
Obviously, the third branch is unreachable, because the if condition can
never be satisfied, so we can never see only column list (c). But that's
fine IMO. When phrased using the CASE expressions (as in tablesync) it's
probably somewhat less cumbersome.
I think it's easier to think about this using "data redaction" example
where you specify which columns can be replicated under what condition.
Obviously, that's "orthogonal" in the sense that we specify column list
for a row filer condition, not row filter for a column. But in principle
it's the same thing, just different grammar.
And in that case it makes perfect sense that you don't blindly combine
the column lists from all publications, because that'd defeat the whole
point of filtering columns based on row filters.
Imagine have a table with customers from different regions, and you want
to replicate the data somewhere else, but for some reason you can only
replicate details for one particular region, and subset of columns for
everyone else. So you'd do something like this:
CREATE PUBLICATION p1 FOR TABLE customers (... all columns ...)
WHERE region = 'USA';
CREATE PUBLICATION p1 FOR TABLE customers (... subset of columns ...)
WHERE region != 'USA';
I think ignoring the row filters and just merging the column lists makes
no sense for this use case.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-04-29 05:05 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-01 21:42 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-02 20:34 ` Peter Eisentraut <[email protected]>
2022-05-03 19:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Peter Eisentraut @ 2022-05-02 20:34 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pg Hackers <[email protected]>
On 01.05.22 23:42, Tomas Vondra wrote:
> Imagine have a table with customers from different regions, and you want
> to replicate the data somewhere else, but for some reason you can only
> replicate details for one particular region, and subset of columns for
> everyone else. So you'd do something like this:
>
> CREATE PUBLICATION p1 FOR TABLE customers (... all columns ...)
> WHERE region = 'USA';
>
> CREATE PUBLICATION p1 FOR TABLE customers (... subset of columns ...)
> WHERE region != 'USA';
>
> I think ignoring the row filters and just merging the column lists makes
> no sense for this use case.
I'm thinking now the underlying problem is that we shouldn't combine
column lists at all. Examples like the above where you want to redact
values somehow are better addressed with something like triggers or an
actual "column filter" that works dynamically or some other mechanism.
The main purpose, in my mind, of column lists is if the tables
statically have different shapes on publisher and subscriber. Perhaps
for space reasons or regulatory reasons you don't want to replicate
everything. But then it doesn't make sense to combine column lists. If
you decide over here that the subscriber table has this shape and over
there that the subscriber table has that other shape, then the
combination of the two will be a table that has neither shape and so
will not work for anything.
I think in general we should be much more restrictive in how we combine
publications. Unless we are really sure it makes sense, we should
disallow it. Users can always make a new publication with different
settings and subscribe to that directly.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-04-29 05:05 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-01 21:42 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 20:34 ` Re: bogus: logical replication rows/cols combinations Peter Eisentraut <[email protected]>
@ 2022-05-03 19:40 ` Tomas Vondra <[email protected]>
2022-05-04 13:56 ` Re: bogus: logical replication rows/cols combinations Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-03 19:40 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pg Hackers <[email protected]>
On 5/2/22 22:34, Peter Eisentraut wrote:
> On 01.05.22 23:42, Tomas Vondra wrote:
>> Imagine have a table with customers from different regions, and you want
>> to replicate the data somewhere else, but for some reason you can only
>> replicate details for one particular region, and subset of columns for
>> everyone else. So you'd do something like this:
>>
>> CREATE PUBLICATION p1 FOR TABLE customers (... all columns ...)
>> WHERE region = 'USA';
>>
>> CREATE PUBLICATION p1 FOR TABLE customers (... subset of columns ...)
>> WHERE region != 'USA';
>>
>> I think ignoring the row filters and just merging the column lists makes
>> no sense for this use case.
>
> I'm thinking now the underlying problem is that we shouldn't combine
> column lists at all. Examples like the above where you want to redact
> values somehow are better addressed with something like triggers or an
> actual "column filter" that works dynamically or some other mechanism.
>
So what's wrong with merging the column lists as implemented in the v2
patch, posted a couple days ago?
I don't think triggers are a suitable alternative, as it executes on the
subscriber node. So you have to first copy the data to the remote node,
where it gets filtered. With column filters the data gets redacted on
the publisher.
> The main purpose, in my mind, of column lists is if the tables
> statically have different shapes on publisher and subscriber. Perhaps
> for space reasons or regulatory reasons you don't want to replicate
> everything. But then it doesn't make sense to combine column lists. If
> you decide over here that the subscriber table has this shape and over
> there that the subscriber table has that other shape, then the
> combination of the two will be a table that has neither shape and so
> will not work for anything.
>
Yeah. If we intend to use column lists only to adapt to a different
schema on the subscriber node, then maybe it'd be fine to not merge
column lists. It'd probably be reasonable to allow at least cases with
multiple publications using the same column list, though. In that case
there's no ambiguity.
> I think in general we should be much more restrictive in how we combine
> publications. Unless we are really sure it makes sense, we should
> disallow it. Users can always make a new publication with different
> settings and subscribe to that directly.
I agree with that in principle - correct first, flexibility second. If
the behavior is not correct, it doesn't matter how flexible it is.
I still think the data redaction use case is valid/interesting, but if
we want to impose some restrictions I'm OK with that, as long as it's
done in a way that we can relax in the future to allow that use case
(that is, without introducing any incompatibilities).
However, what's the definition of "correctness" in this context? Without
that it's hard to say if the restrictions make the behavior any more
correct. It'd be unfortunate to impose restritions, which will prevent
some use cases, only to discover we haven't actually made it correct.
For example, is it enough to restrict column lists, or does it need to
restrict e.g. row filters too? And does it need to consider other stuff,
like publications replicating different actions?
For example, if we allow different column lists (or row filters) for
different actions (one publication for insert, another one for update),
we still have the strange behavior described before.
And if we force users to use separate subscriptions, I'm not sure that
really improves the situation for users who actually need that. They'll
do that, and aside from all the problems they'll also face issues with
timing between the two concurrent subscriptions, having to decode stuff
multiple times, etc.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-04-29 05:05 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-01 21:42 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 20:34 ` Re: bogus: logical replication rows/cols combinations Peter Eisentraut <[email protected]>
2022-05-03 19:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-04 13:56 ` Peter Eisentraut <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Peter Eisentraut @ 2022-05-04 13:56 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pg Hackers <[email protected]>
On 03.05.22 21:40, Tomas Vondra wrote:
> So what's wrong with merging the column lists as implemented in the v2
> patch, posted a couple days ago?
Merging the column lists is ok if all other publication attributes
match. Otherwise, I think not.
> I don't think triggers are a suitable alternative, as it executes on the
> subscriber node. So you have to first copy the data to the remote node,
> where it gets filtered. With column filters the data gets redacted on
> the publisher.
Right, triggers are not currently a solution. But you could imagine a
redaction filter system that runs on the publisher that modifies rows
before they are sent out.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
@ 2022-05-02 09:35 Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 11:23 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Tomas Vondra @ 2022-05-02 09:35 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/2/22 07:31, Amit Kapila wrote:
> On Mon, May 2, 2022 at 3:27 AM Tomas Vondra
> <[email protected]> wrote:
>>
>> On 4/30/22 12:11, Amit Kapila wrote:
>>> On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera <[email protected]> wrote:
>>>>
>>>> My proposal is that if users want to define multiple publications, and
>>>> their definitions conflict in a way that would behave ridiculously (==
>>>> bound to cause data inconsistencies eventually), an error should be
>>>> thrown. Maybe we will not be able to catch all bogus cases, but we can
>>>> be prepared for the most obvious ones, and patch later when we find
>>>> others.
>>>>
>>>
>>> I agree with throwing errors for obvious/known bogus cases but do we
>>> want to throw errors or restrict the combining of column lists when
>>> row filters are present in all cases? See some examples [1 ] where it
>>> may be valid to combine them.
>>>
>>
>> I think there are three challenges:
>>
>> (a) Deciding what's an obvious bug or an unsupported case (e.g. because
>> it's not clear what's the correct behavior / way to merge column lists).
>>
>> (b) When / where to detect the issue.
>>
>> (c) Making sure this does not break/prevent existing use cases.
>>
>>
>> As I said before [1], I think the issue stems from essentially allowing
>> DML to have different row filters / column lists. So we could forbid
>> publications to specify WITH (publish=...) and one of the two features,
>>
>
> I don't think this is feasible for row filters because that would mean
> publishing all actions because we have a restriction that all columns
> referenced in the row filter expression are part of the REPLICA
> IDENTITY index. This restriction is only valid for updates/deletes, so
> if we allow all pubactions then this will be imposed on inserts as
> well. A similar restriction is there for column lists as well, so I
> don't think we can do it there as well. Do you have some idea to
> address it?
>
No, I haven't thought about how exactly to implement this, and I have
not thought about how to deal with the replica identity issues. My
thoughts were that we'd only really need this for tables with row
filters and/or column lists, treating it as a cost of those features.
But yeah, it seems annoying.
>> or make sure subscription does not combine multiple such publications.
>>
>
> Yeah, or don't allow to define such publications in the first place so
> that different subscriptions can't combine them but I guess that might
> forbid some useful cases as well where publication may not get
> combined with other publications.
>
But how would you check that? You don't know which publications will be
combined by a subscription until you create the subscription, right?
>> The second option has the annoying consequence that it makes this
>> useless for the "data redaction" use case I described in [2], because
>> that relies on combining multiple publications.
>>
>
> True, but as a workaround users can create different subscriptions for
> different publications.
>
Won't that replicate duplicate data, when the row filters re not
mutually exclusive?
>> Furthermore, what if the publications change after the subscriptions get
>> created? Will we be able to detect the error etc.?
>>
>
> I think from that apart from 'Create Subscription', the same check
> needs to be added for Alter Subscription ... Refresh, Alter
> Subscription ... Enable.
>
> In the publication side, we need an additional check in Alter
> Publication ... SET table variant. One idea is that we get all other
> publications for which the corresponding relation is defined. And then
> if we find anything which we don't want to allow then we can throw an
> error. This will forbid some useful cases as well as mentioned above.
> So, the other possibility is to expose all publications for a
> walsender, and then we can find the exact set of publications where
> the current publication is used with other publications and we can
> check only those publications. So, if we have three walsenders
> (walsnd1: pub1, pub2; walsnd2 pub2; walsnd3: pub2, pub3) in the system
> and we are currently altering publication pub1 then we need to check
> only pub3 for any conflicting conditions. Yet another simple way could
> be that we don't allow to change column list via Alter Publication ...
> Set variant because the other variants anyway need REFRESH publication
> which we have covered.
>
> I think it is tricky to decide what exactly we want to forbid, so, we
> may want to follow something simple like if the column list and row
> filters for a table are different in the required set of publications
> then we treat it as an unsupported case. I think this will prohibit
> some useful cases but should probably forbid the cases we are worried
> about here.
>
I don't have a clear idea on what the right tradeoff is :-(
Maybe we're digressing a bit from the stuff Alvaro complained about
initially. Arguably the existing column list behavior is surprising and
would not work with reasonable use cases. So let's fix it.
But maybe you're right validating row filters is a step too far. Yes,
users may define strange combinations of publications, but is that
really an issue we have to solve?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-02 10:17 ` Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Alvaro Herrera @ 2022-05-02 10:17 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 2022-May-02, Tomas Vondra wrote:
> On 5/2/22 07:31, Amit Kapila wrote:
> > Yeah, or don't allow to define such publications in the first place so
> > that different subscriptions can't combine them but I guess that might
> > forbid some useful cases as well where publication may not get
> > combined with other publications.
>
> But how would you check that? You don't know which publications will be
> combined by a subscription until you create the subscription, right?
... and I think this poses a problem: if the publisher has multiple
publications and the subscriber later uses those to create a combined
subscription, we can check at CREATE SUBSCRIPTION time that they can be
combined correctly. But if the publisher decides to change the
publications changing the rules and they are no longer consistent, can
we throw an error at ALTER PUBLICATION point? If the publisher can
detect that they are being used together by some subscription, then
maybe we can check consistency in the publication side and everything is
all right. But I'm not sure that the publisher knows who is subscribed
to what, so this might not be an option.
The latter ultimately means that we aren't sure that a combined
subscription is safe. And in turn this means that a pg_dump of such a
database cannot be restored (because the CREATE SUBSCRIPTION will be
rejected as being inconsistent).
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-02 10:23 ` Tomas Vondra <[email protected]>
2022-05-02 10:55 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Tomas Vondra @ 2022-05-02 10:23 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/2/22 12:17, Alvaro Herrera wrote:
> On 2022-May-02, Tomas Vondra wrote:
>> On 5/2/22 07:31, Amit Kapila wrote:
>
>>> Yeah, or don't allow to define such publications in the first place so
>>> that different subscriptions can't combine them but I guess that might
>>> forbid some useful cases as well where publication may not get
>>> combined with other publications.
>>
>> But how would you check that? You don't know which publications will be
>> combined by a subscription until you create the subscription, right?
>
> ... and I think this poses a problem: if the publisher has multiple
> publications and the subscriber later uses those to create a combined
> subscription, we can check at CREATE SUBSCRIPTION time that they can be
> combined correctly. But if the publisher decides to change the
> publications changing the rules and they are no longer consistent, can
> we throw an error at ALTER PUBLICATION point? If the publisher can
> detect that they are being used together by some subscription, then
> maybe we can check consistency in the publication side and everything is
> all right. But I'm not sure that the publisher knows who is subscribed
> to what, so this might not be an option.
>
AFAIK we don't track that (publication/subscription mapping). The
publications are listed in publication_names parameter of the
START_REPLICATION command.
> The latter ultimately means that we aren't sure that a combined
> subscription is safe. And in turn this means that a pg_dump of such a
> database cannot be restored (because the CREATE SUBSCRIPTION will be
> rejected as being inconsistent).
>
We could do this check when executing the START_REPLICATION command, no?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-02 10:55 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: Alvaro Herrera @ 2022-05-02 10:55 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 2022-May-02, Tomas Vondra wrote:
> On 5/2/22 12:17, Alvaro Herrera wrote:
> > The latter ultimately means that we aren't sure that a combined
> > subscription is safe. And in turn this means that a pg_dump of such a
> > database cannot be restored (because the CREATE SUBSCRIPTION will be
> > rejected as being inconsistent).
>
> We could do this check when executing the START_REPLICATION command, no?
Ah! That sounds like it might work: we throw WARNINGs are CREATE
SUBSCRIPTION (so that users are immediately aware in case something is
going to fail later, but the objects are still created and they can fix
the publications afterwards), but the real ERROR is in START_REPLICATION.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Uno puede defenderse de los ataques; contra los elogios se esta indefenso"
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-02 11:14 ` Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-02 11:14 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 2, 2022 at 3:53 PM Tomas Vondra
<[email protected]> wrote:
>
> On 5/2/22 12:17, Alvaro Herrera wrote:
> > On 2022-May-02, Tomas Vondra wrote:
> >> On 5/2/22 07:31, Amit Kapila wrote:
> >
> >>> Yeah, or don't allow to define such publications in the first place so
> >>> that different subscriptions can't combine them but I guess that might
> >>> forbid some useful cases as well where publication may not get
> >>> combined with other publications.
> >>
> >> But how would you check that? You don't know which publications will be
> >> combined by a subscription until you create the subscription, right?
> >
Yeah, I was thinking to check for all publications where the same
relation is published but as mentioned that may not be a very good
option as that would unnecessarily block many valid cases.
> > ... and I think this poses a problem: if the publisher has multiple
> > publications and the subscriber later uses those to create a combined
> > subscription, we can check at CREATE SUBSCRIPTION time that they can be
> > combined correctly. But if the publisher decides to change the
> > publications changing the rules and they are no longer consistent, can
> > we throw an error at ALTER PUBLICATION point? If the publisher can
> > detect that they are being used together by some subscription, then
> > maybe we can check consistency in the publication side and everything is
> > all right. But I'm not sure that the publisher knows who is subscribed
> > to what, so this might not be an option.
> >
>
> AFAIK we don't track that (publication/subscription mapping). The
> publications are listed in publication_names parameter of the
> START_REPLICATION command.
>
We don't do that currently but we can as mentioned in my previous
email [1]. Let me write the relevant part again. We need to expose all
publications for a walsender, and then we can find the exact set of
publications where the current publication is used with other
publications and we can check only those publications. So, if we have
three walsenders (walsnd1: pub1, pub2; walsnd2 pub2; walsnd3: pub2,
pub3) in the system and we are currently altering publication pub1
then we need to check only pub3 for any conflicting conditions.
I think it is possible to expose a list of publications for each
walsender as it is stored in each walsenders
LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
can have one such LogicalDecodingContext and we can probably share it
via shared memory?
[1] - https://www.postgresql.org/message-id/CAA4eK1LGX-ig%3D%3DQyL%2B%3D%3DnKvcAS3qFU7%3DNiKL77ukUT-Q_4Xnc...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-02 11:44 ` Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:53 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 3 replies; 59+ messages in thread
From: Alvaro Herrera @ 2022-05-02 11:44 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 2022-May-02, Amit Kapila wrote:
> We don't do that currently but we can as mentioned in my previous
> email [1]. Let me write the relevant part again. We need to expose all
> publications for a walsender, and then we can find the exact set of
> publications where the current publication is used with other
> publications and we can check only those publications. So, if we have
> three walsenders (walsnd1: pub1, pub2; walsnd2 pub2; walsnd3: pub2,
> pub3) in the system and we are currently altering publication pub1
> then we need to check only pub3 for any conflicting conditions.
Hmm ... so what happens in the current system, if you have a running
walsender and modify the publication concurrently? Will the subscriber
start getting the changes with the new publication definition, at some
arbitrary point in the middle of their stream? If that's what we do,
maybe we should have a signalling system which disconnects all
walsenders using that publication, so that they can connect and receive
the new definition.
I don't see anything in the publication DDL that interacts with
walsenders -- perhaps I'm overlooking something.
> I think it is possible to expose a list of publications for each
> walsender as it is stored in each walsenders
> LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
> can have one such LogicalDecodingContext and we can probably share it
> via shared memory?
I guess we need to create a DSM each time a walsender opens a
connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
connect to all DSMs of all running walsenders and see if they are
reading from it. Is that what you have in mind? Alternatively, we
could have one DSM per publication with a PID array of all walsenders
that are sending it (each walsender needs to add its PID as it starts).
The latter might be better.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"La rebeldía es la virtud original del hombre" (Arthur Schopenhauer)
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-02 17:36 ` Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-02 17:36 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/2/22 13:44, Alvaro Herrera wrote:
> On 2022-May-02, Amit Kapila wrote:
>
>> We don't do that currently but we can as mentioned in my previous
>> email [1]. Let me write the relevant part again. We need to expose all
>> publications for a walsender, and then we can find the exact set of
>> publications where the current publication is used with other
>> publications and we can check only those publications. So, if we have
>> three walsenders (walsnd1: pub1, pub2; walsnd2 pub2; walsnd3: pub2,
>> pub3) in the system and we are currently altering publication pub1
>> then we need to check only pub3 for any conflicting conditions.
>
> Hmm ... so what happens in the current system, if you have a running
> walsender and modify the publication concurrently? Will the subscriber
> start getting the changes with the new publication definition, at some
> arbitrary point in the middle of their stream? If that's what we do,
> maybe we should have a signalling system which disconnects all
> walsenders using that publication, so that they can connect and receive
> the new definition.
>
> I don't see anything in the publication DDL that interacts with
> walsenders -- perhaps I'm overlooking something.
>
pgoutput.c is relies on relcache callbacks to get notified of changes.
See the stuff that touches replicate_valid and publications_valid. So
the walsender should notice the changes immediately.
Maybe you have some particular case in mind, though?
>> I think it is possible to expose a list of publications for each
>> walsender as it is stored in each walsenders
>> LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
>> can have one such LogicalDecodingContext and we can probably share it
>> via shared memory?
>
> I guess we need to create a DSM each time a walsender opens a
> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
> connect to all DSMs of all running walsenders and see if they are
> reading from it. Is that what you have in mind? Alternatively, we
> could have one DSM per publication with a PID array of all walsenders
> that are sending it (each walsender needs to add its PID as it starts).
> The latter might be better.
>
I don't quite follow what we're trying to build here. The walsender
already knows which publications it works with - how else would
pgoutput.c know that? So the walsender should be able to validate the
stuff it's supposed to replicate is OK.
Why would we need to know publications replicated by other walsenders?
And what if the subscriber is not connected at the moment? In that case
there'll be no walsender.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-02 17:51 ` Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Alvaro Herrera @ 2022-05-02 17:51 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 2022-May-02, Tomas Vondra wrote:
> pgoutput.c is relies on relcache callbacks to get notified of changes.
> See the stuff that touches replicate_valid and publications_valid. So
> the walsender should notice the changes immediately.
Hmm, I suppose that makes any changes easy enough to detect. We don't
need a separate signalling mechanism.
But it does mean that the walsender needs to test the consistency of
[rowfilter, column list, published actions] whenever they change for any
of the current publications and it is working for more than one, and
disconnect if the combination no longer complies with the rules. By the
next time the replica tries to connect, START_REPLICATION will throw the
error.
> Why would we need to know publications replicated by other walsenders?
> And what if the subscriber is not connected at the moment? In that case
> there'll be no walsender.
Sure, if the replica is not connected then there's no issue -- as you
say, that replica will fail at START_REPLICATION time.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"La gente vulgar sólo piensa en pasar el tiempo;
el que tiene talento, en aprovecharlo"
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-02 18:40 ` Tomas Vondra <[email protected]>
2022-05-03 03:30 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-02 18:40 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/2/22 19:51, Alvaro Herrera wrote:
> On 2022-May-02, Tomas Vondra wrote:
>
>> pgoutput.c is relies on relcache callbacks to get notified of changes.
>> See the stuff that touches replicate_valid and publications_valid. So
>> the walsender should notice the changes immediately.
>
> Hmm, I suppose that makes any changes easy enough to detect. We don't
> need a separate signalling mechanism.
>
> But it does mean that the walsender needs to test the consistency of
> [rowfilter, column list, published actions] whenever they change for any
> of the current publications and it is working for more than one, and
> disconnect if the combination no longer complies with the rules. By the
> next time the replica tries to connect, START_REPLICATION will throw the
> error.
>
>> Why would we need to know publications replicated by other walsenders?
>> And what if the subscriber is not connected at the moment? In that case
>> there'll be no walsender.
>
> Sure, if the replica is not connected then there's no issue -- as you
> say, that replica will fail at START_REPLICATION time.
>
Right, I got confused a bit.
Anyway, I think the main challenge is defining what exactly we want to
check, in order to ensure "sensible" behavior, without preventing way
too many sensible use cases.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-03 03:30 ` Amit Kapila <[email protected]>
2022-05-06 03:23 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-03 03:30 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Tue, May 3, 2022 at 12:10 AM Tomas Vondra
<[email protected]> wrote:
>
> On 5/2/22 19:51, Alvaro Herrera wrote:
> >> Why would we need to know publications replicated by other walsenders?
> >> And what if the subscriber is not connected at the moment? In that case
> >> there'll be no walsender.
> >
> > Sure, if the replica is not connected then there's no issue -- as you
> > say, that replica will fail at START_REPLICATION time.
> >
>
> Right, I got confused a bit.
>
> Anyway, I think the main challenge is defining what exactly we want to
> check, in order to ensure "sensible" behavior, without preventing way
> too many sensible use cases.
>
I could think of below two options:
1. Forbid any case where column list is different for the same table
when combining publications.
2. Forbid if the column list and row filters for a table are different
in the set of publications we are planning to combine. This means we
will allow combining column lists when row filters are not present or
when column list is the same (we don't get anything additional by
combining but the idea is we won't forbid such cases) and row filters
are different.
Now, I think the points in favor of (1) are that the main purpose of
introducing a column list are: (a) the structure/schema of the
subscriber is different from the publisher, (b) want to hide sensitive
columns data. In both cases, it should be fine if we follow (1) and
from Peter E.'s latest email [1] he also seems to be indicating the
same. If we want to be slightly more relaxed then we can probably (2).
We can decide on something else as well but I feel it should be such
that it is easy to explain.
[1] - https://www.postgresql.org/message-id/47dd2cb9-4e96-169f-15ac-f9407fb54d43%40enterprisedb.com
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:30 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-06 03:23 ` [email protected] <[email protected]>
2022-05-06 12:26 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: [email protected] @ 2022-05-06 03:23 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Tuesday, May 3, 2022 11:31 AM Amit Kapila <[email protected]> wrote:
>
> On Tue, May 3, 2022 at 12:10 AM Tomas Vondra
> <[email protected]> wrote:
> >
> > On 5/2/22 19:51, Alvaro Herrera wrote:
> > >> Why would we need to know publications replicated by other
> walsenders?
> > >> And what if the subscriber is not connected at the moment? In that case
> > >> there'll be no walsender.
> > >
> > > Sure, if the replica is not connected then there's no issue -- as you
> > > say, that replica will fail at START_REPLICATION time.
> > >
> >
> > Right, I got confused a bit.
> >
> > Anyway, I think the main challenge is defining what exactly we want to
> > check, in order to ensure "sensible" behavior, without preventing way
> > too many sensible use cases.
> >
>
> I could think of below two options:
> 1. Forbid any case where column list is different for the same table
> when combining publications.
> 2. Forbid if the column list and row filters for a table are different
> in the set of publications we are planning to combine. This means we
> will allow combining column lists when row filters are not present or
> when column list is the same (we don't get anything additional by
> combining but the idea is we won't forbid such cases) and row filters
> are different.
>
> Now, I think the points in favor of (1) are that the main purpose of
> introducing a column list are: (a) the structure/schema of the
> subscriber is different from the publisher, (b) want to hide sensitive
> columns data. In both cases, it should be fine if we follow (1) and
> from Peter E.'s latest email [1] he also seems to be indicating the
> same. If we want to be slightly more relaxed then we can probably (2).
> We can decide on something else as well but I feel it should be such
> that it is easy to explain.
I also think it makes sense to add a restriction like (1). I am planning to
implement the restriction if no one objects.
Best regards,
Hou zj
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:30 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-06 03:23 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-06 12:26 ` Tomas Vondra <[email protected]>
2022-05-06 13:40 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-06 12:26 UTC (permalink / raw)
To: [email protected] <[email protected]>; Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/6/22 05:23, [email protected] wrote:
> On Tuesday, May 3, 2022 11:31 AM Amit Kapila <[email protected]> wrote:
>>
>> On Tue, May 3, 2022 at 12:10 AM Tomas Vondra
>> <[email protected]> wrote:
>>>
>>> On 5/2/22 19:51, Alvaro Herrera wrote:
>>>>> Why would we need to know publications replicated by other
>> walsenders?
>>>>> And what if the subscriber is not connected at the moment? In that case
>>>>> there'll be no walsender.
>>>>
>>>> Sure, if the replica is not connected then there's no issue -- as you
>>>> say, that replica will fail at START_REPLICATION time.
>>>>
>>>
>>> Right, I got confused a bit.
>>>
>>> Anyway, I think the main challenge is defining what exactly we want to
>>> check, in order to ensure "sensible" behavior, without preventing way
>>> too many sensible use cases.
>>>
>>
>> I could think of below two options:
>> 1. Forbid any case where column list is different for the same table
>> when combining publications.
>> 2. Forbid if the column list and row filters for a table are different
>> in the set of publications we are planning to combine. This means we
>> will allow combining column lists when row filters are not present or
>> when column list is the same (we don't get anything additional by
>> combining but the idea is we won't forbid such cases) and row filters
>> are different.
>>
>> Now, I think the points in favor of (1) are that the main purpose of
>> introducing a column list are: (a) the structure/schema of the
>> subscriber is different from the publisher, (b) want to hide sensitive
>> columns data. In both cases, it should be fine if we follow (1) and
>> from Peter E.'s latest email [1] he also seems to be indicating the
>> same. If we want to be slightly more relaxed then we can probably (2).
>> We can decide on something else as well but I feel it should be such
>> that it is easy to explain.
>
> I also think it makes sense to add a restriction like (1). I am planning to
> implement the restriction if no one objects.
>
I'm not going to block that approach if that's the consensus here,
though I'm not convinced.
Let me point out (1) does *not* work for data redaction use case,
certainly not the example Alvaro and me presented, because that relies
on a combination of row filters and column filters. Requiring all column
lists to be the same (and not specific to row filter) prevents that
example from working. Yes, you can create multiple subscriptions, but
that brings it's own set of challenges too.
I doubt forcing users to use the more complex setup is good idea, and
combining the column lists per [1] seems sound to me.
That being said, the good thing is this restriction seems it might be
relaxed in the future to work per [1], without causing any backwards
compatibility issues.
Should we do something similar for row filters, though? It seems quite
weird we're so concerned about unexpected behavior due to combining
column lists (despite having a patch that makes it behave sanely), and
at the same time wave off similarly strange behavior due to combining
row filters because "that's what you get if you define the publications
in a strange way".
regards
[1]
https://www.postgresql.org/message-id/5a85b8b7-fc1c-364b-5c62-0bb3e1e25824%40enterprisedb.com
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:30 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-06 03:23 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-06 12:26 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-06 13:40 ` Amit Kapila <[email protected]>
2022-05-06 13:57 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-06 13:40 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: [email protected] <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Fri, May 6, 2022 at 5:56 PM Tomas Vondra
<[email protected]> wrote:
>
> >>
> >> I could think of below two options:
> >> 1. Forbid any case where column list is different for the same table
> >> when combining publications.
> >> 2. Forbid if the column list and row filters for a table are different
> >> in the set of publications we are planning to combine. This means we
> >> will allow combining column lists when row filters are not present or
> >> when column list is the same (we don't get anything additional by
> >> combining but the idea is we won't forbid such cases) and row filters
> >> are different.
> >>
> >> Now, I think the points in favor of (1) are that the main purpose of
> >> introducing a column list are: (a) the structure/schema of the
> >> subscriber is different from the publisher, (b) want to hide sensitive
> >> columns data. In both cases, it should be fine if we follow (1) and
> >> from Peter E.'s latest email [1] he also seems to be indicating the
> >> same. If we want to be slightly more relaxed then we can probably (2).
> >> We can decide on something else as well but I feel it should be such
> >> that it is easy to explain.
> >
> > I also think it makes sense to add a restriction like (1). I am planning to
> > implement the restriction if no one objects.
> >
>
> I'm not going to block that approach if that's the consensus here,
> though I'm not convinced.
>
> Let me point out (1) does *not* work for data redaction use case,
> certainly not the example Alvaro and me presented, because that relies
> on a combination of row filters and column filters.
>
This should just forbid the case presented by Alvaro in his first
email in this thread [1].
> Requiring all column
> lists to be the same (and not specific to row filter) prevents that
> example from working. Yes, you can create multiple subscriptions, but
> that brings it's own set of challenges too.
>
> I doubt forcing users to use the more complex setup is good idea, and
> combining the column lists per [1] seems sound to me.
>
> That being said, the good thing is this restriction seems it might be
> relaxed in the future to work per [1], without causing any backwards
> compatibility issues.
>
These are my thoughts as well. Even, if we decide to go via the column
list merging approach (in selective cases), we need to do some
performance testing of that approach as it does much more work per
tuple. It is possible that the impact is not much but still worth
evaluating, so let's try to see the patch to prohibit combining the
column lists then we can decide.
> Should we do something similar for row filters, though? It seems quite
> weird we're so concerned about unexpected behavior due to combining
> column lists (despite having a patch that makes it behave sanely), and
> at the same time wave off similarly strange behavior due to combining
> row filters because "that's what you get if you define the publications
> in a strange way".
>
During development, we found that we can't combine the row-filters for
'insert' and 'update'/'delete' because of replica identity
restrictions, so we have kept them separate. But if we came across
other such things then we can either try to fix those or forbid them.
[1] - https://www.postgresql.org/message-id/202204251548.mudq7jbqnh7r%40alvherre.pgsql
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:30 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-06 03:23 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-06 12:26 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-06 13:40 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-06 13:57 ` Tomas Vondra <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2022-05-06 13:57 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/6/22 15:40, Amit Kapila wrote:
> On Fri, May 6, 2022 at 5:56 PM Tomas Vondra
> <[email protected]> wrote:
>>
>>>>
>>>> I could think of below two options:
>>>> 1. Forbid any case where column list is different for the same table
>>>> when combining publications.
>>>> 2. Forbid if the column list and row filters for a table are different
>>>> in the set of publications we are planning to combine. This means we
>>>> will allow combining column lists when row filters are not present or
>>>> when column list is the same (we don't get anything additional by
>>>> combining but the idea is we won't forbid such cases) and row filters
>>>> are different.
>>>>
>>>> Now, I think the points in favor of (1) are that the main purpose of
>>>> introducing a column list are: (a) the structure/schema of the
>>>> subscriber is different from the publisher, (b) want to hide sensitive
>>>> columns data. In both cases, it should be fine if we follow (1) and
>>>> from Peter E.'s latest email [1] he also seems to be indicating the
>>>> same. If we want to be slightly more relaxed then we can probably (2).
>>>> We can decide on something else as well but I feel it should be such
>>>> that it is easy to explain.
>>>
>>> I also think it makes sense to add a restriction like (1). I am planning to
>>> implement the restriction if no one objects.
>>>
>>
>> I'm not going to block that approach if that's the consensus here,
>> though I'm not convinced.
>>
>> Let me point out (1) does *not* work for data redaction use case,
>> certainly not the example Alvaro and me presented, because that relies
>> on a combination of row filters and column filters.
>>
>
> This should just forbid the case presented by Alvaro in his first
> email in this thread [1].
>
>> Requiring all column
>> lists to be the same (and not specific to row filter) prevents that
>> example from working. Yes, you can create multiple subscriptions, but
>> that brings it's own set of challenges too.
>>
>> I doubt forcing users to use the more complex setup is good idea, and
>> combining the column lists per [1] seems sound to me.
>>
>> That being said, the good thing is this restriction seems it might be
>> relaxed in the future to work per [1], without causing any backwards
>> compatibility issues.
>>
>
> These are my thoughts as well. Even, if we decide to go via the column
> list merging approach (in selective cases), we need to do some
> performance testing of that approach as it does much more work per
> tuple. It is possible that the impact is not much but still worth
> evaluating, so let's try to see the patch to prohibit combining the
> column lists then we can decide.
>
Surely we could do some performance testing now. I doubt it's very
expensive - sure, you can construct cases with many row filters / column
lists, but how likely is that in practice?
Moreover, it's not like this would affect existing setups, so even if
it's a bit expensive, we may interpret that as cost of the feature.
>> Should we do something similar for row filters, though? It seems quite
>> weird we're so concerned about unexpected behavior due to combining
>> column lists (despite having a patch that makes it behave sanely), and
>> at the same time wave off similarly strange behavior due to combining
>> row filters because "that's what you get if you define the publications
>> in a strange way".
>>
>
> During development, we found that we can't combine the row-filters for
> 'insert' and 'update'/'delete' because of replica identity
> restrictions, so we have kept them separate. But if we came across
> other such things then we can either try to fix those or forbid them.
>
I understand how we got to the current state. I'm just saying that this
allows defining separate publications for insert, update and delete
actions, and set different row filters for each of them. Which results
in behavior that is hard to explain/understand, especially when it comes
to tablesync.
It seems quite strange to prohibit merging column lists because there
might be some strange behavior that no one described, and allow setups
with different row filters that definitely have strange behavior.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-03 03:53 ` Amit Kapila <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-03 03:53 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2022-May-02, Amit Kapila wrote:
>
> > I think it is possible to expose a list of publications for each
> > walsender as it is stored in each walsenders
> > LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
> > can have one such LogicalDecodingContext and we can probably share it
> > via shared memory?
>
> I guess we need to create a DSM each time a walsender opens a
> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
> connect to all DSMs of all running walsenders and see if they are
> reading from it. Is that what you have in mind?
>
Yes, something on these lines. We need a way to get the list of
publications each walsender is publishing data for.
> Alternatively, we
> could have one DSM per publication with a PID array of all walsenders
> that are sending it (each walsender needs to add its PID as it starts).
>
I think for this we need to check DSM for all the publications and I
feel in general publications should be more than the number of
walsenders, so the previous approach seems better to me. However, any
one of these or similar ideas should be okay.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-07 05:36 ` Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-07 05:36 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2022-May-02, Amit Kapila wrote:
>
>
> > I think it is possible to expose a list of publications for each
> > walsender as it is stored in each walsenders
> > LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
> > can have one such LogicalDecodingContext and we can probably share it
> > via shared memory?
>
> I guess we need to create a DSM each time a walsender opens a
> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
> connect to all DSMs of all running walsenders and see if they are
> reading from it. Is that what you have in mind? Alternatively, we
> could have one DSM per publication with a PID array of all walsenders
> that are sending it (each walsender needs to add its PID as it starts).
> The latter might be better.
>
While thinking about using DSM here, I came across one of your commits
f2f9fcb303 which seems to indicate that it is not a good idea to rely
on it but I think you have changed dynamic shared memory to fixed
shared memory usage because that was more suitable rather than DSM is
not portable. Because I see a commit bcbd940806 where we have removed
the 'none' option of dynamic_shared_memory_type. So, I think it should
be okay to use DSM in this context. What do you think?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-08 18:11 ` Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-08 18:11 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/7/22 07:36, Amit Kapila wrote:
> On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera <[email protected]> wrote:
>>
>> On 2022-May-02, Amit Kapila wrote:
>>
>>
>>> I think it is possible to expose a list of publications for each
>>> walsender as it is stored in each walsenders
>>> LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
>>> can have one such LogicalDecodingContext and we can probably share it
>>> via shared memory?
>>
>> I guess we need to create a DSM each time a walsender opens a
>> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
>> connect to all DSMs of all running walsenders and see if they are
>> reading from it. Is that what you have in mind? Alternatively, we
>> could have one DSM per publication with a PID array of all walsenders
>> that are sending it (each walsender needs to add its PID as it starts).
>> The latter might be better.
>>
>
> While thinking about using DSM here, I came across one of your commits
> f2f9fcb303 which seems to indicate that it is not a good idea to rely
> on it but I think you have changed dynamic shared memory to fixed
> shared memory usage because that was more suitable rather than DSM is
> not portable. Because I see a commit bcbd940806 where we have removed
> the 'none' option of dynamic_shared_memory_type. So, I think it should
> be okay to use DSM in this context. What do you think?
>
Why would any of this be needed?
ALTER PUBLICATION will invalidate the RelationSyncEntry entries in all
walsenders, no? So AFAICS it should be enough to enforce the limitations
in get_rel_sync_entry, which is necessary anyway because the subscriber
may not be connected when ALTER PUBLICATION gets executed.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-09 03:45 ` Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-09 03:45 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Sun, May 8, 2022 at 11:41 PM Tomas Vondra
<[email protected]> wrote:
>
> On 5/7/22 07:36, Amit Kapila wrote:
> > On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera <[email protected]> wrote:
> >>
> >> On 2022-May-02, Amit Kapila wrote:
> >>
> >>
> >>> I think it is possible to expose a list of publications for each
> >>> walsender as it is stored in each walsenders
> >>> LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
> >>> can have one such LogicalDecodingContext and we can probably share it
> >>> via shared memory?
> >>
> >> I guess we need to create a DSM each time a walsender opens a
> >> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
> >> connect to all DSMs of all running walsenders and see if they are
> >> reading from it. Is that what you have in mind? Alternatively, we
> >> could have one DSM per publication with a PID array of all walsenders
> >> that are sending it (each walsender needs to add its PID as it starts).
> >> The latter might be better.
> >>
> >
> > While thinking about using DSM here, I came across one of your commits
> > f2f9fcb303 which seems to indicate that it is not a good idea to rely
> > on it but I think you have changed dynamic shared memory to fixed
> > shared memory usage because that was more suitable rather than DSM is
> > not portable. Because I see a commit bcbd940806 where we have removed
> > the 'none' option of dynamic_shared_memory_type. So, I think it should
> > be okay to use DSM in this context. What do you think?
> >
>
> Why would any of this be needed?
>
> ALTER PUBLICATION will invalidate the RelationSyncEntry entries in all
> walsenders, no? So AFAICS it should be enough to enforce the limitations
> in get_rel_sync_entry,
>
Yes, that should be sufficient to enforce limitations in
get_rel_sync_entry() but it will lead to the following behavior:
a. The Alter Publication command will be successful but later in the
logs, the error will be logged and the user needs to check it and take
appropriate action. Till that time the walsender will be in an error
loop which means it will restart and again lead to the same error till
the user takes some action.
b. As we use historic snapshots, so even after the user takes action
say by changing publication, it won't be reflected. So, the option for
the user would be to drop their subscription.
Am, I missing something? If not, then are we okay with such behavior?
If yes, then I think it would be much easier implementation-wise and
probably advisable at this point. We can document it so that users are
careful and can take necessary action if they get into such a
situation. Any way we can improve this in future as you also suggested
earlier.
> which is necessary anyway because the subscriber
> may not be connected when ALTER PUBLICATION gets executed.
>
If we are not okay with the resultant behavior of detecting this in
get_rel_sync_entry(), then we can solve this in some other way as
Alvaro has indicated in one of his responses which is to detect that
at start replication time probably in the subscriber-side.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-10 19:05 ` Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2022-05-10 19:05 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/9/22 05:45, Amit Kapila wrote:
> On Sun, May 8, 2022 at 11:41 PM Tomas Vondra
> <[email protected]> wrote:
>>
>> On 5/7/22 07:36, Amit Kapila wrote:
>>> On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera <[email protected]> wrote:
>>>>
>>>> On 2022-May-02, Amit Kapila wrote:
>>>>
>>>>
>>>>> I think it is possible to expose a list of publications for each
>>>>> walsender as it is stored in each walsenders
>>>>> LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
>>>>> can have one such LogicalDecodingContext and we can probably share it
>>>>> via shared memory?
>>>>
>>>> I guess we need to create a DSM each time a walsender opens a
>>>> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
>>>> connect to all DSMs of all running walsenders and see if they are
>>>> reading from it. Is that what you have in mind? Alternatively, we
>>>> could have one DSM per publication with a PID array of all walsenders
>>>> that are sending it (each walsender needs to add its PID as it starts).
>>>> The latter might be better.
>>>>
>>>
>>> While thinking about using DSM here, I came across one of your commits
>>> f2f9fcb303 which seems to indicate that it is not a good idea to rely
>>> on it but I think you have changed dynamic shared memory to fixed
>>> shared memory usage because that was more suitable rather than DSM is
>>> not portable. Because I see a commit bcbd940806 where we have removed
>>> the 'none' option of dynamic_shared_memory_type. So, I think it should
>>> be okay to use DSM in this context. What do you think?
>>>
>>
>> Why would any of this be needed?
>>
>> ALTER PUBLICATION will invalidate the RelationSyncEntry entries in all
>> walsenders, no? So AFAICS it should be enough to enforce the limitations
>> in get_rel_sync_entry,
>>
>
> Yes, that should be sufficient to enforce limitations in
> get_rel_sync_entry() but it will lead to the following behavior:
> a. The Alter Publication command will be successful but later in the
> logs, the error will be logged and the user needs to check it and take
> appropriate action. Till that time the walsender will be in an error
> loop which means it will restart and again lead to the same error till
> the user takes some action.
> b. As we use historic snapshots, so even after the user takes action
> say by changing publication, it won't be reflected. So, the option for
> the user would be to drop their subscription.
>
> Am, I missing something? If not, then are we okay with such behavior?
> If yes, then I think it would be much easier implementation-wise and
> probably advisable at this point. We can document it so that users are
> careful and can take necessary action if they get into such a
> situation. Any way we can improve this in future as you also suggested
> earlier.
>
>> which is necessary anyway because the subscriber
>> may not be connected when ALTER PUBLICATION gets executed.
>>
>
> If we are not okay with the resultant behavior of detecting this in
> get_rel_sync_entry(), then we can solve this in some other way as
> Alvaro has indicated in one of his responses which is to detect that
> at start replication time probably in the subscriber-side.
>
IMO that behavior is acceptable. We have to do that check anyway, and
the subscription may start failing after ALTER PUBLICATION for a number
of other reasons anyway so the user needs/should check the logs.
And if needed, we can improve this and start doing the proactive-checks
during ALTER PUBLICATION too.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-11 03:33 ` Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-11 03:33 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Wed, May 11, 2022 at 12:35 AM Tomas Vondra
<[email protected]> wrote:
>
> On 5/9/22 05:45, Amit Kapila wrote:
> > On Sun, May 8, 2022 at 11:41 PM Tomas Vondra
> > <[email protected]> wrote:
> >>
> >> On 5/7/22 07:36, Amit Kapila wrote:
> >>> On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera <[email protected]> wrote:
> >>>>
> >>>> On 2022-May-02, Amit Kapila wrote:
> >>>>
> >>>>
> >>>>> I think it is possible to expose a list of publications for each
> >>>>> walsender as it is stored in each walsenders
> >>>>> LogicalDecodingContext->output_plugin_private. AFAIK, each walsender
> >>>>> can have one such LogicalDecodingContext and we can probably share it
> >>>>> via shared memory?
> >>>>
> >>>> I guess we need to create a DSM each time a walsender opens a
> >>>> connection, at START_REPLICATION time. Then ALTER PUBLICATION needs to
> >>>> connect to all DSMs of all running walsenders and see if they are
> >>>> reading from it. Is that what you have in mind? Alternatively, we
> >>>> could have one DSM per publication with a PID array of all walsenders
> >>>> that are sending it (each walsender needs to add its PID as it starts).
> >>>> The latter might be better.
> >>>>
> >>>
> >>> While thinking about using DSM here, I came across one of your commits
> >>> f2f9fcb303 which seems to indicate that it is not a good idea to rely
> >>> on it but I think you have changed dynamic shared memory to fixed
> >>> shared memory usage because that was more suitable rather than DSM is
> >>> not portable. Because I see a commit bcbd940806 where we have removed
> >>> the 'none' option of dynamic_shared_memory_type. So, I think it should
> >>> be okay to use DSM in this context. What do you think?
> >>>
> >>
> >> Why would any of this be needed?
> >>
> >> ALTER PUBLICATION will invalidate the RelationSyncEntry entries in all
> >> walsenders, no? So AFAICS it should be enough to enforce the limitations
> >> in get_rel_sync_entry,
> >>
> >
> > Yes, that should be sufficient to enforce limitations in
> > get_rel_sync_entry() but it will lead to the following behavior:
> > a. The Alter Publication command will be successful but later in the
> > logs, the error will be logged and the user needs to check it and take
> > appropriate action. Till that time the walsender will be in an error
> > loop which means it will restart and again lead to the same error till
> > the user takes some action.
> > b. As we use historic snapshots, so even after the user takes action
> > say by changing publication, it won't be reflected. So, the option for
> > the user would be to drop their subscription.
> >
> > Am, I missing something? If not, then are we okay with such behavior?
> > If yes, then I think it would be much easier implementation-wise and
> > probably advisable at this point. We can document it so that users are
> > careful and can take necessary action if they get into such a
> > situation. Any way we can improve this in future as you also suggested
> > earlier.
> >
> >> which is necessary anyway because the subscriber
> >> may not be connected when ALTER PUBLICATION gets executed.
> >>
> >
> > If we are not okay with the resultant behavior of detecting this in
> > get_rel_sync_entry(), then we can solve this in some other way as
> > Alvaro has indicated in one of his responses which is to detect that
> > at start replication time probably in the subscriber-side.
> >
>
> IMO that behavior is acceptable.
>
Fair enough, then we should go with a simpler approach to detect it in
pgoutput.c (get_rel_sync_entry).
> We have to do that check anyway, and
> the subscription may start failing after ALTER PUBLICATION for a number
> of other reasons anyway so the user needs/should check the logs.
>
I think ALTER PUBLICATION won't ever lead to failure in walsender.
Sure, users can do something due to which subscriber-side failures can
happen due to constraint failures. Do you have some specific cases in
mind?
> And if needed, we can improve this and start doing the proactive-checks
> during ALTER PUBLICATION too.
>
Agreed.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-11 07:25 ` [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: [email protected] @ 2022-05-11 07:25 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Wednesday, May 11, 2022 11:33 AM Amit Kapila <[email protected]> wrote:
> On Wed, May 11, 2022 at 12:35 AM Tomas Vondra
> <[email protected]> wrote:
> >
> > On 5/9/22 05:45, Amit Kapila wrote:
> > > On Sun, May 8, 2022 at 11:41 PM Tomas Vondra
> > > <[email protected]> wrote:
> > >>
> > >> On 5/7/22 07:36, Amit Kapila wrote:
> > >>> On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera
> <[email protected]> wrote:
> > >>>>
> > >>>> On 2022-May-02, Amit Kapila wrote:
> > >>>>
> > >>>>
> > >>>>> I think it is possible to expose a list of publications for each
> > >>>>> walsender as it is stored in each walsenders
> > >>>>> LogicalDecodingContext->output_plugin_private. AFAIK, each
> > >>>>> LogicalDecodingContext->walsender
> > >>>>> can have one such LogicalDecodingContext and we can probably
> > >>>>> share it via shared memory?
> > >>>>
> > >>>> I guess we need to create a DSM each time a walsender opens a
> > >>>> connection, at START_REPLICATION time. Then ALTER PUBLICATION
> > >>>> needs to connect to all DSMs of all running walsenders and see if
> > >>>> they are reading from it. Is that what you have in mind?
> > >>>> Alternatively, we could have one DSM per publication with a PID
> > >>>> array of all walsenders that are sending it (each walsender needs to
> add its PID as it starts).
> > >>>> The latter might be better.
> > >>>>
> > >>>
> > >>> While thinking about using DSM here, I came across one of your
> > >>> commits
> > >>> f2f9fcb303 which seems to indicate that it is not a good idea to
> > >>> rely on it but I think you have changed dynamic shared memory to
> > >>> fixed shared memory usage because that was more suitable rather
> > >>> than DSM is not portable. Because I see a commit bcbd940806 where
> > >>> we have removed the 'none' option of dynamic_shared_memory_type.
> > >>> So, I think it should be okay to use DSM in this context. What do you
> think?
> > >>>
> > >>
> > >> Why would any of this be needed?
> > >>
> > >> ALTER PUBLICATION will invalidate the RelationSyncEntry entries in
> > >> all walsenders, no? So AFAICS it should be enough to enforce the
> > >> limitations in get_rel_sync_entry,
> > >>
> > >
> > > Yes, that should be sufficient to enforce limitations in
> > > get_rel_sync_entry() but it will lead to the following behavior:
> > > a. The Alter Publication command will be successful but later in the
> > > logs, the error will be logged and the user needs to check it and
> > > take appropriate action. Till that time the walsender will be in an
> > > error loop which means it will restart and again lead to the same
> > > error till the user takes some action.
> > > b. As we use historic snapshots, so even after the user takes action
> > > say by changing publication, it won't be reflected. So, the option
> > > for the user would be to drop their subscription.
> > >
> > > Am, I missing something? If not, then are we okay with such behavior?
> > > If yes, then I think it would be much easier implementation-wise and
> > > probably advisable at this point. We can document it so that users
> > > are careful and can take necessary action if they get into such a
> > > situation. Any way we can improve this in future as you also
> > > suggested earlier.
> > >
> > >> which is necessary anyway because the subscriber may not be
> > >> connected when ALTER PUBLICATION gets executed.
> > >>
> > >
> > > If we are not okay with the resultant behavior of detecting this in
> > > get_rel_sync_entry(), then we can solve this in some other way as
> > > Alvaro has indicated in one of his responses which is to detect that
> > > at start replication time probably in the subscriber-side.
> > >
> >
> > IMO that behavior is acceptable.
> >
>
> Fair enough, then we should go with a simpler approach to detect it in
> pgoutput.c (get_rel_sync_entry).
OK, here is the patch that try to check column list in that way. The patch also
check the column list when CREATE SUBSCRIPTION and when starting initial copy.
Best regards,
Hou zj
Attachments:
[application/octet-stream] 0001-Disallow-combining-publication-when-column-list-is-d.patch (20.1K, ../../OS0PR01MB5716A594C58DE4FFD1F8100B94C89@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-0001-Disallow-combining-publication-when-column-list-is-d.patch)
download | inline diff:
From f7f68fd11211ffbe5770fb565ac80620dd6ccea9 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Wed, 11 May 2022 14:29:05 +0800
Subject: [PATCH] Disallow combining publication when column list is different
for the same table
the main purpose of introducing a column list are statically have different
shapes on publisher and subscriber or hide sensitive columns data. In both
cases, it doesn't seems make sense to combine column lists. So disallow the
cases where column list is different for the same table when combining
publications.
---
src/backend/commands/subscriptioncmds.c | 37 +++++++--
src/backend/replication/logical/tablesync.c | 61 +++++++++-----
src/backend/replication/pgoutput/pgoutput.c | 78 ++++++++---------
src/test/subscription/t/031_column_list.pl | 124 +++++++++-------------------
4 files changed, 154 insertions(+), 146 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b94236f..8650cc5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1753,7 +1753,8 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
- * publisher connection.
+ * publisher connection. Also get the column list for each table and check if
+ * column lists are the same in different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1762,34 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, INT2VECTOROID};
List *tablelist = NIL;
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
- " FROM pg_catalog.pg_publication_tables t\n"
+ appendStringInfoString(&cmd,
+ "SELECT DISTINCT t.schemaname,\n"
+ " t.tablename,\n"
+ " (CASE WHEN (array_length(pr.prattrs, 1) = t.relnatts)\n"
+ " THEN NULL ELSE pr.prattrs END)\n"
+ " FROM (SELECT P.pubname AS pubname,\n"
+ " N.nspname AS schemaname,\n"
+ " C.relname AS tablename,\n"
+ " P.oid AS pubid,\n"
+ " C.oid AS reloid,\n"
+ " C.relnatts\n"
+ " FROM pg_publication P,\n"
+ " LATERAL pg_get_publication_tables(P.pubname) GPT,\n"
+ " pg_class C JOIN pg_namespace N\n"
+ " ON (N.oid = C.relnamespace)\n"
+ " WHERE C.oid = GPT.relid) t\n"
+ " LEFT OUTER JOIN pg_publication_rel pr\n"
+ " ON (t.pubid = pr.prpubid AND\n"
+ " pr.prrelid = reloid)\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, 3, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1813,14 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ if (list_member(tablelist, rv))
+ ereport(WARNING,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 49ceec3..8ac4171 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -771,7 +771,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,21 +786,20 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that
- * makes it easier to combine lists by simply adding the attnums
- * to a new bitmap (without having to parse the int2vector data).
- * This preserves NULL values, so that if one of the publications
- * has no column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(pr.prattrs, 1) = c.relnatts)"
+ " THEN NULL ELSE pr.prattrs END)"
" FROM pg_publication p"
" LEFT OUTER JOIN pg_publication_rel pr"
" ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
" LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
lrel->remoteid,
@@ -815,27 +814,49 @@ fetch_remote_table_info(char *nspname, char *relname,
errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
nspname, relname, pubres->err)));
+ first = true;
+
/*
- * Merge the column lists (from different publications) by creating
- * a single bitmap with all the attnums. If we find a NULL value,
- * that means one of the publications has no column list for the
- * table we're syncing.
+ * Traverse the column lists from different publications and build a
+ * single bitmap with the attnums.
+ *
+ * During the loop, check that if all the column lists are the same and
+ * report an error if not.
+ *
+ * If we find a NULL value, that means one of the publications has no
+ * column list for the table we're syncing.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
+ Bitmapset *cols = NULL;
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ cols = bms_add_member(cols, elems[i]);
}
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ /* NULL means empty column list. */
+ if (first)
+ {
+ included_cols = cols;
+ first = false;
+ }
+ else if (!bms_equal(included_cols, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index b197bfd..7ca09de 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -803,16 +803,13 @@ pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
* Make sure the per-entry memory context exists.
*/
static void
-pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
+pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry,
+ Relation relation)
{
- Relation relation;
-
/* The context may already exist, in which case bail out. */
if (entry->entry_cxt)
return;
- relation = RelationIdGetRelation(entry->publish_as_relid);
-
entry->entry_cxt = AllocSetContextCreate(data->cachectx,
"entry private context",
ALLOCSET_SMALL_SIZES);
@@ -941,7 +938,7 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- pgoutput_ensure_entry_cxt(data, entry);
+ pgoutput_ensure_entry_cxt(data, entry, relation);
/*
* Now all the filters for all pubactions are known. Combine them when
@@ -978,14 +975,20 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
+ * build a bitmap using the column lists.
*
- * All the given publication-table mappings must be checked.
+ * Note that we don't support the case where column list is different for
+ * the same table when combining publications. But we still need to check
+ * all the given publication-table mappings and report an error if any
+ * publications have different column list.
*
- * Multiple publications might have multiple column lists for this relation.
+ * Multiple publications might have multiple column lists for this
+ * relation.
*
* FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
* list" so it takes precedence.
@@ -995,12 +998,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as if
@@ -1008,6 +1006,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1033,39 +1033,43 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
/*
* Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
*/
if (!pub_no_list) /* when not null */
{
- pgoutput_ensure_entry_cxt(data, entry);
+ pgoutput_ensure_entry_cxt(data, entry, relation);
+
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index bdcf3e4..3dfe7d1 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -21,6 +21,8 @@ $node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $offset = 0;
+
sub wait_for_subscription_sync
{
my ($node) = @_;
@@ -334,12 +336,12 @@ is($result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql('postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -358,8 +360,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure all the columns get fully replicated
$node_publisher->safe_psql('postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
INSERT INTO tab5 VALUES (4, 44, 444, 4444);
@@ -368,39 +369,11 @@ $node_publisher->safe_psql('postgres', qq(
$node_publisher->wait_for_catchup('sub1');
is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
- 'overlapping publications with overlapping column lists');
-
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql('postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql('postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql('postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
+ 'insert on column tab5.d is not replicated');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -822,51 +795,18 @@ is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a,
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-
-$node_publisher->safe_psql('postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql('postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql('postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications is FOR ALL TABLES, it means
+# replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql('postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql('postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -890,12 +830,11 @@ $node_publisher->wait_for_catchup('sub1');
is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES IN
+# SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql('postgres', qq(
DROP SUBSCRIPTION sub1;
@@ -905,7 +844,7 @@ $node_subscriber->safe_psql('postgres', qq(
$node_publisher->safe_psql('postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -927,8 +866,7 @@ $node_publisher->wait_for_catchup('sub1');
is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
-
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
# published through partition root, we should only apply the column list
@@ -979,7 +917,7 @@ is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql('postgres', qq(
@@ -990,7 +928,7 @@ $node_publisher->safe_psql('postgres', qq(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1124,6 +1062,26 @@ is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
4||),
'publication containing both parent and child relation');
+# TEST: With a table included in multiple publications with different column
+# lists, we should catch the error in the log
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2 WITH (copy_data = false);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+my $logfile = slurp_file($node_subscriber->logfile, $offset);
+ok( $logfile =~
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ 'different column lists detected');
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
2.7.2.windows.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-12 06:45 ` Amit Kapila <[email protected]>
2022-05-12 08:32 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-12 06:45 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Wed, May 11, 2022 at 12:55 PM [email protected]
<[email protected]> wrote:
>
> On Wednesday, May 11, 2022 11:33 AM Amit Kapila <[email protected]> wrote:
> >
> > Fair enough, then we should go with a simpler approach to detect it in
> > pgoutput.c (get_rel_sync_entry).
>
> OK, here is the patch that try to check column list in that way. The patch also
> check the column list when CREATE SUBSCRIPTION and when starting initial copy.
>
Few comments:
===============
1.
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
- " FROM pg_catalog.pg_publication_tables t\n"
+ appendStringInfoString(&cmd,
+ "SELECT DISTINCT t.schemaname,\n"
+ " t.tablename,\n"
+ " (CASE WHEN (array_length(pr.prattrs, 1) = t.relnatts)\n"
+ " THEN NULL ELSE pr.prattrs END)\n"
+ " FROM (SELECT P.pubname AS pubname,\n"
+ " N.nspname AS schemaname,\n"
+ " C.relname AS tablename,\n"
+ " P.oid AS pubid,\n"
+ " C.oid AS reloid,\n"
+ " C.relnatts\n"
+ " FROM pg_publication P,\n"
+ " LATERAL pg_get_publication_tables(P.pubname) GPT,\n"
+ " pg_class C JOIN pg_namespace N\n"
+ " ON (N.oid = C.relnamespace)\n"
+ " WHERE C.oid = GPT.relid) t\n"
+ " LEFT OUTER JOIN pg_publication_rel pr\n"
+ " ON (t.pubid = pr.prpubid AND\n"
+ " pr.prrelid = reloid)\n"
Can we modify pg_publication_tables to get the row filter and column
list and then use it directly instead of constructing this query?
2.
+ if (list_member(tablelist, rv))
+ ereport(WARNING,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in
different publications",
+ nspname, relname));
+ else
Can we write comments to explain why we are using WARNING here instead of ERROR?
3.
static void
-pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
+pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry,
+ Relation relation)
What is the need to change this interface as part of this patch?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-12 08:32 ` Amit Kapila <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-12 08:32 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Thu, May 12, 2022 at 12:15 PM Amit Kapila <[email protected]> wrote:
>
> On Wed, May 11, 2022 at 12:55 PM [email protected]
> <[email protected]> wrote:
> >
> > On Wednesday, May 11, 2022 11:33 AM Amit Kapila <[email protected]> wrote:
> > >
> > > Fair enough, then we should go with a simpler approach to detect it in
> > > pgoutput.c (get_rel_sync_entry).
> >
> > OK, here is the patch that try to check column list in that way. The patch also
> > check the column list when CREATE SUBSCRIPTION and when starting initial copy.
> >
>
> Few comments:
> ===============
...
One more point, I think we should update the docs for this.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-13 06:02 ` [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: [email protected] @ 2022-05-13 06:02 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Thursday, May 12, 2022 2:45 PM Amit Kapila <[email protected]> wrote:
>
> On Wed, May 11, 2022 at 12:55 PM [email protected]
> <[email protected]> wrote:
> >
> > On Wednesday, May 11, 2022 11:33 AM Amit Kapila
> <[email protected]> wrote:
> > >
> > > Fair enough, then we should go with a simpler approach to detect it
> > > in pgoutput.c (get_rel_sync_entry).
> >
> > OK, here is the patch that try to check column list in that way. The
> > patch also check the column list when CREATE SUBSCRIPTION and when
> starting initial copy.
> >
>
> Few comments:
> ===============
> 1.
> initStringInfo(&cmd);
> - appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname,
> t.tablename\n"
> - " FROM pg_catalog.pg_publication_tables t\n"
> + appendStringInfoString(&cmd,
> + "SELECT DISTINCT t.schemaname,\n"
> + " t.tablename,\n"
> + " (CASE WHEN (array_length(pr.prattrs, 1) = t.relnatts)\n"
> + " THEN NULL ELSE pr.prattrs END)\n"
> + " FROM (SELECT P.pubname AS pubname,\n"
> + " N.nspname AS schemaname,\n"
> + " C.relname AS tablename,\n"
> + " P.oid AS pubid,\n"
> + " C.oid AS reloid,\n"
> + " C.relnatts\n"
> + " FROM pg_publication P,\n"
> + " LATERAL pg_get_publication_tables(P.pubname) GPT,\n"
> + " pg_class C JOIN pg_namespace N\n"
> + " ON (N.oid = C.relnamespace)\n"
> + " WHERE C.oid = GPT.relid) t\n"
> + " LEFT OUTER JOIN pg_publication_rel pr\n"
> + " ON (t.pubid = pr.prpubid AND\n"
> + " pr.prrelid = reloid)\n"
>
> Can we modify pg_publication_tables to get the row filter and column list and
> then use it directly instead of constructing this query?
Agreed. If we can get columnlist and rowfilter from pg_publication_tables, it
will be more convenient. And I think users that want to fetch the columnlist
and rowfilter of table can also benefit from it.
> 2.
> + if (list_member(tablelist, rv))
> + ereport(WARNING,
> + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("cannot use different column lists for table \"%s.%s\" in
> different publications",
> + nspname, relname));
> + else
>
> Can we write comments to explain why we are using WARNING here instead of
> ERROR?
>
> 3.
> static void
> -pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
> +pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry,
> + Relation relation)
>
> What is the need to change this interface as part of this patch?
Attach the new version patch which addressed these comments and update the
document. 0001 patch is to extent the view and 0002 patch is to add restriction
for column list.
Best regards,
Hou zj
Attachments:
[application/octet-stream] 0002-Disallow-combining-publication-when-column-list-is-d.patch (18.2K, ../../OS0PR01MB571698DEFCE9ACC6DC40510794CA9@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-0002-Disallow-combining-publication-when-column-list-is-d.patch)
download | inline diff:
From e1fb8c867353262a43419ee1892bbd64f4d613d7 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 13:19:49 +0800
Subject: [PATCH] Disallow combining publication when column list is different
for the same table
the main purpose of introducing a column list are statically have different
shapes on publisher and subscriber or hide sensitive columns data. In both
cases, it doesn't seems make sense to combine column lists. So disallow the
cases where column list is different for the same table when combining
publications.
---
src/backend/commands/subscriptioncmds.c | 24 ++++--
src/backend/replication/logical/tablesync.c | 61 ++++++++-----
src/backend/replication/pgoutput/pgoutput.c | 75 ++++++++--------
src/test/subscription/t/031_column_list.pl | 127 +++++++++-------------------
4 files changed, 137 insertions(+), 150 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 690cdaa..c19eb81 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1753,7 +1753,8 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
- * publisher connection.
+ * publisher connection. Also get the column list for each table and check if
+ * column lists are the same in different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1762,18 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, INT2VECTOROID};
List *tablelist = NIL;
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename, \n"
+ " t.columnlist\n"
" FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, 3, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1797,19 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ /*
+ * We only throw a warning here so that the subcription can still be
+ * created and let user aware that something is going to fail later and
+ * they can fix the publications afterwards.
+ */
+ if (list_member(tablelist, rv))
+ ereport(WARNING,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 994c7a0..42de832 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -771,7 +771,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,19 +786,17 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that makes
- * it easier to combine lists by simply adding the attnums to a new
- * bitmap (without having to parse the int2vector data). This
- * preserves NULL values, so that if one of the publications has no
- * column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
+ " THEN NULL ELSE gpt.attrs END)"
" FROM pg_publication p,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
pub_names.data);
@@ -812,27 +810,48 @@ fetch_remote_table_info(char *nspname, char *relname,
errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
nspname, relname, pubres->err)));
+ first = true;
+
/*
- * Merge the column lists (from different publications) by creating a
- * single bitmap with all the attnums. If we find a NULL value, that
- * means one of the publications has no column list for the table
- * we're syncing.
+ * Traverse the column lists from different publications and build a
+ * single bitmap with the attnums.
+ *
+ * During the loop, check that if all the column lists are the same and
+ * report an error if not.
+ *
+ * If we find a NULL value, that means one of the publications has no
+ * column list for the table we're syncing.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
+ Bitmapset *cols = NULL;
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ cols = bms_add_member(cols, elems[i]);
}
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ if (first)
+ {
+ included_cols = cols;
+ first = false;
+ }
+ else if (!bms_equal(included_cols, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 42c06af..4f58df2 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -979,12 +979,17 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
+ * build a bitmap using the column lists.
*
- * All the given publication-table mappings must be checked.
+ * Note that we don't support the case where column list is different for
+ * the same table when combining publications. But we still need to check
+ * all the given publication-table mappings and report an error if any
+ * publications have different column list.
*
* Multiple publications might have multiple column lists for this
* relation.
@@ -997,12 +1002,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as
@@ -1011,6 +1011,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1024,51 +1026,48 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (HeapTupleIsValid(cftuple))
{
- /*
- * Lookup the column list attribute.
- *
- * Note: We update the pub_no_list value directly, because if
- * the value is NULL, we have no list (and vice versa).
- */
+ /* Lookup the column list attribute. */
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
- /*
- * Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
- */
+ /* Build the column list bitmap in the per-entry context. */
if (!pub_no_list) /* when not null */
{
pgoutput_ensure_entry_cxt(data, entry);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
+
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 19812e1..b454a05 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -21,6 +21,8 @@ $node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $offset = 0;
+
sub wait_for_subscription_sync
{
my ($node) = @_;
@@ -361,13 +363,13 @@ is( $result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -388,8 +390,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure the columns in column list get fully replicated
$node_publisher->safe_psql(
'postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
@@ -399,42 +400,12 @@ $node_publisher->safe_psql(
$node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
'overlapping publications with overlapping column lists');
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql(
- 'postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -900,57 +871,21 @@ is( $node_subscriber->safe_psql(
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-$node_publisher->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql(
- 'postgres', "SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications is FOR ALL TABLES, it
+# means replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql(
'postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -976,12 +911,11 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES
+# IN SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql(
'postgres', qq(
@@ -993,7 +927,7 @@ $node_publisher->safe_psql(
'postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -1017,7 +951,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
@@ -1074,7 +1008,7 @@ is( $node_subscriber->safe_psql(
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (with all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql(
@@ -1086,7 +1020,7 @@ $node_publisher->safe_psql(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1233,6 +1167,27 @@ is( $node_subscriber->safe_psql(
'publication containing both parent and child relation');
+# TEST: With a table included in multiple publications with different column
+# lists, we should catch the error in the log
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2 WITH (copy_data = false);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+my $logfile = slurp_file($node_subscriber->logfile, $offset);
+ok( $logfile =~
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ 'different column lists detected');
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
2.7.2.windows.1
[application/octet-stream] 0001-extent-pg_publication_tables.patch (12.6K, ../../OS0PR01MB571698DEFCE9ACC6DC40510794CA9@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-0001-extent-pg_publication_tables.patch)
download | inline diff:
From 16d36b556d066480cda4dd170e13183aaf9d8173 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 11:35:14 +0800
Subject: [PATCH] extent pg_publication_tables
Extend the pg_publication_tables view and pg_get_publication_tables function
so that they can return the column list and row fitler of the table. It will
make it easier for users and developers to fetch the column lists and row filters.
---
doc/src/sgml/catalogs.sgml | 20 +++++++++++
src/backend/catalog/pg_publication.c | 52 ++++++++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 4 ++-
src/backend/replication/logical/tablesync.c | 14 +++-----
src/include/catalog/pg_proc.dat | 6 ++--
src/test/regress/expected/publication.out | 42 +++++++++++------------
src/test/regress/expected/rules.out | 6 ++--
7 files changed, 106 insertions(+), 38 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a533a21..fd0c61b 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -11687,6 +11687,26 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
Name of table
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>columnlist</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>.<structfield>prattrs</structfield>)
+ </para>
+ <para>
+ Column list of table
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rowfilter</structfield> <type>text</type>
+ (references <link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>.<structfield>prqual</structfield>)
+ </para>
+ <para>
+ Row filter of table
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index e2c8bcb..3fe607e 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1082,6 +1082,7 @@ get_publication_name(Oid pubid, bool missing_ok)
Datum
pg_get_publication_tables(PG_FUNCTION_ARGS)
{
+#define NUM_PUBLICATOIN_TABLES_ELEM 3
FuncCallContext *funcctx;
char *pubname = text_to_cstring(PG_GETARG_TEXT_PP(0));
Publication *publication;
@@ -1090,6 +1091,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
/* stuff done only on the first call of the function */
if (SRF_IS_FIRSTCALL())
{
+ TupleDesc tupdesc;
MemoryContext oldcontext;
/* create a function context for cross-call persistence */
@@ -1136,6 +1138,16 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
tables = filter_partitions(tables);
}
+ /* Construct a tuple descriptor for the result rows. */
+ tupdesc = CreateTemplateTupleDesc(NUM_PUBLICATOIN_TABLES_ELEM);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "relid",
+ OIDOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "attrs",
+ INT2VECTOROID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "qual",
+ PG_NODE_TREEOID, -1, 0);
+
+ funcctx->tuple_desc = BlessTupleDesc(tupdesc);
funcctx->user_fctx = (void *) tables;
MemoryContextSwitchTo(oldcontext);
@@ -1147,9 +1159,47 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (funcctx->call_cntr < list_length(tables))
{
+ HeapTuple pubtuple = NULL;
+ HeapTuple rettuple;
Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+ Datum values[NUM_PUBLICATOIN_TABLES_ELEM];
+ bool nulls[NUM_PUBLICATOIN_TABLES_ELEM];
+
+ /*
+ * Form tuple with appropriate data.
+ */
+ MemSet(nulls, 0, sizeof(nulls));
+ MemSet(values, 0, sizeof(values));
+
+ publication = GetPublicationByName(pubname, false);
+
+ values[0] = ObjectIdGetDatum(relid);
+
+ pubtuple = SearchSysCacheCopy2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(publication->oid));
+
+ if (HeapTupleIsValid(pubtuple))
+ {
+ /* Lookup the column list attribute. */
+ values[1] = SysCacheGetAttr(PUBLICATIONRELMAP, pubtuple,
+ Anum_pg_publication_rel_prattrs,
+ &(nulls[1]));
+
+ /* Null indicates no filter. */
+ values[2] = SysCacheGetAttr(PUBLICATIONRELMAP, pubtuple,
+ Anum_pg_publication_rel_prqual,
+ &(nulls[2]));
+ }
+ else
+ {
+ nulls[1] = true;
+ nulls[2] = true;
+ }
+
+ rettuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
- SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(rettuple));
}
SRF_RETURN_DONE(funcctx);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0fc614e..f758f15 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -368,7 +368,9 @@ CREATE VIEW pg_publication_tables AS
SELECT
P.pubname AS pubname,
N.nspname AS schemaname,
- C.relname AS tablename
+ C.relname AS tablename,
+ GPT.attrs AS columnlist,
+ pg_get_expr(GPT.qual, GPT.relid) AS rowfilter
FROM pg_publication P,
LATERAL pg_get_publication_tables(P.pubname) GPT,
pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b03e0f5..994c7a0 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -795,15 +795,12 @@ fetch_remote_table_info(char *nspname, char *relname,
resetStringInfo(&cmd);
appendStringInfo(&cmd,
"SELECT DISTINCT unnest"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
- " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " FROM pg_publication p,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
" WHERE gpt.relid = %u"
" AND p.pubname IN ( %s )",
lrel->remoteid,
- lrel->remoteid,
pub_names.data);
pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
@@ -965,15 +962,12 @@ fetch_remote_table_info(char *nspname, char *relname,
/* Check for row filters. */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT pg_get_expr(pr.prqual, pr.prrelid)"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ "SELECT DISTINCT pg_get_expr(gpt.qual, gpt.relid)"
+ " FROM pg_publication p,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
" WHERE gpt.relid = %u"
" AND p.pubname IN ( %s )",
lrel->remoteid,
- lrel->remoteid,
pub_names.data);
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 1, qualRow);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f..cf6235d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11675,9 +11675,9 @@
# publications
{ oid => '6119', descr => 'get OIDs of tables in a publication',
proname => 'pg_get_publication_tables', prorows => '1000', proretset => 't',
- provolatile => 's', prorettype => 'oid', proargtypes => 'text',
- proallargtypes => '{text,oid}', proargmodes => '{i,o}',
- proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_tables' },
+ provolatile => 's', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes => '{i,o,o,o}',
+ proargnames => '{pubname,relid,attrs,qual}', prosrc => 'pg_get_publication_tables' },
{ oid => '6121',
descr => 'returns whether a relation can be part of a publication',
proname => 'pg_relation_is_publishable', provolatile => 's',
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 398c0f3..4ebafb3 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -1585,52 +1585,52 @@ CREATE TABLE sch2.tbl1_part1 PARTITION OF sch1.tbl1 FOR VALUES FROM (1) to (10);
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+------------+------------+-----------
+ pub | sch2 | tbl1_part1 | |
(1 row)
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+------------+------------+-----------
+ pub | sch2 | tbl1_part1 | |
(1 row)
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+-----------
- pub | sch1 | tbl1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+-----------+------------+-----------
+ pub | sch1 | tbl1 | |
(1 row)
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+------------+------------+-----------
+ pub | sch2 | tbl1_part1 | |
(1 row)
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+------------+------------+-----------
+ pub | sch2 | tbl1_part1 | |
(1 row)
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+------------+------------+-----------
+ pub | sch2 | tbl1_part1 | |
(1 row)
DROP PUBLICATION pub;
@@ -1643,9 +1643,9 @@ CREATE TABLE sch1.tbl1_part3 (a int) PARTITION BY RANGE(a);
ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (30);
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+-----------
- pub | sch1 | tbl1
+ pubname | schemaname | tablename | columnlist | rowfilter
+---------+------------+-----------+------------+-----------
+ pub | sch1 | tbl1 | |
(1 row)
RESET client_min_messages;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8..c7f8300 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1437,9 +1437,11 @@ pg_prepared_xacts| SELECT p.transaction,
LEFT JOIN pg_database d ON ((p.dbid = d.oid)));
pg_publication_tables| SELECT p.pubname,
n.nspname AS schemaname,
- c.relname AS tablename
+ c.relname AS tablename,
+ gpt.attrs AS columnlist,
+ pg_get_expr(gpt.qual, gpt.relid) AS rowfilter
FROM pg_publication p,
- LATERAL pg_get_publication_tables((p.pubname)::text) gpt(relid),
+ LATERAL pg_get_publication_tables((p.pubname)::text) gpt(relid, attrs, qual),
(pg_class c
JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
WHERE (c.oid = gpt.relid);
--
2.7.2.windows.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-16 06:10 ` Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-16 06:10 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Fri, May 13, 2022 at 11:32 AM [email protected]
<[email protected]> wrote:
>
> On Thursday, May 12, 2022 2:45 PM Amit Kapila <[email protected]> wrote:
> >
> > On Wed, May 11, 2022 at 12:55 PM [email protected]
> > <[email protected]> wrote:
> >
> > Few comments:
> > ===============
> > 1.
> > initStringInfo(&cmd);
> > - appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname,
> > t.tablename\n"
> > - " FROM pg_catalog.pg_publication_tables t\n"
> > + appendStringInfoString(&cmd,
> > + "SELECT DISTINCT t.schemaname,\n"
> > + " t.tablename,\n"
> > + " (CASE WHEN (array_length(pr.prattrs, 1) = t.relnatts)\n"
> > + " THEN NULL ELSE pr.prattrs END)\n"
> > + " FROM (SELECT P.pubname AS pubname,\n"
> > + " N.nspname AS schemaname,\n"
> > + " C.relname AS tablename,\n"
> > + " P.oid AS pubid,\n"
> > + " C.oid AS reloid,\n"
> > + " C.relnatts\n"
> > + " FROM pg_publication P,\n"
> > + " LATERAL pg_get_publication_tables(P.pubname) GPT,\n"
> > + " pg_class C JOIN pg_namespace N\n"
> > + " ON (N.oid = C.relnamespace)\n"
> > + " WHERE C.oid = GPT.relid) t\n"
> > + " LEFT OUTER JOIN pg_publication_rel pr\n"
> > + " ON (t.pubid = pr.prpubid AND\n"
> > + " pr.prrelid = reloid)\n"
> >
> > Can we modify pg_publication_tables to get the row filter and column list and
> > then use it directly instead of constructing this query?
>
> Agreed. If we can get columnlist and rowfilter from pg_publication_tables, it
> will be more convenient. And I think users that want to fetch the columnlist
> and rowfilter of table can also benefit from it.
>
After the change for this, we will give an error on combining
publications where one of the publications specifies all columns in
the table and the other doesn't provide any columns. We should not
give an error as both mean all columns.
>
> Attach the new version patch which addressed these comments and update the
> document. 0001 patch is to extent the view and 0002 patch is to add restriction
> for column list.
>
Few comments:
=================
1.
postgres=# select * from pg_publication_tables;
pubname | schemaname | tablename | columnlist | rowfilter
---------+------------+-----------+------------+-----------
pub1 | public | t1 | |
pub2 | public | t1 | 1 2 | (c3 < 10)
(2 rows)
I think it is better to display column names for columnlist in the
exposed view similar to attnames in the pg_stats_ext view. I think
that will make it easier for users to understand this information.
2.
{ oid => '6119', descr => 'get OIDs of tables in a publication',
proname => 'pg_get_publication_tables', prorows => '1000', proretset => 't',
- provolatile => 's', prorettype => 'oid', proargtypes => 'text',
- proallargtypes => '{text,oid}', proargmodes => '{i,o}',
- proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_tables' },
+ provolatile => 's', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes
=> '{i,o,o,o}',
I think we should change the "descr" to something like: 'get
information of tables in a publication'
3.
+
+ /*
+ * We only throw a warning here so that the subcription can still be
+ * created and let user aware that something is going to fail later and
+ * they can fix the publications afterwards.
+ */
+ if (list_member(tablelist, rv))
+ ereport(WARNING,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in
different publications",
+ nspname, relname));
Can we extend this comment to explain the case where after Alter
Publication, if the user dumps and restores back the subscription,
there is a possibility that "CREATE SUBSCRIPTION" won't work if we
give ERROR here instead of WARNING?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-16 12:34 ` [email protected] <[email protected]>
2022-05-17 03:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 06:49 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 06:52 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
1 sibling, 3 replies; 59+ messages in thread
From: [email protected] @ 2022-05-16 12:34 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Monday, May 16, 2022 2:10 PM Amit Kapila <[email protected]>
>
> On Fri, May 13, 2022 at 11:32 AM [email protected]
> <[email protected]> wrote:
> >
> > On Thursday, May 12, 2022 2:45 PM Amit Kapila <[email protected]>
> wrote:
> > >
> > > On Wed, May 11, 2022 at 12:55 PM [email protected]
> > > <[email protected]> wrote:
> > >
> > > Few comments:
> > > ===============
> > > 1.
> > > initStringInfo(&cmd);
> > > - appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname,
> > > t.tablename\n"
> > > - " FROM pg_catalog.pg_publication_tables t\n"
> > > + appendStringInfoString(&cmd,
> > > + "SELECT DISTINCT t.schemaname,\n"
> > > + " t.tablename,\n"
> > > + " (CASE WHEN (array_length(pr.prattrs, 1) =
> t.relnatts)\n"
> > > + " THEN NULL ELSE pr.prattrs END)\n"
> > > + " FROM (SELECT P.pubname AS pubname,\n"
> > > + " N.nspname AS schemaname,\n"
> > > + " C.relname AS tablename,\n"
> > > + " P.oid AS pubid,\n"
> > > + " C.oid AS reloid,\n"
> > > + " C.relnatts\n"
> > > + " FROM pg_publication P,\n"
> > > + " LATERAL pg_get_publication_tables(P.pubname) GPT,\n"
> > > + " pg_class C JOIN pg_namespace N\n"
> > > + " ON (N.oid = C.relnamespace)\n"
> > > + " WHERE C.oid = GPT.relid) t\n"
> > > + " LEFT OUTER JOIN pg_publication_rel pr\n"
> > > + " ON (t.pubid = pr.prpubid AND\n"
> > > + " pr.prrelid = reloid)\n"
> > >
> > > Can we modify pg_publication_tables to get the row filter and column list
> and
> > > then use it directly instead of constructing this query?
> >
> > Agreed. If we can get columnlist and rowfilter from pg_publication_tables, it
> > will be more convenient. And I think users that want to fetch the columnlist
> > and rowfilter of table can also benefit from it.
> >
>
> After the change for this, we will give an error on combining
> publications where one of the publications specifies all columns in
> the table and the other doesn't provide any columns. We should not
> give an error as both mean all columns.
Thanks for the comments. Fixed.
> >
> > Attach the new version patch which addressed these comments and update
> the
> > document. 0001 patch is to extent the view and 0002 patch is to add
> restriction
> > for column list.
> >
>
> Few comments:
> =================
> 1.
> postgres=# select * from pg_publication_tables;
> pubname | schemaname | tablename | columnlist | rowfilter
> ---------+------------+-----------+------------+-----------
> pub1 | public | t1 | |
> pub2 | public | t1 | 1 2 | (c3 < 10)
> (2 rows)
>
> I think it is better to display column names for columnlist in the
> exposed view similar to attnames in the pg_stats_ext view. I think
> that will make it easier for users to understand this information.
Agreed and changed.
> 2.
> { oid => '6119', descr => 'get OIDs of tables in a publication',
> proname => 'pg_get_publication_tables', prorows => '1000', proretset =>
> 't',
> - provolatile => 's', prorettype => 'oid', proargtypes => 'text',
> - proallargtypes => '{text,oid}', proargmodes => '{i,o}',
> - proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_tables' },
> + provolatile => 's', prorettype => 'record', proargtypes => 'text',
> + proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes
> => '{i,o,o,o}',
>
> I think we should change the "descr" to something like: 'get
> information of tables in a publication'
Changed.
> 3.
> +
> + /*
> + * We only throw a warning here so that the subcription can still be
> + * created and let user aware that something is going to fail later and
> + * they can fix the publications afterwards.
> + */
> + if (list_member(tablelist, rv))
> + ereport(WARNING,
> + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("cannot use different column lists for table \"%s.%s\" in
> different publications",
> + nspname, relname));
>
> Can we extend this comment to explain the case where after Alter
> Publication, if the user dumps and restores back the subscription,
> there is a possibility that "CREATE SUBSCRIPTION" won't work if we
> give ERROR here instead of WARNING?
After rethinking about this, It seems ok to report an ERROR here as the pg_dump
of subscription always set (connect = false). So, we won't hit the check when
restore the dump which means the restore can be successful even if user change
the publication afterwards. Based on this, I have changed the warning to error.
Attach the new version patch.
Best regards,
Hou zj
Attachments:
[application/octet-stream] v2-0002-Prohibit-combining-publications-with-different-colum.patch (19.5K, ../../OS0PR01MB5716D2CF3F2E39A6246A76CE94CF9@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v2-0002-Prohibit-combining-publications-with-different-colum.patch)
download | inline diff:
From aa3fa3a3de41b37f292308654c697a8c23ebb6a7 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 13:19:49 +0800
Subject: [PATCH] Prohibit combining publications with different column lists.
The main purpose of introducing a column list is to have statically
different shapes on publisher and subscriber or hide sensitive column
data. In both cases, it doesn't seems to make sense to combine column
lists.
So, we disallow the cases where the column list is different for the same
table when combining publications. It can be later extended to combine the
column lists for selective cases where required.
Reported-by: Alvaro Herrera
Author: Hou Zhijie
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/alter_publication.sgml | 12 +++-
doc/src/sgml/ref/create_subscription.sgml | 5 ++
src/backend/commands/subscriptioncmds.c | 23 ++++--
src/backend/replication/logical/tablesync.c | 60 ++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 77 ++++++++++----------
src/test/subscription/t/031_column_list.pl | 105 +++++-----------------------
6 files changed, 127 insertions(+), 155 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49..f03933a 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -116,7 +116,17 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<para>
Optionally, a column list can be specified. See <xref
- linkend="sql-createpublication"/> for details.
+ linkend="sql-createpublication"/> for details. Note that a subscription
+ having several publications in which the same table has been published
+ with different column lists is not supported. So, changing the column
+ lists of the tables being subscribed could cause inconsistency of column
+ lists among publications in which case <command>ALTER PUBLICATION</command>
+ command will be successful but later the WalSender in publisher or the
+ subscriber may throw an error. In this scenario, the user needs to
+ recreate the subscription after adjusting the column list or drop the
+ problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
+ it back after adjusting the column list.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 203bb41..f6f82a0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -356,6 +356,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<para>
+ Subscription having several publications in which the same table has been
+ published with different column lists is not supported.
+ </para>
+
+ <para>
We allow non-existent publications to be specified so that users can add
those later. This means
<link linkend="catalog-pg-subscription"><structname>pg_subscription</structname></link>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 690cdaa..991b2c1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1753,7 +1753,8 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
- * publisher connection.
+ * publisher connection. Also get the column list for each table and check if
+ * column lists are the same in different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1762,18 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, NAMEARRAYOID};
List *tablelist = NIL;
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename, \n"
+ " t.attnames\n"
" FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, 3, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1797,18 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ /*
+ * We don't support the case where column list is different for the
+ * same table in different publications.
+ */
+ if (list_member(tablelist, rv))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 994c7a0..9ce4b9f 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -771,7 +771,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,19 +786,17 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that makes
- * it easier to combine lists by simply adding the attnums to a new
- * bitmap (without having to parse the int2vector data). This
- * preserves NULL values, so that if one of the publications has no
- * column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
+ " THEN NULL ELSE gpt.attrs END)"
" FROM pg_publication p,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
pub_names.data);
@@ -813,26 +811,42 @@ fetch_remote_table_info(char *nspname, char *relname,
nspname, relname, pubres->err)));
/*
- * Merge the column lists (from different publications) by creating a
- * single bitmap with all the attnums. If we find a NULL value, that
- * means one of the publications has no column list for the table
- * we're syncing.
+ * We don't support the case where column list is different for the
+ * same table when combining publications. So there should be only one
+ * row returned. Although we already checked this when creating
+ * subscription, we still need to check here in case the column list
+ * was changed afterwards.
+ */
+ if (tuplestore_tuple_count(pubres->tuplestore) > 1)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+
+ /*
+ * Get the column list and build a single bitmap with the attnums.
+ *
+ * If we find a NULL value, it means all the columns should be
+ * replicated.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
- }
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ for (natt = 0; natt < nelems; natt++)
+ included_cols = bms_add_member(included_cols, elems[natt]);
+ }
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 42c06af..3d888a5 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -979,30 +979,30 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
+ * build a bitmap using the column lists.
*
- * All the given publication-table mappings must be checked.
+ * Note that we don't support the case where column list is different for
+ * the same table when combining publications. But we still need to check
+ * all the given publication-table mappings and report an error if any
+ * publications have different column list.
*
* Multiple publications might have multiple column lists for this
* relation.
*
* FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
- * list" so it takes precedence.
+ * list".
*/
foreach(lc, publications)
{
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as
@@ -1011,6 +1011,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1024,51 +1026,48 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (HeapTupleIsValid(cftuple))
{
- /*
- * Lookup the column list attribute.
- *
- * Note: We update the pub_no_list value directly, because if
- * the value is NULL, we have no list (and vice versa).
- */
+ /* Lookup the column list attribute. */
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
- /*
- * Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
- */
+ /* Build the column list bitmap in the per-entry context. */
if (!pub_no_list) /* when not null */
{
pgoutput_ensure_entry_cxt(data, entry);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
+
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 19812e1..22be6fd 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -361,13 +361,13 @@ is( $result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -388,8 +388,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure the columns in column list get fully replicated
$node_publisher->safe_psql(
'postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
@@ -399,42 +398,12 @@ $node_publisher->safe_psql(
$node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
'overlapping publications with overlapping column lists');
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql(
- 'postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -900,57 +869,21 @@ is( $node_subscriber->safe_psql(
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-$node_publisher->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql(
- 'postgres', "SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications which is FOR ALL TABLES, it
+# means replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql(
'postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -976,12 +909,11 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES
+# IN SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql(
'postgres', qq(
@@ -993,7 +925,7 @@ $node_publisher->safe_psql(
'postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -1017,7 +949,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
@@ -1074,7 +1006,7 @@ is( $node_subscriber->safe_psql(
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (with all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql(
@@ -1086,7 +1018,7 @@ $node_publisher->safe_psql(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1232,7 +1164,6 @@ is( $node_subscriber->safe_psql(
4||),
'publication containing both parent and child relation');
-
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
2.7.2.windows.1
[application/octet-stream] v2-0001-Extend-pg_publication_tables-to-display-column-list-.patch (14.8K, ../../OS0PR01MB5716D2CF3F2E39A6246A76CE94CF9@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v2-0001-Extend-pg_publication_tables-to-display-column-list-.patch)
download | inline diff:
From 11809a7d7052d0b894f55c22f0660a5fdc217082 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 11:35:14 +0800
Subject: [PATCH 1/2] Extend pg_publication_tables to display column list and
row filter.
Commit 923def9a53 and 52e4f0cd47 allowed to specify column lists and row
filters for publication tables. This commit extends the
pg_publication_tables view and pg_get_publication_tables function to
display that information.
This information will be useful to users and we also need this for the
later commit that prohibits combining multiple publications with different
column lists for the same table.
Author: Hou Zhijie
Reviewed By: Amit Kapila
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/catalogs.sgml | 27 +++++++++++++--
src/backend/catalog/pg_publication.c | 52 ++++++++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 10 +++++-
src/backend/replication/logical/tablesync.c | 14 +++-----
src/include/catalog/pg_proc.dat | 8 ++---
src/test/regress/expected/publication.out | 42 +++++++++++------------
src/test/regress/expected/rules.out | 13 ++++++--
7 files changed, 124 insertions(+), 42 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a533a21..008f81a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9691,7 +9691,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry><link linkend="view-pg-publication-tables"><structname>pg_publication_tables</structname></link></entry>
- <entry>publications and their associated tables</entry>
+ <entry>publications and information of their associated tables</entry>
</row>
<row>
@@ -11635,8 +11635,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The view <structname>pg_publication_tables</structname> provides
- information about the mapping between publications and the tables they
- contain. Unlike the underlying catalog
+ information about the mapping between publications and information of
+ tables they contain. Unlike the underlying catalog
<link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>,
this view expands publications defined as <literal>FOR ALL TABLES</literal>
and <literal>FOR ALL TABLES IN SCHEMA</literal>, so for such publications
@@ -11687,6 +11687,27 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
Name of table
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>attnames</structfield> <type>name[]</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attname</structfield>)
+ </para>
+ <para>
+ Names of table columns included in the publication. This contains all
+ the columns of table when user didn't specify column list for the
+ table.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rowfilter</structfield> <type>text</type>
+ </para>
+ <para>
+ Expression for the table's publication qualifying condition.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index e2c8bcb..3fe607e 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1082,6 +1082,7 @@ get_publication_name(Oid pubid, bool missing_ok)
Datum
pg_get_publication_tables(PG_FUNCTION_ARGS)
{
+#define NUM_PUBLICATOIN_TABLES_ELEM 3
FuncCallContext *funcctx;
char *pubname = text_to_cstring(PG_GETARG_TEXT_PP(0));
Publication *publication;
@@ -1090,6 +1091,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
/* stuff done only on the first call of the function */
if (SRF_IS_FIRSTCALL())
{
+ TupleDesc tupdesc;
MemoryContext oldcontext;
/* create a function context for cross-call persistence */
@@ -1136,6 +1138,16 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
tables = filter_partitions(tables);
}
+ /* Construct a tuple descriptor for the result rows. */
+ tupdesc = CreateTemplateTupleDesc(NUM_PUBLICATOIN_TABLES_ELEM);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "relid",
+ OIDOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "attrs",
+ INT2VECTOROID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "qual",
+ PG_NODE_TREEOID, -1, 0);
+
+ funcctx->tuple_desc = BlessTupleDesc(tupdesc);
funcctx->user_fctx = (void *) tables;
MemoryContextSwitchTo(oldcontext);
@@ -1147,9 +1159,47 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (funcctx->call_cntr < list_length(tables))
{
+ HeapTuple pubtuple = NULL;
+ HeapTuple rettuple;
Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+ Datum values[NUM_PUBLICATOIN_TABLES_ELEM];
+ bool nulls[NUM_PUBLICATOIN_TABLES_ELEM];
+
+ /*
+ * Form tuple with appropriate data.
+ */
+ MemSet(nulls, 0, sizeof(nulls));
+ MemSet(values, 0, sizeof(values));
+
+ publication = GetPublicationByName(pubname, false);
+
+ values[0] = ObjectIdGetDatum(relid);
+
+ pubtuple = SearchSysCacheCopy2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(publication->oid));
+
+ if (HeapTupleIsValid(pubtuple))
+ {
+ /* Lookup the column list attribute. */
+ values[1] = SysCacheGetAttr(PUBLICATIONRELMAP, pubtuple,
+ Anum_pg_publication_rel_prattrs,
+ &(nulls[1]));
+
+ /* Null indicates no filter. */
+ values[2] = SysCacheGetAttr(PUBLICATIONRELMAP, pubtuple,
+ Anum_pg_publication_rel_prqual,
+ &(nulls[2]));
+ }
+ else
+ {
+ nulls[1] = true;
+ nulls[2] = true;
+ }
+
+ rettuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
- SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(rettuple));
}
SRF_RETURN_DONE(funcctx);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0fc614e..fedaed5 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -368,7 +368,15 @@ CREATE VIEW pg_publication_tables AS
SELECT
P.pubname AS pubname,
N.nspname AS schemaname,
- C.relname AS tablename
+ C.relname AS tablename,
+ ( SELECT array_agg(a.attname ORDER BY a.attnum)
+ FROM unnest(CASE WHEN GPT.attrs IS NOT NULL THEN GPT.attrs
+ ELSE (SELECT array_agg(g) FROM generate_series(1, C.relnatts) g)
+ END) k
+ JOIN pg_attribute a
+ ON (a.attrelid = GPT.relid AND a.attnum = k)
+ ) AS attnames,
+ pg_get_expr(GPT.qual, GPT.relid) AS rowfilter
FROM pg_publication P,
LATERAL pg_get_publication_tables(P.pubname) GPT,
pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b03e0f5..994c7a0 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -795,15 +795,12 @@ fetch_remote_table_info(char *nspname, char *relname,
resetStringInfo(&cmd);
appendStringInfo(&cmd,
"SELECT DISTINCT unnest"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
- " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " FROM pg_publication p,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
" WHERE gpt.relid = %u"
" AND p.pubname IN ( %s )",
lrel->remoteid,
- lrel->remoteid,
pub_names.data);
pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
@@ -965,15 +962,12 @@ fetch_remote_table_info(char *nspname, char *relname,
/* Check for row filters. */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT pg_get_expr(pr.prqual, pr.prrelid)"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ "SELECT DISTINCT pg_get_expr(gpt.qual, gpt.relid)"
+ " FROM pg_publication p,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
" WHERE gpt.relid = %u"
" AND p.pubname IN ( %s )",
lrel->remoteid,
- lrel->remoteid,
pub_names.data);
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 1, qualRow);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f..87aa571 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11673,11 +11673,11 @@
prosrc => 'pg_show_replication_origin_status' },
# publications
-{ oid => '6119', descr => 'get OIDs of tables in a publication',
+{ oid => '6119', descr => 'get information of tables in a publication',
proname => 'pg_get_publication_tables', prorows => '1000', proretset => 't',
- provolatile => 's', prorettype => 'oid', proargtypes => 'text',
- proallargtypes => '{text,oid}', proargmodes => '{i,o}',
- proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_tables' },
+ provolatile => 's', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes => '{i,o,o,o}',
+ proargnames => '{pubname,relid,attrs,qual}', prosrc => 'pg_get_publication_tables' },
{ oid => '6121',
descr => 'returns whether a relation can be part of a publication',
proname => 'pg_relation_is_publishable', provolatile => 's',
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 398c0f3..274b37d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -1585,52 +1585,52 @@ CREATE TABLE sch2.tbl1_part1 PARTITION OF sch1.tbl1 FOR VALUES FROM (1) to (10);
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+-----------
- pub | sch1 | tbl1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+-----------+----------+-----------
+ pub | sch1 | tbl1 | {a} |
(1 row)
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
DROP PUBLICATION pub;
@@ -1643,9 +1643,9 @@ CREATE TABLE sch1.tbl1_part3 (a int) PARTITION BY RANGE(a);
ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (30);
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+-----------
- pub | sch1 | tbl1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+-----------+----------+-----------
+ pub | sch1 | tbl1 | {a} |
(1 row)
RESET client_min_messages;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8..fc3cde3 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1437,9 +1437,18 @@ pg_prepared_xacts| SELECT p.transaction,
LEFT JOIN pg_database d ON ((p.dbid = d.oid)));
pg_publication_tables| SELECT p.pubname,
n.nspname AS schemaname,
- c.relname AS tablename
+ c.relname AS tablename,
+ ( SELECT array_agg(a.attname ORDER BY a.attnum) AS array_agg
+ FROM (unnest(
+ CASE
+ WHEN (gpt.attrs IS NOT NULL) THEN (gpt.attrs)::integer[]
+ ELSE ( SELECT array_agg(g.g) AS array_agg
+ FROM generate_series(1, (c.relnatts)::integer) g(g))
+ END) k(k)
+ JOIN pg_attribute a ON (((a.attrelid = gpt.relid) AND (a.attnum = k.k))))) AS attnames,
+ pg_get_expr(gpt.qual, gpt.relid) AS rowfilter
FROM pg_publication p,
- LATERAL pg_get_publication_tables((p.pubname)::text) gpt(relid),
+ LATERAL pg_get_publication_tables((p.pubname)::text) gpt(relid, attrs, qual),
(pg_class c
JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
WHERE (c.oid = gpt.relid);
--
2.7.2.windows.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-17 03:25 ` [email protected] <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: [email protected] @ 2022-05-17 03:25 UTC (permalink / raw)
To: [email protected] <[email protected]>; Amit Kapila <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 16, 2022 8:34 PM [email protected] <[email protected]> wrote:
>
> Attach the new version patch.
>
Thanks for your patch. Here are some comments:
1. (0001 patch)
/*
* Returns Oids of tables in a publication.
*/
Datum
pg_get_publication_tables(PG_FUNCTION_ARGS)
Should we modify the comment of pg_get_publication_tables() to "Returns
information of tables in a publication"?
2. (0002 patch)
+ * Note that we don't support the case where column list is different for
+ * the same table when combining publications. But we still need to check
+ * all the given publication-table mappings and report an error if any
+ * publications have different column list.
*
* Multiple publications might have multiple column lists for this
* relation.
I think it would be better if we swap the order of these two paragraphs.
Regards,
Shi yu
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-17 06:49 ` [email protected] <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: [email protected] @ 2022-05-17 06:49 UTC (permalink / raw)
To: [email protected] <[email protected]>; Amit Kapila <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Monday, May 16, 2022 9:34 PM [email protected] <[email protected]> wrote:
> Attach the new version patch.
Hi,
I have few minor comments.
For v2-0001.
(1) Unnecessary period at the end of column explanation
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rowfilter</structfield> <type>text</type>
+ </para>
+ <para>
+ Expression for the table's publication qualifying condition.
+ </para></entry>
+ </row>
It seems when we write a simple noun to explain a column,
we don't need to put a period at the end of the explanation.
Kindly change
FROM:
"Expression for the table's publication qualifying condition."
TO:
"Expression for the table's publication qualifying condition"
For v2-0002.
(a) typo in the commit message
Kindly change
FROM:
"In both cases, it doesn't seems to make sense to combine column lists."
TO:
"In both cases, it doesn't seem to make sense to combine column lists."
or "In both cases, it doesn't make sense to combine column lists."
(b) fetch_table_list
+ if (list_member(tablelist, rv))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
Kindly add tests for new error paths, when we add them.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-17 06:52 ` Amit Kapila <[email protected]>
2022-05-17 09:10 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-17 06:52 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 16, 2022 at 6:04 PM [email protected]
<[email protected]> wrote:
>
> Attach the new version patch.
>
Few minor comments:
==================
1.
+ <para>
+ Names of table columns included in the publication. This contains all
+ the columns of table when user didn't specify column list for the
+ table.
+ </para></entry>
Can we slightly change it to: "Names of table columns included in the
publication. This contains all the columns of the table when the user
didn't specify the column list for the table."
2. Below comments needs to be removed from tablesync.c as we don't
combine column lists after this patch.
* For initial synchronization, column lists can be ignored in following
* cases:
*
* 1) one of the subscribed publications for the table hasn't specified
* any column list
*
* 2) one of the subscribed publications has puballtables set to true
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
3.
Note that we don't support the case where column list is different for
+ * the same table when combining publications. But we still need to check
+ * all the given publication-table mappings and report an error if any
+ * publications have different column list.
Can we change this comment to: "Note that we don't support the case
where the column list is different for the same table when combining
publications. But one can later change the publication so we still
need to check all the given publication-table mappings and report an
error if any publications have a different column list."?
4. Can we add a test for different column lists if it is not already there?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 06:52 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-17 09:10 ` [email protected] <[email protected]>
2022-05-18 02:28 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: [email protected] @ 2022-05-17 09:10 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Tuesday, May 17, 2022 2:53 PM Amit Kapila <[email protected]> wrote:
>
> Few minor comments:
> ==================
> 1.
> + <para>
> + Names of table columns included in the publication. This contains all
> + the columns of table when user didn't specify column list for the
> + table.
> + </para></entry>
>
> Can we slightly change it to: "Names of table columns included in the
> publication. This contains all the columns of the table when the user
> didn't specify the column list for the table."
>
> 2. Below comments needs to be removed from tablesync.c as we don't
> combine column lists after this patch.
>
> * For initial synchronization, column lists can be ignored in following
> * cases:
> *
> * 1) one of the subscribed publications for the table hasn't specified
> * any column list
> *
> * 2) one of the subscribed publications has puballtables set to true
> *
> * 3) one of the subscribed publications is declared as ALL TABLES IN
> * SCHEMA that includes this relation
>
> 3.
> Note that we don't support the case where column list is different for
> + * the same table when combining publications. But we still need to check
> + * all the given publication-table mappings and report an error if any
> + * publications have different column list.
>
> Can we change this comment to: "Note that we don't support the case
> where the column list is different for the same table when combining
> publications. But one can later change the publication so we still
> need to check all the given publication-table mappings and report an
> error if any publications have a different column list."?
>
> 4. Can we add a test for different column lists if it is not already there?
Thanks for the comments.
Attach the new version patch which addressed all the above comments and
comments from Shi yu[1] and Osumi-san[2].
[1] https://www.postgresql.org/message-id/OSZPR01MB6310F32344884F9C12F45071FDCE9%40OSZPR01MB6310.jpnprd0...
[2] https://www.postgresql.org/message-id/TYCPR01MB83736AEC2493FCBB75CC7556EDCE9%40TYCPR01MB8373.jpnprd0...
Best regards,
Hou zj
Attachments:
[application/octet-stream] v3-0001-Extend-pg_publication_tables-to-display-column-list-.patch (15.0K, ../../OS0PR01MB571647F2739F710E3D1B045D94CE9@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v3-0001-Extend-pg_publication_tables-to-display-column-list-.patch)
download | inline diff:
From 87104fadcbdba46874a52e72e844ed4c3a73de1d Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 11:35:14 +0800
Subject: [PATCH] Extend pg_publication_tables to display column list and row
filter.
Commit 923def9a53 and 52e4f0cd47 allowed to specify column lists and row
filters for publication tables. This commit extends the
pg_publication_tables view and pg_get_publication_tables function to
display that information.
This information will be useful to users and we also need this for the
later commit that prohibits combining multiple publications with different
column lists for the same table.
Author: Hou Zhijie
Reviewed By: Amit Kapila, Alvaro Herrera, Shi Yu, Takamichi Osumi
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/catalogs.sgml | 27 +++++++++++++--
src/backend/catalog/pg_publication.c | 54 +++++++++++++++++++++++++++--
src/backend/catalog/system_views.sql | 10 +++++-
src/backend/replication/logical/tablesync.c | 14 +++-----
src/include/catalog/pg_proc.dat | 8 ++---
src/test/regress/expected/publication.out | 42 +++++++++++-----------
src/test/regress/expected/rules.out | 13 +++++--
7 files changed, 125 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a533a21..d96c72e 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9691,7 +9691,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry><link linkend="view-pg-publication-tables"><structname>pg_publication_tables</structname></link></entry>
- <entry>publications and their associated tables</entry>
+ <entry>publications and information of their associated tables</entry>
</row>
<row>
@@ -11635,8 +11635,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The view <structname>pg_publication_tables</structname> provides
- information about the mapping between publications and the tables they
- contain. Unlike the underlying catalog
+ information about the mapping between publications and information of
+ tables they contain. Unlike the underlying catalog
<link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>,
this view expands publications defined as <literal>FOR ALL TABLES</literal>
and <literal>FOR ALL TABLES IN SCHEMA</literal>, so for such publications
@@ -11687,6 +11687,27 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
Name of table
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>attnames</structfield> <type>name[]</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attname</structfield>)
+ </para>
+ <para>
+ Names of table columns included in the publication. This contains all
+ the columns of the table when the user didn't specify the column list
+ for the table.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rowfilter</structfield> <type>text</type>
+ </para>
+ <para>
+ Expression for the table's publication qualifying condition
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index e2c8bcb..152b4ba 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1077,11 +1077,12 @@ get_publication_name(Oid pubid, bool missing_ok)
}
/*
- * Returns Oids of tables in a publication.
+ * Returns information of tables in a publication.
*/
Datum
pg_get_publication_tables(PG_FUNCTION_ARGS)
{
+#define NUM_PUBLICATOIN_TABLES_ELEM 3
FuncCallContext *funcctx;
char *pubname = text_to_cstring(PG_GETARG_TEXT_PP(0));
Publication *publication;
@@ -1090,6 +1091,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
/* stuff done only on the first call of the function */
if (SRF_IS_FIRSTCALL())
{
+ TupleDesc tupdesc;
MemoryContext oldcontext;
/* create a function context for cross-call persistence */
@@ -1136,6 +1138,16 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
tables = filter_partitions(tables);
}
+ /* Construct a tuple descriptor for the result rows. */
+ tupdesc = CreateTemplateTupleDesc(NUM_PUBLICATOIN_TABLES_ELEM);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "relid",
+ OIDOID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "attrs",
+ INT2VECTOROID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "qual",
+ PG_NODE_TREEOID, -1, 0);
+
+ funcctx->tuple_desc = BlessTupleDesc(tupdesc);
funcctx->user_fctx = (void *) tables;
MemoryContextSwitchTo(oldcontext);
@@ -1147,9 +1159,47 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (funcctx->call_cntr < list_length(tables))
{
+ HeapTuple pubtuple = NULL;
+ HeapTuple rettuple;
Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+ Datum values[NUM_PUBLICATOIN_TABLES_ELEM];
+ bool nulls[NUM_PUBLICATOIN_TABLES_ELEM];
+
+ /*
+ * Form tuple with appropriate data.
+ */
+ MemSet(nulls, 0, sizeof(nulls));
+ MemSet(values, 0, sizeof(values));
+
+ publication = GetPublicationByName(pubname, false);
+
+ values[0] = ObjectIdGetDatum(relid);
+
+ pubtuple = SearchSysCacheCopy2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(publication->oid));
+
+ if (HeapTupleIsValid(pubtuple))
+ {
+ /* Lookup the column list attribute. */
+ values[1] = SysCacheGetAttr(PUBLICATIONRELMAP, pubtuple,
+ Anum_pg_publication_rel_prattrs,
+ &(nulls[1]));
+
+ /* Null indicates no filter. */
+ values[2] = SysCacheGetAttr(PUBLICATIONRELMAP, pubtuple,
+ Anum_pg_publication_rel_prqual,
+ &(nulls[2]));
+ }
+ else
+ {
+ nulls[1] = true;
+ nulls[2] = true;
+ }
+
+ rettuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
- SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(rettuple));
}
SRF_RETURN_DONE(funcctx);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0fc614e..fedaed5 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -368,7 +368,15 @@ CREATE VIEW pg_publication_tables AS
SELECT
P.pubname AS pubname,
N.nspname AS schemaname,
- C.relname AS tablename
+ C.relname AS tablename,
+ ( SELECT array_agg(a.attname ORDER BY a.attnum)
+ FROM unnest(CASE WHEN GPT.attrs IS NOT NULL THEN GPT.attrs
+ ELSE (SELECT array_agg(g) FROM generate_series(1, C.relnatts) g)
+ END) k
+ JOIN pg_attribute a
+ ON (a.attrelid = GPT.relid AND a.attnum = k)
+ ) AS attnames,
+ pg_get_expr(GPT.qual, GPT.relid) AS rowfilter
FROM pg_publication P,
LATERAL pg_get_publication_tables(P.pubname) GPT,
pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b03e0f5..994c7a0 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -795,15 +795,12 @@ fetch_remote_table_info(char *nspname, char *relname,
resetStringInfo(&cmd);
appendStringInfo(&cmd,
"SELECT DISTINCT unnest"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
- " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " FROM pg_publication p,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
" WHERE gpt.relid = %u"
" AND p.pubname IN ( %s )",
lrel->remoteid,
- lrel->remoteid,
pub_names.data);
pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
@@ -965,15 +962,12 @@ fetch_remote_table_info(char *nspname, char *relname,
/* Check for row filters. */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT pg_get_expr(pr.prqual, pr.prrelid)"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ "SELECT DISTINCT pg_get_expr(gpt.qual, gpt.relid)"
+ " FROM pg_publication p,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
" WHERE gpt.relid = %u"
" AND p.pubname IN ( %s )",
lrel->remoteid,
- lrel->remoteid,
pub_names.data);
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 1, qualRow);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f..87aa571 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11673,11 +11673,11 @@
prosrc => 'pg_show_replication_origin_status' },
# publications
-{ oid => '6119', descr => 'get OIDs of tables in a publication',
+{ oid => '6119', descr => 'get information of tables in a publication',
proname => 'pg_get_publication_tables', prorows => '1000', proretset => 't',
- provolatile => 's', prorettype => 'oid', proargtypes => 'text',
- proallargtypes => '{text,oid}', proargmodes => '{i,o}',
- proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_tables' },
+ provolatile => 's', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes => '{i,o,o,o}',
+ proargnames => '{pubname,relid,attrs,qual}', prosrc => 'pg_get_publication_tables' },
{ oid => '6121',
descr => 'returns whether a relation can be part of a publication',
proname => 'pg_relation_is_publishable', provolatile => 's',
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 398c0f3..274b37d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -1585,52 +1585,52 @@ CREATE TABLE sch2.tbl1_part1 PARTITION OF sch1.tbl1 FOR VALUES FROM (1) to (10);
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+-----------
- pub | sch1 | tbl1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+-----------+----------+-----------
+ pub | sch1 | tbl1 | {a} |
(1 row)
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+------------
- pub | sch2 | tbl1_part1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+------------+----------+-----------
+ pub | sch2 | tbl1_part1 | {a} |
(1 row)
DROP PUBLICATION pub;
@@ -1643,9 +1643,9 @@ CREATE TABLE sch1.tbl1_part3 (a int) PARTITION BY RANGE(a);
ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (30);
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
- pubname | schemaname | tablename
----------+------------+-----------
- pub | sch1 | tbl1
+ pubname | schemaname | tablename | attnames | rowfilter
+---------+------------+-----------+----------+-----------
+ pub | sch1 | tbl1 | {a} |
(1 row)
RESET client_min_messages;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8..fc3cde3 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1437,9 +1437,18 @@ pg_prepared_xacts| SELECT p.transaction,
LEFT JOIN pg_database d ON ((p.dbid = d.oid)));
pg_publication_tables| SELECT p.pubname,
n.nspname AS schemaname,
- c.relname AS tablename
+ c.relname AS tablename,
+ ( SELECT array_agg(a.attname ORDER BY a.attnum) AS array_agg
+ FROM (unnest(
+ CASE
+ WHEN (gpt.attrs IS NOT NULL) THEN (gpt.attrs)::integer[]
+ ELSE ( SELECT array_agg(g.g) AS array_agg
+ FROM generate_series(1, (c.relnatts)::integer) g(g))
+ END) k(k)
+ JOIN pg_attribute a ON (((a.attrelid = gpt.relid) AND (a.attnum = k.k))))) AS attnames,
+ pg_get_expr(gpt.qual, gpt.relid) AS rowfilter
FROM pg_publication p,
- LATERAL pg_get_publication_tables((p.pubname)::text) gpt(relid),
+ LATERAL pg_get_publication_tables((p.pubname)::text) gpt(relid, attrs, qual),
(pg_class c
JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
WHERE (c.oid = gpt.relid);
--
2.7.2.windows.1
[application/octet-stream] v3-0002-Prohibit-combining-publications-with-different-colum.patch (22.0K, ../../OS0PR01MB571647F2739F710E3D1B045D94CE9@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v3-0002-Prohibit-combining-publications-with-different-colum.patch)
download | inline diff:
From 91a4774b9f351cb09f461ee920be9031cc4e073e Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 13:19:49 +0800
Subject: [PATCH] Prohibit combining publications with different column lists.
The main purpose of introducing a column list is to have statically
different shapes on publisher and subscriber or hide sensitive column
data. In both cases, it doesn't seem to make sense to combine column
lists.
So, we disallow the cases where the column list is different for the same
table when combining publications. It can be later extended to combine the
column lists for selective cases where required.
Reported-by: Alvaro Herrera
Author: Hou Zhijie
Reviewed-by: Amit Kapila, Alvaro Herrera, Shi Yu, Takamichi Osumi
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/alter_publication.sgml | 12 ++-
doc/src/sgml/ref/create_subscription.sgml | 5 +
src/backend/commands/subscriptioncmds.c | 23 ++++-
src/backend/replication/logical/tablesync.c | 72 ++++++-------
src/backend/replication/pgoutput/pgoutput.c | 80 +++++++--------
src/test/subscription/t/031_column_list.pl | 150 ++++++++++++----------------
6 files changed, 176 insertions(+), 166 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49..f03933a 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -116,7 +116,17 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<para>
Optionally, a column list can be specified. See <xref
- linkend="sql-createpublication"/> for details.
+ linkend="sql-createpublication"/> for details. Note that a subscription
+ having several publications in which the same table has been published
+ with different column lists is not supported. So, changing the column
+ lists of the tables being subscribed could cause inconsistency of column
+ lists among publications in which case <command>ALTER PUBLICATION</command>
+ command will be successful but later the WalSender in publisher or the
+ subscriber may throw an error. In this scenario, the user needs to
+ recreate the subscription after adjusting the column list or drop the
+ problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
+ it back after adjusting the column list.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 203bb41..f6f82a0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -356,6 +356,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<para>
+ Subscription having several publications in which the same table has been
+ published with different column lists is not supported.
+ </para>
+
+ <para>
We allow non-existent publications to be specified so that users can add
those later. This means
<link linkend="catalog-pg-subscription"><structname>pg_subscription</structname></link>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 690cdaa..991b2c1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1753,7 +1753,8 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
- * publisher connection.
+ * publisher connection. Also get the column list for each table and check if
+ * column lists are the same in different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1762,18 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, NAMEARRAYOID};
List *tablelist = NIL;
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename, \n"
+ " t.attnames\n"
" FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, 3, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1797,18 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ /*
+ * We don't support the case where column list is different for the
+ * same table in different publications.
+ */
+ if (list_member(tablelist, rv))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 994c7a0..afc5bcd 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -753,17 +753,6 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Get column lists for each relation.
*
- * For initial synchronization, column lists can be ignored in following
- * cases:
- *
- * 1) one of the subscribed publications for the table hasn't specified
- * any column list
- *
- * 2) one of the subscribed publications has puballtables set to true
- *
- * 3) one of the subscribed publications is declared as ALL TABLES IN
- * SCHEMA that includes this relation
- *
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
@@ -771,7 +760,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,19 +775,17 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that makes
- * it easier to combine lists by simply adding the attnums to a new
- * bitmap (without having to parse the int2vector data). This
- * preserves NULL values, so that if one of the publications has no
- * column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
+ " THEN NULL ELSE gpt.attrs END)"
" FROM pg_publication p,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
pub_names.data);
@@ -813,26 +800,43 @@ fetch_remote_table_info(char *nspname, char *relname,
nspname, relname, pubres->err)));
/*
- * Merge the column lists (from different publications) by creating a
- * single bitmap with all the attnums. If we find a NULL value, that
- * means one of the publications has no column list for the table
- * we're syncing.
+ * We don't support the case where the column list is different for the
+ * same table when combining publications. So there should be only one
+ * row returned. Although we already checked this when creating
+ * subscription, we still need to check here in case the column list
+ * was changed after creating the subscription and before the sync
+ * worker is started.
+ */
+ if (tuplestore_tuple_count(pubres->tuplestore) > 1)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+
+ /*
+ * Get the column list and build a single bitmap with the attnums.
+ *
+ * If we find a NULL value, it means all the columns should be
+ * replicated.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
- }
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (natt = 0; natt < nelems; natt++)
+ included_cols = bms_add_member(included_cols, elems[natt]);
+ }
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 42c06af..e9079bd 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -979,30 +979,31 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
- *
- * All the given publication-table mappings must be checked.
+ * build a bitmap using the column lists.
*
* Multiple publications might have multiple column lists for this
* relation.
*
+ * Note that we don't support the case where the column list is different
+ * for the same table when combining publications. But one can later change
+ * the publication so we still need to check all the given
+ * publication-table mappings and report an error if any publications have
+ * a different column list.
+ *
* FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
- * list" so it takes precedence.
+ * list".
*/
foreach(lc, publications)
{
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as
@@ -1011,6 +1012,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1024,51 +1027,48 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (HeapTupleIsValid(cftuple))
{
- /*
- * Lookup the column list attribute.
- *
- * Note: We update the pub_no_list value directly, because if
- * the value is NULL, we have no list (and vice versa).
- */
+ /* Lookup the column list attribute. */
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
- /*
- * Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
- */
+ /* Build the column list bitmap in the per-entry context. */
if (!pub_no_list) /* when not null */
{
pgoutput_ensure_entry_cxt(data, entry);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
+
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 19812e1..7f031bc 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -20,6 +20,7 @@ $node_subscriber->append_conf('postgresql.conf',
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $offset = 0;
sub wait_for_subscription_sync
{
@@ -361,13 +362,13 @@ is( $result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -388,8 +389,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure the columns in column list get fully replicated
$node_publisher->safe_psql(
'postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
@@ -399,42 +399,12 @@ $node_publisher->safe_psql(
$node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
'overlapping publications with overlapping column lists');
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql(
- 'postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -900,57 +870,21 @@ is( $node_subscriber->safe_psql(
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql(
- 'postgres', "SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications which is FOR ALL TABLES, it
+# means replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql(
'postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -976,12 +910,11 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES
+# IN SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql(
'postgres', qq(
@@ -993,7 +926,7 @@ $node_publisher->safe_psql(
'postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -1017,7 +950,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
@@ -1074,7 +1007,7 @@ is( $node_subscriber->safe_psql(
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (with all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql(
@@ -1086,7 +1019,7 @@ $node_publisher->safe_psql(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1233,6 +1166,51 @@ is( $node_subscriber->safe_psql(
'publication containing both parent and child relation');
+# TEST: With a table included in multiple publications with different column
+# lists, we should catch the error when creating the subscription.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+));
+
+my ($cmdret, $stdout, $stderr) = $node_subscriber->psql(
+ 'postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+ok( $stderr =~
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ 'different column lists detected');
+
+# TEST: If the column list is changed after creating the subscription, we
+# should catch the error reported by walsender.
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, b);
+ INSERT INTO test_mix_1 VALUES(1, 1, 1);
+));
+
+$offset = $node_publisher->wait_for_log(
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ $offset);
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
2.7.2.windows.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 06:52 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-17 09:10 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-18 02:28 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-18 02:28 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Tue, May 17, 2022 at 2:40 PM [email protected]
<[email protected]> wrote:
>
> Attach the new version patch which addressed all the above comments and
> comments from Shi yu[1] and Osumi-san[2].
>
Thanks, your first patch looks good to me. I'll commit that tomorrow
unless there are more comments on the same. The second one is also in
good shape but I would like to test it a bit more and also see if
others have any suggestions/objections on the same.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-16 13:20 ` Alvaro Herrera <[email protected]>
2022-05-17 03:26 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
1 sibling, 2 replies; 59+ messages in thread
From: Alvaro Herrera @ 2022-05-16 13:20 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 2022-May-16, Amit Kapila wrote:
> > Agreed. If we can get columnlist and rowfilter from pg_publication_tables, it
> > will be more convenient. And I think users that want to fetch the columnlist
> > and rowfilter of table can also benefit from it.
>
> After the change for this, we will give an error on combining
> publications where one of the publications specifies all columns in
> the table and the other doesn't provide any columns. We should not
> give an error as both mean all columns.
But don't we need to behave the same way for both column lists and row
filters? I understand that some cases with different row filters for
different publications have shown to have weird behavior, so I think
it'd make sense to restrict it in the same way. That would allow us to
extend the behavior in a sensible way when we develop that, instead of
setting in stone now behavior that we regret later.
> Few comments:
> =================
> 1.
> postgres=# select * from pg_publication_tables;
> pubname | schemaname | tablename | columnlist | rowfilter
> ---------+------------+-----------+------------+-----------
> pub1 | public | t1 | |
> pub2 | public | t1 | 1 2 | (c3 < 10)
> (2 rows)
>
> I think it is better to display column names for columnlist in the
> exposed view similar to attnames in the pg_stats_ext view. I think
> that will make it easier for users to understand this information.
+1
> I think we should change the "descr" to something like: 'get
> information of tables in a publication'
+1
> 3.
> +
> + /*
> + * We only throw a warning here so that the subcription can still be
> + * created and let user aware that something is going to fail later and
> + * they can fix the publications afterwards.
> + */
> + if (list_member(tablelist, rv))
> + ereport(WARNING,
> + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("cannot use different column lists for table \"%s.%s\" in
> different publications",
> + nspname, relname));
>
> Can we extend this comment to explain the case where after Alter
> Publication, if the user dumps and restores back the subscription,
> there is a possibility that "CREATE SUBSCRIPTION" won't work if we
> give ERROR here instead of WARNING?
Yeah, and not only the comment — I think we need to have more in the
warning message itself. How about:
ERROR: cannot use different column lists for table "..." in different publications
DETAIL: The subscription "..." cannot currently be used for replication.
I think this whole affair is a bit sad TBH and I'm sure it'll give us
some grief -- similar to replication slots becoming inactive and nobody
noticing. A user changing a publication in a way that prevents some
replica from working and the warnings are hidden, they could have
trouble noticing that the replica is stuck.
But I have no better ideas.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-17 03:26 ` Amit Kapila <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-17 03:26 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: [email protected] <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 16, 2022 at 6:50 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2022-May-16, Amit Kapila wrote:
>
> > > Agreed. If we can get columnlist and rowfilter from pg_publication_tables, it
> > > will be more convenient. And I think users that want to fetch the columnlist
> > > and rowfilter of table can also benefit from it.
> >
> > After the change for this, we will give an error on combining
> > publications where one of the publications specifies all columns in
> > the table and the other doesn't provide any columns. We should not
> > give an error as both mean all columns.
>
> But don't we need to behave the same way for both column lists and row
> filters? I understand that some cases with different row filters for
> different publications have shown to have weird behavior, so I think
> it'd make sense to restrict it in the same way.
>
I think the case where we are worried about row filter behavior is for
initial table sync where we ignore publication actions and that is
true with and without row filters. See email [1]. We are planning to
document that behavior as a separate patch. The idea we have used for
row filters is similar to what IBM DB2 [2] and Oracle [3] uses where
they allow combining filters with pub-action (operation (insert,
update, delete) in their case).
I think both column lists and row filters have a different purpose and
we shouldn't try to make them behave in the same way. The main purpose
of introducing a column list is to have statically different shapes on
publisher and subscriber or hide sensitive column data. In both cases,
it doesn't seem to make sense to combine column lists and we didn't
find any other database doing so. OTOH, for row filters, it makes
sense to combine filters for each pub-action as both IBM DB2 and
Oracle seems to be doing.
>
> > 3.
> > +
> > + /*
> > + * We only throw a warning here so that the subcription can still be
> > + * created and let user aware that something is going to fail later and
> > + * they can fix the publications afterwards.
> > + */
> > + if (list_member(tablelist, rv))
> > + ereport(WARNING,
> > + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> > + errmsg("cannot use different column lists for table \"%s.%s\" in
> > different publications",
> > + nspname, relname));
> >
> > Can we extend this comment to explain the case where after Alter
> > Publication, if the user dumps and restores back the subscription,
> > there is a possibility that "CREATE SUBSCRIPTION" won't work if we
> > give ERROR here instead of WARNING?
>
> Yeah, and not only the comment — I think we need to have more in the
> warning message itself.
>
But as mentioned by Hou-San in his last email (pg_dump of subscription
always set (connect = false) which means it won't try to fetch column
list), I think we don't need to give a WARNING here, instead, we can
use ERROR. So, do we need the extra DETAIL (The subscription "..."
cannot currently be used for replication.) as that is implicit for the
ERROR case?
>
> I think this whole affair is a bit sad TBH and I'm sure it'll give us
> some grief -- similar to replication slots becoming inactive and nobody
> noticing. A user changing a publication in a way that prevents some
> replica from working and the warnings are hidden, they could have
> trouble noticing that the replica is stuck.
>
I agree and it seems this is the best we can do for now.
[1] - https://www.postgresql.org/message-id/CAA4eK1L_98LF7Db4yFY1PhKKRzoT83xtN41jTS5X%2B8OeGrAkLw%40mail.g...
[2] - https://www.ibm.com/docs/en/idr/11.4.0?topic=rows-log-record-variables
[3] - https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/selecting-and-filtering-rows.html#GUID-...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
@ 2022-05-19 05:03 ` Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-19 05:03 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: [email protected] <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 16, 2022 at 6:50 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2022-May-16, Amit Kapila wrote:
>
> > Few comments:
> > =================
> > 1.
> > postgres=# select * from pg_publication_tables;
> > pubname | schemaname | tablename | columnlist | rowfilter
> > ---------+------------+-----------+------------+-----------
> > pub1 | public | t1 | |
> > pub2 | public | t1 | 1 2 | (c3 < 10)
> > (2 rows)
> >
> > I think it is better to display column names for columnlist in the
> > exposed view similar to attnames in the pg_stats_ext view. I think
> > that will make it easier for users to understand this information.
>
> +1
>
I have committed the first patch after fixing this part. It seems Tom
is not very happy doing this after beta-1 [1]. The reason we get this
information via this view (and underlying function) is that it
simplifies the queries on the subscriber-side as you can see in the
second patch. The query change is as below:
@@ -1761,17 +1762,18 @@ fetch_table_list(WalReceiverConn *wrconn, List
*publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, NAMEARRAYOID};
List *tablelist = NIL;
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename, \n"
+ " t.attnames\n"
" FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
Now, there is another way to change this query as well as done by
Hou-San in his first version [2] of the patch. The changed query with
that approach will be something like:
@@ -1761,17 +1762,34 @@ fetch_table_list(WalReceiverConn *wrconn, List
*publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, INT2VECTOROID};
List *tablelist = NIL;
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
- " FROM pg_catalog.pg_publication_tables t\n"
+ appendStringInfoString(&cmd,
+ "SELECT DISTINCT t.schemaname,\n"
+ " t.tablename,\n"
+ " (CASE WHEN (array_length(pr.prattrs, 1) = t.relnatts)\n"
+ " THEN NULL ELSE pr.prattrs END)\n"
+ " FROM (SELECT P.pubname AS pubname,\n"
+ " N.nspname AS schemaname,\n"
+ " C.relname AS tablename,\n"
+ " P.oid AS pubid,\n"
+ " C.oid AS reloid,\n"
+ " C.relnatts\n"
+ " FROM pg_publication P,\n"
+ " LATERAL pg_get_publication_tables(P.pubname) GPT,\n"
+ " pg_class C JOIN pg_namespace N\n"
+ " ON (N.oid = C.relnamespace)\n"
+ " WHERE C.oid = GPT.relid) t\n"
+ " LEFT OUTER JOIN pg_publication_rel pr\n"
+ " ON (t.pubid = pr.prpubid AND\n"
+ " pr.prrelid = reloid)\n"
It appeared slightly complex and costly to me, so I have given the
suggestion to change it as we have now in the second patch as shown
above. Now, I can think of below ways to proceed here:
a. Revert the change in view (and underlying function) as done in
commit 0ff20288e1 and consider the alternate way (using a slightly
complex query) to fix. Then maybe for PG-16, we can simplify it by
changing the underlying function and view.
b. Proceed with the current approach of using a simplified query.
What do you think?
[1] - https://www.postgresql.org/message-id/91075.1652929852%40sss.pgh.pa.us
[2] - https://www.postgresql.org/message-id/OS0PR01MB5716A594C58DE4FFD1F8100B94C89%40OS0PR01MB5716.jpnprd0...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-19 12:07 ` Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Justin Pryzby @ 2022-05-19 12:07 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; [email protected]; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]
On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote:
> I have committed the first patch after fixing this part. It seems Tom
> is not very happy doing this after beta-1 [1]. The reason we get this
> information via this view (and underlying function) is that it
> simplifies the queries on the subscriber-side as you can see in the
> second patch. The query change is as below:
> [1] - https://www.postgresql.org/message-id/91075.1652929852%40sss.pgh.pa.us
I think Tom's concern is that adding information to a view seems like adding a
feature that hadn't previously been contemplated.
(Catalog changes themselves are not prohibited during the beta period).
> a. Revert the change in view (and underlying function) as done in
> commit 0ff20288e1 and consider the alternate way (using a slightly
> complex query) to fix. Then maybe for PG-16, we can simplify it by
> changing the underlying function and view.
But, ISTM that it makes no sense to do it differently for v15 just to avoid the
appearance of adding a new feature, only to re-do it in 2 weeks for v16...
So (from a passive observer) +0.1 to keep the current patch.
I have some minor language fixes to that patch.
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index d96c72e5310..82aa84e96e1 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9691,7 +9691,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry><link linkend="view-pg-publication-tables"><structname>pg_publication_tables</structname></link></entry>
- <entry>publications and information of their associated tables</entry>
+ <entry>publications and information about their associated tables</entry>
</row>
<row>
@@ -11635,7 +11635,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The view <structname>pg_publication_tables</structname> provides
- information about the mapping between publications and information of
+ information about the mapping between publications and information about
tables they contain. Unlike the underlying catalog
<link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>,
this view expands publications defined as <literal>FOR ALL TABLES</literal>
@@ -11695,7 +11695,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
Names of table columns included in the publication. This contains all
- the columns of the table when the user didn't specify the column list
+ the columns of the table when the user didn't specify a column list
for the table.
</para></entry>
</row>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 8c7fca62de3..2f706f638ce 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1077,7 +1077,7 @@ get_publication_name(Oid pubid, bool missing_ok)
}
/*
- * Returns information of tables in a publication.
+ * Returns information about tables in a publication.
*/
Datum
pg_get_publication_tables(PG_FUNCTION_ARGS)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87aa571a331..86f13293090 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11673,7 +11673,7 @@
prosrc => 'pg_show_replication_origin_status' },
# publications
-{ oid => '6119', descr => 'get information of tables in a publication',
+{ oid => '6119', descr => 'get information about tables in a publication',
proname => 'pg_get_publication_tables', prorows => '1000', proretset => 't',
provolatile => 's', prorettype => 'record', proargtypes => 'text',
proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes => '{i,o,o,o}',
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
@ 2022-05-19 14:24 ` Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Tom Lane @ 2022-05-19 14:24 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; [email protected]; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote:
>> I have committed the first patch after fixing this part. It seems Tom
>> is not very happy doing this after beta-1 [1]. The reason we get this
>> information via this view (and underlying function) is that it
>> simplifies the queries on the subscriber-side as you can see in the
>> second patch. The query change is as below:
>> [1] - https://www.postgresql.org/message-id/91075.1652929852%40sss.pgh.pa.us
> I think Tom's concern is that adding information to a view seems like adding a
> feature that hadn't previously been contemplated.
> (Catalog changes themselves are not prohibited during the beta period).
It certainly smells like a new feature, but my concern was more around the
post-beta catalog change. We do those only if really forced to, and the
explanation in the commit message didn't satisfy me as to why it was
necessary. This explanation isn't much better --- if we're trying to
prohibit a certain class of publication definitions, what good does it do
to check that on the subscriber side? Even more to the point, how can we
have a subscriber do that by relying on view columns that don't exist in
older versions? I'm also quite concerned about anything that involves
subscribers examining row filter conditions; that sounds like a pretty
direct route to bugs involving unsolvability and the halting problem.
(But I've not read very much of this thread ... been a bit under the
weather the last couple weeks. Maybe this actually is a sane solution.
It just doesn't sound like one at this level of detail.)
regards, tom lane
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
@ 2022-05-20 03:06 ` Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-24 09:49 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-20 03:06 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Hou, Zhijie/侯 志杰 <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, May 19, 2022 at 7:54 PM Tom Lane <[email protected]> wrote:
>
> Justin Pryzby <[email protected]> writes:
> > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote:
> >> I have committed the first patch after fixing this part. It seems Tom
> >> is not very happy doing this after beta-1 [1]. The reason we get this
> >> information via this view (and underlying function) is that it
> >> simplifies the queries on the subscriber-side as you can see in the
> >> second patch. The query change is as below:
> >> [1] - https://www.postgresql.org/message-id/91075.1652929852%40sss.pgh.pa.us
>
> > I think Tom's concern is that adding information to a view seems like adding a
> > feature that hadn't previously been contemplated.
> > (Catalog changes themselves are not prohibited during the beta period).
>
> It certainly smells like a new feature, but my concern was more around the
> post-beta catalog change. We do those only if really forced to, and the
> explanation in the commit message didn't satisfy me as to why it was
> necessary. This explanation isn't much better --- if we're trying to
> prohibit a certain class of publication definitions, what good does it do
> to check that on the subscriber side?
>
It is required on the subscriber side because prohibition is only for
the cases where multiple publications are combined. We disallow the
cases where the column list is different for the same table when
combining publications. For example:
Publisher-side:
Create table tab(c1 int, c2 int);
Create Publication pub1 for table tab(c1);
Create Publication pub1 for table tab(c2);
Subscriber-side:
Create Subscription sub1 Connection 'dbname=postgres' Publication pub1, pub2;
We want to prohibit such cases. So, it would be better to check at the
time of 'Create Subscription' to validate such combinations and
prohibit them. To achieve that we extended the existing function
pg_get_publication_tables() and view pg_publication_tables to expose
the column list and verify such a combination. We primarily need
column list information for this prohibition but it appeared natural
to expose the row filter.
As mentioned in my previous email, we can fetch the required
information directly from system table pg_publication_rel and extend
the query in fetch_table_list to achieve the desired purpose but
extending the existing function/view for this appears to be a simpler
way.
> Even more to the point, how can we
> have a subscriber do that by relying on view columns that don't exist in
> older versions?
>
We need a version check like (if
(walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)) for that.
> I'm also quite concerned about anything that involves
> subscribers examining row filter conditions; that sounds like a pretty
> direct route to bugs involving unsolvability and the halting problem.
>
We examine only the column list for the purpose of this prohibition.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-24 05:33 ` [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: [email protected] @ 2022-05-24 05:33 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Tom Lane <[email protected]>
On Friday, May 20, 2022 11:06 AM Amit Kapila <[email protected]> wrote:
>
> On Thu, May 19, 2022 at 7:54 PM Tom Lane <[email protected]> wrote:
>
> > Even more to the point, how can we
> > have a subscriber do that by relying on view columns that don't exist
> > in older versions?
> >
>
> We need a version check like (if
> (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)) for that.
Thanks for pointing it out. Here is the new version patch which add this version check.
Best regards,
Hou zj
Attachments:
[application/octet-stream] v4-0002-Prohibit-combining-publications-with-different-colum.patch (22.3K, ../../OS0PR01MB5716AD7C0FE7386630BDBAAB94D79@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v4-0002-Prohibit-combining-publications-with-different-colum.patch)
download | inline diff:
From 080d3e8ac5d9ec44b52f0a03e6d204d7f451a779 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 13 May 2022 13:19:49 +0800
Subject: [PATCH] Prohibit combining publications with different column lists.
The main purpose of introducing a column list is to have statically
different shapes on publisher and subscriber or hide sensitive column
data. In both cases, it doesn't seem to make sense to combine column
lists.
So, we disallow the cases where the column list is different for the same
table when combining publications. It can be later extended to combine the
column lists for selective cases where required.
Reported-by: Alvaro Herrera
Author: Hou Zhijie
Reviewed-by: Amit Kapila, Alvaro Herrera, Shi Yu, Takamichi Osumi
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/alter_publication.sgml | 12 ++-
doc/src/sgml/ref/create_subscription.sgml | 5 +
src/backend/commands/subscriptioncmds.c | 30 ++++--
src/backend/replication/logical/tablesync.c | 72 ++++++-------
src/backend/replication/pgoutput/pgoutput.c | 80 +++++++--------
src/test/subscription/t/031_column_list.pl | 150 ++++++++++++----------------
6 files changed, 182 insertions(+), 167 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49..f03933a 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -116,7 +116,17 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<para>
Optionally, a column list can be specified. See <xref
- linkend="sql-createpublication"/> for details.
+ linkend="sql-createpublication"/> for details. Note that a subscription
+ having several publications in which the same table has been published
+ with different column lists is not supported. So, changing the column
+ lists of the tables being subscribed could cause inconsistency of column
+ lists among publications in which case <command>ALTER PUBLICATION</command>
+ command will be successful but later the WalSender in publisher or the
+ subscriber may throw an error. In this scenario, the user needs to
+ recreate the subscription after adjusting the column list or drop the
+ problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
+ it back after adjusting the column list.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 203bb41..f6f82a0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -356,6 +356,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<para>
+ Subscription having several publications in which the same table has been
+ published with different column lists is not supported.
+ </para>
+
+ <para>
We allow non-existent publications to be specified so that users can add
those later. This means
<link linkend="catalog-pg-subscription"><structname>pg_subscription</structname></link>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 690cdaa..641caf7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1753,7 +1753,8 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
- * publisher connection.
+ * publisher connection. Also get the column list for each table and check if
+ * column lists are the same in different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1762,23 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, NAMEARRAYOID};
List *tablelist = NIL;
+ bool check_columnlist = (walrcv_server_version(wrconn) >= 150000);
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
- " FROM pg_catalog.pg_publication_tables t\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename \n");
+
+ /* Get column lists for each relation if the publisher supports it */
+ if (check_columnlist)
+ appendStringInfoString(&cmd, ", t.attnames\n");
+
+ appendStringInfoString(&cmd, "FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, check_columnlist ? 3 : 2, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1802,18 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ /*
+ * We don't support the case where column list is different for the
+ * same table in different publications.
+ */
+ if (check_columnlist && list_member(tablelist, rv))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 994c7a0..e4633bd 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -753,17 +753,6 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Get column lists for each relation.
*
- * For initial synchronization, column lists can be ignored in following
- * cases:
- *
- * 1) one of the subscribed publications for the table hasn't specified
- * any column list
- *
- * 2) one of the subscribed publications has puballtables set to true
- *
- * 3) one of the subscribed publications is declared as ALL TABLES IN
- * SCHEMA that includes this relation
- *
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
@@ -771,7 +760,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,19 +775,17 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that makes
- * it easier to combine lists by simply adding the attnums to a new
- * bitmap (without having to parse the int2vector data). This
- * preserves NULL values, so that if one of the publications has no
- * column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
+ " THEN NULL ELSE gpt.attrs END)"
" FROM pg_publication p,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
pub_names.data);
@@ -813,26 +800,43 @@ fetch_remote_table_info(char *nspname, char *relname,
nspname, relname, pubres->err)));
/*
- * Merge the column lists (from different publications) by creating a
- * single bitmap with all the attnums. If we find a NULL value, that
- * means one of the publications has no column list for the table
- * we're syncing.
+ * We don't support the case where the column list is different for the
+ * same table when combining publications. So there should be only one
+ * row returned. Although we already checked this when creating
+ * subscription, we still need to check here in case the column list
+ * was changed after creating the subscription and before the sync
+ * worker is started.
+ */
+ if (tuplestore_tuple_count(pubres->tuplestore) > 1)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+
+ /*
+ * Get the column list and build a single bitmap with the attnums.
+ *
+ * If we find a NULL value, it means all the columns should be
+ * replicated.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
- }
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (natt = 0; natt < nelems; natt++)
+ included_cols = bms_add_member(included_cols, elems[natt]);
+ }
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 42c06af..e9079bd 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -979,30 +979,31 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
- *
- * All the given publication-table mappings must be checked.
+ * build a bitmap using the column lists.
*
* Multiple publications might have multiple column lists for this
* relation.
*
+ * Note that we don't support the case where the column list is different
+ * for the same table when combining publications. But one can later change
+ * the publication so we still need to check all the given
+ * publication-table mappings and report an error if any publications have
+ * a different column list.
+ *
* FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
- * list" so it takes precedence.
+ * list".
*/
foreach(lc, publications)
{
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as
@@ -1011,6 +1012,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1024,51 +1027,48 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (HeapTupleIsValid(cftuple))
{
- /*
- * Lookup the column list attribute.
- *
- * Note: We update the pub_no_list value directly, because if
- * the value is NULL, we have no list (and vice versa).
- */
+ /* Lookup the column list attribute. */
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
- /*
- * Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
- */
+ /* Build the column list bitmap in the per-entry context. */
if (!pub_no_list) /* when not null */
{
pgoutput_ensure_entry_cxt(data, entry);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
+
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 19812e1..7f031bc 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -20,6 +20,7 @@ $node_subscriber->append_conf('postgresql.conf',
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $offset = 0;
sub wait_for_subscription_sync
{
@@ -361,13 +362,13 @@ is( $result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -388,8 +389,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure the columns in column list get fully replicated
$node_publisher->safe_psql(
'postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
@@ -399,42 +399,12 @@ $node_publisher->safe_psql(
$node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
'overlapping publications with overlapping column lists');
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql(
- 'postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -900,57 +870,21 @@ is( $node_subscriber->safe_psql(
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql(
- 'postgres', "SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications which is FOR ALL TABLES, it
+# means replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql(
'postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -976,12 +910,11 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES
+# IN SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql(
'postgres', qq(
@@ -993,7 +926,7 @@ $node_publisher->safe_psql(
'postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -1017,7 +950,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
@@ -1074,7 +1007,7 @@ is( $node_subscriber->safe_psql(
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (with all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql(
@@ -1086,7 +1019,7 @@ $node_publisher->safe_psql(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1233,6 +1166,51 @@ is( $node_subscriber->safe_psql(
'publication containing both parent and child relation');
+# TEST: With a table included in multiple publications with different column
+# lists, we should catch the error when creating the subscription.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+));
+
+my ($cmdret, $stdout, $stderr) = $node_subscriber->psql(
+ 'postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+ok( $stderr =~
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ 'different column lists detected');
+
+# TEST: If the column list is changed after creating the subscription, we
+# should catch the error reported by walsender.
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, b);
+ INSERT INTO test_mix_1 VALUES(1, 1, 1);
+));
+
+$offset = $node_publisher->wait_for_log(
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ $offset);
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
2.7.2.windows.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-05-27 05:47 ` Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-27 05:47 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Tom Lane <[email protected]>
On Tue, May 24, 2022 at 11:03 AM [email protected]
<[email protected]> wrote:
>
> On Friday, May 20, 2022 11:06 AM Amit Kapila <[email protected]> wrote:
>
> Thanks for pointing it out. Here is the new version patch which add this version check.
>
I have added/edited a few comments and ran pgindent. The attached
looks good to me. I'll push this early next week unless there are more
comments/suggestions.
--
With Regards,
Amit Kapila.
Attachments:
[application/octet-stream] v5-0001-Prohibit-combining-publications-with-different-co.patch (22.4K, ../../CAA4eK1JDuOb96naN7akLpXz6G2ZGnGyg73w5Xt8ncAhWNF5Giw@mail.gmail.com/2-v5-0001-Prohibit-combining-publications-with-different-co.patch)
download | inline diff:
From 201721776782809c3bb99bb6ff8fbdedb02483ab Mon Sep 17 00:00:00 2001
From: Amit Kapila <[email protected]>
Date: Fri, 27 May 2022 08:30:33 +0530
Subject: [PATCH v5] Prohibit combining publications with different column
lists.
The main purpose of introducing a column list is to have statically
different shapes on publisher and subscriber or hide sensitive column
data. In both cases, it doesn't seem to make sense to combine column
lists.
So, we disallow the cases where the column list is different for the same
table when combining publications. It can be later extended to combine the
column lists for selective cases where required.
Reported-by: Alvaro Herrera
Author: Hou Zhijie
Reviewed-by: Amit Kapila, Alvaro Herrera, Shi Yu, Takamichi Osumi
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/alter_publication.sgml | 12 ++-
doc/src/sgml/ref/create_subscription.sgml | 5 +
src/backend/commands/subscriptioncmds.c | 28 +++++-
src/backend/replication/logical/tablesync.c | 72 ++++++-------
src/backend/replication/pgoutput/pgoutput.c | 80 +++++++--------
src/test/subscription/t/031_column_list.pl | 150 ++++++++++++----------------
6 files changed, 181 insertions(+), 166 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49..f03933a 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -116,7 +116,17 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<para>
Optionally, a column list can be specified. See <xref
- linkend="sql-createpublication"/> for details.
+ linkend="sql-createpublication"/> for details. Note that a subscription
+ having several publications in which the same table has been published
+ with different column lists is not supported. So, changing the column
+ lists of the tables being subscribed could cause inconsistency of column
+ lists among publications in which case <command>ALTER PUBLICATION</command>
+ command will be successful but later the WalSender in publisher or the
+ subscriber may throw an error. In this scenario, the user needs to
+ recreate the subscription after adjusting the column list or drop the
+ problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
+ it back after adjusting the column list.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 203bb41..f6f82a0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -356,6 +356,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<para>
+ Subscription having several publications in which the same table has been
+ published with different column lists is not supported.
+ </para>
+
+ <para>
We allow non-existent publications to be specified so that users can add
those later. This means
<link linkend="catalog-pg-subscription"><structname>pg_subscription</structname></link>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 690cdaa..83e6eae 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1754,6 +1754,11 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
* publisher connection.
+ *
+ * Note that we don't support the case where the column list is different for
+ * the same table in different publications to avoid sending unwanted column
+ * information for some of the rows. This can happen when both the column
+ * list and row filter are specified for different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1766,23 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, NAMEARRAYOID};
List *tablelist = NIL;
+ bool check_columnlist = (walrcv_server_version(wrconn) >= 150000);
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
- " FROM pg_catalog.pg_publication_tables t\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename \n");
+
+ /* Get column lists for each relation if the publisher supports it */
+ if (check_columnlist)
+ appendStringInfoString(&cmd, ", t.attnames\n");
+
+ appendStringInfoString(&cmd, "FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, check_columnlist ? 3 : 2, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1806,14 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ if (check_columnlist && list_member(tablelist, rv))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 994c7a0..670c6fc 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -753,17 +753,6 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Get column lists for each relation.
*
- * For initial synchronization, column lists can be ignored in following
- * cases:
- *
- * 1) one of the subscribed publications for the table hasn't specified
- * any column list
- *
- * 2) one of the subscribed publications has puballtables set to true
- *
- * 3) one of the subscribed publications is declared as ALL TABLES IN
- * SCHEMA that includes this relation
- *
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
@@ -771,7 +760,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,19 +775,17 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that makes
- * it easier to combine lists by simply adding the attnums to a new
- * bitmap (without having to parse the int2vector data). This
- * preserves NULL values, so that if one of the publications has no
- * column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
+ " THEN NULL ELSE gpt.attrs END)"
" FROM pg_publication p,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
pub_names.data);
@@ -813,26 +800,43 @@ fetch_remote_table_info(char *nspname, char *relname,
nspname, relname, pubres->err)));
/*
- * Merge the column lists (from different publications) by creating a
- * single bitmap with all the attnums. If we find a NULL value, that
- * means one of the publications has no column list for the table
- * we're syncing.
+ * We don't support the case where the column list is different for
+ * the same table when combining publications. See comments atop
+ * fetch_table_list. So there should be only one row returned.
+ * Although we already checked this when creating the subscription, we
+ * still need to check here in case the column list was changed after
+ * creating the subscription and before the sync worker is started.
+ */
+ if (tuplestore_tuple_count(pubres->tuplestore) > 1)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+
+ /*
+ * Get the column list and build a single bitmap with the attnums.
+ *
+ * If we find a NULL value, it means all the columns should be
+ * replicated.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
- }
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (natt = 0; natt < nelems; natt++)
+ included_cols = bms_add_member(included_cols, elems[natt]);
+ }
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 42c06af..8deae57 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -979,30 +979,31 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
- *
- * All the given publication-table mappings must be checked.
+ * build a bitmap using the column lists.
*
* Multiple publications might have multiple column lists for this
* relation.
*
+ * Note that we don't support the case where the column list is different
+ * for the same table when combining publications. See comments atop
+ * fetch_table_list. But one can later change the publication so we still
+ * need to check all the given publication-table mappings and report an
+ * error if any publications have a different column list.
+ *
* FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
- * list" so it takes precedence.
+ * list".
*/
foreach(lc, publications)
{
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as
@@ -1011,6 +1012,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1024,51 +1027,48 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (HeapTupleIsValid(cftuple))
{
- /*
- * Lookup the column list attribute.
- *
- * Note: We update the pub_no_list value directly, because if
- * the value is NULL, we have no list (and vice versa).
- */
+ /* Lookup the column list attribute. */
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
- /*
- * Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
- */
+ /* Build the column list bitmap in the per-entry context. */
if (!pub_no_list) /* when not null */
{
pgoutput_ensure_entry_cxt(data, entry);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
+
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 19812e1..7f031bc 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -20,6 +20,7 @@ $node_subscriber->append_conf('postgresql.conf',
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $offset = 0;
sub wait_for_subscription_sync
{
@@ -361,13 +362,13 @@ is( $result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -388,8 +389,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure the columns in column list get fully replicated
$node_publisher->safe_psql(
'postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
@@ -399,42 +399,12 @@ $node_publisher->safe_psql(
$node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
'overlapping publications with overlapping column lists');
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql(
- 'postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -900,57 +870,21 @@ is( $node_subscriber->safe_psql(
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql(
- 'postgres', "SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications which is FOR ALL TABLES, it
+# means replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql(
'postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -976,12 +910,11 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES
+# IN SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql(
'postgres', qq(
@@ -993,7 +926,7 @@ $node_publisher->safe_psql(
'postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -1017,7 +950,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
@@ -1074,7 +1007,7 @@ is( $node_subscriber->safe_psql(
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (with all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql(
@@ -1086,7 +1019,7 @@ $node_publisher->safe_psql(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1233,6 +1166,51 @@ is( $node_subscriber->safe_psql(
'publication containing both parent and child relation');
+# TEST: With a table included in multiple publications with different column
+# lists, we should catch the error when creating the subscription.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+));
+
+my ($cmdret, $stdout, $stderr) = $node_subscriber->psql(
+ 'postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+ok( $stderr =~
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ 'different column lists detected');
+
+# TEST: If the column list is changed after creating the subscription, we
+# should catch the error reported by walsender.
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, b);
+ INSERT INTO test_mix_1 VALUES(1, 1, 1);
+));
+
+$offset = $node_publisher->wait_for_log(
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ $offset);
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
1.8.3.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-27 05:53 ` Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Justin Pryzby @ 2022-05-27 05:53 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected]; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Tom Lane <[email protected]>
On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote:
> On Tue, May 24, 2022 at 11:03 AM [email protected] <[email protected]> wrote:
> >
> > On Friday, May 20, 2022 11:06 AM Amit Kapila <[email protected]> wrote:
> >
> > Thanks for pointing it out. Here is the new version patch which add this version check.
>
> I have added/edited a few comments and ran pgindent. The attached
> looks good to me. I'll push this early next week unless there are more
> comments/suggestions.
A minor doc review.
Note that I also sent some doc comments at [email protected].
+ lists among publications in which case <command>ALTER PUBLICATION</command>
+ command will be successful but later the WalSender in publisher or the
COMMA in which
remove "command" ?
s/in publisher/on the publisher/
+ Subscription having several publications in which the same table has been
+ published with different column lists is not supported.
Either "Subscriptions having .. are not supported"; or,
"A subscription having .. is not supported".
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
@ 2022-05-27 07:34 ` [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: [email protected] @ 2022-05-27 07:34 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Tom Lane <[email protected]>
On Friday, May 27, 2022 1:54 PM Justin Pryzby <[email protected]> wrote:
>
> On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote:
> > On Tue, May 24, 2022 at 11:03 AM [email protected]
> <[email protected]> wrote:
> > >
> > > On Friday, May 20, 2022 11:06 AM Amit Kapila <[email protected]>
> wrote:
> > >
> > > Thanks for pointing it out. Here is the new version patch which add this
> version check.
> >
> > I have added/edited a few comments and ran pgindent. The attached
> > looks good to me. I'll push this early next week unless there are more
> > comments/suggestions.
>
> A minor doc review.
> Note that I also sent some doc comments at
> [email protected].
>
> + lists among publications in which case <command>ALTER
> PUBLICATION</command>
> + command will be successful but later the WalSender in publisher
> + or the
>
> COMMA in which
>
> remove "command" ?
>
> s/in publisher/on the publisher/
>
> + Subscription having several publications in which the same table has been
> + published with different column lists is not supported.
>
> Either "Subscriptions having .. are not supported"; or, "A subscription having ..
> is not supported".
Thanks for the comments. Here is the new version patch set which fixes these.
Best regards,
Hou zj
Attachments:
[application/octet-stream] 0001-language-fixes-on-HEAD-from-Justin.patch (3.0K, ../../OS0PR01MB5716FFBE5AACA8707C3420E994D89@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-0001-language-fixes-on-HEAD-from-Justin.patch)
download | inline diff:
From 83a6e85100a173124396f8d4ac29ff170319f0a4 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Fri, 27 May 2022 15:24:52 +0800
Subject: [PATCH] language fixes from Justin
---
doc/src/sgml/catalogs.sgml | 6 +++---
src/backend/catalog/pg_publication.c | 2 +-
src/include/catalog/pg_proc.dat | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c00c93d..9808180 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9701,7 +9701,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry><link linkend="view-pg-publication-tables"><structname>pg_publication_tables</structname></link></entry>
- <entry>publications and information of their associated tables</entry>
+ <entry>publications and information about their associated tables</entry>
</row>
<row>
@@ -11645,7 +11645,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The view <structname>pg_publication_tables</structname> provides
- information about the mapping between publications and information of
+ information about the mapping between publications and information about
tables they contain. Unlike the underlying catalog
<link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>,
this view expands publications defined as <literal>FOR ALL TABLES</literal>
@@ -11705,7 +11705,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
Names of table columns included in the publication. This contains all
- the columns of the table when the user didn't specify the column list
+ the columns of the table when the user didn't specify a column list
for the table.
</para></entry>
</row>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 8c7fca6..2f706f6 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1077,7 +1077,7 @@ get_publication_name(Oid pubid, bool missing_ok)
}
/*
- * Returns information of tables in a publication.
+ * Returns information about tables in a publication.
*/
Datum
pg_get_publication_tables(PG_FUNCTION_ARGS)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87aa571..86f1329 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11673,7 +11673,7 @@
prosrc => 'pg_show_replication_origin_status' },
# publications
-{ oid => '6119', descr => 'get information of tables in a publication',
+{ oid => '6119', descr => 'get information about tables in a publication',
proname => 'pg_get_publication_tables', prorows => '1000', proretset => 't',
provolatile => 's', prorettype => 'record', proargtypes => 'text',
proallargtypes => '{text,oid,int2vector,pg_node_tree}', proargmodes => '{i,o,o,o}',
--
2.7.2.windows.1
[application/octet-stream] v6-0001-Prohibit-combining-publications-with-different-co.patch (22.4K, ../../OS0PR01MB5716FFBE5AACA8707C3420E994D89@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v6-0001-Prohibit-combining-publications-with-different-co.patch)
download | inline diff:
From 201721776782809c3bb99bb6ff8fbdedb02483ab Mon Sep 17 00:00:00 2001
From: Amit Kapila <[email protected]>
Date: Fri, 27 May 2022 08:30:33 +0530
Subject: [PATCH v5] Prohibit combining publications with different column
lists.
The main purpose of introducing a column list is to have statically
different shapes on publisher and subscriber or hide sensitive column
data. In both cases, it doesn't seem to make sense to combine column
lists.
So, we disallow the cases where the column list is different for the same
table when combining publications. It can be later extended to combine the
column lists for selective cases where required.
Reported-by: Alvaro Herrera
Author: Hou Zhijie
Reviewed-by: Amit Kapila, Alvaro Herrera, Shi Yu, Takamichi Osumi, Justin Pryzby
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/alter_publication.sgml | 12 ++-
doc/src/sgml/ref/create_subscription.sgml | 5 +
src/backend/commands/subscriptioncmds.c | 28 +++++-
src/backend/replication/logical/tablesync.c | 72 ++++++-------
src/backend/replication/pgoutput/pgoutput.c | 80 +++++++--------
src/test/subscription/t/031_column_list.pl | 150 ++++++++++++----------------
6 files changed, 181 insertions(+), 166 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49..f03933a 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -116,7 +116,17 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<para>
Optionally, a column list can be specified. See <xref
- linkend="sql-createpublication"/> for details.
+ linkend="sql-createpublication"/> for details. Note that a subscription
+ having several publications in which the same table has been published
+ with different column lists is not supported. So, changing the column
+ lists of the tables being subscribed could cause inconsistency of column
+ lists among publications, in which case <command>ALTER PUBLICATION</command>
+ will be successful but later the WalSender on the publisher or the
+ subscriber may throw an error. In this scenario, the user needs to
+ recreate the subscription after adjusting the column list or drop the
+ problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
+ it back after adjusting the column list.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 203bb41..f6f82a0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -356,6 +356,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<para>
+ Subscriptions having several publications in which the same table has been
+ published with different column lists are not supported.
+ </para>
+
+ <para>
We allow non-existent publications to be specified so that users can add
those later. This means
<link linkend="catalog-pg-subscription"><structname>pg_subscription</structname></link>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 690cdaa..83e6eae 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1754,6 +1754,11 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
/*
* Get the list of tables which belong to specified publications on the
* publisher connection.
+ *
+ * Note that we don't support the case where the column list is different for
+ * the same table in different publications to avoid sending unwanted column
+ * information for some of the rows. This can happen when both the column
+ * list and row filter are specified for different publications.
*/
static List *
fetch_table_list(WalReceiverConn *wrconn, List *publications)
@@ -1761,17 +1766,23 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[2] = {TEXTOID, TEXTOID};
+ Oid tableRow[3] = {TEXTOID, TEXTOID, NAMEARRAYOID};
List *tablelist = NIL;
+ bool check_columnlist = (walrcv_server_version(wrconn) >= 150000);
initStringInfo(&cmd);
- appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename\n"
- " FROM pg_catalog.pg_publication_tables t\n"
+ appendStringInfoString(&cmd, "SELECT DISTINCT t.schemaname, t.tablename \n");
+
+ /* Get column lists for each relation if the publisher supports it */
+ if (check_columnlist)
+ appendStringInfoString(&cmd, ", t.attnames\n");
+
+ appendStringInfoString(&cmd, "FROM pg_catalog.pg_publication_tables t\n"
" WHERE t.pubname IN (");
get_publications_str(publications, &cmd, true);
appendStringInfoChar(&cmd, ')');
- res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ res = walrcv_exec(wrconn, cmd.data, check_columnlist ? 3 : 2, tableRow);
pfree(cmd.data);
if (res->status != WALRCV_OK_TUPLES)
@@ -1795,7 +1806,14 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
Assert(!isnull);
rv = makeRangeVar(nspname, relname, -1);
- tablelist = lappend(tablelist, rv);
+
+ if (check_columnlist && list_member(tablelist, rv))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+ else
+ tablelist = lappend(tablelist, rv);
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 994c7a0..670c6fc 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -753,17 +753,6 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Get column lists for each relation.
*
- * For initial synchronization, column lists can be ignored in following
- * cases:
- *
- * 1) one of the subscribed publications for the table hasn't specified
- * any column list
- *
- * 2) one of the subscribed publications has puballtables set to true
- *
- * 3) one of the subscribed publications is declared as ALL TABLES IN
- * SCHEMA that includes this relation
- *
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
@@ -771,7 +760,7 @@ fetch_remote_table_info(char *nspname, char *relname,
{
WalRcvExecResult *pubres;
TupleTableSlot *slot;
- Oid attrsRow[] = {INT2OID};
+ Oid attrsRow[] = {INT2VECTOROID};
StringInfoData pub_names;
bool first = true;
@@ -786,19 +775,17 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* Fetch info about column lists for the relation (from all the
- * publications). We unnest the int2vector values, because that makes
- * it easier to combine lists by simply adding the attnums to a new
- * bitmap (without having to parse the int2vector data). This
- * preserves NULL values, so that if one of the publications has no
- * column list, we'll know that.
+ * publications).
*/
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT DISTINCT unnest"
+ "SELECT DISTINCT"
+ " (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
+ " THEN NULL ELSE gpt.attrs END)"
" FROM pg_publication p,"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " LEFT OUTER JOIN unnest(gpt.attrs) ON TRUE"
- " WHERE gpt.relid = %u"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt,"
+ " pg_class c"
+ " WHERE gpt.relid = %u AND c.oid = gpt.relid"
" AND p.pubname IN ( %s )",
lrel->remoteid,
pub_names.data);
@@ -813,26 +800,43 @@ fetch_remote_table_info(char *nspname, char *relname,
nspname, relname, pubres->err)));
/*
- * Merge the column lists (from different publications) by creating a
- * single bitmap with all the attnums. If we find a NULL value, that
- * means one of the publications has no column list for the table
- * we're syncing.
+ * We don't support the case where the column list is different for
+ * the same table when combining publications. See comments atop
+ * fetch_table_list. So there should be only one row returned.
+ * Although we already checked this when creating the subscription, we
+ * still need to check here in case the column list was changed after
+ * creating the subscription and before the sync worker is started.
+ */
+ if (tuplestore_tuple_count(pubres->tuplestore) > 1)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ nspname, relname));
+
+ /*
+ * Get the column list and build a single bitmap with the attnums.
+ *
+ * If we find a NULL value, it means all the columns should be
+ * replicated.
*/
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
Datum cfval = slot_getattr(slot, 1, &isnull);
- /* NULL means empty column list, so we're done. */
- if (isnull)
+ if (!isnull)
{
- bms_free(included_cols);
- included_cols = NULL;
- break;
- }
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
+ arr = DatumGetArrayTypeP(cfval);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (natt = 0; natt < nelems; natt++)
+ included_cols = bms_add_member(included_cols, elems[natt]);
+ }
ExecClearTuple(slot);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 42c06af..8deae57 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -979,30 +979,31 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
RelationSyncEntry *entry)
{
ListCell *lc;
+ bool first = true;
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
/*
* Find if there are any column lists for this relation. If there are,
- * build a bitmap merging all the column lists.
- *
- * All the given publication-table mappings must be checked.
+ * build a bitmap using the column lists.
*
* Multiple publications might have multiple column lists for this
* relation.
*
+ * Note that we don't support the case where the column list is different
+ * for the same table when combining publications. See comments atop
+ * fetch_table_list. But one can later change the publication so we still
+ * need to check all the given publication-table mappings and report an
+ * error if any publications have a different column list.
+ *
* FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
- * list" so it takes precedence.
+ * list".
*/
foreach(lc, publications)
{
Publication *pub = lfirst(lc);
HeapTuple cftuple = NULL;
Datum cfdatum = 0;
-
- /*
- * Assume there's no column list. Only if we find pg_publication_rel
- * entry with a column list we'll switch it to false.
- */
- bool pub_no_list = true;
+ Bitmapset *cols = NULL;
/*
* If the publication is FOR ALL TABLES then it is treated the same as
@@ -1011,6 +1012,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
*/
if (!pub->alltables)
{
+ bool pub_no_list = true;
+
/*
* Check for the presence of a column list in this publication.
*
@@ -1024,51 +1027,48 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (HeapTupleIsValid(cftuple))
{
- /*
- * Lookup the column list attribute.
- *
- * Note: We update the pub_no_list value directly, because if
- * the value is NULL, we have no list (and vice versa).
- */
+ /* Lookup the column list attribute. */
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
- /*
- * Build the column list bitmap in the per-entry context.
- *
- * We need to merge column lists from all publications, so we
- * update the same bitmapset. If the column list is null, we
- * interpret it as replicating all columns.
- */
+ /* Build the column list bitmap in the per-entry context. */
if (!pub_no_list) /* when not null */
{
pgoutput_ensure_entry_cxt(data, entry);
- entry->columns = pub_collist_to_bitmapset(entry->columns,
- cfdatum,
- entry->entry_cxt);
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
+ entry->entry_cxt);
+
+ /*
+ * If column list includes all the columns of the table,
+ * set it to NULL.
+ */
+ if (bms_num_members(cols) == RelationGetNumberOfAttributes(relation))
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
}
+
+ ReleaseSysCache(cftuple);
}
}
- /*
- * Found a publication with no column list, so we're done. But first
- * discard column list we might have from preceding publications.
- */
- if (pub_no_list)
+ if (first)
{
- if (cftuple)
- ReleaseSysCache(cftuple);
-
- bms_free(entry->columns);
- entry->columns = NULL;
-
- break;
+ entry->columns = cols;
+ first = false;
}
-
- ReleaseSysCache(cftuple);
+ else if (!bms_equal(entry->columns, cols))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different column lists for table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
} /* loop all subscribed publications */
+
+ RelationClose(relation);
}
/*
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 19812e1..7f031bc 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -20,6 +20,7 @@ $node_subscriber->append_conf('postgresql.conf',
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $offset = 0;
sub wait_for_subscription_sync
{
@@ -361,13 +362,13 @@ is( $result, qq(1|abc
2|xyz), 'update on column tab2.c is not replicated');
-# TEST: add a table to two publications with different column lists, and
+# TEST: add a table to two publications with same column lists, and
# create a single subscription replicating both publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
- CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, b);
-- insert a couple initial records
INSERT INTO tab5 VALUES (1, 11, 111, 1111);
@@ -388,8 +389,7 @@ wait_for_subscription_sync($node_subscriber);
$node_publisher->wait_for_catchup('sub1');
-# insert data and make sure all the columns (union of the columns lists)
-# get fully replicated
+# insert data and make sure the columns in column list get fully replicated
$node_publisher->safe_psql(
'postgres', qq(
INSERT INTO tab5 VALUES (3, 33, 333, 3333);
@@ -399,42 +399,12 @@ $node_publisher->safe_psql(
$node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111
-2|22|2222
-3|33|3333
-4|44|4444),
+ qq(1|11|
+2|22|
+3|33|
+4|44|),
'overlapping publications with overlapping column lists');
-# and finally, remove the column list for one of the publications, which
-# means replicating all columns (removing the column list), but first add
-# the missing column to the table on subscriber
-$node_publisher->safe_psql(
- 'postgres', qq(
- ALTER PUBLICATION pub3 SET TABLE tab5;
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
- ALTER TABLE tab5 ADD COLUMN c INT;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO tab5 VALUES (5, 55, 555, 5555);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a"),
- qq(1|11|1111|
-2|22|2222|
-3|33|3333|
-4|44|4444|
-5|55|5555|555),
- 'overlapping publications with overlapping column lists');
# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
@@ -900,57 +870,21 @@ is( $node_subscriber->safe_psql(
3|),
'partitions with different replica identities not replicated correctly');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. So with column lists (a,b) and (a,c) we
-# should replicate (a,b,c).
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
- CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
-
- -- initial data
- INSERT INTO test_mix_1 VALUES (1, 2, 3);
-));
-
-$node_subscriber->safe_psql(
- 'postgres', qq(
- CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
- ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
-));
-
-wait_for_subscription_sync($node_subscriber);
-
-$node_publisher->safe_psql(
- 'postgres', qq(
- INSERT INTO test_mix_1 VALUES (4, 5, 6);
-));
-
-$node_publisher->wait_for_catchup('sub1');
-
-is( $node_subscriber->safe_psql(
- 'postgres', "SELECT * FROM test_mix_1 ORDER BY a"),
- qq(1|2|3
-4|5|6),
- 'a mix of publications should use a union of column list');
-
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES, we should replicate all columns.
+# TEST: With a table included in the publications which is FOR ALL TABLES, it
+# means replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
$node_publisher->safe_psql(
'postgres', qq(
- DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7,
test_part, test_part_a, test_part_b, test_part_c, test_part_d;
));
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b, c);
CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
-- initial data
@@ -976,12 +910,11 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_2"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
-# TEST: With a table included in multiple publications, we should use a
-# union of the column lists. If any of the publications is FOR ALL
-# TABLES IN SCHEMA, we should replicate all columns.
+# TEST: With a table included in the publication which is FOR ALL TABLES
+# IN SCHEMA, it means replicate all columns.
$node_subscriber->safe_psql(
'postgres', qq(
@@ -993,7 +926,7 @@ $node_publisher->safe_psql(
'postgres', qq(
DROP TABLE test_mix_2;
CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
- CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b, c);
CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
-- initial data
@@ -1017,7 +950,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql('postgres', "SELECT * FROM test_mix_3"),
qq(1|2|3
4|5|6),
- 'a mix of publications should use a union of column list');
+ 'all columns should be replicated');
# TEST: Check handling of publish_via_partition_root - if a partition is
@@ -1074,7 +1007,7 @@ is( $node_subscriber->safe_psql(
# TEST: Multiple publications which publish schema of parent table and
# partition. The partition is published through two publications, once
# through a schema (so no column list) containing the parent, and then
-# also directly (with a columns list). The expected outcome is there is
+# also directly (with all columns). The expected outcome is there is
# no column list.
$node_publisher->safe_psql(
@@ -1086,7 +1019,7 @@ $node_publisher->safe_psql(
CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
- CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+ CREATE PUBLICATION pub2 FOR TABLE t_1(a, b, c);
-- initial data
INSERT INTO s1.t VALUES (1, 2, 3);
@@ -1233,6 +1166,51 @@ is( $node_subscriber->safe_psql(
'publication containing both parent and child relation');
+# TEST: With a table included in multiple publications with different column
+# lists, we should catch the error when creating the subscription.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+));
+
+my ($cmdret, $stdout, $stderr) = $node_subscriber->psql(
+ 'postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+ok( $stderr =~
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ 'different column lists detected');
+
+# TEST: If the column list is changed after creating the subscription, we
+# should catch the error reported by walsender.
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub_mix_1 SET TABLE test_mix_1 (a, b);
+ INSERT INTO test_mix_1 VALUES(1, 1, 1);
+));
+
+$offset = $node_publisher->wait_for_log(
+ qr/cannot use different column lists for table "public.test_mix_1" in different publications/,
+ $offset);
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
1.8.3.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
@ 2022-06-02 11:58 ` Amit Kapila <[email protected]>
2022-06-06 05:42 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-06-02 11:58 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Tom Lane <[email protected]>
On Fri, May 27, 2022 at 1:04 PM [email protected]
<[email protected]> wrote:
>
> On Friday, May 27, 2022 1:54 PM Justin Pryzby <[email protected]> wrote:
> >
> > On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote:
> > > On Tue, May 24, 2022 at 11:03 AM [email protected]
> > <[email protected]> wrote:
> > > >
> > > > On Friday, May 20, 2022 11:06 AM Amit Kapila <[email protected]>
> > wrote:
> > > >
> > > > Thanks for pointing it out. Here is the new version patch which add this
> > version check.
> > >
> > > I have added/edited a few comments and ran pgindent. The attached
> > > looks good to me. I'll push this early next week unless there are more
> > > comments/suggestions.
> >
> > A minor doc review.
> > Note that I also sent some doc comments at
> > [email protected].
> >
> > + lists among publications in which case <command>ALTER
> > PUBLICATION</command>
> > + command will be successful but later the WalSender in publisher
> > + or the
> >
> > COMMA in which
> >
> > remove "command" ?
> >
> > s/in publisher/on the publisher/
> >
> > + Subscription having several publications in which the same table has been
> > + published with different column lists is not supported.
> >
> > Either "Subscriptions having .. are not supported"; or, "A subscription having ..
> > is not supported".
>
> Thanks for the comments. Here is the new version patch set which fixes these.
>
I have pushed the bug-fix patch. I'll look at the language
improvements patch next.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-06-06 05:42 ` Peter Smith <[email protected]>
2022-06-08 03:25 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Peter Smith @ 2022-06-06 05:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Tom Lane <[email protected]>
On Thu, Jun 2, 2022 at 9:58 PM Amit Kapila <[email protected]> wrote:
>
> On Fri, May 27, 2022 at 1:04 PM [email protected]
> <[email protected]> wrote:
> >
> > On Friday, May 27, 2022 1:54 PM Justin Pryzby <[email protected]> wrote:
> > >
> > > On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote:
> > > > On Tue, May 24, 2022 at 11:03 AM [email protected]
> > > <[email protected]> wrote:
> > > > >
> > > > > On Friday, May 20, 2022 11:06 AM Amit Kapila <[email protected]>
> > > wrote:
> > > > >
> > > > > Thanks for pointing it out. Here is the new version patch which add this
> > > version check.
> > > >
> > > > I have added/edited a few comments and ran pgindent. The attached
> > > > looks good to me. I'll push this early next week unless there are more
> > > > comments/suggestions.
> > >
> > > A minor doc review.
> > > Note that I also sent some doc comments at
> > > [email protected].
> > >
> > > + lists among publications in which case <command>ALTER
> > > PUBLICATION</command>
> > > + command will be successful but later the WalSender in publisher
> > > + or the
> > >
> > > COMMA in which
> > >
> > > remove "command" ?
> > >
> > > s/in publisher/on the publisher/
> > >
> > > + Subscription having several publications in which the same table has been
> > > + published with different column lists is not supported.
> > >
> > > Either "Subscriptions having .. are not supported"; or, "A subscription having ..
> > > is not supported".
> >
> > Thanks for the comments. Here is the new version patch set which fixes these.
> >
>
> I have pushed the bug-fix patch. I'll look at the language
> improvements patch next.
I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says:
@@ -11673,7 +11673,7 @@
prosrc => 'pg_show_replication_origin_status' },
# publications
-{ oid => '6119', descr => 'get information of tables in a publication',
+{ oid => '6119', descr => 'get information about tables in a publication',
~~~
But, this grammar website [1] says:
What Does Of Mean
As defined by Cambridge dictionary Of is basically used “to show
possession, belonging, or origin”.
What Does About Mean
Similarly about primarily indicates ‘On the subject of; concerning’ as
defined by the Oxford dictionary. Or about in brief highlights some
fact ‘on the subject of, or connected with’
The main difference between of and about is that of implies a
possessive quality while about implies concerning or on the subject of
something.
~~~
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-06-06 05:42 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
@ 2022-06-08 03:25 ` Justin Pryzby <[email protected]>
2022-06-08 05:35 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Justin Pryzby @ 2022-06-08 03:25 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected]; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Tom Lane <[email protected]>
On Mon, Jun 06, 2022 at 03:42:31PM +1000, Peter Smith wrote:
> I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says:
>
> @@ -11673,7 +11673,7 @@
> prosrc => 'pg_show_replication_origin_status' },
>
> # publications
> -{ oid => '6119', descr => 'get information of tables in a publication',
> +{ oid => '6119', descr => 'get information about tables in a publication',
>
> ~~~
>
> But, this grammar website [1] says:
...
> From which I guess
>
> 1. 'get information of tables in a publication' ~= 'get information
> belonging to tables in a publication'
But the information doesn't "belong to" the tables.
The information is "regarding" the tables (or "associated with" or "concerned
with" or "respecting" or "on the subject of" the tables).
I think my change is correct based on the grammar definition, as well as its
intuitive "feel".
--
Justin
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-06-06 05:42 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
2022-06-08 03:25 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
@ 2022-06-08 05:35 ` Peter Smith <[email protected]>
2022-06-13 03:24 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Peter Smith @ 2022-06-08 05:35 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected]; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Tom Lane <[email protected]>
On Wed, Jun 8, 2022 at 1:25 PM Justin Pryzby <[email protected]> wrote:
>
> On Mon, Jun 06, 2022 at 03:42:31PM +1000, Peter Smith wrote:
> > I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says:
> >
> > @@ -11673,7 +11673,7 @@
> > prosrc => 'pg_show_replication_origin_status' },
> >
> > # publications
> > -{ oid => '6119', descr => 'get information of tables in a publication',
> > +{ oid => '6119', descr => 'get information about tables in a publication',
> >
> > ~~~
> >
> > But, this grammar website [1] says:
> ...
> > From which I guess
> >
> > 1. 'get information of tables in a publication' ~= 'get information
> > belonging to tables in a publication'
>
> But the information doesn't "belong to" the tables.
>
> The information is "regarding" the tables (or "associated with" or "concerned
> with" or "respecting" or "on the subject of" the tables).
>
> I think my change is correct based on the grammar definition, as well as its
> intuitive "feel".
>
Actually, I have no problem with this being worded either way. My
point was mostly to question if it was really worth changing it at
this time - e.g. I think there is a reluctance to change anything to
do with the catalogs during beta (even when a catversion bump may not
be required).
I agree that "about" seems better if the text said, "get information
about tables". But it does not say that - it says "get information
about tables in a publication" which I felt made a subtle difference.
e.g.1 "... on the subject of / concerned with tables."
- sounds like attributes about each table (col names, row filter etc)
versus
e.g.2 "... on the subject of / concerned with tables in a publication."
- sounds less like information PER table, and more like information
about the table membership of the publication.
~~
Any ambiguities can be eliminated if this text was just fixed to be
consistent with the wording of catalogs.sgml:
e.g. "publications and information about their associated tables"
But then this comes full circle back to my question if during beta is
a good time to be making such a change.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-06-06 05:42 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
2022-06-08 03:25 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-06-08 05:35 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
@ 2022-06-13 03:24 ` Amit Kapila <[email protected]>
2022-06-16 04:23 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-06-13 03:24 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Hou, Zhijie/侯 志杰 <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Tom Lane <[email protected]>
On Wed, Jun 8, 2022 at 11:05 AM Peter Smith <[email protected]> wrote:
>
> On Wed, Jun 8, 2022 at 1:25 PM Justin Pryzby <[email protected]> wrote:
> >
> > On Mon, Jun 06, 2022 at 03:42:31PM +1000, Peter Smith wrote:
> > > I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says:
> > >
> > > @@ -11673,7 +11673,7 @@
> > > prosrc => 'pg_show_replication_origin_status' },
> > >
> > > # publications
> > > -{ oid => '6119', descr => 'get information of tables in a publication',
> > > +{ oid => '6119', descr => 'get information about tables in a publication',
> > >
> > > ~~~
> > >
> > > But, this grammar website [1] says:
> > ...
> > > From which I guess
> > >
> > > 1. 'get information of tables in a publication' ~= 'get information
> > > belonging to tables in a publication'
> >
> > But the information doesn't "belong to" the tables.
> >
> > The information is "regarding" the tables (or "associated with" or "concerned
> > with" or "respecting" or "on the subject of" the tables).
> >
> > I think my change is correct based on the grammar definition, as well as its
> > intuitive "feel".
> >
>
> Actually, I have no problem with this being worded either way. My
> point was mostly to question if it was really worth changing it at
> this time - e.g. I think there is a reluctance to change anything to
> do with the catalogs during beta (even when a catversion bump may not
> be required).
>
> I agree that "about" seems better if the text said, "get information
> about tables". But it does not say that - it says "get information
> about tables in a publication" which I felt made a subtle difference.
>
> e.g.1 "... on the subject of / concerned with tables."
> - sounds like attributes about each table (col names, row filter etc)
>
> versus
>
> e.g.2 "... on the subject of / concerned with tables in a publication."
> - sounds less like information PER table, and more like information
> about the table membership of the publication.
>
> ~~
>
> Any ambiguities can be eliminated if this text was just fixed to be
> consistent with the wording of catalogs.sgml:
> e.g. "publications and information about their associated tables"
>
I don't know if this is better than the current text for this view:
'get information of tables in a publication' and unless we have a
consensus on any change here, I think it is better to retain the
current text as it is.
I would like to close the Open item listed corresponding to this
thread [1] as the fix for the reported issue is committed
(fd0b9dcebd). Do let me know if you or others think otherwise?
[1] - https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-06-06 05:42 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
2022-06-08 03:25 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-06-08 05:35 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
2022-06-13 03:24 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-06-16 04:23 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2022-06-16 04:23 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Hou, Zhijie/侯 志杰 <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Tom Lane <[email protected]>
On Mon, Jun 13, 2022 at 8:54 AM Amit Kapila <[email protected]> wrote:
>
> I would like to close the Open item listed corresponding to this
> thread [1] as the fix for the reported issue is committed
> (fd0b9dcebd). Do let me know if you or others think otherwise?
>
Seeing no objections, I have closed this item.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-24 09:49 ` Amit Kapila <[email protected]>
2022-05-26 03:26 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-24 09:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Hou, Zhijie/侯 志杰 <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Fri, May 20, 2022 at 8:36 AM Amit Kapila <[email protected]> wrote:
>
> On Thu, May 19, 2022 at 7:54 PM Tom Lane <[email protected]> wrote:
> >
> > Justin Pryzby <[email protected]> writes:
> > > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote:
> > >> I have committed the first patch after fixing this part. It seems Tom
> > >> is not very happy doing this after beta-1 [1]. The reason we get this
> > >> information via this view (and underlying function) is that it
> > >> simplifies the queries on the subscriber-side as you can see in the
> > >> second patch. The query change is as below:
> > >> [1] - https://www.postgresql.org/message-id/91075.1652929852%40sss.pgh.pa.us
> >
> > > I think Tom's concern is that adding information to a view seems like adding a
> > > feature that hadn't previously been contemplated.
> > > (Catalog changes themselves are not prohibited during the beta period).
> >
> > It certainly smells like a new feature, but my concern was more around the
> > post-beta catalog change. We do those only if really forced to, and the
> > explanation in the commit message didn't satisfy me as to why it was
> > necessary. This explanation isn't much better --- if we're trying to
> > prohibit a certain class of publication definitions, what good does it do
> > to check that on the subscriber side?
> >
>
> It is required on the subscriber side because prohibition is only for
> the cases where multiple publications are combined. We disallow the
> cases where the column list is different for the same table when
> combining publications. For example:
>
> Publisher-side:
> Create table tab(c1 int, c2 int);
> Create Publication pub1 for table tab(c1);
> Create Publication pub1 for table tab(c2);
>
> Subscriber-side:
> Create Subscription sub1 Connection 'dbname=postgres' Publication pub1, pub2;
>
> We want to prohibit such cases. So, it would be better to check at the
> time of 'Create Subscription' to validate such combinations and
> prohibit them. To achieve that we extended the existing function
> pg_get_publication_tables() and view pg_publication_tables to expose
> the column list and verify such a combination. We primarily need
> column list information for this prohibition but it appeared natural
> to expose the row filter.
>
I still feel that the current approach to extend the underlying
function and view is a better idea but if you and or others are not
convinced then we can try to achieve it by extending the existing
query on the subscriber side as mentioned in my previous email [1].
Kindly let me know your opinion?
[1] - https://www.postgresql.org/message-id/CAA4eK1KfL%3Dez5fKPB-0Nrgf7wiqN9bXP-YHHj2YH5utXAmjYug%40mail.g...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 09:49 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-26 03:26 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2022-05-26 03:26 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Hou, Zhijie/侯 志杰 <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Tue, May 24, 2022 at 3:19 PM Amit Kapila <[email protected]> wrote:
>
> On Fri, May 20, 2022 at 8:36 AM Amit Kapila <[email protected]> wrote:
> >
> > On Thu, May 19, 2022 at 7:54 PM Tom Lane <[email protected]> wrote:
> > >
> > > Justin Pryzby <[email protected]> writes:
> > > > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote:
> > > >> I have committed the first patch after fixing this part. It seems Tom
> > > >> is not very happy doing this after beta-1 [1]. The reason we get this
> > > >> information via this view (and underlying function) is that it
> > > >> simplifies the queries on the subscriber-side as you can see in the
> > > >> second patch. The query change is as below:
> > > >> [1] - https://www.postgresql.org/message-id/91075.1652929852%40sss.pgh.pa.us
> > >
> > > > I think Tom's concern is that adding information to a view seems like adding a
> > > > feature that hadn't previously been contemplated.
> > > > (Catalog changes themselves are not prohibited during the beta period).
> > >
> > > It certainly smells like a new feature, but my concern was more around the
> > > post-beta catalog change. We do those only if really forced to, and the
> > > explanation in the commit message didn't satisfy me as to why it was
> > > necessary. This explanation isn't much better --- if we're trying to
> > > prohibit a certain class of publication definitions, what good does it do
> > > to check that on the subscriber side?
> > >
> >
> > It is required on the subscriber side because prohibition is only for
> > the cases where multiple publications are combined. We disallow the
> > cases where the column list is different for the same table when
> > combining publications. For example:
> >
> > Publisher-side:
> > Create table tab(c1 int, c2 int);
> > Create Publication pub1 for table tab(c1);
> > Create Publication pub1 for table tab(c2);
> >
> > Subscriber-side:
> > Create Subscription sub1 Connection 'dbname=postgres' Publication pub1, pub2;
> >
> > We want to prohibit such cases. So, it would be better to check at the
> > time of 'Create Subscription' to validate such combinations and
> > prohibit them. To achieve that we extended the existing function
> > pg_get_publication_tables() and view pg_publication_tables to expose
> > the column list and verify such a combination. We primarily need
> > column list information for this prohibition but it appeared natural
> > to expose the row filter.
> >
>
> I still feel that the current approach to extend the underlying
> function and view is a better idea but if you and or others are not
> convinced then we can try to achieve it by extending the existing
> query on the subscriber side as mentioned in my previous email [1].
> Kindly let me know your opinion?
>
Unless someone has objections or thinks otherwise, I am planning to
proceed with the approach of extending the function/view (patch for
which is already committed) and using it to prohibit the combinations
of publications having different column lists as is done in the
currently proposed patch [1].
[1] - https://www.postgresql.org/message-id/OS0PR01MB5716AD7C0FE7386630BDBAAB94D79%40OS0PR01MB5716.jpnprd0...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
@ 2022-05-02 11:23 ` Amit Kapila <[email protected]>
2022-05-02 18:37 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2022-05-02 11:23 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On Mon, May 2, 2022 at 3:05 PM Tomas Vondra
<[email protected]> wrote:
>
> On 5/2/22 07:31, Amit Kapila wrote:
> > On Mon, May 2, 2022 at 3:27 AM Tomas Vondra
> > <[email protected]> wrote:
> >>
>
> >> The second option has the annoying consequence that it makes this
> >> useless for the "data redaction" use case I described in [2], because
> >> that relies on combining multiple publications.
> >>
> >
> > True, but as a workaround users can create different subscriptions for
> > different publications.
> >
>
> Won't that replicate duplicate data, when the row filters re not
> mutually exclusive?
>
True, but this is a recommendation for mutually exclusive data, and as
far as I can understand the example given by you [1] and Alvaro has
mutually exclusive conditions. In your example, one of the
publications has a condition (region = 'USA') and the other
publication has a condition (region != 'USA'), so will there be a
problem in using different subscriptions for such cases?
[1] - https://www.postgresql.org/message-id/[email protected]
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 11:23 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
@ 2022-05-02 18:37 ` Tomas Vondra <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2022-05-02 18:37 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Peter Eisentraut <[email protected]>; Pg Hackers <[email protected]>
On 5/2/22 13:23, Amit Kapila wrote:
> On Mon, May 2, 2022 at 3:05 PM Tomas Vondra
> <[email protected]> wrote:
>>
>> On 5/2/22 07:31, Amit Kapila wrote:
>>> On Mon, May 2, 2022 at 3:27 AM Tomas Vondra
>>> <[email protected]> wrote:
>>>>
>>
>>>> The second option has the annoying consequence that it makes this
>>>> useless for the "data redaction" use case I described in [2], because
>>>> that relies on combining multiple publications.
>>>>
>>>
>>> True, but as a workaround users can create different subscriptions for
>>> different publications.
>>>
>>
>> Won't that replicate duplicate data, when the row filters re not
>> mutually exclusive?
>>
>
> True, but this is a recommendation for mutually exclusive data, and as
> far as I can understand the example given by you [1] and Alvaro has
> mutually exclusive conditions. In your example, one of the
> publications has a condition (region = 'USA') and the other
> publication has a condition (region != 'USA'), so will there be a
> problem in using different subscriptions for such cases?
>
I kept that example intentionally simple, but I'm sure we could come up
with more complex use cases. Following the "data redaction" idea, we
could also apply the "deny all" approach, and do something like this:
-- replicate the minimal column list by default (replica identity)
CREATE PUBLICATION p1 FOR TABLE t (id, region);
-- replicate more columns for the selected region
CREATE PUBLICATION p2 FOR TABLE t (...) WHERE (region = 'USA')
Now, I admit this is something I just made up, but I think it seems like
a pretty common approach.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: bogus: logical replication rows/cols combinations
@ 2022-05-02 16:30 Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Alvaro Herrera @ 2022-05-02 16:30 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Pg Hackers <[email protected]>
On 2022-Apr-28, Tomas Vondra wrote:
> SELECT
> (CASE WHEN (a < 0) OR (a > 0) THEN a ELSE NULL END) AS a,
> (CASE WHEN (a > 0) THEN b ELSE NULL END) AS b,
> (CASE WHEN (a < 0) THEN c ELSE NULL END) AS c
> FROM uno WHERE (a < 0) OR (a > 0)
BTW, looking at the new COPY commands, the idea of "COPY table_foo
(PUBLICATION pub1, pub2)" is looking more and more attractive, as a
replacement for having the replica cons up an ad-hoc subquery to COPY
from. Something to think about for pg16, maybe.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"You're _really_ hosed if the person doing the hiring doesn't understand
relational systems: you end up with a whole raft of programmers, none of
whom has had a Date with the clue stick." (Andrew Sullivan)
^ permalink raw reply [nested|flat] 59+ messages in thread
end of thread, other threads:[~2022-06-16 04:23 UTC | newest]
Thread overview: 59+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-03 18:00 [PATCH 5/6] BRIN minmax-multi indexes Tomas Vondra <[email protected]>
2022-04-27 06:12 ` Re: bogus: logical replication rows/cols combinations Michael Paquier <[email protected]>
2022-04-29 05:05 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-01 21:42 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 20:34 ` Re: bogus: logical replication rows/cols combinations Peter Eisentraut <[email protected]>
2022-05-03 19:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-04 13:56 ` Re: bogus: logical replication rows/cols combinations Peter Eisentraut <[email protected]>
2022-05-02 09:35 Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:17 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 10:23 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 10:55 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 11:14 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:44 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 17:36 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 17:51 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-02 18:40 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:30 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-06 03:23 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-06 12:26 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-06 13:40 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-06 13:57 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-03 03:53 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-07 05:36 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-08 18:11 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-09 03:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-10 19:05 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-11 03:33 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-11 07:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-12 06:45 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-12 08:32 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-13 06:02 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-16 06:10 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 12:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 03:25 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 06:49 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-17 06:52 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-17 09:10 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-18 02:28 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-16 13:20 ` Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
2022-05-17 03:26 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 05:03 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-19 12:07 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-19 14:24 ` Re: bogus: logical replication rows/cols combinations Tom Lane <[email protected]>
2022-05-20 03:06 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 05:33 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-05-27 05:47 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-27 05:53 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-05-27 07:34 ` RE: bogus: logical replication rows/cols combinations [email protected] <[email protected]>
2022-06-02 11:58 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-06-06 05:42 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
2022-06-08 03:25 ` Re: bogus: logical replication rows/cols combinations Justin Pryzby <[email protected]>
2022-06-08 05:35 ` Re: bogus: logical replication rows/cols combinations Peter Smith <[email protected]>
2022-06-13 03:24 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-06-16 04:23 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-24 09:49 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-26 03:26 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 11:23 ` Re: bogus: logical replication rows/cols combinations Amit Kapila <[email protected]>
2022-05-02 18:37 ` Re: bogus: logical replication rows/cols combinations Tomas Vondra <[email protected]>
2022-05-02 16:30 Re: bogus: logical replication rows/cols combinations Alvaro Herrera <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox