public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 8/9] Define multi-minmax oclasses for types without distance
5+ messages / 4 participants
[nested] [flat]
* [PATCH 8/9] Define multi-minmax oclasses for types without distance
@ 2021-02-03 18:12 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Tomas Vondra @ 2021-02-03 18:12 UTC (permalink / raw)
The existing multi-minmax opclasses rely on "distance" function when
deciding how to build the ranges, covering all the values. In principle,
those opclasses try to minimize the lengths of ranges, i.e. maximize the
lengths of "gaps" between them.
For some data types it's hard to construct a distance function - types
like "text" or "name" generally serve as labels, and may have ordering
only. So this uses a different approach, based on the observation that
it's the outliers that "break" BRIN minmax indexes, i.e. the lowest and
highest values. So simply sort the values, keep the (K-2) extreme values
on either tail, and build a single range representing the values in the
middle. Of course, the question is whether this is actually useful, but
that's hard to judge, and it's the best we can do.
---
src/backend/access/brin/brin_minmax_multi.c | 75 ++++++++++++
src/include/catalog/pg_amop.dat | 119 ++++++++++++++++++++
src/include/catalog/pg_amproc.dat | 113 +++++++++++++++++++
src/include/catalog/pg_opclass.dat | 21 ++++
src/include/catalog/pg_opfamily.dat | 14 +++
5 files changed, 342 insertions(+)
diff --git a/src/backend/access/brin/brin_minmax_multi.c b/src/backend/access/brin/brin_minmax_multi.c
index 90d17e5008..02aa618b49 100644
--- a/src/backend/access/brin/brin_minmax_multi.c
+++ b/src/backend/access/brin/brin_minmax_multi.c
@@ -1210,6 +1210,9 @@ build_distances(FmgrInfo *distanceFn, Oid colloid,
Assert(ncranges >= 2);
+ if (!distanceFn)
+ return NULL;
+
distances = (DistanceValue *) palloc0(sizeof(DistanceValue) * (ncranges - 1));
/*
@@ -1298,6 +1301,70 @@ count_values(CombineRange *cranges, int ncranges)
}
#endif
+
+static int
+reduce_combine_ranges_simple(CombineRange *cranges, int ncranges,
+ int max_values, FmgrInfo *cmp, Oid colloid)
+{
+ int i;
+ int nvalues;
+ Datum *values;
+
+ compare_context cxt;
+
+ /* number of values to keep on each tail */
+ int tail = (max_values - 2) / 2;
+ int m = Min(tail / 2, ncranges - 1 / 2);
+
+ /* allocate space for the boundary values */
+ nvalues = 0;
+ values = (Datum *) palloc(sizeof(Datum) * max_values);
+
+ for (i = 0; i < m; i++)
+ {
+ /* head */
+ values[nvalues++] = cranges[i].minval;
+ values[nvalues++] = cranges[i].maxval;
+
+ /* tail */
+ values[nvalues++] = cranges[ncranges - 1 - i].minval;
+ values[nvalues++] = cranges[ncranges - 1 - i].maxval;
+ }
+
+ /* middle part */
+ values[nvalues++] = cranges[m].maxval;
+ values[nvalues++] = cranges[ncranges - 1 - m].minval;
+
+ /* We should have even number of range values. */
+ Assert(nvalues % 2 == 0);
+
+ /* sort the values */
+ cxt.colloid = colloid;
+ cxt.cmpFn = cmp;
+
+ /*
+ * Sort the values using the comparator function, and form ranges
+ * from the sorted result.
+ */
+ qsort_arg(values, nvalues, sizeof(Datum),
+ compare_values, (void *) &cxt);
+
+ /* We have nvalues boundary values, which means nvalues/2 ranges. */
+ for (i = 0; i < (nvalues / 2); i++)
+ {
+ cranges[i].minval = values[2*i];
+ cranges[i].maxval = values[2*i + 1];
+
+ /* if the boundary values are the same, it's a collapsed range */
+ cranges[i].collapsed = (compare_values(&values[2*i],
+ &values[2*i+1],
+ &cxt) == 0);
+ }
+
+ return (nvalues / 2);
+}
+
+
/*
* reduce_combine_ranges
* reduce the ranges until the number of values is low enough
@@ -1366,6 +1433,14 @@ reduce_combine_ranges(CombineRange *cranges, int ncranges,
if (keep >= ndistances)
return ncranges;
+ /*
+ * Without distances, we use a simple approach keeping as many
+ * outliers as possible.
+ */
+ if (!distances)
+ return reduce_combine_ranges_simple(cranges, ncranges, max_values,
+ cmp, colloid);
+
/* sort the values */
cxt.colloid = colloid;
cxt.cmpFn = cmp;
diff --git a/src/include/catalog/pg_amop.dat b/src/include/catalog/pg_amop.dat
index 762ac3acef..4ab4aff073 100644
--- a/src/include/catalog/pg_amop.dat
+++ b/src/include/catalog/pg_amop.dat
@@ -1814,6 +1814,23 @@
amoprighttype => 'bytea', amopstrategy => '5', amopopr => '>(bytea,bytea)',
amopmethod => 'brin' },
+# minmax multi bytea
+{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea',
+ amoprighttype => 'bytea', amopstrategy => '1', amopopr => '<(bytea,bytea)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea',
+ amoprighttype => 'bytea', amopstrategy => '2', amopopr => '<=(bytea,bytea)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea',
+ amoprighttype => 'bytea', amopstrategy => '3', amopopr => '=(bytea,bytea)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea',
+ amoprighttype => 'bytea', amopstrategy => '4', amopopr => '>=(bytea,bytea)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bytea_minmax_multi_ops', amoplefttype => 'bytea',
+ amoprighttype => 'bytea', amopstrategy => '5', amopopr => '>(bytea,bytea)',
+ amopmethod => 'brin' },
+
# bloom bytea
{ amopfamily => 'brin/bytea_bloom_ops', amoplefttype => 'bytea',
amoprighttype => 'bytea', amopstrategy => '1', amopopr => '=(bytea,bytea)',
@@ -1836,6 +1853,23 @@
amoprighttype => 'char', amopstrategy => '5', amopopr => '>(char,char)',
amopmethod => 'brin' },
+# minmax multi "char"
+{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char',
+ amoprighttype => 'char', amopstrategy => '1', amopopr => '<(char,char)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char',
+ amoprighttype => 'char', amopstrategy => '2', amopopr => '<=(char,char)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char',
+ amoprighttype => 'char', amopstrategy => '3', amopopr => '=(char,char)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char',
+ amoprighttype => 'char', amopstrategy => '4', amopopr => '>=(char,char)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/char_minmax_multi_ops', amoplefttype => 'char',
+ amoprighttype => 'char', amopstrategy => '5', amopopr => '>(char,char)',
+ amopmethod => 'brin' },
+
# bloom "char"
{ amopfamily => 'brin/char_bloom_ops', amoplefttype => 'char',
amoprighttype => 'char', amopstrategy => '1', amopopr => '=(char,char)',
@@ -1858,6 +1892,23 @@
amoprighttype => 'name', amopstrategy => '5', amopopr => '>(name,name)',
amopmethod => 'brin' },
+# minmax multi name
+{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name',
+ amoprighttype => 'name', amopstrategy => '1', amopopr => '<(name,name)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name',
+ amoprighttype => 'name', amopstrategy => '2', amopopr => '<=(name,name)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name',
+ amoprighttype => 'name', amopstrategy => '3', amopopr => '=(name,name)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name',
+ amoprighttype => 'name', amopstrategy => '4', amopopr => '>=(name,name)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/name_minmax_multi_ops', amoplefttype => 'name',
+ amoprighttype => 'name', amopstrategy => '5', amopopr => '>(name,name)',
+ amopmethod => 'brin' },
+
# bloom name
{ amopfamily => 'brin/name_bloom_ops', amoplefttype => 'name',
amoprighttype => 'name', amopstrategy => '1', amopopr => '=(name,name)',
@@ -2210,6 +2261,23 @@
amoprighttype => 'text', amopstrategy => '5', amopopr => '>(text,text)',
amopmethod => 'brin' },
+# minmax multi text
+{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text',
+ amoprighttype => 'text', amopstrategy => '1', amopopr => '<(text,text)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text',
+ amoprighttype => 'text', amopstrategy => '2', amopopr => '<=(text,text)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text',
+ amoprighttype => 'text', amopstrategy => '3', amopopr => '=(text,text)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text',
+ amoprighttype => 'text', amopstrategy => '4', amopopr => '>=(text,text)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/text_minmax_multi_ops', amoplefttype => 'text',
+ amoprighttype => 'text', amopstrategy => '5', amopopr => '>(text,text)',
+ amopmethod => 'brin' },
+
# bloom text
{ amopfamily => 'brin/text_bloom_ops', amoplefttype => 'text',
amoprighttype => 'text', amopstrategy => '1', amopopr => '=(text,text)',
@@ -2592,6 +2660,23 @@
amoprighttype => 'bpchar', amopstrategy => '5', amopopr => '>(bpchar,bpchar)',
amopmethod => 'brin' },
+# minmax multi character
+{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar',
+ amoprighttype => 'bpchar', amopstrategy => '1', amopopr => '<(bpchar,bpchar)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar',
+ amoprighttype => 'bpchar', amopstrategy => '2',
+ amopopr => '<=(bpchar,bpchar)', amopmethod => 'brin' },
+{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar',
+ amoprighttype => 'bpchar', amopstrategy => '3', amopopr => '=(bpchar,bpchar)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar',
+ amoprighttype => 'bpchar', amopstrategy => '4',
+ amopopr => '>=(bpchar,bpchar)', amopmethod => 'brin' },
+{ amopfamily => 'brin/bpchar_minmax_multi_ops', amoplefttype => 'bpchar',
+ amoprighttype => 'bpchar', amopstrategy => '5', amopopr => '>(bpchar,bpchar)',
+ amopmethod => 'brin' },
+
# bloom character
{ amopfamily => 'brin/bpchar_bloom_ops', amoplefttype => 'bpchar',
amoprighttype => 'bpchar', amopstrategy => '1', amopopr => '=(bpchar,bpchar)',
@@ -3061,6 +3146,23 @@
amoprighttype => 'bit', amopstrategy => '5', amopopr => '>(bit,bit)',
amopmethod => 'brin' },
+# minmax multi bit
+{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit',
+ amoprighttype => 'bit', amopstrategy => '1', amopopr => '<(bit,bit)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit',
+ amoprighttype => 'bit', amopstrategy => '2', amopopr => '<=(bit,bit)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit',
+ amoprighttype => 'bit', amopstrategy => '3', amopopr => '=(bit,bit)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit',
+ amoprighttype => 'bit', amopstrategy => '4', amopopr => '>=(bit,bit)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/bit_minmax_multi_ops', amoplefttype => 'bit',
+ amoprighttype => 'bit', amopstrategy => '5', amopopr => '>(bit,bit)',
+ amopmethod => 'brin' },
+
# minmax bit varying
{ amopfamily => 'brin/varbit_minmax_ops', amoplefttype => 'varbit',
amoprighttype => 'varbit', amopstrategy => '1', amopopr => '<(varbit,varbit)',
@@ -3078,6 +3180,23 @@
amoprighttype => 'varbit', amopstrategy => '5', amopopr => '>(varbit,varbit)',
amopmethod => 'brin' },
+# minmax multi bit varying
+{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit',
+ amoprighttype => 'varbit', amopstrategy => '1', amopopr => '<(varbit,varbit)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit',
+ amoprighttype => 'varbit', amopstrategy => '2',
+ amopopr => '<=(varbit,varbit)', amopmethod => 'brin' },
+{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit',
+ amoprighttype => 'varbit', amopstrategy => '3', amopopr => '=(varbit,varbit)',
+ amopmethod => 'brin' },
+{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit',
+ amoprighttype => 'varbit', amopstrategy => '4',
+ amopopr => '>=(varbit,varbit)', amopmethod => 'brin' },
+{ amopfamily => 'brin/varbit_minmax_multi_ops', amoplefttype => 'varbit',
+ amoprighttype => 'varbit', amopstrategy => '5', amopopr => '>(varbit,varbit)',
+ amopmethod => 'brin' },
+
# minmax numeric
{ amopfamily => 'brin/numeric_minmax_ops', amoplefttype => 'numeric',
amoprighttype => 'numeric', amopstrategy => '1',
diff --git a/src/include/catalog/pg_amproc.dat b/src/include/catalog/pg_amproc.dat
index 51403716b1..69a7ed682c 100644
--- a/src/include/catalog/pg_amproc.dat
+++ b/src/include/catalog/pg_amproc.dat
@@ -805,6 +805,22 @@
{ amprocfamily => 'brin/bytea_minmax_ops', amproclefttype => 'bytea',
amprocrighttype => 'bytea', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi bytea
+{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea',
+ amprocrighttype => 'bytea', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea',
+ amprocrighttype => 'bytea', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea',
+ amprocrighttype => 'bytea', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea',
+ amprocrighttype => 'bytea', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/bytea_minmax_multi_ops', amproclefttype => 'bytea',
+ amprocrighttype => 'bytea', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# bloom bytea
{ amprocfamily => 'brin/bytea_bloom_ops', amproclefttype => 'bytea',
amprocrighttype => 'bytea', amprocnum => '1',
@@ -836,6 +852,22 @@
{ amprocfamily => 'brin/char_minmax_ops', amproclefttype => 'char',
amprocrighttype => 'char', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi "char"
+{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char',
+ amprocrighttype => 'char', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char',
+ amprocrighttype => 'char', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char',
+ amprocrighttype => 'char', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char',
+ amprocrighttype => 'char', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/char_minmax_multi_ops', amproclefttype => 'char',
+ amprocrighttype => 'char', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# bloom "char"
{ amprocfamily => 'brin/char_bloom_ops', amproclefttype => 'char',
amprocrighttype => 'char', amprocnum => '1',
@@ -867,6 +899,22 @@
{ amprocfamily => 'brin/name_minmax_ops', amproclefttype => 'name',
amprocrighttype => 'name', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi name
+{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name',
+ amprocrighttype => 'name', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name',
+ amprocrighttype => 'name', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name',
+ amprocrighttype => 'name', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name',
+ amprocrighttype => 'name', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/name_minmax_multi_ops', amproclefttype => 'name',
+ amprocrighttype => 'name', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# bloom name
{ amprocfamily => 'brin/name_bloom_ops', amproclefttype => 'name',
amprocrighttype => 'name', amprocnum => '1',
@@ -1197,6 +1245,22 @@
{ amprocfamily => 'brin/text_minmax_ops', amproclefttype => 'text',
amprocrighttype => 'text', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi text
+{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text',
+ amprocrighttype => 'text', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text',
+ amprocrighttype => 'text', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text',
+ amprocrighttype => 'text', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text',
+ amprocrighttype => 'text', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/text_minmax_multi_ops', amproclefttype => 'text',
+ amprocrighttype => 'text', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# bloom text
{ amprocfamily => 'brin/text_bloom_ops', amproclefttype => 'text',
amprocrighttype => 'text', amprocnum => '1',
@@ -1663,6 +1727,23 @@
amprocrighttype => 'bpchar', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi character
+{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar',
+ amprocrighttype => 'bpchar', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar',
+ amprocrighttype => 'bpchar', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar',
+ amprocrighttype => 'bpchar', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar',
+ amprocrighttype => 'bpchar', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/bpchar_minmax_multi_ops', amproclefttype => 'bpchar',
+ amprocrighttype => 'bpchar', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# bloom character
{ amprocfamily => 'brin/bpchar_bloom_ops', amproclefttype => 'bpchar',
amprocrighttype => 'bpchar', amprocnum => '1',
@@ -2180,6 +2261,21 @@
{ amprocfamily => 'brin/bit_minmax_ops', amproclefttype => 'bit',
amprocrighttype => 'bit', amprocnum => '4', amproc => 'brin_minmax_union' },
+# minmax multi bit
+{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit',
+ amprocrighttype => 'bit', amprocnum => '1', amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit',
+ amprocrighttype => 'bit', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit',
+ amprocrighttype => 'bit', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit',
+ amprocrighttype => 'bit', amprocnum => '4', amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/bit_minmax_multi_ops', amproclefttype => 'bit',
+ amprocrighttype => 'bit', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# minmax bit varying
{ amprocfamily => 'brin/varbit_minmax_ops', amproclefttype => 'varbit',
amprocrighttype => 'varbit', amprocnum => '1',
@@ -2194,6 +2290,23 @@
amprocrighttype => 'varbit', amprocnum => '4',
amproc => 'brin_minmax_union' },
+# minmax multi bit varying
+{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit',
+ amprocrighttype => 'varbit', amprocnum => '1',
+ amproc => 'brin_minmax_multi_opcinfo' },
+{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit',
+ amprocrighttype => 'varbit', amprocnum => '2',
+ amproc => 'brin_minmax_multi_add_value' },
+{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit',
+ amprocrighttype => 'varbit', amprocnum => '3',
+ amproc => 'brin_minmax_multi_consistent' },
+{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit',
+ amprocrighttype => 'varbit', amprocnum => '4',
+ amproc => 'brin_minmax_multi_union' },
+{ amprocfamily => 'brin/varbit_minmax_multi_ops', amproclefttype => 'varbit',
+ amprocrighttype => 'varbit', amprocnum => '5',
+ amproc => 'brin_minmax_multi_options' },
+
# minmax numeric
{ amprocfamily => 'brin/numeric_minmax_ops', amproclefttype => 'numeric',
amprocrighttype => 'numeric', amprocnum => '1',
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index da25befefe..823f1b01fe 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -266,18 +266,27 @@
{ opcmethod => 'brin', opcname => 'bytea_minmax_ops',
opcfamily => 'brin/bytea_minmax_ops', opcintype => 'bytea',
opckeytype => 'bytea' },
+{ opcmethod => 'brin', opcname => 'bytea_minmax_multi_ops',
+ opcfamily => 'brin/bytea_minmax_multi_ops', opcintype => 'bytea',
+ opckeytype => 'bytea', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'bytea_bloom_ops',
opcfamily => 'brin/bytea_bloom_ops', opcintype => 'bytea',
opckeytype => 'bytea', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'char_minmax_ops',
opcfamily => 'brin/char_minmax_ops', opcintype => 'char',
opckeytype => 'char' },
+{ opcmethod => 'brin', opcname => 'char_minmax_multi_ops',
+ opcfamily => 'brin/char_minmax_multi_ops', opcintype => 'char',
+ opckeytype => 'char', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'char_bloom_ops',
opcfamily => 'brin/char_bloom_ops', opcintype => 'char',
opckeytype => 'char', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'name_minmax_ops',
opcfamily => 'brin/name_minmax_ops', opcintype => 'name',
opckeytype => 'name' },
+{ opcmethod => 'brin', opcname => 'name_minmax_multi_ops',
+ opcfamily => 'brin/name_minmax_multi_ops', opcintype => 'name',
+ opckeytype => 'name', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'name_bloom_ops',
opcfamily => 'brin/name_bloom_ops', opcintype => 'name',
opckeytype => 'name', opcdefault => 'f' },
@@ -311,6 +320,9 @@
{ opcmethod => 'brin', opcname => 'text_minmax_ops',
opcfamily => 'brin/text_minmax_ops', opcintype => 'text',
opckeytype => 'text' },
+{ opcmethod => 'brin', opcname => 'text_minmax_multi_ops',
+ opcfamily => 'brin/text_minmax_multi_ops', opcintype => 'text',
+ opckeytype => 'text', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'text_bloom_ops',
opcfamily => 'brin/text_bloom_ops', opcintype => 'text',
opckeytype => 'text', opcdefault => 'f' },
@@ -381,6 +393,9 @@
{ opcmethod => 'brin', opcname => 'bpchar_minmax_ops',
opcfamily => 'brin/bpchar_minmax_ops', opcintype => 'bpchar',
opckeytype => 'bpchar' },
+{ opcmethod => 'brin', opcname => 'bpchar_minmax_multi_ops',
+ opcfamily => 'brin/bpchar_minmax_multi_ops', opcintype => 'bpchar',
+ opckeytype => 'bpchar', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'bpchar_bloom_ops',
opcfamily => 'brin/bpchar_bloom_ops', opcintype => 'bpchar',
opckeytype => 'bpchar', opcdefault => 'f' },
@@ -440,9 +455,15 @@
opckeytype => 'timetz', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'bit_minmax_ops',
opcfamily => 'brin/bit_minmax_ops', opcintype => 'bit', opckeytype => 'bit' },
+{ opcmethod => 'brin', opcname => 'bit_minmax_multi_ops',
+ opcfamily => 'brin/bit_minmax_multi_ops', opcintype => 'bit',
+ opckeytype => 'bit', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'varbit_minmax_ops',
opcfamily => 'brin/varbit_minmax_ops', opcintype => 'varbit',
opckeytype => 'varbit' },
+{ opcmethod => 'brin', opcname => 'varbit_minmax_multi_ops',
+ opcfamily => 'brin/varbit_minmax_multi_ops', opcintype => 'varbit',
+ opckeytype => 'varbit', opcdefault => 'f' },
{ opcmethod => 'brin', opcname => 'numeric_minmax_ops',
opcfamily => 'brin/numeric_minmax_ops', opcintype => 'numeric',
opckeytype => 'numeric' },
diff --git a/src/include/catalog/pg_opfamily.dat b/src/include/catalog/pg_opfamily.dat
index ba9231ac8c..ffb20e72ab 100644
--- a/src/include/catalog/pg_opfamily.dat
+++ b/src/include/catalog/pg_opfamily.dat
@@ -192,6 +192,8 @@
opfmethod => 'brin', opfname => 'numeric_minmax_multi_ops' },
{ oid => '4056',
opfmethod => 'brin', opfname => 'text_minmax_ops' },
+{ oid => '9967',
+ opfmethod => 'brin', opfname => 'text_minmax_multi_ops' },
{ oid => '9902',
opfmethod => 'brin', opfname => 'text_bloom_ops' },
{ oid => '9903',
@@ -210,14 +212,20 @@
opfmethod => 'brin', opfname => 'datetime_bloom_ops' },
{ oid => '4062',
opfmethod => 'brin', opfname => 'char_minmax_ops' },
+{ oid => '9968',
+ opfmethod => 'brin', opfname => 'char_minmax_multi_ops' },
{ oid => '9906',
opfmethod => 'brin', opfname => 'char_bloom_ops' },
{ oid => '4064',
opfmethod => 'brin', opfname => 'bytea_minmax_ops' },
+{ oid => '9969',
+ opfmethod => 'brin', opfname => 'bytea_minmax_multi_ops' },
{ oid => '9907',
opfmethod => 'brin', opfname => 'bytea_bloom_ops' },
{ oid => '4065',
opfmethod => 'brin', opfname => 'name_minmax_ops' },
+{ oid => '9970',
+ opfmethod => 'brin', opfname => 'name_minmax_multi_ops' },
{ oid => '9908',
opfmethod => 'brin', opfname => 'name_bloom_ops' },
{ oid => '4068',
@@ -260,6 +268,8 @@
opfmethod => 'brin', opfname => 'network_bloom_ops' },
{ oid => '4076',
opfmethod => 'brin', opfname => 'bpchar_minmax_ops' },
+{ oid => '9971',
+ opfmethod => 'brin', opfname => 'bpchar_minmax_multi_ops' },
{ oid => '9915',
opfmethod => 'brin', opfname => 'bpchar_bloom_ops' },
{ oid => '4077',
@@ -276,8 +286,12 @@
opfmethod => 'brin', opfname => 'interval_bloom_ops' },
{ oid => '4079',
opfmethod => 'brin', opfname => 'bit_minmax_ops' },
+{ oid => '9972',
+ opfmethod => 'brin', opfname => 'bit_minmax_multi_ops' },
{ oid => '4080',
opfmethod => 'brin', opfname => 'varbit_minmax_ops' },
+{ oid => '9973',
+ opfmethod => 'brin', opfname => 'varbit_minmax_multi_ops' },
{ oid => '4081',
opfmethod => 'brin', opfname => 'uuid_minmax_ops' },
{ oid => '9938',
--
2.26.2
--------------4B194FF8F3EA3786FF9EAE1F
Content-Type: text/x-patch; charset=UTF-8;
name="0007-Remove-the-special-batch-mode-use-a-larger--20210203.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename*0="0007-Remove-the-special-batch-mode-use-a-larger--20210203.pa";
filename*1="tch"
^ permalink raw reply [nested|flat] 5+ messages in thread
* Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done
@ 2025-01-17 01:05 Roman Eskin <[email protected]>
2025-01-18 16:31 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Andrey M. Borodin <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Roman Eskin @ 2025-01-17 01:05 UTC (permalink / raw)
To: [email protected]
Hello pgsql-hackers,
It appears to exist an issue with timeline switching in archive
recovery mode if the startup process is interrupted by some external
reason (for ex. system power loss) in a specific tight time window
(after a new history file is created but before end-of-recovery record
is done). If this happens, and the Postgres instance is launched
again, it carries out the common crash recovery, starts successfully
to 'in production' state, but continues the previous timeline. Thus,
we are having inconsistent WAL state, where we are writing WAL segment
files to old timeline and having presented new orphaned timeline in
the history file.
This issue can reveal itself in streaming replication scenario, for
ex.:
1. Assume we have instance_1 (Master) and instance_2 (Replica);
2. instance_1 (Master) fails for any reason, instance_2 (Replica)
starts to promote. But while instance_2 is still in `StartupXLOG()`
somewhere between `writeTimeLineHistory()` and
`CreateEndOfRecoveryRecord()`, we are getting, for example, power
loss on instance_2 server.
3. Once instance_2 server is up again, instance_2 is restarted. The
problem with timeline already exists here, but very likely to be
unnoticed, as instance_2 (now Master) is accepting connections and
working.
4. Bring back instance_1 as Replica.
5. Switch back instance_1 and instance_2 to the original
configuration. And here, when we try to start instance_2 as Replica,
we'll get a FATAL:
"FATAL: requested timeline 2 is not a child of this server's history
DETAIL: Latest checkpoint is at 0/303FF90 on timeline 1, but in the
history of the requested timeline, the server forked off from that
timeline at 0/3023538."
I agree that one have to be very unlucky to get into such situation
with 2 consequent failures and to hit the specific time window, but if
this happens, it will look quite severe and hard to debug. We faced
this situation on our synthetic tests.
To reproduce the issue on latest code from Postgres master branch in a
stable manner, I used the attached
'test-code-for-timeline-issue.patch'. It postpones execution in
`StartupXLOG()` if we need to hit the requested time window. Detailed
steps to reproduce I've used with the patch are in the attached file
'steps - NG scenario.txt'.
Possible way to fix this issue, that we now consider, is to postpone
the removal of signal files ("recovery.signal" and "standby.signal")
until the end-of-recovery record is done, and thus retry interrupted
promotion in case previous one failed. Looks like, the only
insignificant negative outcome for this approach is possible
assignment of one more timeline identity. But we would like to get
more feedback from the community. Do you foresee any possible
drawbacks for this approach? Please advise.
Many thanks in advance!
P.S. The topic above is a bit similar to this old one -
https://www.postgresql.org/message-id/flat/CABUevEz09XY2EevA2dLjPCY-C5UO4Hq%3DXxmXLmF6ipNFecbShQ%40m....
Best regards,
Roman Eskin
TEST_PG_PORT_MASTER=5433
TEST_PG_PORT_REPLICA=5434
TEST_PG_DIR_MASTER=~/postgres_data_master
TEST_PG_DIR_REPLICA=~/postgres_data_replica
# Setup master
mkdir $TEST_PG_DIR_MASTER
pg_ctl -D $TEST_PG_DIR_MASTER initdb
sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/g" $TEST_PG_DIR_MASTER/postgresql.conf
sed -i "s/#port = 5432/port = $TEST_PG_PORT_MASTER/g" $TEST_PG_DIR_MASTER/postgresql.conf
pg_ctl -D $TEST_PG_DIR_MASTER -l logfile start
# Setup replica
pg_basebackup -h localhost --checkpoint=fast -D $TEST_PG_DIR_REPLICA -R -X stream --port=$TEST_PG_PORT_MASTER
sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/g" $TEST_PG_DIR_REPLICA/postgresql.conf
sed -i "s/port = $TEST_PG_PORT_MASTER/port = $TEST_PG_PORT_REPLICA/g" $TEST_PG_DIR_REPLICA/postgresql.conf
pg_ctl -D $TEST_PG_DIR_REPLICA -l logfile_replica start
# Check replication is working
psql postgres --port=$TEST_PG_PORT_MASTER -c "
create table t (
a integer,
b text,
d timestamp without time zone
);
insert into t select i, 'test-master-2-replica'||i , now() from generate_series(1, 5)i;
"
sleep 2
psql postgres --port=$TEST_PG_PORT_REPLICA -c "select * from t;";
# Shutdown master and switch to replica
pg_ctl -D $TEST_PG_DIR_MASTER stop
# enable test changes that will postpone CreateEndOfRecoveryRecord()
touch $TEST_PG_DIR_REPLICA/delay_recovery.signal
psql postgres --port=$TEST_PG_PORT_REPLICA -c "SELECT pg_promote();" &
# Shutdown immediately Postgres (in real life that could be an accident power loss, or any other system level problem...)
sleep 5 # sleep a while to ensure that history file was created
pg_ctl -D $TEST_PG_DIR_REPLICA stop -m i
sleep 5
# cleanup - we do not need the file anymore
rm $TEST_PG_DIR_REPLICA/delay_recovery.signal
# start replica again
pg_ctl -D $TEST_PG_DIR_REPLICA -l logfile_replica start
# after this point replica is started, and working, but the timeline is still 1 !!!
# it can be checked with `pg_controldata $TEST_PG_DIR_REPLICA`
# and we will get problems later once we switch the replica back to its original role (refer to steps below).
# Bring back master as replica
touch $TEST_PG_DIR_MASTER/standby.signal
sed -i "s/#primary_conninfo = ''/primary_conninfo = 'host=localhost port=$TEST_PG_PORT_REPLICA'/g" $TEST_PG_DIR_MASTER/postgresql.conf
pg_ctl -D $TEST_PG_DIR_MASTER -l logfile start
# Check replication is working
psql postgres --port=$TEST_PG_PORT_REPLICA -c "
create table t2 (
a integer,
b text,
d timestamp without time zone
);
insert into t2 select i, 'test-replica-2-master'||i , now() from generate_series(1, 5)i;
"
sleep 2
psql postgres --port=$TEST_PG_PORT_MASTER -c "select * from t2;";
# Switch back
pg_ctl -D $TEST_PG_DIR_REPLICA stop
psql postgres --port=$TEST_PG_PORT_MASTER -c "SELECT pg_promote();"
touch $TEST_PG_DIR_REPLICA/standby.signal
pg_ctl -D $TEST_PG_DIR_REPLICA -l logfile_replica start
# attempt to start replica here will result in:
# 'pg_ctl: could not start server'
# 'Examine the log output.'
# and in the log:
# 2025-01-16 17:30:27.174 +10 [4806] FATAL: requested timeline 2 is not a child of this server's history
# 2025-01-16 17:30:27.174 +10 [4806] DETAIL: Latest checkpoint is at 0/303FF90 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/3023538.
#
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index bf3dbda901..bdfc92cd6e 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6126,6 +6126,19 @@ StartupXLOG(void)
Insert->fullPageWrites = lastFullPageWrites;
UpdateFullPageWrites();
+ /*
+ * Code below is for test purposes only to reveal possible timeline issue.
+ */
+ {
+ struct stat st;
+
+ elog(LOG, "Start delay in StartupXLOG()");
+
+ while (stat("delay_recovery.signal", &st) == 0)
+ pg_usleep(10000L); /* sleep for 10 ms*/
+
+ elog(LOG, "Stop delay in StartupXLOG()");
+ }
/*
* Emit checkpoint or end-of-recovery record in XLOG, if required.
*/
Attachments:
[text/plain] steps - NG scenario.txt (3.3K, ../../[email protected]/2-steps%20-%20NG%20scenario.txt)
download | inline:
TEST_PG_PORT_MASTER=5433
TEST_PG_PORT_REPLICA=5434
TEST_PG_DIR_MASTER=~/postgres_data_master
TEST_PG_DIR_REPLICA=~/postgres_data_replica
# Setup master
mkdir $TEST_PG_DIR_MASTER
pg_ctl -D $TEST_PG_DIR_MASTER initdb
sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/g" $TEST_PG_DIR_MASTER/postgresql.conf
sed -i "s/#port = 5432/port = $TEST_PG_PORT_MASTER/g" $TEST_PG_DIR_MASTER/postgresql.conf
pg_ctl -D $TEST_PG_DIR_MASTER -l logfile start
# Setup replica
pg_basebackup -h localhost --checkpoint=fast -D $TEST_PG_DIR_REPLICA -R -X stream --port=$TEST_PG_PORT_MASTER
sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/g" $TEST_PG_DIR_REPLICA/postgresql.conf
sed -i "s/port = $TEST_PG_PORT_MASTER/port = $TEST_PG_PORT_REPLICA/g" $TEST_PG_DIR_REPLICA/postgresql.conf
pg_ctl -D $TEST_PG_DIR_REPLICA -l logfile_replica start
# Check replication is working
psql postgres --port=$TEST_PG_PORT_MASTER -c "
create table t (
a integer,
b text,
d timestamp without time zone
);
insert into t select i, 'test-master-2-replica'||i , now() from generate_series(1, 5)i;
"
sleep 2
psql postgres --port=$TEST_PG_PORT_REPLICA -c "select * from t;";
# Shutdown master and switch to replica
pg_ctl -D $TEST_PG_DIR_MASTER stop
# enable test changes that will postpone CreateEndOfRecoveryRecord()
touch $TEST_PG_DIR_REPLICA/delay_recovery.signal
psql postgres --port=$TEST_PG_PORT_REPLICA -c "SELECT pg_promote();" &
# Shutdown immediately Postgres (in real life that could be an accident power loss, or any other system level problem...)
sleep 5 # sleep a while to ensure that history file was created
pg_ctl -D $TEST_PG_DIR_REPLICA stop -m i
sleep 5
# cleanup - we do not need the file anymore
rm $TEST_PG_DIR_REPLICA/delay_recovery.signal
# start replica again
pg_ctl -D $TEST_PG_DIR_REPLICA -l logfile_replica start
# after this point replica is started, and working, but the timeline is still 1 !!!
# it can be checked with `pg_controldata $TEST_PG_DIR_REPLICA`
# and we will get problems later once we switch the replica back to its original role (refer to steps below).
# Bring back master as replica
touch $TEST_PG_DIR_MASTER/standby.signal
sed -i "s/#primary_conninfo = ''/primary_conninfo = 'host=localhost port=$TEST_PG_PORT_REPLICA'/g" $TEST_PG_DIR_MASTER/postgresql.conf
pg_ctl -D $TEST_PG_DIR_MASTER -l logfile start
# Check replication is working
psql postgres --port=$TEST_PG_PORT_REPLICA -c "
create table t2 (
a integer,
b text,
d timestamp without time zone
);
insert into t2 select i, 'test-replica-2-master'||i , now() from generate_series(1, 5)i;
"
sleep 2
psql postgres --port=$TEST_PG_PORT_MASTER -c "select * from t2;";
# Switch back
pg_ctl -D $TEST_PG_DIR_REPLICA stop
psql postgres --port=$TEST_PG_PORT_MASTER -c "SELECT pg_promote();"
touch $TEST_PG_DIR_REPLICA/standby.signal
pg_ctl -D $TEST_PG_DIR_REPLICA -l logfile_replica start
# attempt to start replica here will result in:
# 'pg_ctl: could not start server'
# 'Examine the log output.'
# and in the log:
# 2025-01-16 17:30:27.174 +10 [4806] FATAL: requested timeline 2 is not a child of this server's history
# 2025-01-16 17:30:27.174 +10 [4806] DETAIL: Latest checkpoint is at 0/303FF90 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/3023538.
#
[text/plain] test-code-for-timeline-issue.patch (702B, ../../[email protected]/3-test-code-for-timeline-issue.patch)
download | inline diff:
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index bf3dbda901..bdfc92cd6e 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6126,6 +6126,19 @@ StartupXLOG(void)
Insert->fullPageWrites = lastFullPageWrites;
UpdateFullPageWrites();
+ /*
+ * Code below is for test purposes only to reveal possible timeline issue.
+ */
+ {
+ struct stat st;
+
+ elog(LOG, "Start delay in StartupXLOG()");
+
+ while (stat("delay_recovery.signal", &st) == 0)
+ pg_usleep(10000L); /* sleep for 10 ms*/
+
+ elog(LOG, "Stop delay in StartupXLOG()");
+ }
/*
* Emit checkpoint or end-of-recovery record in XLOG, if required.
*/
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done
2025-01-17 01:05 Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
@ 2025-01-18 16:31 ` Andrey M. Borodin <[email protected]>
2025-01-21 11:47 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Andrey M. Borodin @ 2025-01-18 16:31 UTC (permalink / raw)
To: Roman Eskin <[email protected]>; +Cc: [email protected]
Hi Roman!
Thanks for raising the issue. I think the root cause is that many systems imply that higher number of timeline id means more recent timeline write. This invariant is not uphold. It's not even more recent timeline start.
"latest timeline" effectively means "random timeline".
> On 17 Jan 2025, at 06:05, Roman Eskin <[email protected]> wrote:
>
> 5. Switch back instance_1 and instance_2 to the original
> configuration. And here, when we try to start instance_2 as Replica,
> we'll get a FATAL:
> "FATAL: requested timeline 2 is not a child of this server's history
> DETAIL: Latest checkpoint is at 0/303FF90 on timeline 1, but in the
> history of the requested timeline, the server forked off from that
> timeline at 0/3023538."
I think here you can just specify target timeline for the standby instance_1 and it will continue recovery from instance_2.
Having say that, I must admit that we observe something similar approximately 2 times a week, tried several fixes, but still have to live with it.
In our case we have a "resetup" cron job, which will automatically rebuild replica from backup if Postgres cannot start recovery for some hours.
So in our case this looks like extra 3 hours of standby downtime.
I'm not sure if this is a result of pgconsul not setting up target timeline or some other error...
Persisting recovery signal file for some _timeout_ seems super dangerous to me. In distributed systems every extra _timeout_ is a source of complexity, uncertainty and despair.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done
2025-01-17 01:05 Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
2025-01-18 16:31 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Andrey M. Borodin <[email protected]>
@ 2025-01-21 11:47 ` Roman Eskin <[email protected]>
2025-01-28 09:51 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Andrey Borodin <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Roman Eskin @ 2025-01-21 11:47 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: [email protected]
Hi Andrey,
Thank you for your feedback!
> I think here you can just specify target timeline for the standby instance_1 and it will continue recovery from instance_2.
Most likely yes, but nevertheless it looks more like a W/A.
> Persisting recovery signal file for some _timeout_ seems super dangerous to me. In distributed systems every extra _timeout_ is a source of complexity, uncertainty and despair.
The approach is not about persisting the signal files for some timeout.
Currently the files are removed in StartupXLOG() before
writeTimeLineHistory() and PerformRecoveryXLogAction() are called. The
suggestion is to move the file removal after PerformRecoveryXLogAction()
inside StartupXLOG().
Best regards,
Roman Eskin
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done
2025-01-17 01:05 Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
2025-01-18 16:31 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Andrey M. Borodin <[email protected]>
2025-01-21 11:47 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
@ 2025-01-28 09:51 ` Andrey Borodin <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Andrey Borodin @ 2025-01-28 09:51 UTC (permalink / raw)
To: Roman Eskin <[email protected]>; +Cc: pgsql-hackers <[email protected]>
> On 21 Jan 2025, at 16:47, Roman Eskin <[email protected]> wrote:
>
>>
>> Persisting recovery signal file for some _timeout_ seems super dangerous to me. In distributed systems every extra _timeout_ is a source of complexity, uncertainty and despair.
>
> The approach is not about persisting the signal files for some timeout. Currently the files are removed in StartupXLOG() before writeTimeLineHistory() and PerformRecoveryXLogAction() are called. The suggestion is to move the file removal after PerformRecoveryXLogAction() inside StartupXLOG().
Sending node to repeated promote-fail cycle without resolving root cause seems like even less appealing idea.
If something prevented promotion, why we should retry by this particular method?
Even in case of transient failure which you described - power loss - it does not sound like a very good idea to retry promotion after returning online. The user will get unexpected splitbrain.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2025-01-28 09:51 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-03 18:12 [PATCH 8/9] Define multi-minmax oclasses for types without distance Tomas Vondra <[email protected]>
2025-01-17 01:05 Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
2025-01-18 16:31 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Andrey M. Borodin <[email protected]>
2025-01-21 11:47 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Roman Eskin <[email protected]>
2025-01-28 09:51 ` Re: Timeline issue if StartupXLOG() is interrupted right before end-of-recovery record is done Andrey Borodin <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox