public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v13 6/8] Row pattern recognition patch (docs).
5+ messages / 4 participants
[nested] [flat]
* [PATCH v13 6/8] Row pattern recognition patch (docs).
@ 2024-01-22 09:45 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Tatsuo Ishii @ 2024-01-22 09:45 UTC (permalink / raw)
---
doc/src/sgml/advanced.sgml | 80 ++++++++++++++++++++++++++++++++++++
doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++
doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++-
3 files changed, 170 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml
index 755c9f1485..cf18dd887e 100644
--- a/doc/src/sgml/advanced.sgml
+++ b/doc/src/sgml/advanced.sgml
@@ -537,6 +537,86 @@ WHERE pos < 3;
<literal>rank</literal> less than 3.
</para>
+ <para>
+ Row pattern common syntax can be used to perform row pattern recognition
+ in a query. Row pattern common syntax includes two sub
+ clauses: <literal>DEFINE</literal>
+ and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines
+ definition variables along with an expression. The expression must be a
+ logical expression, which means it must
+ return <literal>TRUE</literal>, <literal>FALSE</literal>
+ or <literal>NULL</literal>. The expression may comprise column references
+ and functions. Window functions, aggregate functions and subqueries are
+ not allowed. An example of <literal>DEFINE</literal> is as follows.
+
+<programlisting>
+DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+</programlisting>
+
+ Note that <function>PREV</function> returns the price column in the
+ previous row if it's called in a context of row pattern recognition. So in
+ the second line the definition variable "UP" is <literal>TRUE</literal>
+ when the price column in the current row is greater than the price column
+ in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when
+ the price column in the current row is lower than the price column in the
+ previous row.
+ </para>
+ <para>
+ Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be
+ used. <literal>PATTERN</literal> defines a sequence of rows that satisfies
+ certain conditions. For example following <literal>PATTERN</literal>
+ defines that a row starts with the condition "LOWPRICE", then one or more
+ rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that
+ "+" means one or more matches. Also you can use "*", which means zero or
+ more matches. If a sequence of rows which satisfies the PATTERN is found,
+ in the starting row of the sequence of rows all window functions and
+ aggregates are shown in the target list. Note that aggregations only look
+ into the matched rows, rather than whole frame. In the second or
+ subsequent rows all window functions and aggregates are NULL. For rows
+ that do not match the PATTERN, all window functions and aggregates are
+ shown AS NULL too, except count which shows 0. This is because the
+ unmatched rows are in an empty frame. Example of
+ a <literal>SELECT</literal> using the <literal>DEFINE</literal>
+ and <literal>PATTERN</literal> clause is as follows.
+
+<programlisting>
+SELECT company, tdate, price,
+ first_value(price) OVER w,
+ max(price) OVER w,
+ count(price) OVER w
+FROM stock,
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+</programlisting>
+<screen>
+ company | tdate | price | first_value | max | count
+----------+------------+-------+-------------+-----+-------
+ company1 | 2023-07-01 | 100 | 100 | 200 | 4
+ company1 | 2023-07-02 | 200 | | |
+ company1 | 2023-07-03 | 150 | | |
+ company1 | 2023-07-04 | 140 | | |
+ company1 | 2023-07-05 | 150 | | | 0
+ company1 | 2023-07-06 | 90 | 90 | 130 | 4
+ company1 | 2023-07-07 | 110 | | |
+ company1 | 2023-07-08 | 130 | | |
+ company1 | 2023-07-09 | 120 | | |
+ company1 | 2023-07-10 | 130 | | | 0
+</screen>
+ </para>
+
<para>
When a query involves multiple window functions, it is possible to write
out each one with a separate <literal>OVER</literal> clause, but this is
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 210c7c0b02..8422aa6b93 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21941,6 +21941,7 @@ SELECT count(*) FROM sometable;
returns <literal>NULL</literal> if there is no such row.
</para></entry>
</row>
+
</tbody>
</tgroup>
</table>
@@ -21980,6 +21981,59 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ Row pattern recognition navigation functions are listed in
+ <xref linkend="functions-rpr-navigation-table"/>. These functions
+ can be used to describe DEFINE clause of Row pattern recognition.
+ </para>
+
+ <table id="functions-rpr-navigation-table">
+ <title>Row Pattern Navigation Functions</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ Function
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>prev</primary>
+ </indexterm>
+ <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Returns the column value at the previous row;
+ returns NULL if there is no previous row in the window frame.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>next</primary>
+ </indexterm>
+ <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Returns the column value at the next row;
+ returns NULL if there is no next row in the window frame.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
<note>
<para>
The SQL standard defines a <literal>RESPECT NULLS</literal> or
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 9917df7839..1575fc2167 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl
The <replaceable class="parameter">frame_clause</replaceable> can be one of
<synopsis>
-{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
-{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
+{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
+{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
</synopsis>
where <replaceable>frame_start</replaceable>
@@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS
a given peer group will be in the frame or excluded from it.
</para>
+ <para>
+ The
+ optional <replaceable class="parameter">row_pattern_common_syntax</replaceable>
+ defines the <firstterm>row pattern recognition condition</firstterm> for
+ this
+ window. <replaceable class="parameter">row_pattern_common_syntax</replaceable>
+ includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST
+ ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls
+ how to proceed to next row position after a match
+ found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the
+ default) next row position is next to the last row of previous match. On
+ the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next
+ row position is always next to the last row of previous
+ match. <literal>DEFINE</literal> defines definition variables along with a
+ boolean expression. <literal>PATTERN</literal> defines a sequence of rows
+ that satisfies certain conditions using variables defined
+ in <literal>DEFINE</literal> clause. If the variable is not defined in
+ the <literal>DEFINE</literal> clause, it is implicitly assumed
+ following is defined in the <literal>DEFINE</literal> clause.
+
+<synopsis>
+<literal>variable_name</literal> AS TRUE
+</synopsis>
+
+ Note that the maximu number of variables defined
+ in <literal>DEFINE</literal> clause is 26.
+
+<synopsis>
+[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ]
+PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...]
+DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...]
+</synopsis>
+ </para>
+
<para>
The purpose of a <literal>WINDOW</literal> clause is to specify the
behavior of <firstterm>window functions</firstterm> appearing in the query's
--
2.25.1
----Next_Part(Mon_Jan_22_19_26_18_2024_011)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v13-0007-Row-pattern-recognition-patch-tests.patch"
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid?
@ 2024-06-19 17:30 Peter Geoghegan <[email protected]>
2024-06-19 17:39 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Peter Geoghegan @ 2024-06-19 17:30 UTC (permalink / raw)
To: Yura Sokolov <[email protected]>; +Cc: Maxim Orlov <[email protected]>; Anton A. Melnikov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jun 19, 2024 at 1:00 PM Yura Sokolov <[email protected]> wrote:
> So it is quite different code paths, and one could not be used
> to decline or justify other.
The point is that we shouldn't need to rely on what is formally a
hint. It might be useful to use the hint to decide whether or not
freezing now actually makes sense, but that isn't the same thing as
relying on the hint (we could make the same decision for a number of
different reasons).
> More over, certainly test on HEAP_XMAX_INVALID could be used
> there in heap_prepare_freeze_tuple to set
> freeze_xmax = true;
> Why didn't you do it?
You might as well ask me why I didn't do any number of other things. I
actually wrote a patch that made FreezeMultiXactId() more aggressive
about this sort of thing (setting HEAP_XMAX_INVALID) that targeted
Postgres 16. That worked by noticing that every member XID was at
least before OldestXmin, even when the MXID itself was >= OldestMxact.
That didn't go anywhere, even though it was a perfectly valid
optimization.
It's true that FreezeMultiXactId() optimizations like this are
possible. So what?
> It is not a bug per se.
> But:
> - it is missed opportunity for optimization,
> - it is inconsistency in data handling.
> Inconsistency leads to bugs when people attempt to modify code.
In what sense is there an inconsistency?
I think that you must mean that we need to do the same thing for the
!MultiXactIdIsValid() case and the already-HEAP_XMAX_INVALID case. But
I don't think that that's any meaningful kind of inconsistency. It's
*also* what we do with plain XIDs. If anything, the problem is that
we're *too* consistent (ideally we *would* treat MultiXacts
differently).
> Yes, we changed completely different place mistakenly relying on
> consistent reaction on this "hint", and that leads to bug in our
> patch.
Oooops!
> > HEAP_XMAX_INVALID is just a hint.
> >
>
> WTF is "just a hint"?
> I thought, hint is "yep, you can ignore it. But we already did some job
> and stored its result as this hint. And if you don't ignore this hint,
> then you can skip doing the job we did already".
>
> So every time we ignore hint, we miss opportunity for optimization.
> Why the hell we shouldn't optimize when we safely can?
This is the first email that anybody has used the word "optimization".
We've been discussing this as if it was a bug. You introduced the
topic of optimization 3 seconds ago. Remember?
> If we couldn't rely on hint, then hint is completely meaningless.
We don't actually trust the hints in any way. We always run checks
inside heap_pre_freeze_checks(), rather than assuming that the hints
are accurate.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid?
2024-06-19 17:30 Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
@ 2024-06-19 17:39 ` Alvaro Herrera <[email protected]>
2024-06-19 18:06 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Alvaro Herrera @ 2024-06-19 17:39 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: Yura Sokolov <[email protected]>; Maxim Orlov <[email protected]>; Anton A. Melnikov <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2024-Jun-19, Peter Geoghegan wrote:
> On Wed, Jun 19, 2024 at 1:00 PM Yura Sokolov <[email protected]> wrote:
> > So it is quite different code paths, and one could not be used
> > to decline or justify other.
>
> The point is that we shouldn't need to rely on what is formally a
> hint. It might be useful to use the hint to decide whether or not
> freezing now actually makes sense, but that isn't the same thing as
> relying on the hint (we could make the same decision for a number of
> different reasons).
FWIW I don't think HEAP_XMAX_INVALID as purely a hint.
HEAP_XMAX_COMMITTED is a hint, for sure, as is HEAP_XMIN_COMMITTED on
its own; but as far as I recall, the INVALID flags must persist once
set. Consider the HEAP_XMIN_COMMITTED+ HEAP_XMIN_INVALID combination,
which we use to represent HEAP_XMIN_FROZEN; if that didn't persist, we'd
have a pretty serious data corruption issue, because we don't reset the
Xmin field when freezing the tuple. So if we fail to keep the flag, the
tuple is no longer frozen. (My point here is that some infomask bits
are hints, but not all them are only hints.) So XMAX_INVALID gives
certainty that the Xmax value must not be read. That is to say, I think
there are (or there were) situations in which we set the bit but don't
bother to reset the actual Xmax field. We should never try to read the
Xmax flag if the bit is set.
I think the problem being investigated in this thread is that
HEAP_XMAX_IS_MULTI is being treated as persistent, that is, it can only
be set if the xmax is not invalid, but apparently that's not always the
case (or we wouldn't be having this conversation).
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid?
2024-06-19 17:30 Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
2024-06-19 17:39 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Alvaro Herrera <[email protected]>
@ 2024-06-19 18:06 ` Peter Geoghegan <[email protected]>
2024-07-29 02:48 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Anton A. Melnikov <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Peter Geoghegan @ 2024-06-19 18:06 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Yura Sokolov <[email protected]>; Maxim Orlov <[email protected]>; Anton A. Melnikov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jun 19, 2024 at 1:39 PM Alvaro Herrera <[email protected]> wrote:
> FWIW I don't think HEAP_XMAX_INVALID as purely a hint.
> HEAP_XMAX_COMMITTED is a hint, for sure, as is HEAP_XMIN_COMMITTED on
> its own; but as far as I recall, the INVALID flags must persist once
> set.
Seems we disagree on some pretty fundamental things in this area, then.
> Consider the HEAP_XMIN_COMMITTED+ HEAP_XMIN_INVALID combination,
> which we use to represent HEAP_XMIN_FROZEN; if that didn't persist, we'd
> have a pretty serious data corruption issue, because we don't reset the
> Xmin field when freezing the tuple.
That's definitely true, but that's a case where the setting happens during a
WAL-logged operation. It doesn't involve HEAP_XMAX_INVALID at all.
FWIW I also don't think that even HEAP_XMIN_INVALID should be
considered anything more than a hint when it appears on its own. Only
HEAP_XMIN_FROZEN (by which I mean HEAP_XMIN_COMMITTED +
HEAP_XMIN_INVALID) are non-hint xact status infomask bits.
It's slightly annoying that we have this HEAP_XMIN_FROZEN case where a
hint bit isn't just a hint, but that's just a historical detail. And I
suppose that the same thing can be said of HEAP_XMAX_IS_MULTI itself
(we have no other way of distinguishing a Multi from an Xid, so
clearly that also has to be treated as a persistent non-hint by everybody).
> So if we fail to keep the flag, the
> tuple is no longer frozen. (My point here is that some infomask bits
> are hints, but not all them are only hints.) So XMAX_INVALID gives
> certainty that the Xmax value must not be read.
"Certainty" seems far too strong here.
> That is to say, I think
> there are (or there were) situations in which we set the bit but don't
> bother to reset the actual Xmax field.
I'm sure that that's true, but that doesn't seem at all equivalent to
what you said about XMAX_INVALID "giving certainty" about the tuple.
> We should never try to read the
> Xmax flag if the bit is set.
But that's exactly what FreezeMultiXactId does. It doesn't pay
attention to XMAX_INVALID (only to !MultiXactIdIsValid()).
Yura is apparently arguing that FreezeMultiXactId should notice
XMAX_INVALID and then tell its caller to "FRM_INVALIDATE_XMAX". That
does seem like a valid optimization. But if we were to do that then
we'd likely do it in a way that still resulted in ordinary processing
of the multi (it would not work by immediately setting
"FRM_INVALIDATE_XMAX" in the !MultiXactIdIsValid() path). That
approach to the optimization makes the most sense, because we'd likely
want to preserve the existing FreezeMultiXactId sanity checks.
> I think the problem being investigated in this thread is that
> HEAP_XMAX_IS_MULTI is being treated as persistent, that is, it can only
> be set if the xmax is not invalid, but apparently that's not always the
> case (or we wouldn't be having this conversation).
A multixact/HEAP_XMAX_IS_MULTI xmax doesn't start out as invalid.
Treating HEAP_XMAX_IS_MULTI as persistent doesn't mean that we should
treat XMAX_INVALID as consistent. In particular, XMAX_INVALID isn't
equivalent to !MultiXactIdIsValid() (you can make a similar statement
about xmax XIDs).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid?
2024-06-19 17:30 Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
2024-06-19 17:39 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Alvaro Herrera <[email protected]>
2024-06-19 18:06 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
@ 2024-07-29 02:48 ` Anton A. Melnikov <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Anton A. Melnikov @ 2024-07-29 02:48 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Yura Sokolov <[email protected]>; Maxim Orlov <[email protected]>; PostgreSQL Hackers <[email protected]>
On 19.06.2024 21:06, Peter Geoghegan wrote:
> On Wed, Jun 19, 2024 at 1:39 PM Alvaro Herrera <[email protected]> wrote:
>> FWIW I don't think HEAP_XMAX_INVALID as purely a hint.
>> HEAP_XMAX_COMMITTED is a hint, for sure, as is HEAP_XMIN_COMMITTED on
>> its own; but as far as I recall, the INVALID flags must persist once
>> set.
>
> Seems we disagree on some pretty fundamental things in this area, then.
To resolve this situation seems it is necessary to agree on what
is a "hint bit" exactly means and how to use it.
For example, in this way:
1) Definition. The "hint bit" if it is set represents presence of the property of some object (e.g. tuple).
The value of a hint bit can be derived again at any time. So it is acceptable for a hint
bit to be lost during some operations.
2) Purpose. (It has already been defined by Yura Sokolov in one of the previous letters)
Some work (e.g CPU + mem usage) must be done to check the property of some object.
Checking the hint bit, if it is set, saves this work.
So the purpose of the hint bit is optimization.
3) Use. By default code that needs to check some property of the object
must firstly check the corresponding hint bit. If hint is set, determine that the property
is present. If hint is not set, do the work to check this property of the object and set
hint bit if that property is present.
Also, non-standard behavior is allowed, when the hint bit is ignored and the work on property
check will be performed unconditionally for some reasons. In this case the code must contain
a comment with an explanation of this reason.
And maybe for clarity, explicitly say that some bit is a hint right in its definition?
For instance, use HEAP_XMIN_COMMITTED_HINT instead of HEAP_XMIN_COMMITTED.
Remarks and concerns are gratefully welcome.
With the best regards,
--
Anton A. Melnikov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-07-29 02:48 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-01-22 09:45 [PATCH v13 6/8] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]>
2024-06-19 17:30 Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
2024-06-19 17:39 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Alvaro Herrera <[email protected]>
2024-06-19 18:06 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Peter Geoghegan <[email protected]>
2024-07-29 02:48 ` Re: Maybe don't process multi xmax in FreezeMultiXactId() if it is already marked as invalid? Anton A. Melnikov <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox