public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v19 6/8] Row pattern recognition patch (docs).
6+ messages / 3 participants
[nested] [flat]
* [PATCH v19 6/8] Row pattern recognition patch (docs).
@ 2024-05-14 23:26 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw)
---
doc/src/sgml/advanced.sgml | 82 ++++++++++++++++++++++++++++++++++++
doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++
doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++-
3 files changed, 172 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml
index 755c9f1485..b0b1d1c51e 100644
--- a/doc/src/sgml/advanced.sgml
+++ b/doc/src/sgml/advanced.sgml
@@ -537,6 +537,88 @@ WHERE pos < 3;
<literal>rank</literal> less than 3.
</para>
+ <para>
+ Row pattern common syntax can be used to perform row pattern recognition
+ in a query. The row pattern common syntax includes two sub
+ clauses: <literal>DEFINE</literal>
+ and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines
+ definition variables along with an expression. The expression must be a
+ logical expression, which means it must
+ return <literal>TRUE</literal>, <literal>FALSE</literal>
+ or <literal>NULL</literal>. The expression may comprise column references
+ and functions. Window functions, aggregate functions and subqueries are
+ not allowed. An example of <literal>DEFINE</literal> is as follows.
+
+<programlisting>
+DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+</programlisting>
+
+ Note that <function>PREV</function> returns the price column in the
+ previous row if it's called in a context of row pattern recognition. Thus in
+ the second line the definition variable "UP" is <literal>TRUE</literal>
+ when the price column in the current row is greater than the price column
+ in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when
+ the price column in the current row is lower than the price column in the
+ previous row.
+ </para>
+ <para>
+ Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be
+ used. <literal>PATTERN</literal> defines a sequence of rows that satisfies
+ certain conditions. For example following <literal>PATTERN</literal>
+ defines that a row starts with the condition "LOWPRICE", then one or more
+ rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that
+ "+" means one or more matches. Also you can use "*", which means zero or
+ more matches. If a sequence of rows which satisfies the PATTERN is found,
+ in the starting row of the sequence of rows all window functions and
+ aggregates are shown in the target list. Note that aggregations only look
+ into the matched rows, rather than whole frame. On the second or
+ subsequent rows all window functions are NULL. Aggregates are NULL or 0
+ (count case) depending on its aggregation definition. For rows that do not
+ match on the PATTERN, all window functions and aggregates are shown AS
+ NULL too, except count showing 0. This is because the rows do not match,
+ thus they are in an empty frame. Example of a <literal>SELECT</literal>
+ using the <literal>DEFINE</literal> and <literal>PATTERN</literal> clause
+ is as follows.
+
+<programlisting>
+SELECT company, tdate, price,
+ first_value(price) OVER w,
+ max(price) OVER w,
+ count(price) OVER w
+FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+</programlisting>
+<screen>
+ company | tdate | price | first_value | max | count
+----------+------------+-------+-------------+-----+-------
+ company1 | 2023-07-01 | 100 | 100 | 200 | 4
+ company1 | 2023-07-02 | 200 | | | 0
+ company1 | 2023-07-03 | 150 | | | 0
+ company1 | 2023-07-04 | 140 | | | 0
+ company1 | 2023-07-05 | 150 | | | 0
+ company1 | 2023-07-06 | 90 | 90 | 130 | 4
+ company1 | 2023-07-07 | 110 | | | 0
+ company1 | 2023-07-08 | 130 | | | 0
+ company1 | 2023-07-09 | 120 | | | 0
+ company1 | 2023-07-10 | 130 | | | 0
+(10 rows)
+</screen>
+ </para>
+
<para>
When a query involves multiple window functions, it is possible to write
out each one with a separate <literal>OVER</literal> clause, but this is
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 17c44bc338..8dbab31300 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23124,6 +23124,7 @@ SELECT count(*) FROM sometable;
returns <literal>NULL</literal> if there is no such row.
</para></entry>
</row>
+
</tbody>
</tgroup>
</table>
@@ -23163,6 +23164,59 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ Row pattern recognition navigation functions are listed in
+ <xref linkend="functions-rpr-navigation-table"/>. These functions
+ can be used to describe DEFINE clause of Row pattern recognition.
+ </para>
+
+ <table id="functions-rpr-navigation-table">
+ <title>Row Pattern Navigation Functions</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ Function
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>prev</primary>
+ </indexterm>
+ <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Returns the column value at the previous row;
+ returns NULL if there is no previous row in the window frame.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>next</primary>
+ </indexterm>
+ <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Returns the column value at the next row;
+ returns NULL if there is no next row in the window frame.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
<note>
<para>
The SQL standard defines a <literal>RESPECT NULLS</literal> or
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 066aed44e6..8f18718d58 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl
The <replaceable class="parameter">frame_clause</replaceable> can be one of
<synopsis>
-{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
-{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
+{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
+{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
</synopsis>
where <replaceable>frame_start</replaceable>
@@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS
a given peer group will be in the frame or excluded from it.
</para>
+ <para>
+ The
+ optional <replaceable class="parameter">row_pattern_common_syntax</replaceable>
+ defines the <firstterm>row pattern recognition condition</firstterm> for
+ this
+ window. <replaceable class="parameter">row_pattern_common_syntax</replaceable>
+ includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST
+ ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls
+ how to proceed to next row position after a match
+ found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the
+ default) next row position is next to the last row of previous match. On
+ the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next
+ row position is always next to the last row of previous
+ match. <literal>DEFINE</literal> defines definition variables along with a
+ boolean expression. <literal>PATTERN</literal> defines a sequence of rows
+ that satisfies certain conditions using variables defined
+ in <literal>DEFINE</literal> clause. If the variable is not defined in
+ the <literal>DEFINE</literal> clause, it is implicitly assumed
+ following is defined in the <literal>DEFINE</literal> clause.
+
+<synopsis>
+<literal>variable_name</literal> AS TRUE
+</synopsis>
+
+ Note that the maximu number of variables defined
+ in <literal>DEFINE</literal> clause is 26.
+
+<synopsis>
+[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ]
+PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...]
+DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...]
+</synopsis>
+ </para>
+
<para>
The purpose of a <literal>WINDOW</literal> clause is to specify the
behavior of <firstterm>window functions</firstterm> appearing in the query's
--
2.25.1
----Next_Part(Wed_May_15_09_02_03_2024_008)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v19-0007-Row-pattern-recognition-patch-tests.patch"
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: RFC: Additional Directory for Extensions
@ 2024-11-12 13:25 Peter Eisentraut <[email protected]>
2024-11-12 14:44 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Peter Eisentraut @ 2024-11-12 13:25 UTC (permalink / raw)
To: David E. Wheeler <[email protected]>; +Cc: Craig Ringer <[email protected]>; Gabriele Bartolini <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers
On 11.11.24 19:15, David E. Wheeler wrote:
>> In fact, most of the patch is refactoring the routines in dfmgr.c to not hardcode dynamic_library_path but allow searching for any file in any path. Once a control file is found, the other extension support files (script files and auxiliary control files) are looked for in the same directory.
>
> What about shared libraries files?
Nothing changes about shared library files. They are looked up in
dynamic_library_path or any hardcoded file name.
>> This works pretty much fine for the use cases that have been presented here, including installing extensions outside of the core installation tree (for CNPG and Postgres.app) and for testing uninstalled extensions (for Debian).
>
> If I understand correctly, shared modules still lie in dynamic_library_path, yes? That makes things a bit more complicated, as the CNPG use case has to set up multiple persistent volumes to persist files put into various directories, notably sharedir and pkglibdir.
No, you can also install them into a common directory and mount that
one. For example, you install the extension at build time into
/tmp/foo/{lib,share/extension}, you package that up as a disk image,
mount it at /opt/extensions/myext, and then you can point
extension_control_path at /opt/extensions/myext/lib and
dynamic_library_path at /opt/extensions/myext/share/extension.
>> - The biggest problem is that many extensions set in their control file
>>
>> module_pathname = '$libdir/foo'
>>
>> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way. The obvious solution is that extensions change this to just 'foo'. But this will require a lot updating work for many extensions, or a lot of patching by packagers.
>
> Since that’s set at build/install time, couldn’t the definition of `$libdir` here be changed to mean “the directory into which it’s being installed right now?”. Doesn’t seem necessary to search a path if the specific location is set at install time.
No, this is not set at build or install time. This is for typical
extensions hardcoded, and $libdir is resolved by the PostgreSQL server
at run time.
> Perhaps I misunderstand, but I would like to talk through the implications of a more radical rethinking of extension file location along the lines of the other thread[2] and the RFC I’m working up based on them both[1], especially since there are a few other use cases that inform it.
I'm aware of that thread, but I think that is looking like a much larger
project than what I'm proposing here.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: RFC: Additional Directory for Extensions
2024-11-12 13:25 Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
@ 2024-11-12 14:44 ` David E. Wheeler <[email protected]>
2024-11-18 19:19 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: David E. Wheeler @ 2024-11-12 14:44 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Craig Ringer <[email protected]>; Gabriele Bartolini <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers
On Nov 12, 2024, at 08:25, Peter Eisentraut <[email protected]> wrote:
> No, you can also install them into a common directory and mount that one. For example, you install the extension at build time into /tmp/foo/{lib,share/extension}, you package that up as a disk image, mount it at /opt/extensions/myext, and then you can point extension_control_path at /opt/extensions/myext/lib and dynamic_library_path at /opt/extensions/myext/share/extension.
Ah, I see, then you just have to set both GUCs to subdirectories of the one volume.
>> Since that’s set at build/install time, couldn’t the definition of `$libdir` here be changed to mean “the directory into which it’s being installed right now?”. Doesn’t seem necessary to search a path if the specific location is set at install time.
>
> No, this is not set at build or install time. This is for typical extensions hardcoded, and $libdir is resolved by the PostgreSQL server at run time.
I see, so that they could be moved and, as long as dynamic_library_path is updated, would still be findable.
So back to your original caveat:
>>> - The biggest problem is that many extensions set in their control file
>>>
>>> module_pathname = '$libdir/foo'
>>>
>>> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way. The obvious solution is that extensions change this to just 'foo'. But this will require a lot updating work for many extensions, or a lot of patching by packagers.
Yeah, '$libdir/foo' has been the documented way to do it for quite some time, as I recall. Perhaps the behavior of the MODULE_PATHNAME replacement function could be changed to omit $libdir when writing the SQL files?
>> Perhaps I misunderstand, but I would like to talk through the implications of a more radical rethinking of extension file location along the lines of the other thread[2] and the RFC I’m working up based on them both[1], especially since there are a few other use cases that inform it.
>
> I'm aware of that thread, but I think that is looking like a much larger project than what I'm proposing here.
Fair enough. Once we get to some consensus on a design there (and I’ve continued to iterate on it elsewhere[1]), I doubt it’d take much to use this patch as the first step toward it.
Best,
David
[1]: https://github.com/theory/justatheory/pull/7/files
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: RFC: Additional Directory for Extensions
2024-11-12 13:25 Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
2024-11-12 14:44 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
@ 2024-11-18 19:19 ` David E. Wheeler <[email protected]>
2024-11-20 09:05 ` Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: David E. Wheeler @ 2024-11-18 19:19 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Craig Ringer <[email protected]>; Gabriele Bartolini <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers
Hi Peter,
Making another pass at this proposal, I’m a bit confused by this issue:
On Nov 12, 2024, at 09:44, David E. Wheeler <[email protected]> wrote:
>>>> - The biggest problem is that many extensions set in their control file
>>>>
>>>> module_pathname = '$libdir/foo'
>>>>
>>>> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way. The obvious solution is that extensions change this to just 'foo'. But this will require a lot updating work for many extensions, or a lot of patching by packagers.
>
> Yeah, '$libdir/foo' has been the documented way to do it for quite some time, as I recall. Perhaps the behavior of the MODULE_PATHNAME replacement function could be changed to omit $libdir when writing the SQL files?
Elsewhere you write:
> Nothing changes about shared library files. They are looked up in dynamic_library_path or any hardcoded file name.
And also point out that the way to install them is:
```
make install datadir=/else/where/share pkglibdir=/else/where/lib
```
So as long as dynamic_library_path includes /else/where/lib it should work, just as before, no?
Best,
David
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: RFC: Additional Directory for Extensions
2024-11-12 13:25 Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
2024-11-12 14:44 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
2024-11-18 19:19 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
@ 2024-11-20 09:05 ` Peter Eisentraut <[email protected]>
2024-11-20 18:04 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
0 siblings, 1 reply; 6+ messages in thread
From: Peter Eisentraut @ 2024-11-20 09:05 UTC (permalink / raw)
To: David E. Wheeler <[email protected]>; +Cc: Craig Ringer <[email protected]>; Gabriele Bartolini <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers
On 18.11.24 20:19, David E. Wheeler wrote:
>>>>> - The biggest problem is that many extensions set in their control file
>>>>>
>>>>> module_pathname = '$libdir/foo'
>>>>>
>>>>> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way. The obvious solution is that extensions change this to just 'foo'. But this will require a lot updating work for many extensions, or a lot of patching by packagers.
>>
>> Yeah, '$libdir/foo' has been the documented way to do it for quite some time, as I recall. Perhaps the behavior of the MODULE_PATHNAME replacement function could be changed to omit $libdir when writing the SQL files?
>
> Elsewhere you write:
>
>> Nothing changes about shared library files. They are looked up in dynamic_library_path or any hardcoded file name.
>
> And also point out that the way to install them is:
>
> ```
> make install datadir=/else/where/share pkglibdir=/else/where/lib
> ```
>
> So as long as dynamic_library_path includes /else/where/lib it should work, just as before, no?
The path is only consulted if the specified name does not contain a
slash. So if you do LOAD 'foo', the path is consulted, but if you do
LOAD '$libdir/foo', it is not. The problem I'm describing is that most
extensions use the latter style, per current recommendation in the
documentation.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: RFC: Additional Directory for Extensions
2024-11-12 13:25 Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
2024-11-12 14:44 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
2024-11-18 19:19 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
2024-11-20 09:05 ` Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
@ 2024-11-20 18:04 ` David E. Wheeler <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: David E. Wheeler @ 2024-11-20 18:04 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Craig Ringer <[email protected]>; Gabriele Bartolini <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers
On Nov 20, 2024, at 04:05, Peter Eisentraut <[email protected]> wrote:
> The path is only consulted if the specified name does not contain a slash. So if you do LOAD 'foo', the path is consulted, but if you do LOAD '$libdir/foo', it is not. The problem I'm describing is that most extensions use the latter style, per current recommendation in the documentation.
I see; some details here:
https://www.postgresql.org/docs/current/xfunc-c.html#XFUNC-C-DYNLOAD
And I suppose the `directory` control file variable and `MODULEDIR` make variable make that necessary.
Maybe $libdir should be stripped out when installing extensions to work with this patch?
Best,
David
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2024-11-20 18:04 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-05-14 23:26 [PATCH v19 6/8] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]>
2024-11-12 13:25 Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
2024-11-12 14:44 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
2024-11-18 19:19 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
2024-11-20 09:05 ` Re: RFC: Additional Directory for Extensions Peter Eisentraut <[email protected]>
2024-11-20 18:04 ` Re: RFC: Additional Directory for Extensions David E. Wheeler <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox