public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v2 5/7] Row pattern recognition patch (docs).
5+ messages / 4 participants
[nested] [flat]
* [PATCH v2 5/7] Row pattern recognition patch (docs).
@ 2023-06-26 08:05 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw)
---
doc/src/sgml/advanced.sgml | 52 +++++++++++++++++++++++++++
doc/src/sgml/func.sgml | 69 ++++++++++++++++++++++++++++++++++++
doc/src/sgml/ref/select.sgml | 29 +++++++++++++--
3 files changed, 148 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml
index 755c9f1485..e9bbd5bc7c 100644
--- a/doc/src/sgml/advanced.sgml
+++ b/doc/src/sgml/advanced.sgml
@@ -537,6 +537,58 @@ WHERE pos < 3;
<literal>rank</literal> less than 3.
</para>
+ <para>
+ Window function <function>rpr</function> can be used with row pattern
+ common syntax to perform row pattern recognition in a query. Row pattern
+ common syntax includes two sub clauses. <literal>DEFINE</literal> defines
+ definition variables along with an expression. The expression must be a
+ logical expression, which means it must
+ return <literal>TRUE</literal>, <literal>FALSE</literal>
+ or <literal>NULL</literal>. Moreover if the expression comprises a column
+ reference, it must be the argument of <function>rpr</function>. An
+ example of <literal>DEFINE</literal> is as follows.
+
+<programlisting>
+DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+</programlisting>
+
+ Note that <function>PREV</function> returns price column in the previous
+ row if it's called in a context of row pattern recognition. So in the
+ second line means the definition variable "UP" is <literal>TRUE</literal>
+ when price column in the current row is greater than the price column in
+ the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when
+ price column in the current row is lower than the price column in the
+ previous row.
+ </para>
+ <para>
+ Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be
+ used. <literal>PATTERN</literal> defines a sequence of rows that satisfies
+ certain conditions. For example following <literal>PATTERN</literal>
+ defines that a row starts with the condition "LOWPRICE", then one or more
+ rows satisfy "UP" and finally one or more rows satisfy "DOWN". If a
+ sequence of rows found, rpr returns the column at the starting row.
+ Example of a <literal>SELECT</literal> using the <literal>DEFINE</literal>
+ and <literal>PATTERN</literal> clause is as follows.
+
+<programlisting>
+SELECT company, tdate, price, rpr(price) OVER w FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+</programlisting>
+ </para>
+
<para>
When a query involves multiple window functions, it is possible to write
out each one with a separate <literal>OVER</literal> clause, but this is
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 5a47ce4343..8069c58ca5 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21648,6 +21648,22 @@ SELECT count(*) FROM sometable;
returns <literal>NULL</literal> if there is no such row.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>rpr</primary>
+ </indexterm>
+ <function>rpr</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Perform row pattern recognition using column specified
+ by <parameter>value</parameter> and returns the value of the column if
+ current row is the first matching row;
+ returns <literal>NULL</literal> otherwise.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -21687,6 +21703,59 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ Row pattern recognition navigation functions are listed in
+ <xref linkend="functions-rpr-navigation-table"/>. These functions
+ can be used to describe DEFINE clause of Row pattern recognition.
+ </para>
+
+ <table id="functions-rpr-navigation-table">
+ <title>Row Pattern Navigation Functions</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ Function
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>prev</primary>
+ </indexterm>
+ <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Returns the column value at the previous row;
+ returns NULL if there is no previous row in the window frame.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>next</primary>
+ </indexterm>
+ <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <returnvalue>anyelement</returnvalue>
+ </para>
+ <para>
+ Returns the column value at the next row;
+ returns NULL if there is no next row in the window frame.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
<note>
<para>
The SQL standard defines a <literal>RESPECT NULLS</literal> or
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 0ee0cc7e64..16478a3950 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -966,8 +966,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl
The <replaceable class="parameter">frame_clause</replaceable> can be one of
<synopsis>
-{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
-{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
+{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
+{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
</synopsis>
where <replaceable>frame_start</replaceable>
@@ -1074,6 +1074,31 @@ EXCLUDE NO OTHERS
a given peer group will be in the frame or excluded from it.
</para>
+ <para>
+ The
+ optional <replaceable class="parameter">row_pattern_common_syntax</replaceable>
+ defines the <firstterm>row pattern recognition condition</firstterm> for
+ this
+ window. <replaceable class="parameter">row_pattern_common_syntax</replaceable>
+ includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST
+ ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls
+ how to proceed to next row position after a match
+ found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the
+ default) next row position is next to the last row of previous match. On
+ the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next
+ row position is always next to the last row of previous
+ match. <literal>DEFINE</literal> defines definition variables along with a
+ boolean expression. <literal>PATTERN</literal> defines a sequence of rows
+ that satisfies certain conditions using variables defined
+ in <literal>DEFINE</literal> clause.
+
+<synopsis>
+[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ]
+PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...]
+DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...]
+</synopsis>
+ </para>
+
<para>
The purpose of a <literal>WINDOW</literal> clause is to specify the
behavior of <firstterm>window functions</firstterm> appearing in the query's
--
2.25.1
----Next_Part(Mon_Jun_26_17_45_07_2023_724)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v2-0006-Row-pattern-recognition-patch-tests.patch"
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: plpgsql memory leaks
@ 2024-01-12 13:53 Michael Banck <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Michael Banck @ 2024-01-12 13:53 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
Hi,
On Fri, Jan 12, 2024 at 01:35:24PM +0100, Pavel Stehule wrote:
> pá 12. 1. 2024 v 11:54 odesílatel Michael Banck <[email protected]> napsal:
> > Which version of Postgres is this and on which platform/distribution?
>
> It was tested on master branch (pg 17) on Fedora 39
>
> > Did you try keep jit on but set jit_inline_above_cost to 0?
> >
> > The back-branches have a fix for the above case, i.e. llvmjit memleaks
> > that can be worked-around by setting jit_inline_above_cost=0.
I got that wrong, it needs to be -1 to disable it.
But if you are already running the master branch, it is probably a
separate issue.
Michael
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: plpgsql memory leaks
@ 2024-01-12 19:09 Pavel Stehule <[email protected]>
parent: Michael Banck <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Pavel Stehule @ 2024-01-12 19:09 UTC (permalink / raw)
To: Michael Banck <[email protected]>; +Cc: pgsql-hackers
pá 12. 1. 2024 v 14:53 odesílatel Michael Banck <[email protected]> napsal:
> Hi,
>
> On Fri, Jan 12, 2024 at 01:35:24PM +0100, Pavel Stehule wrote:
> > pá 12. 1. 2024 v 11:54 odesílatel Michael Banck <[email protected]> napsal:
> > > Which version of Postgres is this and on which platform/distribution?
> >
> > It was tested on master branch (pg 17) on Fedora 39
> >
> > > Did you try keep jit on but set jit_inline_above_cost to 0?
> > >
> > > The back-branches have a fix for the above case, i.e. llvmjit memleaks
> > > that can be worked-around by setting jit_inline_above_cost=0.
>
> I got that wrong, it needs to be -1 to disable it.
>
> But if you are already running the master branch, it is probably a
> separate issue.
>
I tested code
CREATE OR REPLACE FUNCTION public.fx(iter integer)
RETURNS void
LANGUAGE plpgsql
AS $function$
declare
c cursor(m bigint) for select distinct i from generate_series(1, m) g(i);
t bigint;
s bigint;
begin
for i in 1..iter
loop
s := 0;
for r in c(i*10000)
loop
s := s + r.i;
end loop;
raise notice '%=%', i, s;
end loop;
end;
$function$
default master branch - res 190MB ram
jit_inline_above_cost = -1 doesn't helps
disabling JIT doesn't helps too,
so it looks like the wrong hypothesis , and the problem is maybe somewhere
else :-/
Regards
Pavel
>
>
> Michael
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: plpgsql memory leaks
@ 2024-01-12 21:25 Tom Lane <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Tom Lane @ 2024-01-12 21:25 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Michael Banck <[email protected]>; pgsql-hackers
Pavel Stehule <[email protected]> writes:
> default master branch - res 190MB ram
> jit_inline_above_cost = -1 doesn't helps
> disabling JIT doesn't helps too,
> so it looks like the wrong hypothesis , and the problem is maybe somewhere
> else :-/
I see no leak with these examples on HEAD, either with or without
--enable-llvm --- the process size stays quite stable according
to "top". I wonder if you are using some extension that's
contributing to the problem.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: plpgsql memory leaks
@ 2024-01-13 05:25 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Pavel Stehule @ 2024-01-13 05:25 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Michael Banck <[email protected]>; pgsql-hackers
Hi
pá 12. 1. 2024 v 22:25 odesílatel Tom Lane <[email protected]> napsal:
> Pavel Stehule <[email protected]> writes:
> > default master branch - res 190MB ram
> > jit_inline_above_cost = -1 doesn't helps
> > disabling JIT doesn't helps too,
>
> > so it looks like the wrong hypothesis , and the problem is maybe
> somewhere
> > else :-/
>
> I see no leak with these examples on HEAD, either with or without
> --enable-llvm --- the process size stays quite stable according
> to "top". I wonder if you are using some extension that's
> contributing to the problem.
>
memory info after DO $$ BEGIN END $$;
(2024-01-13 05:36:46) postgres=# do $$ begin end $$;
DO
(2024-01-13 05:37:16) postgres=# select meminfo();
NOTICE: Total non-mmapped bytes (arena): 1114112
NOTICE: # of free chunks (ordblks): 11
NOTICE: # of free fastbin blocks (smblks): 0
NOTICE: # of mapped regions (hblks): 2
NOTICE: Bytes in mapped regions (hblkhd): 401408
NOTICE: Max. total allocated space (usmblks): 0
NOTICE: Free bytes held in fastbins (fsmblks): 0
NOTICE: Total allocated space (uordblks): 1039216
NOTICE: Total free space (fordblks): 74896
NOTICE: Topmost releasable block (keepcost): 67360
after script execution
NOTICE: ("1165 kB","1603 kB","438 kB")
NOTICE: Total non-mmapped bytes (arena): 22548480
NOTICE: # of free chunks (ordblks): 25
NOTICE: # of free fastbin blocks (smblks): 0
NOTICE: # of mapped regions (hblks): 2
NOTICE: Bytes in mapped regions (hblkhd): 401408
NOTICE: Max. total allocated space (usmblks): 0
NOTICE: Free bytes held in fastbins (fsmblks): 0
NOTICE: Total allocated space (uordblks): 1400224
NOTICE: Total free space (fordblks): 21148256
NOTICE: Topmost releasable block (keepcost): 20908384
so attached memory is 20MB - but is almost free. The sum of memory context
is very stable without leaks (used 1165kB).
but when I modify the script to
CREATE OR REPLACE FUNCTION public.fx(iter integer)
RETURNS void
LANGUAGE plpgsql
AS $function$
declare
c cursor(m bigint) for select distinct i from generate_series(1, m) g(i);
t bigint;
s bigint;
begin
for i in 1..iter
loop
open c(m := i * 10000);
s := 0;
loop
fetch c into t;
exit when not found;
s := s + t;
end loop;
raise notice '===========before close';
raise notice '%', (select (pg_size_pretty(sum(used_bytes)),
pg_size_pretty(sum(total_bytes)), pg_size_pretty(sum(free_bytes))) from
pg_get_backend_memory_contexts());
--perform meminfo();
raise notice '-----------after close';
close c;
raise notice '%=%', i, s;
raise notice '%', (select (pg_size_pretty(sum(used_bytes)),
pg_size_pretty(sum(total_bytes)), pg_size_pretty(sum(free_bytes))) from
pg_get_backend_memory_contexts());
--perform meminfo();
end loop;
end;
$function$
meminfo is simple extension - see the attachment, I got interesting things
NOTICE: ===========before close
NOTICE: ("149 MB","154 MB","5586 kB")
NOTICE: Total non-mmapped bytes (arena): 132960256
NOTICE: # of free chunks (ordblks): 49
NOTICE: # of free fastbin blocks (smblks): 0
NOTICE: # of mapped regions (hblks): 4
NOTICE: Bytes in mapped regions (hblkhd): 51265536
NOTICE: Max. total allocated space (usmblks): 0
NOTICE: Free bytes held in fastbins (fsmblks): 0
NOTICE: Total allocated space (uordblks): 110730576
NOTICE: Total free space (fordblks): 22229680
NOTICE: Topmost releasable block (keepcost): 133008
so this script really used mbytes memory, but it is related to query
`select distinct i from generate_series(1, m) g(i);`
This maybe is in correlation to my default work mem 64MB - when I set work
mem to 10MB, then it consumes only 15MB
So I was confused because it uses only about 3x work_mem, which is not too
bad.
Regards
Pavel
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-01-13 05:25 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-06-26 08:05 [PATCH v2 5/7] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]>
2024-01-12 13:53 Re: plpgsql memory leaks Michael Banck <[email protected]>
2024-01-12 19:09 ` Re: plpgsql memory leaks Pavel Stehule <[email protected]>
2024-01-12 21:25 ` Re: plpgsql memory leaks Tom Lane <[email protected]>
2024-01-13 05:25 ` Re: plpgsql memory leaks Pavel Stehule <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox