public inbox for [email protected]
help / color / mirror / Atom feedAdd RESPECT/IGNORE NULLS and FROM FIRST/LAST options
91+ messages / 13 participants
[nested] [flat]
* Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-07-13 12:52 Oliver Ford <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Oliver Ford @ 2018-07-13 12:52 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Krasiyan Andreev <[email protected]>
Adds the options RESPECT/IGNORE NULLS (null treatment clause) and FROM
FIRST/LAST to the non-aggregate window functions.
A previous patch
(https://www.postgresql.org/message-id/[email protected]...)
partially implemented this feature. However, that patch worked by
adding the null treatment clause to the window frame's frameOptions
variable, and consequently had the limitation that it wasn't possible
to reuse a window frame definition in a single query where two
functions were called that had different null treatment options. This
meant that the patch was never committed. The attached path takes a
different approach which gets around this limitation.
For example, the following query would not work correctly with the
implementation in the old patch but does with the attached patch:
WITH cte (x) AS (
select null union select 1 union select 2)
SELECT x,
first_value(x) over w as with_default,
first_value(x) respect nulls over w as with_respect,
first_value(x) ignore nulls over w as with_ignore
from cte WINDOW w as (order by x nulls first rows between unbounded
preceding and unbounded following);
x | with_default | with_respect | with_ignore
---+--------------+--------------+-------------
| | | 1
1 | | | 1
2 | | | 1
(3 rows)
== Implementation ==
The patch adds two types to the pg_type catalog: "ignorenulls" and
"fromlast". These types are of the Boolean category, and work as
wrappers around the bool type. They are used as function arguments to
extra versions of the window functions that take additional boolean
arguments. RESPECT NULLS and FROM FIRST are ignored by the parser, but
IGNORE NULLS and FROM LAST lead to the extra versions being called
with arguments to ignore nulls and order from last.
== Testing ==
Updated documentation and added regression tests. All existing tests
pass. This change will need a catversion bump.
Thanks to Krasiyan Andreev for initially testing this patch.
Attachments:
[application/octet-stream] 0001-respect.patch (53.1K, ../../CAGMVOdsbtRwE_4+v8zjH1d9xfovDeQAGLkP_B6k69_VoFEgX-A@mail.gmail.com/2-0001-respect.patch)
download | inline diff:
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index edc9be92a6..cf44aeddcf 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14762,7 +14762,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<function>
lag(<replaceable class="parameter">value</replaceable> <type>anyelement</type>
[, <replaceable class="parameter">offset</replaceable> <type>integer</type>
- [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]])
+ [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]]) [null_treatment]
</function>
</entry>
<entry>
@@ -14791,7 +14791,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<function>
lead(<replaceable class="parameter">value</replaceable> <type>anyelement</type>
[, <replaceable class="parameter">offset</replaceable> <type>integer</type>
- [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]])
+ [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]]) [null_treatment]
</function>
</entry>
<entry>
@@ -14817,7 +14817,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value(<replaceable class="parameter">value</replaceable> <type>any</type>)</function>
+ <function>first_value(<replaceable class="parameter">value</replaceable> <type>any</type>) [null_treatment]</function>
</entry>
<entry>
<type>same type as <replaceable class="parameter">value</replaceable></type>
@@ -14833,7 +14833,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value(<replaceable class="parameter">value</replaceable> <type>any</type>)</function>
+ <function>last_value(<replaceable class="parameter">value</replaceable> <type>any</type>) [null_treatment]</function>
</entry>
<entry>
<type>same type as <replaceable class="parameter">value</replaceable></type>
@@ -14850,7 +14850,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<primary>nth_value</primary>
</indexterm>
<function>
- nth_value(<replaceable class="parameter">value</replaceable> <type>any</type>, <replaceable class="parameter">nth</replaceable> <type>integer</type>)
+ nth_value(<replaceable class="parameter">value</replaceable> <type>any</type>, <replaceable class="parameter">nth</replaceable> <type>integer</type>) [from_first_last] [null_treatment]
</function>
</entry>
<entry>
@@ -14865,6 +14865,23 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
</tbody>
</tgroup>
</table>
+
+ <para>
+ In <xref linkend="functions-window-table"/>, <replaceable>null_treatment</replaceable> is one of:
+ <synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+ </synopsis>
+
+ and <replaceable>from_first_last</replaceable> is one of:
+ <synopsis>
+ FROM FIRST
+ FROM LAST
+ </synopsis>
+ <literal>RESPECT NULLS</literal> specifies the default behavior to include nulls in the result.
+ <literal>IGNORE NULLS</literal> ignores any null values when determining a result.
+ <literal>FROM FIRST</literal> specifies the default ordering, and <literal>FROM LAST</literal> reverses the ordering.
+ </para>
<para>
All of the functions listed in
@@ -14901,22 +14918,6 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
Other frame specifications can be used to obtain other effects.
</para>
- <note>
- <para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
- ordering.)
- </para>
- </note>
-
<para>
<function>cume_dist</function> computes the fraction of partition rows that
are less than or equal to the current row and its peers, while
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index aeb262a5b0..30257157b8 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -494,9 +494,9 @@ T612 Advanced OLAP operations NO some forms supported
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE function YES
-T618 NTH_VALUE function NO function exists, but some options missing
+T618 NTH_VALUE function YES
T619 Nested window functions NO
T620 WINDOW clause: GROUPS option YES
T621 Enhanced numeric functions YES
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 90dfac2cb1..44e8de3c12 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -153,6 +153,7 @@ static Node *makeBitStringConst(char *str, int location);
static Node *makeNullAConst(int location);
static Node *makeAConst(Value *v, int location);
static Node *makeBoolAConst(bool state, int location);
+static Node *makeTypedBoolAConst(bool state, char *type, int location);
static RoleSpec *makeRoleSpec(RoleSpecType type, int location);
static void check_qualified_name(List *names, core_yyscan_t yyscanner);
static List *check_func_name(List *names, core_yyscan_t yyscanner);
@@ -561,7 +562,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> xml_namespace_list
%type <target> xml_namespace_el
-%type <node> func_application func_expr_common_subexpr
+%type <node> func_application func_expr_common_subexpr func_expr_first_last
%type <node> func_expr func_expr_windowless
%type <node> common_table_expr
%type <with> with_clause opt_with_clause
@@ -569,6 +570,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> within_group_clause
%type <node> filter_clause
+%type <ival> from_first_last_null_treatment_clause null_treatment_clause
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
@@ -632,14 +634,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN
EXTENSION EXTERNAL EXTRACT
- FALSE_P FAMILY FETCH FILTER FIRST_P FLOAT_P FOLLOWING FOR
+ FALSE_P FAMILY FETCH FILTER FIRST_P FIRST_VALUE FLOAT_P FOLLOWING FOR
FORCE FOREIGN FORWARD FREEZE FROM FULL FUNCTION FUNCTIONS
GENERATED GLOBAL GRANT GRANTED GREATEST GROUP_P GROUPING GROUPS
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -648,14 +650,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
KEY
- LABEL LANGUAGE LARGE_P LAST_P LATERAL_P
- LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
+ LABEL LAG LANGUAGE LARGE_P LAST_P LAST_VALUE LATERAL_P
+ LEAD LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED
MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE
NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NO NONE
- NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF
+ NOT NOTHING NOTIFY NOTNULL NOWAIT NTH_VALUE NULL_P NULLIF
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
@@ -670,7 +672,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT RESTART RESTRICT RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT SEQUENCE SEQUENCES
@@ -714,6 +716,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
/* Precedence: lowest to highest */
%nonassoc SET /* see relation_expr_opt_alias */
+%nonassoc FIRST_VALUE LAG LAST_VALUE LEAD NTH_VALUE
%left UNION EXCEPT
%left INTERSECT
%left OR
@@ -763,6 +766,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%right UMINUS
%left '[' ']'
%left '(' ')'
+%left FROM FIRST_P LAST_P NULLS_P
%left TYPECAST
%left '.'
/*
@@ -13632,6 +13636,33 @@ func_application: func_name '(' ')'
}
;
+null_treatment_clause:
+ RESPECT NULLS_P { $$ = 0; }
+ | IGNORE_P NULLS_P { $$ = WINFUNC_OPT_IGNORE_NULLS; }
+ ;
+
+from_first_last_null_treatment_clause:
+ FROM FIRST_P null_treatment_clause
+ {
+ $$ = $3;
+ }
+ | FROM LAST_P null_treatment_clause
+ {
+ $$ = WINFUNC_OPT_FROM_LAST | $3;
+ }
+ | null_treatment_clause
+ {
+ $$ = $1;
+ }
+ | FROM FIRST_P
+ {
+ $$ = 0;
+ }
+ | FROM LAST_P
+ {
+ $$ = WINFUNC_OPT_FROM_LAST;
+ }
+ ;
/*
* func_expr and its cousin func_expr_windowless are split out from c_expr just
@@ -13679,6 +13710,133 @@ func_expr: func_application within_group_clause filter_clause over_clause
}
| func_expr_common_subexpr
{ $$ = $1; }
+ | func_expr_first_last over_clause
+ {
+ FuncCall *n = (FuncCall *) $1;
+ n->over = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+func_expr_first_last:
+ FIRST_VALUE '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("first_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ |
+ FIRST_VALUE '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("first_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAG '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("lag")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAG '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("lag")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAST_VALUE '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("last_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAST_VALUE '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("last_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LEAD '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("lead")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LEAD '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("lead")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | NTH_VALUE '(' func_arg_list opt_sort_clause ')' from_first_last_null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Nulls and From First options to bools */
+ if (winFuncArgs & WINFUNC_OPT_FROM_LAST)
+ l = lappend(l, makeTypedBoolAConst(true, "fromlast", @2));
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("nth_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | NTH_VALUE '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("nth_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
;
/*
@@ -15082,6 +15240,7 @@ unreserved_keyword:
| FAMILY
| FILTER
| FIRST_P
+ | FIRST_VALUE
| FOLLOWING
| FORCE
| FORWARD
@@ -15097,6 +15256,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -15117,9 +15277,12 @@ unreserved_keyword:
| ISOLATION
| KEY
| LABEL
+ | LAG
| LANGUAGE
| LARGE_P
| LAST_P
+ | LAST_VALUE
+ | LEAD
| LEAKPROOF
| LEVEL
| LISTEN
@@ -15147,6 +15310,7 @@ unreserved_keyword:
| NOTHING
| NOTIFY
| NOWAIT
+ | NTH_VALUE
| NULLS_P
| OBJECT_P
| OF
@@ -15198,6 +15362,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT
| RESTART
| RESTRICT
| RETURNS
@@ -15670,13 +15835,22 @@ makeAConst(Value *v, int location)
static Node *
makeBoolAConst(bool state, int location)
{
+ return makeTypedBoolAConst(state, "bool", location);
+}
+
+/* makeTypedBoolAConst()
+ * Create an A_Const string node from a boolean and store inside the specified type.
+ */
+static Node *
+makeTypedBoolAConst(bool state, char *type, int location)
+{
A_Const *n = makeNode(A_Const);
n->val.type = T_String;
n->val.val.str = (state ? "t" : "f");
n->location = location;
- return makeTypeCast((Node *)n, SystemTypeName("bool"), -1);
+ return makeTypeCast((Node *)n, SystemTypeName(type), -1);
}
/* makeRoleSpec
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 065238b0fe..12e00ce2eb 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -9226,6 +9226,8 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
int nargs;
List *argnames;
ListCell *l;
+ bool ignorenulls = false,
+ fromlast = false;
if (list_length(wfunc->args) > FUNC_MAX_ARGS)
ereport(ERROR,
@@ -9252,15 +9254,47 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
if (wfunc->winstar)
appendStringInfoChar(buf, '*');
else
- get_rule_expr((Node *) wfunc->args, context, true);
+ {
+ ListCell *arglist;
+ Node *argnode = (Node *) wfunc->args;
+
+ get_rule_expr(argnode, context, true);
+
+ /* Determine if FROM LAST and/or IGNORE NULLS should be appended */
+ foreach(arglist, (List *) argnode)
+ {
+ Node *arg = (Node *) lfirst(arglist);
+ if (nodeTag(arg) == T_Const)
+ {
+ Const *constnode = (Const *) arg;
+ if (constnode->consttype == FROMLASTOID)
+ {
+ /* parser does not save FROM FIRST arguments */
+ fromlast = true;
+ buf->len -= 2;
+ }
+ if (constnode->consttype == IGNORENULLSOID)
+ {
+ /* parser does not save RESPECT NULLS arguments */
+ ignorenulls = true;
+ buf->len -= 2;
+ }
+ }
+ }
+ }
+
+ appendStringInfoChar(buf, ')');
+ if (fromlast)
+ appendStringInfoString(buf, " FROM LAST");
+ if (ignorenulls)
+ appendStringInfoString(buf, " IGNORE NULLS");
if (wfunc->aggfilter != NULL)
{
appendStringInfoString(buf, ") FILTER (WHERE ");
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
-
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, " OVER ");
foreach(l, context->windowClause)
{
@@ -9435,6 +9469,11 @@ get_const_expr(Const *constval, deparse_context *context, int showtype)
appendStringInfoString(buf, "false");
break;
+ case FROMLASTOID:
+ case IGNORENULLSOID:
+ showtype = -1;
+ break;
+
default:
simple_quote_literal(buf, extval);
break;
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 40ba783572..094590334d 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -38,7 +38,14 @@ typedef struct
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
+static Datum leadlag_common_ignore_nulls(FunctionCallInfo fcinfo,
+ bool forward, bool withoffset, bool withdefault);
+static Datum
+window_nth_value_ignorenulls_common(FunctionCallInfo fcinfo, int32 nth,
+ bool fromlast);
+static Datum
+window_nth_value_respectnulls_common(FunctionCallInfo fcinfo, int32 nth,
+ bool fromlast);
/*
* utility routine for *_rank functions.
@@ -328,6 +335,79 @@ leadlag_common(FunctionCallInfo fcinfo,
PG_RETURN_DATUM(result);
}
+static Datum
+leadlag_common_ignore_nulls(FunctionCallInfo fcinfo,
+ bool forward, bool withoffset, bool withdefault)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ int32 offset;
+ Datum result;
+ bool isnull;
+ bool isout = false;
+ int32 notnull_offset = 0, tmp_offset = 0;
+
+ if (withoffset)
+ {
+ offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ if (offset < 0)
+ {
+ offset = abs(offset);
+ forward = !forward;
+ } else if (offset == 0)
+ {
+ result = WinGetFuncArgInPartition(winobj, 0, 0,
+ WINDOW_SEEK_CURRENT,
+ false,
+ &isnull, &isout);
+ if (isnull || isout)
+ PG_RETURN_NULL();
+ else
+ PG_RETURN_DATUM(result);
+ }
+ }
+ else
+ offset = 1;
+
+ while (notnull_offset < offset)
+ {
+ tmp_offset++;
+ result = WinGetFuncArgInPartition(winobj, 0,
+ (forward ? tmp_offset : -tmp_offset),
+ WINDOW_SEEK_CURRENT,
+ false,
+ &isnull, &isout);
+ if (isout)
+ goto out_of_frame;
+ else if (!isnull)
+ notnull_offset++;
+ }
+
+ result = WinGetFuncArgInPartition(winobj, 0,
+ (forward ? tmp_offset : -tmp_offset),
+ WINDOW_SEEK_CURRENT,
+ false,
+ &isnull, &isout);
+ if (isout)
+ goto out_of_frame;
+ else
+ PG_RETURN_DATUM(result);
+
+ out_of_frame:
+ /*
+ * target row is out of the partition; supply default value if
+ * provided. Otherwise return NULL.
+ */
+ if (withdefault)
+ {
+ result = WinGetFuncArgCurrent(winobj, 2, &isnull);
+ PG_RETURN_DATUM(result);
+ }
+ else
+ PG_RETURN_NULL();
+}
+
/*
* lag
* returns the value of VE evaluated on a row that is 1
@@ -363,6 +443,24 @@ window_lag_with_offset_and_default(PG_FUNCTION_ARGS)
return leadlag_common(fcinfo, false, true, true);
}
+Datum
+window_lag_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, false, false, false);
+}
+
+Datum
+window_lag_with_offset_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, false, true, false);
+}
+
+Datum
+window_lag_with_offset_and_default_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, false, true, true);
+}
+
/*
* lead
* returns the value of VE evaluated on a row that is 1
@@ -398,6 +496,24 @@ window_lead_with_offset_and_default(PG_FUNCTION_ARGS)
return leadlag_common(fcinfo, true, true, true);
}
+Datum
+window_lead_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, true, false, false);
+}
+
+Datum
+window_lead_with_offset_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, true, true, false);
+}
+
+Datum
+window_lead_with_offset_and_default_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, true, true, true);
+}
+
/*
* first_value
* return the value of VE evaluated on the first row of the
@@ -419,6 +535,31 @@ window_first_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+Datum
+window_first_value_nulls_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ Datum result;
+ bool isnull,
+ isout;
+ int64 pos;
+
+ isout = false;
+ pos = 0;
+
+ while (!isout)
+ {
+ result = WinGetFuncArgInFrame(winobj, 0,
+ pos, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (!isnull)
+ PG_RETURN_DATUM(result);
+ pos++;
+ }
+
+ PG_RETURN_NULL();
+}
+
/*
* last_value
* return the value of VE evaluated on the last row of the
@@ -440,35 +581,156 @@ window_last_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+Datum
+window_last_value_nulls_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ Datum result;
+ bool isnull,
+ isout;
+ int64 pos;
+
+ isout = false;
+ pos = 0;
+
+ while (!isout)
+ {
+ result = WinGetFuncArgInFrame(winobj, 0,
+ pos, WINDOW_SEEK_TAIL, false,
+ &isnull, &isout);
+ if (!isnull)
+ PG_RETURN_DATUM(result);
+ pos--;
+ }
+
+ PG_RETURN_NULL();
+}
+
/*
* nth_value
* return the value of VE evaluated on the n-th row from the first
* row of the window frame, per spec.
*/
-Datum
-window_nth_value(PG_FUNCTION_ARGS)
+static Datum
+window_nth_value_respectnulls_common(FunctionCallInfo fcinfo, int32 nth,
+ bool fromlast)
{
WindowObject winobj = PG_WINDOW_OBJECT();
bool const_offset;
Datum result;
bool isnull;
- int32 nth;
+ const_offset = get_fn_expr_arg_stable(fcinfo->flinfo, 1);
+
+ if (nth <= 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_NTH_VALUE),
+ errmsg("argument of nth_value must be greater than zero")));
+
+ result = WinGetFuncArgInFrame(winobj,
+ 0,
+ nth - 1,
+ fromlast ? WINDOW_SEEK_TAIL : WINDOW_SEEK_HEAD, const_offset,
+ &isnull,
+ NULL);
+ if (isnull)
+ PG_RETURN_NULL();
+
+ PG_RETURN_DATUM(result);
+}
+
+static Datum
+window_nth_value_ignorenulls_common(FunctionCallInfo fcinfo, int32 nth,
+ bool fromlast)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ Datum result;
+ bool isnull,
+ isout;
+ int32 tmp_offset, notnull_offset = 0;
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (fromlast)
+ tmp_offset = 1;
+ else
+ tmp_offset = -1;
+
if (isnull)
PG_RETURN_NULL();
- const_offset = get_fn_expr_arg_stable(fcinfo->flinfo, 1);
if (nth <= 0)
ereport(ERROR,
(errcode(ERRCODE_INVALID_ARGUMENT_FOR_NTH_VALUE),
errmsg("argument of nth_value must be greater than zero")));
+ while (notnull_offset < nth)
+ {
+ fromlast ? tmp_offset-- : tmp_offset++;
+ result = WinGetFuncArgInFrame(winobj, 0,
+ tmp_offset, fromlast ? WINDOW_SEEK_TAIL : WINDOW_SEEK_HEAD,
+ false, &isnull, &isout);
+ if (isout)
+ PG_RETURN_NULL();
+ if (!isnull)
+ notnull_offset++;
+ }
+
result = WinGetFuncArgInFrame(winobj, 0,
- nth - 1, WINDOW_SEEK_HEAD, const_offset,
- &isnull, NULL);
- if (isnull)
+ tmp_offset, fromlast ? WINDOW_SEEK_TAIL : WINDOW_SEEK_HEAD,
+ false, &isnull, &isout);
+ if (isout || isnull)
PG_RETURN_NULL();
PG_RETURN_DATUM(result);
}
+
+Datum
+window_nth_value(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ bool isnull;
+ int32 nth;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ PG_RETURN_DATUM(window_nth_value_respectnulls_common(fcinfo, nth, false));
+}
+
+Datum
+window_nth_value_with_first_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ bool isnull;
+ int32 nth;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ PG_RETURN_DATUM(window_nth_value_respectnulls_common(fcinfo, nth, true));
+}
+
+Datum
+window_nth_value_with_nulls_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ bool isnull;
+ int32 nth;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ PG_RETURN_DATUM(window_nth_value_ignorenulls_common(fcinfo, nth, false));
+}
+
+Datum
+window_nth_value_with_first_nulls_opts(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ bool isnull;
+ int32 nth;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ PG_RETURN_DATUM(window_nth_value_ignorenulls_common(fcinfo, nth, true));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a14651010f..2fdb509a69 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9432,32 +9432,72 @@
{ oid => '3106', descr => 'fetch the preceding row value',
proname => 'lag', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_lag' },
+{ oid => '4144', descr => 'fetch the preceding row value with nulls option',
+ proname => 'lag', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_lag_nulls_opt' },
{ oid => '3107', descr => 'fetch the Nth preceding row value',
proname => 'lag', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_lag_with_offset' },
+{ oid => '4145', descr => 'fetch the Nth preceding row value with nulls option',
+ proname => 'lag', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 ignorenulls',
+ prosrc => 'window_lag_with_offset_nulls_opt' },
{ oid => '3108', descr => 'fetch the Nth preceding row value with default',
proname => 'lag', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4 anyelement',
prosrc => 'window_lag_with_offset_and_default' },
+{ oid => '4146', descr => 'fetch the Nth preceding row value with default and nulls option',
+ proname => 'lag', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 anyelement ignorenulls',
+ prosrc => 'window_lag_with_offset_and_default_nulls_opt' },
{ oid => '3109', descr => 'fetch the following row value',
proname => 'lead', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_lead' },
+{ oid => '4147', descr => 'fetch the following row value with nulls option',
+ proname => 'lead', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_lead_nulls_opt' },
{ oid => '3110', descr => 'fetch the Nth following row value',
proname => 'lead', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_lead_with_offset' },
+{ oid => '4148', descr => 'fetch the Nth following row value with nulls option',
+ proname => 'lead', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 ignorenulls',
+ prosrc => 'window_lead_with_offset_nulls_opt' },
{ oid => '3111', descr => 'fetch the Nth following row value with default',
proname => 'lead', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4 anyelement',
prosrc => 'window_lead_with_offset_and_default' },
+{ oid => '4149', descr => 'fetch the Nth following row value with default and nulls option',
+ proname => 'lead', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 anyelement ignorenulls',
+ prosrc => 'window_lead_with_offset_and_default_nulls_opt' },
{ oid => '3112', descr => 'fetch the first row value',
proname => 'first_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_first_value' },
+{ oid => '4150', descr => 'fetch the first row value with nulls option',
+ proname => 'first_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_first_value_nulls_opt' },
{ oid => '3113', descr => 'fetch the last row value',
proname => 'last_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_last_value' },
+{ oid => '4151', descr => 'fetch the last row value with nulls option',
+ proname => 'last_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_last_value_nulls_opt' },
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '4152', descr => 'fetch the Nth row value with from first option',
+ proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 fromlast',
+ prosrc => 'window_nth_value_with_first_opt' },
+{ oid => '4153', descr => 'fetch the Nth row value with nulls option',
+ proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 ignorenulls',
+ prosrc => 'window_nth_value_with_nulls_opt' },
+{ oid => '4154', descr => 'fetch the Nth row value with from first and nulls option',
+ proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 fromlast ignorenulls',
+ prosrc => 'window_nth_value_with_first_nulls_opts' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 48e01cd694..f9f6933b18 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -938,5 +938,15 @@
typname => 'anyrange', typlen => '-1', typbyval => 'f', typtype => 'p',
typcategory => 'P', typinput => 'anyrange_in', typoutput => 'anyrange_out',
typreceive => '-', typsend => '-', typalign => 'd', typstorage => 'x' },
+{ oid => '4142',
+ typname => 'ignorenulls', descr => 'boolean wrapper, \'true\'/\'false\'',
+ typlen => '1', typbyval => 't', typtype => 'b', typcategory => 'B',
+ typinput => 'boolin', typoutput => 'boolout', typreceive => 'boolrecv',
+ typsend => 'boolsend', typalign => 'c' },
+{ oid => '4143',
+ typname => 'fromlast', descr => 'boolean wrapper, \'true\'/\'false\'',
+ typlen => '1', typbyval => 't', typtype => 'b', typcategory => 'B',
+ typinput => 'boolin', typoutput => 'boolout', typreceive => 'boolrecv',
+ typsend => 'boolsend', typalign => 'c' },
]
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2a2b17d570..ddd9208d8b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -534,6 +534,12 @@ typedef struct WindowDef
FRAMEOPTION_END_CURRENT_ROW)
/*
+ * From Last and Null Treatment options
+ */
+#define WINFUNC_OPT_FROM_LAST 0x00001 /* FROM LAST */
+#define WINFUNC_OPT_IGNORE_NULLS 0x00002 /* IGNORE NULLS */
+
+/*
* RangeSubselect - subquery appearing in a FROM clause
*/
typedef struct RangeSubselect
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 23db40147b..a043742768 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -164,6 +164,7 @@ PG_KEYWORD("family", FAMILY, UNRESERVED_KEYWORD)
PG_KEYWORD("fetch", FETCH, RESERVED_KEYWORD)
PG_KEYWORD("filter", FILTER, UNRESERVED_KEYWORD)
PG_KEYWORD("first", FIRST_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("first_value", FIRST_VALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("float", FLOAT_P, COL_NAME_KEYWORD)
PG_KEYWORD("following", FOLLOWING, UNRESERVED_KEYWORD)
PG_KEYWORD("for", FOR, RESERVED_KEYWORD)
@@ -190,6 +191,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD)
@@ -223,10 +225,13 @@ PG_KEYWORD("isolation", ISOLATION, UNRESERVED_KEYWORD)
PG_KEYWORD("join", JOIN, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("key", KEY, UNRESERVED_KEYWORD)
PG_KEYWORD("label", LABEL, UNRESERVED_KEYWORD)
+PG_KEYWORD("lag", LAG, UNRESERVED_KEYWORD)
PG_KEYWORD("language", LANGUAGE, UNRESERVED_KEYWORD)
PG_KEYWORD("large", LARGE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("last", LAST_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("last_value", LAST_VALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("lateral", LATERAL_P, RESERVED_KEYWORD)
+PG_KEYWORD("lead", LEAD, UNRESERVED_KEYWORD)
PG_KEYWORD("leading", LEADING, RESERVED_KEYWORD)
PG_KEYWORD("leakproof", LEAKPROOF, UNRESERVED_KEYWORD)
PG_KEYWORD("least", LEAST, COL_NAME_KEYWORD)
@@ -267,6 +272,7 @@ PG_KEYWORD("nothing", NOTHING, UNRESERVED_KEYWORD)
PG_KEYWORD("notify", NOTIFY, UNRESERVED_KEYWORD)
PG_KEYWORD("notnull", NOTNULL, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("nowait", NOWAIT, UNRESERVED_KEYWORD)
+PG_KEYWORD("nth_value", NTH_VALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("null", NULL_P, RESERVED_KEYWORD)
PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD)
PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD)
@@ -336,6 +342,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD)
+PG_KEYWORD("respect", RESPECT, UNRESERVED_KEYWORD)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD)
PG_KEYWORD("returning", RETURNING, RESERVED_KEYWORD)
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index b1419d4bc2..d6f38f5b75 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -73,7 +73,9 @@ WHERE p1.typtype not in ('c','d','p') AND p1.typname NOT LIKE E'\\_%'
3361 | pg_ndistinct
3402 | pg_dependencies
210 | smgr
-(4 rows)
+ 4142 | ignorenulls
+ 4143 | fromlast
+(6 rows)
-- Make sure typarray points to a varlena array type of our own base
SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype,
@@ -166,10 +168,12 @@ WHERE p1.typinput = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p1.typelem != 0 AND p1.typlen < 0) AND NOT
(p2.prorettype = p1.oid AND NOT p2.proretset)
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+-----+---------
- 1790 | refcursor | 46 | textin
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+---------
+ 1790 | refcursor | 46 | textin
+ 4142 | ignorenulls | 1242 | boolin
+ 4143 | fromlast | 1242 | boolin
+(3 rows)
-- Varlena array types will point to array_in
-- Exception as of 8.1: int2vector and oidvector have their own I/O routines
@@ -217,10 +221,12 @@ WHERE p1.typoutput = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p2.oid = 'array_out'::regproc AND
p1.typelem != 0 AND p1.typlen = -1)))
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+-----+---------
- 1790 | refcursor | 47 | textout
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+---------
+ 1790 | refcursor | 47 | textout
+ 4142 | ignorenulls | 1243 | boolout
+ 4143 | fromlast | 1243 | boolout
+(3 rows)
SELECT p1.oid, p1.typname, p2.oid, p2.proname
FROM pg_type AS p1, pg_proc AS p2
@@ -280,10 +286,12 @@ WHERE p1.typreceive = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p1.typelem != 0 AND p1.typlen < 0) AND NOT
(p2.prorettype = p1.oid AND NOT p2.proretset)
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+------+----------
- 1790 | refcursor | 2414 | textrecv
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+----------
+ 1790 | refcursor | 2414 | textrecv
+ 4142 | ignorenulls | 2436 | boolrecv
+ 4143 | fromlast | 2436 | boolrecv
+(3 rows)
-- Varlena array types will point to array_recv
-- Exception as of 8.1: int2vector and oidvector have their own I/O routines
@@ -340,10 +348,12 @@ WHERE p1.typsend = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p2.oid = 'array_send'::regproc AND
p1.typelem != 0 AND p1.typlen = -1)))
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+------+----------
- 1790 | refcursor | 2415 | textsend
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+----------
+ 1790 | refcursor | 2415 | textsend
+ 4142 | ignorenulls | 2437 | boolsend
+ 4143 | fromlast | 2437 | boolsend
+(3 rows)
SELECT p1.oid, p1.typname, p2.oid, p2.proname
FROM pg_type AS p1, pg_proc AS p2
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index 562006a2b8..87b9340129 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -3787,3 +3787,369 @@ SELECT i, b, bool_and(b) OVER w, bool_or(b) OVER w
5 | t | t | t
(5 rows)
+-- FROM LAST and IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ orbit int
+);
+INSERT INTO planets VALUES
+ ('mercury', 88),
+ ('venus', 224),
+ ('earth', NULL),
+ ('mars', NULL),
+ ('jupiter', 4332),
+ ('saturn', 24491),
+ ('uranus', NULL),
+ ('neptune', 60182),
+ ('pluto', 90560);
+ -- test view definitions are preserved
+CREATE TEMP VIEW v_planets AS
+ SELECT
+ name,
+ sum(orbit) OVER (order by orbit) as sum_rows,
+ lag(orbit, 1) RESPECT NULLS OVER (ORDER BY name DESC) AS lagged_by_1,
+ lag(orbit, 2) IGNORE NULLS OVER w AS lagged_by_2,
+ first_value(orbit) IGNORE NULLS OVER w AS first_value_ignore,
+ nth_value(orbit,2) FROM FIRST IGNORE NULLS OVER w AS nth_first_ignore,
+ nth_value(orbit,2) FROM LAST IGNORE NULLS OVER w AS nth_last_ignore
+ FROM planets
+ WINDOW w as (ORDER BY name ASC);
+SELECT pg_get_viewdef('v_planets');
+ pg_get_viewdef
+----------------------------------------------------------------------------------
+ SELECT planets.name, +
+ sum(planets.orbit) OVER (ORDER BY planets.orbit) AS sum_rows, +
+ lag(planets.orbit, 1) OVER (ORDER BY planets.name DESC) AS lagged_by_1, +
+ lag(planets.orbit, 2) IGNORE NULLS OVER w AS lagged_by_2, +
+ first_value(planets.orbit) IGNORE NULLS OVER w AS first_value_ignore, +
+ nth_value(planets.orbit, 2) IGNORE NULLS OVER w AS nth_first_ignore, +
+ nth_value(planets.orbit, 2) FROM LAST IGNORE NULLS OVER w AS nth_last_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY planets.name);
+(1 row)
+
+SELECT name, lag(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury |
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus |
+(9 rows)
+
+SELECT name, lag(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury |
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus |
+(9 rows)
+
+SELECT name, lag(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury | 4332
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus | 24491
+(9 rows)
+
+SELECT name, lead(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth | 4332
+ jupiter |
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn |
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lead(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth | 4332
+ jupiter |
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn |
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lead(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth | 4332
+ jupiter | 88
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn | 224
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lag(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth | 4332
+ jupiter | 88
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn | 224
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lead(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury | 4332
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus | 24491
+(9 rows)
+
+SELECT name, first_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | first_value
+---------+-------------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+SELECT name, first_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | first_value
+---------+-------------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+SELECT name, first_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | first_value
+---------+-------------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, last_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | last_value
+---------+------------
+ earth | 224
+ jupiter | 224
+ mars | 224
+ mercury | 224
+ neptune | 224
+ pluto | 224
+ saturn | 224
+ uranus | 224
+ venus | 224
+(9 rows)
+
+SELECT name, last_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | last_value
+---------+------------
+ earth | 224
+ jupiter | 224
+ mars | 224
+ mercury | 224
+ neptune | 224
+ pluto | 224
+ saturn | 224
+ uranus | 224
+ venus | 224
+(9 rows)
+
+SELECT name, last_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | last_value
+---------+------------
+ earth | 224
+ jupiter | 224
+ mars | 224
+ mercury | 224
+ neptune | 224
+ pluto | 224
+ saturn | 224
+ uranus | 224
+ venus | 224
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 88
+ jupiter | 88
+ mars | 88
+ mercury | 88
+ neptune | 88
+ pluto | 88
+ saturn | 88
+ uranus | 88
+ venus | 88
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) FROM FIRST OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) FROM FIRST IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 88
+ jupiter | 88
+ mars | 88
+ mercury | 88
+ neptune | 88
+ pluto | 88
+ saturn | 88
+ uranus | 88
+ venus | 88
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) FROM FIRST RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) FROM LAST OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) FROM LAST IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 24491
+ jupiter | 24491
+ mars | 24491
+ mercury | 24491
+ neptune | 24491
+ pluto | 24491
+ saturn | 24491
+ uranus | 24491
+ venus | 24491
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) FROM LAST RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view v_planets
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index e2943a38f1..95d5125533 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1241,3 +1241,66 @@ SELECT to_char(SUM(n::float8) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FO
SELECT i, b, bool_and(b) OVER w, bool_or(b) OVER w
FROM (VALUES (1,true), (2,true), (3,false), (4,false), (5,true)) v(i,b)
WINDOW w AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING);
+
+-- FROM LAST and IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ orbit int
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 88),
+ ('venus', 224),
+ ('earth', NULL),
+ ('mars', NULL),
+ ('jupiter', 4332),
+ ('saturn', 24491),
+ ('uranus', NULL),
+ ('neptune', 60182),
+ ('pluto', 90560);
+
+ -- test view definitions are preserved
+CREATE TEMP VIEW v_planets AS
+ SELECT
+ name,
+ sum(orbit) OVER (order by orbit) as sum_rows,
+ lag(orbit, 1) RESPECT NULLS OVER (ORDER BY name DESC) AS lagged_by_1,
+ lag(orbit, 2) IGNORE NULLS OVER w AS lagged_by_2,
+ first_value(orbit) IGNORE NULLS OVER w AS first_value_ignore,
+ nth_value(orbit,2) FROM FIRST IGNORE NULLS OVER w AS nth_first_ignore,
+ nth_value(orbit,2) FROM LAST IGNORE NULLS OVER w AS nth_last_ignore
+ FROM planets
+ WINDOW w as (ORDER BY name ASC);
+SELECT pg_get_viewdef('v_planets');
+
+SELECT name, lag(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lag(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lag(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, lead(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lead(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lead(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, lag(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lead(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, first_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, first_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, first_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, last_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, last_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, last_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, nth_value(orbit, 2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) FROM FIRST OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) FROM FIRST IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) FROM FIRST RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) FROM LAST OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) FROM LAST IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) FROM LAST RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+--cleanup
+DROP TABLE planets CASCADE;
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-07-28 18:59 David Fetter <[email protected]>
parent: Oliver Ford <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: David Fetter @ 2018-07-28 18:59 UTC (permalink / raw)
To: Oliver Ford <[email protected]>; +Cc: pgsql-hackers; Krasiyan Andreev <[email protected]>
On Fri, Jul 13, 2018 at 01:52:00PM +0100, Oliver Ford wrote:
> Adds the options RESPECT/IGNORE NULLS (null treatment clause) and FROM
> FIRST/LAST to the non-aggregate window functions.
Please find attached an updated version for OID drift.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-18 12:05 Krasiyan Andreev <[email protected]>
parent: David Fetter <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Krasiyan Andreev @ 2018-09-18 12:05 UTC (permalink / raw)
To: [email protected]; +Cc: Oliver Ford <[email protected]>; pgsql-hackers
Hi,
Patch applies and compiles, all included tests and building of the docs
pass.
I am using last version from more than two months ago in production
environment with real data and I didn't find any bugs,
so I'm marking this patch as ready for committer in the commitfest app.
На сб, 28.07.2018 г. в 22:00 ч. David Fetter <[email protected]> написа:
> On Fri, Jul 13, 2018 at 01:52:00PM +0100, Oliver Ford wrote:
> > Adds the options RESPECT/IGNORE NULLS (null treatment clause) and FROM
> > FIRST/LAST to the non-aggregate window functions.
>
> Please find attached an updated version for OID drift.
>
> Best,
> David.
> --
> David Fetter <david(at)fetter(dot)org> http://fetter.org/
> Phone: +1 415 235 3778
>
> Remember to vote!
> Consider donating to Postgres: http://www.postgresql.org/about/donate
>
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-22 20:06 Andrew Gierth <[email protected]>
parent: Krasiyan Andreev <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Andrew Gierth @ 2018-09-22 20:06 UTC (permalink / raw)
To: pgsql-hackers; +Cc: [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
>>>>> "Krasiyan" == Krasiyan Andreev <[email protected]> writes:
Krasiyan> Hi,
Krasiyan> Patch applies and compiles, all included tests and building
Krasiyan> of the docs pass. I am using last version from more than two
Krasiyan> months ago in production environment with real data and I
Krasiyan> didn't find any bugs, so I'm marking this patch as ready for
Krasiyan> committer in the commitfest app.
Unfortunately, reviewing it from a committer perspective - I can't
possibly commit this as it stands, and anything I did to it would be
basically a rewrite of much of it.
Some of the problems could be fixed. For example the type names could be
given pg_* prefixes (it's clearly not acceptable to create random
special-purpose boolean subtypes in pg_catalog and _not_ give them such
a prefix), and the precedence hackery in gram.y could have comments
added (gram.y is already bad enough; _anything_ fancy with precedence
has to be described in the comments). But I don't like that hack with
the special types at all, and I think that needs a better solution.
Normally I'd push hard to try and get some solution that's sufficiently
generic to allow user-defined functions to make use of the feature. But
I think the SQL spec people have managed to make that literally
impossible in this case, what with the FROM keyword appearing in the
middle of a production and not followed by anything sufficiently
distinctive to even use for extra token lookahead.
Also, as has been pointed out in a number of previous features, we're
starting to accumulate identifiers that are reserved in subtly different
ways from our basic four-category system (which is itself a significant
elaboration compared to the spec's simple reserved/unreserved
distinction). As I recall this objection was specifically raised for
CUBE, but justified there by the existence of the contrib/cube extension
(and the fact that the standard CUBE() construct is used only in very
specific places in the syntax). This patch would make lead / lag /
first_value / last_value / nth_value syntactically "special" while not
actually reserving them (beyond having them in unreserved_keywords); I
think serious consideration should be given to whether they should
instead become col_name_keywords (which would, I believe, make it
unnecessary to mess with precedence).
Anyone have any thoughts or comments on the above?
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-23 21:13 Tom Lane <[email protected]>
parent: Andrew Gierth <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tom Lane @ 2018-09-23 21:13 UTC (permalink / raw)
To: Andrew Gierth <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
Andrew Gierth <[email protected]> writes:
> Normally I'd push hard to try and get some solution that's sufficiently
> generic to allow user-defined functions to make use of the feature. But
> I think the SQL spec people have managed to make that literally
> impossible in this case, what with the FROM keyword appearing in the
> middle of a production and not followed by anything sufficiently
> distinctive to even use for extra token lookahead.
Yeah. Is there any appetite for a "Just Say No" approach? That is,
refuse to implement the spec's syntax on the grounds that it's too
brain-dead to even consider, and instead provide some less random,
more extensibility-friendly way to accomplish the same thing?
The FROM FIRST/LAST bit seems particularly badly thought through,
because AFAICS it is flat out ambiguous with a normal FROM clause
immediately following the window function call. The only way to
make it not so would be to make FIRST and LAST be fully reserved,
which is neither a good idea nor spec-compliant.
In short, there's a really good case to be made here that the SQL
committee is completely clueless about syntax design, and so we
shouldn't follow this particular pied piper.
> ... This patch would make lead / lag /
> first_value / last_value / nth_value syntactically "special" while not
> actually reserving them (beyond having them in unreserved_keywords); I
> think serious consideration should be given to whether they should
> instead become col_name_keywords (which would, I believe, make it
> unnecessary to mess with precedence).
I agree that messing with the precedence rules is likely to have
unforeseen and undesirable side-effects. Generally, if you need
to create a precedence rule, that's because your grammar is
ambiguous. Precedence fixes that in a well-behaved way only for
cases that actually are very much like operator precedence rules.
Otherwise, you may just be papering over something that isn't
working very well. See e.g. commits 670a6c7a2 and 12b716457
for past cases where we learned that the hard way. (The latter
also points out that if you must have a precedence hack, it's
safer to hack individual rules than to stick precedences onto
terminal symbols.) In the case at hand, since the proposed patch
doesn't make FIRST and LAST be fully reserved, it seems just about
certain that it can be made to misbehave, including failing on
queries that were and should remain legal.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-23 22:22 Andrew Gierth <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Andrew Gierth @ 2018-09-23 22:22 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
>>>>> "Tom" == Tom Lane <[email protected]> writes:
Tom> The FROM FIRST/LAST bit seems particularly badly thought through,
Tom> because AFAICS it is flat out ambiguous with a normal FROM clause
Tom> immediately following the window function call. The only way to
Tom> make it not so would be to make FIRST and LAST be fully reserved,
Tom> which is neither a good idea nor spec-compliant.
In the actual spec syntax it's not ambiguous at all because NTH_VALUE is
a reserved word (as are LEAD, LAG, FIRST_VALUE and LAST_VALUE), and OVER
is a mandatory clause in its syntax, so a FROM appearing before the OVER
must be part of a FROM FIRST/LAST and not introducing a FROM-clause.
In our syntax, if we made NTH_VALUE etc. a col_name_keyword (and thus
not legal as a function name outside its own special syntax) it would
also become unambiguous.
i.e. given this token sequence (with . marking the current posision):
select nth_value(x) . from first ignore
if we know up front that "nth_value" is a window function and not any
other kind of function, we know that we have to shift the "from" rather
than reducing the select-list because we haven't seen an "over" yet.
(Neither "first" nor "ignore" are reserved, so "select foo(x) from first
ignore;" is a valid and complete query, and without reserving the
function name we'd need at least four tokens of lookahead to decide
otherwise.)
This is why I think the col_name_keyword option needs to be given
serious consideration - it still doesn't reserve the names as strongly
as the spec does, but enough to make the standard syntax work without
needing any dubious hacks.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-23 23:10 Tom Lane <[email protected]>
parent: Andrew Gierth <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tom Lane @ 2018-09-23 23:10 UTC (permalink / raw)
To: Andrew Gierth <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
Andrew Gierth <[email protected]> writes:
> "Tom" == Tom Lane <[email protected]> writes:
> Tom> The FROM FIRST/LAST bit seems particularly badly thought through,
> Tom> because AFAICS it is flat out ambiguous with a normal FROM clause
> Tom> immediately following the window function call. The only way to
> Tom> make it not so would be to make FIRST and LAST be fully reserved,
> Tom> which is neither a good idea nor spec-compliant.
> In the actual spec syntax it's not ambiguous at all because NTH_VALUE is
> a reserved word (as are LEAD, LAG, FIRST_VALUE and LAST_VALUE), and OVER
> is a mandatory clause in its syntax, so a FROM appearing before the OVER
> must be part of a FROM FIRST/LAST and not introducing a FROM-clause.
Hmm ...
> In our syntax, if we made NTH_VALUE etc. a col_name_keyword (and thus
> not legal as a function name outside its own special syntax) it would
> also become unambiguous.
> i.e. given this token sequence (with . marking the current posision):
> select nth_value(x) . from first ignore
> if we know up front that "nth_value" is a window function and not any
> other kind of function, we know that we have to shift the "from" rather
> than reducing the select-list because we haven't seen an "over" yet.
I don't really find that to be a desirable solution, because quite aside
from the extensibility problem, it would mean that a lot of errors
become "syntax error" where we formerly gave a more useful message.
This does open up a thought about how to proceed, though. I'd been
trying to think of a way to solve this using base_yylex's ability to
do some internal lookahead and change token types based on that.
If you just think of recognizing FROM FIRST/LAST, you get nowhere
because that's still legal in other contexts. But if you were to
look for FROM followed by FIRST/LAST followed by IGNORE/RESPECT/OVER,
I think that could only validly happen in this syntax. It'd take
some work to extend base_yylex to look ahead 2 tokens not one, but
I'm sure that could be done. (You'd also need a lookahead rule to
match "IGNORE/RESPECT NULLS OVER", but that seems just as doable.)
Then the relevant productions use FROM_LA, IGNORE_LA, RESPECT_LA
instead of the corresponding bare tokens, and the grammar no longer
has an ambiguity problem.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-23 23:46 Andrew Gierth <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Andrew Gierth @ 2018-09-23 23:46 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
>>>>> "Tom" == Tom Lane <[email protected]> writes:
Tom> If you just think of recognizing FROM FIRST/LAST, you get nowhere
Tom> because that's still legal in other contexts. But if you were to
Tom> look for FROM followed by FIRST/LAST followed by
Tom> IGNORE/RESPECT/OVER, I think that could only validly happen in
Tom> this syntax.
No; you need to go four tokens ahead in total, not three. Assuming
nth_value is unreserved, then
select nth_value(x) from first ignore;
is a valid query that has nth_value(x) as an expression, "first" as a
table name and "ignore" as its alias. Only when you see NULLS after
IGNORE, or OVER after FIRST/LAST, do you know that you're looking at
a window function and not a from clause.
So FROM_LA would have to mean "FROM" followed by any of:
FIRST IGNORE NULLS
LAST IGNORE NULLS
FIRST RESPECT NULLS
LAST RESPECT NULLS
FIRST OVER
LAST OVER
Remember that while OVER is reserved, all of FIRST, LAST, RESPECT and
IGNORE are unreserved.
Tom> It'd take some work to extend base_yylex to look ahead 2 tokens
Tom> not one, but I'm sure that could be done. (You'd also need a
Tom> lookahead rule to match "IGNORE/RESPECT NULLS OVER", but that
Tom> seems just as doable.) Then the relevant productions use FROM_LA,
Tom> IGNORE_LA, RESPECT_LA instead of the corresponding bare tokens,
Tom> and the grammar no longer has an ambiguity problem.
Yeah, but at the cost of having to extend base_yylex to go 3 tokens
ahead (not 2) rather than the current single lookahead slot.
Doable, certainly (probably not much harder to do 3 than 2 actually)
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-24 00:20 Tom Lane <[email protected]>
parent: Andrew Gierth <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tom Lane @ 2018-09-24 00:20 UTC (permalink / raw)
To: Andrew Gierth <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
Andrew Gierth <[email protected]> writes:
> "Tom" == Tom Lane <[email protected]> writes:
> Tom> If you just think of recognizing FROM FIRST/LAST, you get nowhere
> Tom> because that's still legal in other contexts. But if you were to
> Tom> look for FROM followed by FIRST/LAST followed by
> Tom> IGNORE/RESPECT/OVER, I think that could only validly happen in
> Tom> this syntax.
> No; you need to go four tokens ahead in total, not three. Assuming
> nth_value is unreserved, then
> select nth_value(x) from first ignore;
> is a valid query that has nth_value(x) as an expression, "first" as a
> table name and "ignore" as its alias.
No, because once IGNORE is a keyword, even unreserved, it's not legal
as an AS-less alias. We'd be breaking queries like that no matter what.
(I know there are people around here who'd like to remove that
restriction, but it's not happening anytime soon IMO.)
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-24 00:26 Andrew Gierth <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Andrew Gierth @ 2018-09-24 00:26 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
>>>>> "Tom" == Tom Lane <[email protected]> writes:
>> select nth_value(x) from first ignore;
Tom> No, because once IGNORE is a keyword, even unreserved, it's not
Tom> legal as an AS-less alias.
That rule only applies in the select-list, not in the FROM clause; table
aliases in FROM are just ColId, so they can be anything except a fully
reserved or type_func_name keyword.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-25 04:16 Andrew Gierth <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 2 replies; 91+ messages in thread
From: Andrew Gierth @ 2018-09-25 04:16 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
So I've tried to rough out a decision tree for the various options on
how this might be implemented (discarding the "use precedence hacks"
option). Opinions? Additions?
(formatted for emacs outline-mode)
* 1. use lexical lookahead
+: relatively straightforward parser changes
+: no new reserved words
+: has the option of working extensibly with all functions
-: base_yylex needs extending to 3 lookahead tokens
** 1.1. Allow from/ignore clause on all (or all non-agg) window function calls
If the clauses are legal on all window functions, what to do about existing
window functions for which the clauses do not make sense?
*** 1.1.1. Ignore the clause when the function isn't aware of it
+: simple
-: somewhat surprising for users perhaps?
*** 1.1.2. Change the behavior of the windowapi in some consistent way
Not sure if this can work.
+: fairly simple(maybe?) and predictable
-: changes the behavior of existing window functions
** 1.2. Allow from/ignore clause on only certain functions
+: avoids any unexpected behavior
-: needs some way to control what functions allow it
*** 1.2.1. Check the function name in parse analysis against a fixed list.
+: simple
-: not extensible
*** 1.2.2. Provide some option in CREATE FUNCTION
+: extensible
-: fairly intrusive, adding stuff to create function and pg_proc
*** 1.2.3. Do something magical with function argument types
+: doesn't need changes in create function / pg_proc
-: it's an ugly hack
* 2. reserve nth_value etc. as functions
+: follows the spec reasonably well
+: less of a hack than extending base_yylex
-: new reserved words
-: more parser rules
-: not extensible
(now goto 1.2.1)
* 3. "just say no" to the spec
e.g. add new functions like lead_ignore_nulls(), or add extra boolean
args to lead() etc. telling them to skip nulls
+: simple
-: doesn't conform to spec
-: using extra args isn't quite the right semantics
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-25 14:07 Tom Lane <[email protected]>
parent: Andrew Gierth <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tom Lane @ 2018-09-25 14:07 UTC (permalink / raw)
To: Andrew Gierth <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
Andrew Gierth <[email protected]> writes:
> So I've tried to rough out a decision tree for the various options on
> how this might be implemented (discarding the "use precedence hacks"
> option). Opinions? Additions?
I think it'd be worth at least drafting an implementation for the
lexical-lookahead fix. I think it's likely that we'll need to extend
base_yylex to do more lookahead in the future even if we don't do it
for this, given the SQL committee's evident love for COBOL-ish syntax
and lack of regard for what you can do in LALR(1).
The questions of how we interface to the individual window functions
are really independent of how we handle the parsing problem. My
first inclination is to just pass the flags down to the window functions
(store them in WindowObject and provide some additional inquiry functions
in windowapi.h) and let them deal with it.
> If the clauses are legal on all window functions, what to do about existing
> window functions for which the clauses do not make sense?
Option 1: do nothing, document that nothing happens if w.f. doesn't
implement it.
Option 2: record whether the inquiry functions got called. At end of
query, error out if they weren't and the options were used.
It's also worth wondering if we couldn't just implement the flags in
some generic fashion and not need to involve the window functions at
all. FROM LAST, for example, could and perhaps should be implemented
by inverting the sort order. Possibly IGNORE NULLS could be implemented
inside the WinGetFuncArgXXX functions? These behaviors might or might
not make much sense with other window functions, but that doesn't seem
like it's our problem.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-27 04:40 Andrew Gierth <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Andrew Gierth @ 2018-09-27 04:40 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
>>>>> "Tom" == Tom Lane <[email protected]> writes:
>> So I've tried to rough out a decision tree for the various options
>> on how this might be implemented (discarding the "use precedence
>> hacks" option). Opinions? Additions?
Tom> I think it'd be worth at least drafting an implementation for the
Tom> lexical-lookahead fix. I think it's likely that we'll need to
Tom> extend base_yylex to do more lookahead in the future even if we
Tom> don't do it for this, given the SQL committee's evident love for
Tom> COBOL-ish syntax and lack of regard for what you can do in
Tom> LALR(1).
That's not _quite_ a fair criticism of the SQL committee; they're not
ignoring the capabilities of parsers, they're just not making their
syntax robust in the presence of the kinds of extensions and
generalizations that we want to do.
Without exception that I know of, every time we've run into a problem
that needed ugly precedence rules or extra lookahead it's been caused by
us not reserving something that is reserved in the spec, or by us
allowing constructs that are not in the spec at all (postfix operators,
especially), or by us deliberately generalizing what the spec allows
(e.g. allowing full expressions where the spec only allows column
references, or allowing extra parens around subselects) or where we've
repurposed syntax from the spec in an incompatible way (as in
ANY(array)).
Anyway, for the time being I will mark this patch as "returned with
feedback".
>> If the clauses are legal on all window functions, what to do about
>> existing window functions for which the clauses do not make sense?
Tom> Option 1: do nothing, document that nothing happens if w.f.
Tom> doesn't implement it.
That was 1.1.1 on my list.
Tom> Option 2: record whether the inquiry functions got called. At end
Tom> of query, error out if they weren't and the options were used.
Erroring at the _end_ of the query seems a bit of a potential surprise.
Tom> It's also worth wondering if we couldn't just implement the flags
Tom> in some generic fashion and not need to involve the window
Tom> functions at all.
That was what I meant by option 1.1.2 on my list.
Tom> FROM LAST, for example, could and perhaps should be implemented by
Tom> inverting the sort order.
Actually that can't work for reasons brought up in the recent discussion
of optimization of window function sorts: if you change the sort order
you potentially disturb the ordering of peer rows, and the spec requires
that an (nth_value(x,n) from last over w) and (otherfunc(x) over w) for
order-equivalent windows "w" must see the peer rows in the same order.
So FROM LAST really does have to keep the original sort order, and count
backwards from the end of the window.
Tom> Possibly IGNORE NULLS could be implemented inside the
Tom> WinGetFuncArgXXX functions? These behaviors might or might not
Tom> make much sense with other window functions, but that doesn't seem
Tom> like it's our problem.
That's about what I was thinking for option 1.1.2, yes.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2018-09-27 04:58 Andrew Gierth <[email protected]>
parent: Krasiyan Andreev <[email protected]>
1 sibling, 0 replies; 91+ messages in thread
From: Andrew Gierth @ 2018-09-27 04:58 UTC (permalink / raw)
To: Oliver Ford <[email protected]>; +Cc: [email protected]; pgsql-hackers; Krasiyan Andreev <[email protected]>
>>>>> "Krasiyan" == Krasiyan Andreev <[email protected]> writes:
Krasiyan> I am using last version from more than two months ago in
Krasiyan> production environment with real data and I didn't find any
Krasiyan> bugs, so I'm marking this patch as ready for committer in the
Krasiyan> commitfest app.
Oliver (or anyone else), do you plan to continue working on this in the
immediate future, in line with the comments from myself and Tom in this
thread? If so I'll bump it to the next CF, otherwise I'll mark it
"returned with feedback".
I'm happy to help out with further work on this patch if needed, time
permitting.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v7 2/6] Document historic behavior about hiding directories and special files
@ 2020-03-06 22:50 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 22:50 UTC (permalink / raw)
Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
doc/src/sgml/func.sgml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
(mtime) of each file in the log directory. By default, only superusers
and members of the <literal>pg_monitor</literal> role can use this function.
Access may be granted to others using <command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal> role
can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
superusers and members of the <literal>pg_monitor</literal> role can
use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--jKBxcB1XkHIR0Eqt
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0003-Document-historic-behavior-about-hiding-directori.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v12 01/11] Document historic behavior about hiding directories and special files
@ 2020-03-06 22:50 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 22:50 UTC (permalink / raw)
Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
doc/src/sgml/func.sgml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
(mtime) of each file in the log directory. By default, only superusers
and members of the <literal>pg_monitor</literal> role can use this function.
Access may be granted to others using <command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal> role
can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
superusers and members of the <literal>pg_monitor</literal> role can
use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--wIc/V6YLA2QdyfT4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0002-Document-historic-behavior-of-links-to-directori.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v11 1/9] Document historic behavior about hiding directories and special files
@ 2020-03-06 22:50 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 22:50 UTC (permalink / raw)
Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
doc/src/sgml/func.sgml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
(mtime) of each file in the log directory. By default, only superusers
and members of the <literal>pg_monitor</literal> role can use this function.
Access may be granted to others using <command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal> role
can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
superusers and members of the <literal>pg_monitor</literal> role can
use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--oTHb8nViIGeoXxdp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0002-Document-historic-behavior-about-hiding-director.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v10 1/9] Document historic behavior about hiding directories and special files
@ 2020-03-06 22:50 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 22:50 UTC (permalink / raw)
Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
doc/src/sgml/func.sgml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
(mtime) of each file in the log directory. By default, only superusers
and members of the <literal>pg_monitor</literal> role can use this function.
Access may be granted to others using <command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal> role
can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
superusers and members of the <literal>pg_monitor</literal> role can
use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--rCwQ2Y43eQY6RBgR
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0002-Document-historic-behavior-about-hiding-director.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v9 02/11] Document historic behavior about hiding directories and special files
@ 2020-03-06 22:50 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 22:50 UTC (permalink / raw)
Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
doc/src/sgml/func.sgml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
(mtime) of each file in the log directory. By default, only superusers
and members of the <literal>pg_monitor</literal> role can use this function.
Access may be granted to others using <command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal> role
can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
superusers and members of the <literal>pg_monitor</literal> role can
use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--32u276st3Jlj2kUU
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0003-Document-historic-behavior-about-hiding-directori.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v8 2/6] Document historic behavior about hiding directories and special files
@ 2020-03-06 22:50 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 22:50 UTC (permalink / raw)
Should backpatch to v10: tmpdir, waldir and archive_statusdir
---
doc/src/sgml/func.sgml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 323366feb6..4c0ea5ab3f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
(mtime) of each file in the log directory. By default, only superusers
and members of the <literal>pg_monitor</literal> role can use this function.
Access may be granted to others using <command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal> role
can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
@@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
superusers and members of the <literal>pg_monitor</literal> role can
use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--k1lZvvs/B4yU6o8G
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0003-Document-historic-behavior-about-hiding-directori.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v7 3/6] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--jKBxcB1XkHIR0Eqt
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0004-pg_ls_tmpdir-to-show-directories.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v12 03/11] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index ace95fe661..2c6142a0e0 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--wIc/V6YLA2QdyfT4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0004-Add-tests-exercizing-pg_ls_tmpdir.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v11 2/9] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--oTHb8nViIGeoXxdp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0003-Add-tests-exercizing-pg_ls_tmpdir.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v10 2/9] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--rCwQ2Y43eQY6RBgR
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0003-Add-tests-exercizing-pg_ls_tmpdir.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v9 03/11] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--32u276st3Jlj2kUU
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0004-Add-pg_ls_dir_metadata-to-list-a-dir-with-file-me.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v8 3/6] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4c0ea5ab3f..fc4d7f0f78 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--k1lZvvs/B4yU6o8G
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0004-pg_ls_tmpdir-to-show-directories-recursively.patch"
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2020-04-30 18:58 Stephen Frost <[email protected]>
parent: Andrew Gierth <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Stephen Frost @ 2020-04-30 18:58 UTC (permalink / raw)
To: Andrew Gierth <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers; [email protected]; Oliver Ford <[email protected]>; Krasiyan Andreev <[email protected]>
Greetings,
This seems to have died out, and that's pretty unfortunate because this
is awfully useful SQL standard syntax that people look for and wish we
had.
* Andrew Gierth ([email protected]) wrote:
> So I've tried to rough out a decision tree for the various options on
> how this might be implemented (discarding the "use precedence hacks"
> option). Opinions? Additions?
>
> (formatted for emacs outline-mode)
>
> * 1. use lexical lookahead
>
> +: relatively straightforward parser changes
> +: no new reserved words
> +: has the option of working extensibly with all functions
>
> -: base_yylex needs extending to 3 lookahead tokens
This sounds awful grotty and challenging to do and get right, and the
alternative (just reserving these, as the spec does) doesn't seem so
draconian as to be that much of an issue.
> * 2. reserve nth_value etc. as functions
>
> +: follows the spec reasonably well
> +: less of a hack than extending base_yylex
>
> -: new reserved words
> -: more parser rules
> -: not extensible
>
For my 2c, at least, reserving these strikes me as entirely reasonable.
Yes, it sucks that we have to partially-reserve some additional
keywords, but such is life. I get that we'll throw syntax errors
sometimes when we might have given a better error, but I think we can
accept that.
> (now goto 1.2.1)
Hmm, not sure this was right? but sure, I'll try...
> *** 1.2.1. Check the function name in parse analysis against a fixed list.
>
> +: simple
> -: not extensible
Seems like this is more-or-less required since we'd be reserving them..?
> *** 1.2.2. Provide some option in CREATE FUNCTION
>
> +: extensible
> -: fairly intrusive, adding stuff to create function and pg_proc
How would this work though, if we reserve the functions as keywords..?
Maybe I'm not entirely following, but wouldn't attempts to use other
functions end up with syntax errors in at least some of the cases,
meaning that having other functions support this wouldn't really work?
I don't particularly like the idea that some built-in functions would
always work but others would work but only some of the time.
> *** 1.2.3. Do something magical with function argument types
>
> +: doesn't need changes in create function / pg_proc
> -: it's an ugly hack
Not really a fan of 'ugly hack'.
> * 3. "just say no" to the spec
>
> e.g. add new functions like lead_ignore_nulls(), or add extra boolean
> args to lead() etc. telling them to skip nulls
>
> +: simple
> -: doesn't conform to spec
> -: using extra args isn't quite the right semantics
Ugh, no thank you.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2020-04-30 19:50 Krasiyan Andreev <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Krasiyan Andreev @ 2020-04-30 19:50 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Andrew Gierth <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers; [email protected]; Oliver Ford <[email protected]>
Thank you very much for feedback and yes, that is very useful SQL syntax.
Maybe you miss my previous answer, but you are right, that patch is
currently dead,
because some important design questions must be discussed here, before
patch rewriting.
I have dropped support of from first/last for nth_value(),
but also I reimplemented it in a different way,
by using negative number for the position argument, to be able to get the
same frame in exact reverse order.
After that patch becomes much more simple and some concerns about
precedence hack has gone.
I have not renamed special bool type "ignorenulls"
(I know that it is not acceptable way for calling extra version
of window functions, but also it makes things very easy and it can reuse
frames),
but I removed the other special bool type "fromlast".
Attached file was for PostgreSQL 13 (master git branch, last commit fest),
everything was working and patch was at the time in very good shape, all
tests was passed.
I read previous review and suggestions from Tom about special bool type and
unreserved keywords and also,
that IGNORE NULLS could be implemented inside the WinGetFuncArgXXX
functions,
but I am not sure how exactly to proceed (some example will be very
helpful).
На чт, 30.04.2020 г. в 21:58 Stephen Frost <[email protected]> написа:
> Greetings,
>
> This seems to have died out, and that's pretty unfortunate because this
> is awfully useful SQL standard syntax that people look for and wish we
> had.
>
> * Andrew Gierth ([email protected]) wrote:
> > So I've tried to rough out a decision tree for the various options on
> > how this might be implemented (discarding the "use precedence hacks"
> > option). Opinions? Additions?
> >
> > (formatted for emacs outline-mode)
> >
> > * 1. use lexical lookahead
> >
> > +: relatively straightforward parser changes
> > +: no new reserved words
> > +: has the option of working extensibly with all functions
> >
> > -: base_yylex needs extending to 3 lookahead tokens
>
> This sounds awful grotty and challenging to do and get right, and the
> alternative (just reserving these, as the spec does) doesn't seem so
> draconian as to be that much of an issue.
>
> > * 2. reserve nth_value etc. as functions
> >
> > +: follows the spec reasonably well
> > +: less of a hack than extending base_yylex
> >
> > -: new reserved words
> > -: more parser rules
> > -: not extensible
> >
>
> For my 2c, at least, reserving these strikes me as entirely reasonable.
> Yes, it sucks that we have to partially-reserve some additional
> keywords, but such is life. I get that we'll throw syntax errors
> sometimes when we might have given a better error, but I think we can
> accept that.
>
> > (now goto 1.2.1)
>
> Hmm, not sure this was right? but sure, I'll try...
>
> > *** 1.2.1. Check the function name in parse analysis against a fixed
> list.
> >
> > +: simple
> > -: not extensible
>
> Seems like this is more-or-less required since we'd be reserving them..?
>
> > *** 1.2.2. Provide some option in CREATE FUNCTION
> >
> > +: extensible
> > -: fairly intrusive, adding stuff to create function and pg_proc
>
> How would this work though, if we reserve the functions as keywords..?
> Maybe I'm not entirely following, but wouldn't attempts to use other
> functions end up with syntax errors in at least some of the cases,
> meaning that having other functions support this wouldn't really work?
> I don't particularly like the idea that some built-in functions would
> always work but others would work but only some of the time.
>
> > *** 1.2.3. Do something magical with function argument types
> >
> > +: doesn't need changes in create function / pg_proc
> > -: it's an ugly hack
>
> Not really a fan of 'ugly hack'.
>
> > * 3. "just say no" to the spec
> >
> > e.g. add new functions like lead_ignore_nulls(), or add extra boolean
> > args to lead() etc. telling them to skip nulls
> >
> > +: simple
> > -: doesn't conform to spec
> > -: using extra args isn't quite the right semantics
>
> Ugh, no thank you.
>
> Thanks!
>
> Stephen
>
Attachments:
[text/x-patch] pg13_ignore_nulls.patch (51.3K, ../../CAN1PwonAnC-KkRyY+DtRmxQ8rjdJw+gcOsHruLr6EnF7zSMH=Q@mail.gmail.com/3-pg13_ignore_nulls.patch)
download | inline diff:
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 28035f1635..3d73c96891 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -15702,7 +15702,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<function>
lag(<replaceable class="parameter">value</replaceable> <type>anyelement</type>
[, <replaceable class="parameter">offset</replaceable> <type>integer</type>
- [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]])
+ [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]]) [null_treatment]
</function>
</entry>
<entry>
@@ -15731,7 +15731,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<function>
lead(<replaceable class="parameter">value</replaceable> <type>anyelement</type>
[, <replaceable class="parameter">offset</replaceable> <type>integer</type>
- [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]])
+ [, <replaceable class="parameter">default</replaceable> <type>anyelement</type> ]]) [null_treatment]
</function>
</entry>
<entry>
@@ -15757,7 +15757,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value(<replaceable class="parameter">value</replaceable> <type>any</type>)</function>
+ <function>first_value(<replaceable class="parameter">value</replaceable> <type>any</type>) [null_treatment]</function>
</entry>
<entry>
<type>same type as <replaceable class="parameter">value</replaceable></type>
@@ -15773,7 +15773,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value(<replaceable class="parameter">value</replaceable> <type>any</type>)</function>
+ <function>last_value(<replaceable class="parameter">value</replaceable> <type>any</type>) [null_treatment]</function>
</entry>
<entry>
<type>same type as <replaceable class="parameter">value</replaceable></type>
@@ -15790,7 +15790,7 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<primary>nth_value</primary>
</indexterm>
<function>
- nth_value(<replaceable class="parameter">value</replaceable> <type>any</type>, <replaceable class="parameter">nth</replaceable> <type>integer</type>)
+ nth_value(<replaceable class="parameter">value</replaceable> <type>any</type>, <replaceable class="parameter">nth</replaceable> <type>integer</type>) [null_treatment]
</function>
</entry>
<entry>
@@ -15806,6 +15806,16 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
</tgroup>
</table>
+ <para>
+ In <xref linkend="functions-window-table"/>, <replaceable>null_treatment</replaceable> is one of:
+ <synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+ </synopsis>
+ <literal>RESPECT NULLS</literal> specifies the default behavior to include nulls in the result.
+ <literal>IGNORE NULLS</literal> ignores any null values when determining a result.
+ </para>
+
<para>
All of the functions listed in
<xref linkend="functions-window-table"/> depend on the sort ordering
@@ -15843,17 +15853,11 @@ SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
- ordering.)
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in <productname>PostgreSQL</productname>:
+ only the default <literal>FROM FIRST</literal> behavior is supported.
+ (You can achieve the result of <literal>FROM LAST</literal> by using negative number for the position argument,
+ as is done in many languages to indicate a <literal>FROM END</literal> index.)
</para>
</note>
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index 9f840ddfd2..a355e379e7 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -508,9 +508,9 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE function YES
-T618 NTH_VALUE function NO function exists, but some options missing
+T618 NTH_VALUE function YES FROM LAST is supported by using negative number for the position argument
T619 Nested window functions NO
T620 WINDOW clause: GROUPS option YES
T621 Enhanced numeric functions YES
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 96e7fdbcfe..db369b1f9a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -154,6 +154,7 @@ static Node *makeBitStringConst(char *str, int location);
static Node *makeNullAConst(int location);
static Node *makeAConst(Value *v, int location);
static Node *makeBoolAConst(bool state, int location);
+static Node *makeTypedBoolAConst(bool state, char *type, int location);
static RoleSpec *makeRoleSpec(RoleSpecType type, int location);
static void check_qualified_name(List *names, core_yyscan_t yyscanner);
static List *check_func_name(List *names, core_yyscan_t yyscanner);
@@ -569,7 +570,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> xml_namespace_list
%type <target> xml_namespace_el
-%type <node> func_application func_expr_common_subexpr
+%type <node> func_application func_expr_common_subexpr func_expr_respect_ignore
%type <node> func_expr func_expr_windowless
%type <node> common_table_expr
%type <with> with_clause opt_with_clause
@@ -577,6 +578,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> within_group_clause
%type <node> filter_clause
+%type <ival> null_treatment_clause
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
@@ -642,14 +644,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN EXPRESSION
EXTENSION EXTERNAL EXTRACT
- FALSE_P FAMILY FETCH FILTER FIRST_P FLOAT_P FOLLOWING FOR
+ FALSE_P FAMILY FETCH FILTER FIRST_P FIRST_VALUE FLOAT_P FOLLOWING FOR
FORCE FOREIGN FORWARD FREEZE FROM FULL FUNCTION FUNCTIONS
GENERATED GLOBAL GRANT GRANTED GREATEST GROUP_P GROUPING GROUPS
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -658,14 +660,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
KEY
- LABEL LANGUAGE LARGE_P LAST_P LATERAL_P
- LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
+ LABEL LAG LANGUAGE LARGE_P LAST_P LAST_VALUE LATERAL_P
+ LEAD LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED
MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE
NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NO NONE
- NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF
+ NOT NOTHING NOTIFY NOTNULL NOWAIT NTH_VALUE NULL_P NULLIF
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
@@ -680,7 +682,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT RESTART RESTRICT RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT SEQUENCE SEQUENCES
@@ -724,6 +726,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
/* Precedence: lowest to highest */
%nonassoc SET /* see relation_expr_opt_alias */
+%nonassoc FIRST_VALUE LAG LAST_VALUE LEAD NTH_VALUE
%left UNION EXCEPT
%left INTERSECT
%left OR
@@ -13769,6 +13772,10 @@ func_application: func_name '(' ')'
}
;
+null_treatment_clause:
+ RESPECT NULLS_P { $$ = 0; }
+ | IGNORE_P NULLS_P { $$ = WINFUNC_OPT_IGNORE_NULLS; }
+ ;
/*
* func_expr and its cousin func_expr_windowless are split out from c_expr just
@@ -13816,8 +13823,133 @@ func_expr: func_application within_group_clause filter_clause over_clause
}
| func_expr_common_subexpr
{ $$ = $1; }
+ | func_expr_respect_ignore over_clause
+ {
+ FuncCall *n = (FuncCall *) $1;
+ n->over = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+func_expr_respect_ignore:
+ FIRST_VALUE '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("first_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | FIRST_VALUE '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("first_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAG '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("lag")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAG '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("lag")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAST_VALUE '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("last_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LAST_VALUE '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("last_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LEAD '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Ignore Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("lead")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | LEAD '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("lead")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | NTH_VALUE '(' func_arg_list opt_sort_clause ')' null_treatment_clause
+ {
+ FuncCall *n;
+ List *l = $3;
+ int winFuncArgs = $6;
+
+ /* Convert Nulls option to bool */
+ if (winFuncArgs & WINFUNC_OPT_IGNORE_NULLS)
+ l = lappend(l, makeTypedBoolAConst(true, "ignorenulls", @2));
+
+ n = makeFuncCall(list_make1(makeString("nth_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
+ | NTH_VALUE '(' func_arg_list opt_sort_clause ')'
+ {
+ FuncCall *n;
+ List *l = $3;
+
+ n = makeFuncCall(list_make1(makeString("nth_value")), l, @1);
+ n->agg_order = $4;
+ $$ = (Node *) n;
+ }
;
+
/*
* As func_expr but does not accept WINDOW functions directly
* (but they can still be contained in arguments for functions etc).
@@ -15225,6 +15357,7 @@ unreserved_keyword:
| FAMILY
| FILTER
| FIRST_P
+ | FIRST_VALUE
| FOLLOWING
| FORCE
| FORWARD
@@ -15240,6 +15373,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -15260,9 +15394,12 @@ unreserved_keyword:
| ISOLATION
| KEY
| LABEL
+ | LAG
| LANGUAGE
| LARGE_P
| LAST_P
+ | LAST_VALUE
+ | LEAD
| LEAKPROOF
| LEVEL
| LISTEN
@@ -15290,6 +15427,7 @@ unreserved_keyword:
| NOTHING
| NOTIFY
| NOWAIT
+ | NTH_VALUE
| NULLS_P
| OBJECT_P
| OF
@@ -15341,6 +15479,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT
| RESTART
| RESTRICT
| RETURNS
@@ -15815,6 +15954,15 @@ makeAConst(Value *v, int location)
*/
static Node *
makeBoolAConst(bool state, int location)
+{
+ return makeTypedBoolAConst(state, "bool", location);
+}
+
+/* makeTypedBoolAConst()
+ * Create an A_Const string node from a boolean and store inside the specified type.
+ */
+static Node *
+makeTypedBoolAConst(bool state, char *type, int location)
{
A_Const *n = makeNode(A_Const);
@@ -15822,7 +15970,7 @@ makeBoolAConst(bool state, int location)
n->val.val.str = (state ? "t" : "f");
n->location = location;
- return makeTypeCast((Node *)n, SystemTypeName("bool"), -1);
+ return makeTypeCast((Node *)n, SystemTypeName(type), -1);
}
/* makeRoleSpec
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 158784474d..ae7e821ab2 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -9437,6 +9437,7 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
int nargs;
List *argnames;
ListCell *l;
+ bool ignorenulls = false;
if (list_length(wfunc->args) > FUNC_MAX_ARGS)
ereport(ERROR,
@@ -9463,7 +9464,32 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
if (wfunc->winstar)
appendStringInfoChar(buf, '*');
else
- get_rule_expr((Node *) wfunc->args, context, true);
+ {
+ ListCell *arglist;
+ Node *argnode = (Node *) wfunc->args;
+
+ get_rule_expr(argnode, context, true);
+
+ /* Determine if IGNORE NULLS should be appended */
+ foreach(arglist, (List *) argnode)
+ {
+ Node *arg = (Node *) lfirst(arglist);
+ if (nodeTag(arg) == T_Const)
+ {
+ Const *constnode = (Const *) arg;
+ if (constnode->consttype == IGNORENULLSOID)
+ {
+ /* parser does not save RESPECT NULLS arguments */
+ ignorenulls = true;
+ buf->len -= 2;
+ }
+ }
+ }
+ }
+
+ appendStringInfoChar(buf, ')');
+ if (ignorenulls)
+ appendStringInfoString(buf, " IGNORE NULLS");
if (wfunc->aggfilter != NULL)
{
@@ -9471,7 +9497,7 @@ get_windowfunc_expr(WindowFunc *wfunc, deparse_context *context)
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, " OVER ");
foreach(l, context->windowClause)
{
@@ -9649,6 +9675,10 @@ get_const_expr(Const *constval, deparse_context *context, int showtype)
appendStringInfoString(buf, "false");
break;
+ case IGNORENULLSOID:
+ showtype = -1;
+ break;
+
default:
simple_quote_literal(buf, extval);
break;
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index f0c8ae686d..bc639b1883 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -38,7 +38,12 @@ typedef struct
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
+static Datum leadlag_common_ignore_nulls(FunctionCallInfo fcinfo,
+ bool forward, bool withoffset, bool withdefault);
+static Datum
+window_nth_value_ignorenulls_common(FunctionCallInfo fcinfo, int32 nth);
+static Datum
+window_nth_value_respectnulls_common(FunctionCallInfo fcinfo, int32 nth);
/*
* utility routine for *_rank functions.
@@ -328,6 +333,79 @@ leadlag_common(FunctionCallInfo fcinfo,
PG_RETURN_DATUM(result);
}
+static Datum
+leadlag_common_ignore_nulls(FunctionCallInfo fcinfo,
+ bool forward, bool withoffset, bool withdefault)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ int32 offset;
+ Datum result;
+ bool isnull;
+ bool isout = false;
+ int32 notnull_offset = 0, tmp_offset = 0;
+
+ if (withoffset)
+ {
+ offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ if (offset < 0)
+ {
+ offset = abs(offset);
+ forward = !forward;
+ } else if (offset == 0)
+ {
+ result = WinGetFuncArgInPartition(winobj, 0, 0,
+ WINDOW_SEEK_CURRENT,
+ false,
+ &isnull, &isout);
+ if (isnull || isout)
+ PG_RETURN_NULL();
+ else
+ PG_RETURN_DATUM(result);
+ }
+ }
+ else
+ offset = 1;
+
+ while (notnull_offset < offset)
+ {
+ tmp_offset++;
+ result = WinGetFuncArgInPartition(winobj, 0,
+ (forward ? tmp_offset : -tmp_offset),
+ WINDOW_SEEK_CURRENT,
+ false,
+ &isnull, &isout);
+ if (isout)
+ goto out_of_frame;
+ else if (!isnull)
+ notnull_offset++;
+ }
+
+ result = WinGetFuncArgInPartition(winobj, 0,
+ (forward ? tmp_offset : -tmp_offset),
+ WINDOW_SEEK_CURRENT,
+ false,
+ &isnull, &isout);
+ if (isout)
+ goto out_of_frame;
+ else
+ PG_RETURN_DATUM(result);
+
+ out_of_frame:
+ /*
+ * target row is out of the partition; supply default value if
+ * provided. Otherwise return NULL.
+ */
+ if (withdefault)
+ {
+ result = WinGetFuncArgCurrent(winobj, 2, &isnull);
+ PG_RETURN_DATUM(result);
+ }
+ else
+ PG_RETURN_NULL();
+}
+
/*
* lag
* returns the value of VE evaluated on a row that is 1
@@ -363,6 +441,24 @@ window_lag_with_offset_and_default(PG_FUNCTION_ARGS)
return leadlag_common(fcinfo, false, true, true);
}
+Datum
+window_lag_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, false, false, false);
+}
+
+Datum
+window_lag_with_offset_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, false, true, false);
+}
+
+Datum
+window_lag_with_offset_and_default_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, false, true, true);
+}
+
/*
* lead
* returns the value of VE evaluated on a row that is 1
@@ -398,6 +494,24 @@ window_lead_with_offset_and_default(PG_FUNCTION_ARGS)
return leadlag_common(fcinfo, true, true, true);
}
+Datum
+window_lead_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, true, false, false);
+}
+
+Datum
+window_lead_with_offset_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, true, true, false);
+}
+
+Datum
+window_lead_with_offset_and_default_nulls_opt(PG_FUNCTION_ARGS)
+{
+ return leadlag_common_ignore_nulls(fcinfo, true, true, true);
+}
+
/*
* first_value
* return the value of VE evaluated on the first row of the
@@ -419,6 +533,31 @@ window_first_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+Datum
+window_first_value_nulls_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ Datum result;
+ bool isnull,
+ isout;
+ int64 pos;
+
+ isout = false;
+ pos = 0;
+
+ while (!isout)
+ {
+ result = WinGetFuncArgInFrame(winobj, 0,
+ pos, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (!isnull)
+ PG_RETURN_DATUM(result);
+ pos++;
+ }
+
+ PG_RETURN_NULL();
+}
+
/*
* last_value
* return the value of VE evaluated on the last row of the
@@ -440,35 +579,149 @@ window_last_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+Datum
+window_last_value_nulls_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ Datum result;
+ bool isnull,
+ isout;
+ int64 pos;
+
+ isout = false;
+ pos = 0;
+
+ while (!isout)
+ {
+ result = WinGetFuncArgInFrame(winobj, 0,
+ pos, WINDOW_SEEK_TAIL, false,
+ &isnull, &isout);
+ if (!isnull)
+ PG_RETURN_DATUM(result);
+ pos--;
+ }
+
+ PG_RETURN_NULL();
+}
+
/*
* nth_value
* return the value of VE evaluated on the n-th row from the first
* row of the window frame, per spec.
*/
-Datum
-window_nth_value(PG_FUNCTION_ARGS)
+static Datum
+window_nth_value_respectnulls_common(FunctionCallInfo fcinfo, int32 nth)
{
WindowObject winobj = PG_WINDOW_OBJECT();
bool const_offset;
Datum result;
bool isnull;
- int32 nth;
+ bool fromlast;
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
const_offset = get_fn_expr_arg_stable(fcinfo->flinfo, 1);
- if (nth <= 0)
+ if (nth == 0)
ereport(ERROR,
- (errcode(ERRCODE_INVALID_ARGUMENT_FOR_NTH_VALUE),
- errmsg("argument of nth_value must be greater than zero")));
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_NTH_VALUE),
+ errmsg("argument of nth_value must be greater or less than zero")));
+ else if (nth < 0)
+ {
+ nth = abs(nth);
+ fromlast = true;
+ }
+ else
+ fromlast = false;
+
+ result = WinGetFuncArgInFrame(winobj,
+ 0,
+ nth - 1,
+ fromlast ? WINDOW_SEEK_TAIL : WINDOW_SEEK_HEAD, const_offset,
+ &isnull,
+ NULL);
- result = WinGetFuncArgInFrame(winobj, 0,
- nth - 1, WINDOW_SEEK_HEAD, const_offset,
- &isnull, NULL);
if (isnull)
PG_RETURN_NULL();
PG_RETURN_DATUM(result);
}
+
+static Datum
+window_nth_value_ignorenulls_common(FunctionCallInfo fcinfo, int32 nth)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ Datum result;
+ bool isnull,
+ isout;
+ bool fromlast;
+ int32 tmp_offset, notnull_offset = 0;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+
+ if (nth == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_NTH_VALUE),
+ errmsg("argument of nth_value must be greater or less than zero")));
+ else if (nth < 0)
+ {
+ nth = abs(nth);
+ fromlast = true;
+ tmp_offset = 1;
+ }
+ else
+ {
+ fromlast = false;
+ tmp_offset = -1;
+ }
+
+ while (notnull_offset < nth)
+ {
+ fromlast ? tmp_offset-- : tmp_offset++;
+ result = WinGetFuncArgInFrame(winobj, 0,
+ tmp_offset, fromlast ? WINDOW_SEEK_TAIL : WINDOW_SEEK_HEAD,
+ false, &isnull, &isout);
+ if (isout)
+ PG_RETURN_NULL();
+ if (!isnull)
+ notnull_offset++;
+ }
+
+ result = WinGetFuncArgInFrame(winobj, 0,
+ tmp_offset, fromlast ? WINDOW_SEEK_TAIL : WINDOW_SEEK_HEAD,
+ false, &isnull, &isout);
+
+ if (isout || isnull)
+ PG_RETURN_NULL();
+
+ PG_RETURN_DATUM(result);
+}
+
+Datum
+window_nth_value(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ bool isnull;
+ int32 nth;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ PG_RETURN_DATUM(window_nth_value_respectnulls_common(fcinfo, nth));
+}
+
+Datum
+window_nth_value_with_nulls_opt(PG_FUNCTION_ARGS)
+{
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ bool isnull;
+ int32 nth;
+
+ nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
+ if (isnull)
+ PG_RETURN_NULL();
+ PG_RETURN_DATUM(window_nth_value_ignorenulls_common(fcinfo, nth));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 07a86c7b7b..f07a9e442d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9539,32 +9539,64 @@
{ oid => '3106', descr => 'fetch the preceding row value',
proname => 'lag', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_lag' },
+{ oid => '4191', descr => 'fetch the preceding row value with nulls option',
+ proname => 'lag', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_lag_nulls_opt' },
{ oid => '3107', descr => 'fetch the Nth preceding row value',
proname => 'lag', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_lag_with_offset' },
+{ oid => '4192', descr => 'fetch the Nth preceding row value with nulls option',
+ proname => 'lag', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 ignorenulls',
+ prosrc => 'window_lag_with_offset_nulls_opt' },
{ oid => '3108', descr => 'fetch the Nth preceding row value with default',
proname => 'lag', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4 anyelement',
prosrc => 'window_lag_with_offset_and_default' },
+{ oid => '4193', descr => 'fetch the Nth preceding row value with default and nulls option',
+ proname => 'lag', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 anyelement ignorenulls',
+ prosrc => 'window_lag_with_offset_and_default_nulls_opt' },
{ oid => '3109', descr => 'fetch the following row value',
proname => 'lead', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_lead' },
+{ oid => '4194', descr => 'fetch the following row value with nulls option',
+ proname => 'lead', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_lead_nulls_opt' },
{ oid => '3110', descr => 'fetch the Nth following row value',
proname => 'lead', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_lead_with_offset' },
+{ oid => '4195', descr => 'fetch the Nth following row value with nulls option',
+ proname => 'lead', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 ignorenulls',
+ prosrc => 'window_lead_with_offset_nulls_opt' },
{ oid => '3111', descr => 'fetch the Nth following row value with default',
proname => 'lead', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4 anyelement',
prosrc => 'window_lead_with_offset_and_default' },
+{ oid => '4196', descr => 'fetch the Nth following row value with default and nulls option',
+ proname => 'lead', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 anyelement ignorenulls',
+ prosrc => 'window_lead_with_offset_and_default_nulls_opt' },
{ oid => '3112', descr => 'fetch the first row value',
proname => 'first_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_first_value' },
+{ oid => '4197', descr => 'fetch the first row value with nulls option',
+ proname => 'first_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_first_value_nulls_opt' },
{ oid => '3113', descr => 'fetch the last row value',
proname => 'last_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement', prosrc => 'window_last_value' },
+{ oid => '4198', descr => 'fetch the last row value with nulls option',
+ proname => 'last_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement ignorenulls', prosrc => 'window_last_value_nulls_opt' },
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '4199', descr => 'fetch the Nth row value with nulls option',
+ proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement int4 ignorenulls',
+ prosrc => 'window_nth_value_with_nulls_opt' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 4cf2b9df7b..3c847e6db5 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -594,5 +594,10 @@
typname => 'anyrange', typlen => '-1', typbyval => 'f', typtype => 'p',
typcategory => 'P', typinput => 'anyrange_in', typoutput => 'anyrange_out',
typreceive => '-', typsend => '-', typalign => 'd', typstorage => 'x' },
+{ oid => '4142',
+ descr => 'boolean wrapper, \'true\'/\'false\'',
+ typname => 'ignorenulls', typlen => '1', typbyval => 't', typtype => 'b',
+ typcategory => 'B', typinput => 'boolin', typoutput => 'boolout',
+ typreceive => 'boolrecv', typsend => 'boolsend', typalign => 'c' },
]
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index da0706add5..0899bddd6e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -532,6 +532,11 @@ typedef struct WindowDef
(FRAMEOPTION_RANGE | FRAMEOPTION_START_UNBOUNDED_PRECEDING | \
FRAMEOPTION_END_CURRENT_ROW)
+/*
+ * Null Treatment option
+ */
+#define WINFUNC_OPT_IGNORE_NULLS 0x00001 /* IGNORE NULLS */
+
/*
* RangeSubselect - subquery appearing in a FROM clause
*/
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index b1184c2d15..a33ce1d0fd 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -164,6 +164,7 @@ PG_KEYWORD("family", FAMILY, UNRESERVED_KEYWORD)
PG_KEYWORD("fetch", FETCH, RESERVED_KEYWORD)
PG_KEYWORD("filter", FILTER, UNRESERVED_KEYWORD)
PG_KEYWORD("first", FIRST_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("first_value", FIRST_VALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("float", FLOAT_P, COL_NAME_KEYWORD)
PG_KEYWORD("following", FOLLOWING, UNRESERVED_KEYWORD)
PG_KEYWORD("for", FOR, RESERVED_KEYWORD)
@@ -190,6 +191,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD)
@@ -223,10 +225,13 @@ PG_KEYWORD("isolation", ISOLATION, UNRESERVED_KEYWORD)
PG_KEYWORD("join", JOIN, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("key", KEY, UNRESERVED_KEYWORD)
PG_KEYWORD("label", LABEL, UNRESERVED_KEYWORD)
+PG_KEYWORD("lag", LAG, UNRESERVED_KEYWORD)
PG_KEYWORD("language", LANGUAGE, UNRESERVED_KEYWORD)
PG_KEYWORD("large", LARGE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("last", LAST_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("last_value", LAST_VALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("lateral", LATERAL_P, RESERVED_KEYWORD)
+PG_KEYWORD("lead", LEAD, UNRESERVED_KEYWORD)
PG_KEYWORD("leading", LEADING, RESERVED_KEYWORD)
PG_KEYWORD("leakproof", LEAKPROOF, UNRESERVED_KEYWORD)
PG_KEYWORD("least", LEAST, COL_NAME_KEYWORD)
@@ -267,6 +272,7 @@ PG_KEYWORD("nothing", NOTHING, UNRESERVED_KEYWORD)
PG_KEYWORD("notify", NOTIFY, UNRESERVED_KEYWORD)
PG_KEYWORD("notnull", NOTNULL, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("nowait", NOWAIT, UNRESERVED_KEYWORD)
+PG_KEYWORD("nth_value", NTH_VALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("null", NULL_P, RESERVED_KEYWORD)
PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD)
PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD)
@@ -336,6 +342,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD)
+PG_KEYWORD("respect", RESPECT, UNRESERVED_KEYWORD)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD)
PG_KEYWORD("returning", RETURNING, RESERVED_KEYWORD)
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index cd7fc03b04..0ac49263ec 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -73,7 +73,8 @@ WHERE p1.typtype not in ('c','d','p') AND p1.typname NOT LIKE E'\\_%'
3361 | pg_ndistinct
3402 | pg_dependencies
5017 | pg_mcv_list
-(4 rows)
+ 4142 | ignorenulls
+(5 rows)
-- Make sure typarray points to a varlena array type of our own base
SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype,
@@ -166,10 +167,11 @@ WHERE p1.typinput = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p1.typelem != 0 AND p1.typlen < 0) AND NOT
(p2.prorettype = p1.oid AND NOT p2.proretset)
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+-----+---------
- 1790 | refcursor | 46 | textin
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+---------
+ 1790 | refcursor | 46 | textin
+ 4142 | ignorenulls | 1242 | boolin
+(2 rows)
-- Varlena array types will point to array_in
-- Exception as of 8.1: int2vector and oidvector have their own I/O routines
@@ -217,10 +219,11 @@ WHERE p1.typoutput = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p2.oid = 'array_out'::regproc AND
p1.typelem != 0 AND p1.typlen = -1)))
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+-----+---------
- 1790 | refcursor | 47 | textout
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+---------
+ 1790 | refcursor | 47 | textout
+ 4142 | ignorenulls | 1243 | boolout
+(2 rows)
SELECT p1.oid, p1.typname, p2.oid, p2.proname
FROM pg_type AS p1, pg_proc AS p2
@@ -280,10 +283,11 @@ WHERE p1.typreceive = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p1.typelem != 0 AND p1.typlen < 0) AND NOT
(p2.prorettype = p1.oid AND NOT p2.proretset)
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+------+----------
- 1790 | refcursor | 2414 | textrecv
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+----------
+ 1790 | refcursor | 2414 | textrecv
+ 4142 | ignorenulls | 2436 | boolrecv
+(2 rows)
-- Varlena array types will point to array_recv
-- Exception as of 8.1: int2vector and oidvector have their own I/O routines
@@ -340,10 +344,11 @@ WHERE p1.typsend = p2.oid AND p1.typtype in ('b', 'p') AND NOT
(p2.oid = 'array_send'::regproc AND
p1.typelem != 0 AND p1.typlen = -1)))
ORDER BY 1;
- oid | typname | oid | proname
-------+-----------+------+----------
- 1790 | refcursor | 2415 | textsend
-(1 row)
+ oid | typname | oid | proname
+------+-------------+------+----------
+ 1790 | refcursor | 2415 | textsend
+ 4142 | ignorenulls | 2437 | boolsend
+(2 rows)
SELECT p1.oid, p1.typname, p2.oid, p2.proname
FROM pg_type AS p1, pg_proc AS p2
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index d5fd4045f9..54ded65906 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -2985,7 +2985,7 @@ LINE 1: SELECT generate_series(1, 100) OVER () FROM empsalary;
SELECT ntile(0) OVER (ORDER BY ten), ten, four FROM tenk1;
ERROR: argument of ntile must be greater than zero
SELECT nth_value(four, 0) OVER (ORDER BY ten), ten, four FROM tenk1;
-ERROR: argument of nth_value must be greater than zero
+ERROR: argument of nth_value must be greater or less than zero
-- filter
SELECT sum(salary), row_number() OVER (ORDER BY depname), sum(
sum(salary) FILTER (WHERE enroll_date > '2007-01-01')
@@ -3863,3 +3863,369 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- RESPECT NULLS and IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ orbit int
+);
+INSERT INTO planets VALUES
+ ('mercury', 88),
+ ('venus', 224),
+ ('earth', NULL),
+ ('mars', NULL),
+ ('jupiter', 4332),
+ ('saturn', 24491),
+ ('uranus', NULL),
+ ('neptune', 60182),
+ ('pluto', 90560);
+ -- test view definitions are preserved
+CREATE TEMP VIEW v_planets AS
+ SELECT
+ name,
+ sum(orbit) OVER (order by orbit) as sum_rows,
+ lag(orbit, 1) RESPECT NULLS OVER (ORDER BY name DESC) AS lagged_by_1,
+ lag(orbit, 2) IGNORE NULLS OVER w AS lagged_by_2,
+ first_value(orbit) IGNORE NULLS OVER w AS first_value_ignore,
+ nth_value(orbit,2) IGNORE NULLS OVER w AS nth_first_ignore,
+ nth_value(orbit,-2) IGNORE NULLS OVER w AS nth_last_ignore
+ FROM planets
+ WINDOW w as (ORDER BY name ASC);
+SELECT pg_get_viewdef('v_planets');
+ pg_get_viewdef
+------------------------------------------------------------------------------------
+ SELECT planets.name, +
+ sum(planets.orbit) OVER (ORDER BY planets.orbit) AS sum_rows, +
+ lag(planets.orbit, 1) OVER (ORDER BY planets.name DESC) AS lagged_by_1, +
+ lag(planets.orbit, 2) IGNORE NULLS OVER w AS lagged_by_2, +
+ first_value(planets.orbit) IGNORE NULLS OVER w AS first_value_ignore, +
+ nth_value(planets.orbit, 2) IGNORE NULLS OVER w AS nth_first_ignore, +
+ nth_value(planets.orbit, '-2'::integer) IGNORE NULLS OVER w AS nth_last_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY planets.name);
+(1 row)
+
+SELECT name, lag(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury |
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus |
+(9 rows)
+
+SELECT name, lag(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury |
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus |
+(9 rows)
+
+SELECT name, lag(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury | 4332
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus | 24491
+(9 rows)
+
+SELECT name, lead(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth | 4332
+ jupiter |
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn |
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lead(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth | 4332
+ jupiter |
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn |
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lead(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth | 4332
+ jupiter | 88
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn | 224
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lag(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lag
+---------+-------
+ earth | 4332
+ jupiter | 88
+ mars | 88
+ mercury | 60182
+ neptune | 90560
+ pluto | 24491
+ saturn | 224
+ uranus | 224
+ venus |
+(9 rows)
+
+SELECT name, lead(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | lead
+---------+-------
+ earth |
+ jupiter |
+ mars | 4332
+ mercury | 4332
+ neptune | 88
+ pluto | 60182
+ saturn | 90560
+ uranus | 24491
+ venus | 24491
+(9 rows)
+
+SELECT name, first_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | first_value
+---------+-------------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+SELECT name, first_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | first_value
+---------+-------------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+SELECT name, first_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | first_value
+---------+-------------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, last_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | last_value
+---------+------------
+ earth | 224
+ jupiter | 224
+ mars | 224
+ mercury | 224
+ neptune | 224
+ pluto | 224
+ saturn | 224
+ uranus | 224
+ venus | 224
+(9 rows)
+
+SELECT name, last_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | last_value
+---------+------------
+ earth | 224
+ jupiter | 224
+ mars | 224
+ mercury | 224
+ neptune | 224
+ pluto | 224
+ saturn | 224
+ uranus | 224
+ venus | 224
+(9 rows)
+
+SELECT name, last_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | last_value
+---------+------------
+ earth | 224
+ jupiter | 224
+ mars | 224
+ mercury | 224
+ neptune | 224
+ pluto | 224
+ saturn | 224
+ uranus | 224
+ venus | 224
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 88
+ jupiter | 88
+ mars | 88
+ mercury | 88
+ neptune | 88
+ pluto | 88
+ saturn | 88
+ uranus | 88
+ venus | 88
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 88
+ jupiter | 88
+ mars | 88
+ mercury | 88
+ neptune | 88
+ pluto | 88
+ saturn | 88
+ uranus | 88
+ venus | 88
+(9 rows)
+
+SELECT name, nth_value(orbit, 2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 4332
+ jupiter | 4332
+ mars | 4332
+ mercury | 4332
+ neptune | 4332
+ pluto | 4332
+ saturn | 4332
+ uranus | 4332
+ venus | 4332
+(9 rows)
+
+SELECT name, nth_value(orbit, -2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+SELECT name, nth_value(orbit, -2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth | 24491
+ jupiter | 24491
+ mars | 24491
+ mercury | 24491
+ neptune | 24491
+ pluto | 24491
+ saturn | 24491
+ uranus | 24491
+ venus | 24491
+(9 rows)
+
+SELECT name, nth_value(orbit, -2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+ name | nth_value
+---------+-----------
+ earth |
+ jupiter |
+ mars |
+ mercury |
+ neptune |
+ pluto |
+ saturn |
+ uranus |
+ venus |
+(9 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view v_planets
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index fe273aa31e..548902d482 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1276,3 +1276,67 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- RESPECT NULLS and IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ orbit int
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 88),
+ ('venus', 224),
+ ('earth', NULL),
+ ('mars', NULL),
+ ('jupiter', 4332),
+ ('saturn', 24491),
+ ('uranus', NULL),
+ ('neptune', 60182),
+ ('pluto', 90560);
+
+ -- test view definitions are preserved
+CREATE TEMP VIEW v_planets AS
+ SELECT
+ name,
+ sum(orbit) OVER (order by orbit) as sum_rows,
+ lag(orbit, 1) RESPECT NULLS OVER (ORDER BY name DESC) AS lagged_by_1,
+ lag(orbit, 2) IGNORE NULLS OVER w AS lagged_by_2,
+ first_value(orbit) IGNORE NULLS OVER w AS first_value_ignore,
+ nth_value(orbit,2) IGNORE NULLS OVER w AS nth_first_ignore,
+ nth_value(orbit,-2) IGNORE NULLS OVER w AS nth_last_ignore
+ FROM planets
+ WINDOW w as (ORDER BY name ASC);
+SELECT pg_get_viewdef('v_planets');
+
+SELECT name, lag(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lag(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lag(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, lead(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lead(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lead(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, lag(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, lead(orbit, -1) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, first_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, first_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, first_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, last_value(orbit) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, last_value(orbit) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, last_value(orbit) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, nth_value(orbit, 2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+SELECT name, nth_value(orbit, 2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, 2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, -2) OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, -2) IGNORE NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+SELECT name, nth_value(orbit, -2) RESPECT NULLS OVER (ORDER BY name range between unbounded preceding and unbounded following) FROM planets ORDER BY name;
+
+--cleanup
+DROP TABLE planets CASCADE;
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-03-09 06:39 Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-03-09 06:39 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
> Attached version removes the non-nulls array. That seems to speed
> everything up. Running the above query with 1 million rows averages 450ms,
> similar when using lead/lag.
Great. However, CFbot complains about the patch:
https://cirrus-ci.com/task/6364194477441024
Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-03-09 20:07 Oliver Ford <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Oliver Ford @ 2025-03-09 20:07 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Sun, Mar 9, 2025 at 6:40 AM Tatsuo Ishii <[email protected]> wrote:
> > Attached version removes the non-nulls array. That seems to speed
> > everything up. Running the above query with 1 million rows averages
> 450ms,
> > similar when using lead/lag.
>
> Great. However, CFbot complains about the patch:
>
> https://cirrus-ci.com/task/6364194477441024
>
>
Attached fixes the headerscheck locally.
Attachments:
[application/octet-stream] 0010-ignore-nulls.patch (47.5K, ../../CAGMVOdvAfKzfjDoA747fGNxYKdXG3rBiqRnwaz=FTcqW1TyOOA@mail.gmail.com/3-0010-ignore-nulls.patch)
download | inline diff:
From cf678af9b4ebf4f31af5c3e9b0632b2c777a9ad1 Mon Sep 17 00:00:00 2001
From: Oliver Ford <[email protected]>
Date: Sun, 9 Mar 2025 16:39:26 +0000
Subject: [PATCH] ignore nulls
---
doc/src/sgml/func.sgml | 38 ++--
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 225 ++++++++++++++++++-
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 ++
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 +++++++++++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++++++
15 files changed, 773 insertions(+), 28 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 51dd8ad6571..d8cf8b039d5 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23442,7 +23442,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23467,7 +23467,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23490,7 +23490,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23504,7 +23504,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23518,7 +23518,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23567,18 +23567,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index 2f250d2c57b..46a8959cb2f 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..149333bb687 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,7 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +97,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -198,6 +200,13 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginpartition(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool *isnull, bool *isout);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
/*
* initialize_windowaggregate
@@ -2619,14 +2628,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2691,7 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3227,212 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * ignorenulls_getfuncarginpartition
+ * For IGNORE NULLS, get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+static Datum
+ignorenulls_getfuncarginpartition(WindowObject winobj, int argno,
+ int relpos, int seektype, bool *isnull,
+ bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ bool gottuple;
+ int64 abs_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ abs_pos = winstate->currentpos;
+ break;
+ case WINDOW_SEEK_HEAD:
+ abs_pos = 0;
+ break;
+ case WINDOW_SEEK_TAIL:
+ spool_tuples(winstate, -1);
+ abs_pos = winstate->spooled_rows - 1 + relpos;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ abs_pos += forward;
+ gottuple = window_gettupleslot(winobj, abs_pos, slot);
+
+ if (!gottuple)
+ {
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+
+ if (!*isnull)
+ ++notnull_offset;
+ } while (notnull_offset < notnull_relpos);
+
+ return datum;
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ int inframe;
+
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ inframe = row_is_in_frame(winstate, abs_pos, slot);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+
+ if (!*isnull)
+ ++notnull_offset;
+
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3388,6 +3601,10 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ return ignorenulls_getfuncarginpartition(winobj, argno, relpos, seektype,
+ isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3476,6 +3693,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 43dfecfb47f..e7091d7468c 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2570,6 +2570,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 271ae26cbaf..19b1e61d10b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -632,7 +632,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -730,7 +730,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -765,7 +765,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15758,7 +15758,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15791,7 +15791,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16387,6 +16388,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17824,6 +17831,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17941,6 +17949,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index d11a8a20eea..63e9a347fed 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11028,7 +11028,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
foreach(l, context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 23c9e3c5abf..b13a7bfe674 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -439,6 +439,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index d0576da3e25..65081616402 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -577,6 +577,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -600,6 +611,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 40cf090ce61..3ba00a39e5d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -377,6 +378,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index 23d1463df22..f596dfff6ef 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5403,3 +5403,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-03-13 07:49 Oliver Ford <[email protected]>
parent: Oliver Ford <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Oliver Ford @ 2025-03-13 07:49 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: Krasiyan Andreev <[email protected]>; Tom Lane <[email protected]>; Vik Fearing <[email protected]>; pgsql-hackers; [email protected]; David Fetter <[email protected]>
On Sun, 9 Mar 2025, 20:07 Oliver Ford, <[email protected]> wrote:
> On Sun, Mar 9, 2025 at 6:40 AM Tatsuo Ishii <[email protected]> wrote:
>
>> > Attached version removes the non-nulls array. That seems to speed
>> > everything up. Running the above query with 1 million rows averages
>> 450ms,
>> > similar when using lead/lag.
>>
>> Great. However, CFbot complains about the patch:
>>
>> https://cirrus-ci.com/task/6364194477441024
>>
>>
> Attached fixes the headerscheck locally.
>
v11 attached because the previous version was broken by commit 8b1b342
>
Attachments:
[text/x-diff] 0011-ignore-nulls.patch (47.5K, ../../CAGMVOduuhBa5b9ZzsJ2EwQZKFJhtjx3j4xwmbvfNo8Oks63mZw@mail.gmail.com/3-0011-ignore-nulls.patch)
download | inline diff:
From bf4ddaa83c1004a96471106babf6a06022c4228c Mon Sep 17 00:00:00 2001
From: Oliver Ford <[email protected]>
Date: Wed, 12 Mar 2025 13:34:30 +0000
Subject: [PATCH] ignore nulls
---
doc/src/sgml/func.sgml | 38 ++--
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 225 ++++++++++++++++++-
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 ++
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 +++++++++++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++++++
15 files changed, 773 insertions(+), 28 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 51dd8ad6571..d8cf8b039d5 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23442,7 +23442,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23467,7 +23467,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23490,7 +23490,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23504,7 +23504,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23518,7 +23518,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23567,18 +23567,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index 2f250d2c57b..46a8959cb2f 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..149333bb687 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,7 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +97,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -198,6 +200,13 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginpartition(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool *isnull, bool *isout);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
/*
* initialize_windowaggregate
@@ -2619,14 +2628,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2691,7 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3227,212 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * ignorenulls_getfuncarginpartition
+ * For IGNORE NULLS, get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+static Datum
+ignorenulls_getfuncarginpartition(WindowObject winobj, int argno,
+ int relpos, int seektype, bool *isnull,
+ bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ bool gottuple;
+ int64 abs_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ abs_pos = winstate->currentpos;
+ break;
+ case WINDOW_SEEK_HEAD:
+ abs_pos = 0;
+ break;
+ case WINDOW_SEEK_TAIL:
+ spool_tuples(winstate, -1);
+ abs_pos = winstate->spooled_rows - 1 + relpos;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ abs_pos += forward;
+ gottuple = window_gettupleslot(winobj, abs_pos, slot);
+
+ if (!gottuple)
+ {
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+
+ if (!*isnull)
+ ++notnull_offset;
+ } while (notnull_offset < notnull_relpos);
+
+ return datum;
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ int inframe;
+
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ inframe = row_is_in_frame(winstate, abs_pos, slot);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+
+ if (!*isnull)
+ ++notnull_offset;
+
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3388,6 +3601,10 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ return ignorenulls_getfuncarginpartition(winobj, argno, relpos, seektype,
+ isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3476,6 +3693,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 43dfecfb47f..e7091d7468c 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2570,6 +2570,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 271ae26cbaf..19b1e61d10b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -632,7 +632,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -730,7 +730,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -765,7 +765,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15758,7 +15758,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15791,7 +15791,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16387,6 +16388,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17824,6 +17831,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17941,6 +17949,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 9e90acedb91..ac97a35ad47 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11080,7 +11080,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 23c9e3c5abf..b13a7bfe674 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -439,6 +439,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index d0576da3e25..65081616402 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -577,6 +577,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -600,6 +611,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 40cf090ce61..3ba00a39e5d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -377,6 +378,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-03-29 07:51 Krasiyan Andreev <[email protected]>
parent: Oliver Ford <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Krasiyan Andreev @ 2025-03-29 07:51 UTC (permalink / raw)
To: Oliver Ford <[email protected]>; +Cc: Tatsuo Ishii <[email protected]>; Tom Lane <[email protected]>; Vik Fearing <[email protected]>; pgsql-hackers; [email protected]; David Fetter <[email protected]>
Hi,
Patch applies and compiles, all included tests passed and after the latest
fixes for non-nulls array, performance is near to lead/lag without support
of "ignore nulls".
I have been using the last version for more than one month in a production
environment with real data and didn't find any bugs, so It is ready for
committer status.
На чт, 13.03.2025 г. в 9:49 Oliver Ford <[email protected]> написа:
>
>
> On Sun, 9 Mar 2025, 20:07 Oliver Ford, <[email protected]> wrote:
>
>> On Sun, Mar 9, 2025 at 6:40 AM Tatsuo Ishii <[email protected]> wrote:
>>
>>> > Attached version removes the non-nulls array. That seems to speed
>>> > everything up. Running the above query with 1 million rows averages
>>> 450ms,
>>> > similar when using lead/lag.
>>>
>>> Great. However, CFbot complains about the patch:
>>>
>>> https://cirrus-ci.com/task/6364194477441024
>>>
>>>
>> Attached fixes the headerscheck locally.
>>
>
> v11 attached because the previous version was broken by commit 8b1b342
>
>>
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-03-29 11:18 Tatsuo Ishii <[email protected]>
parent: Krasiyan Andreev <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-03-29 11:18 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
> Hi,
> Patch applies and compiles, all included tests passed and after the latest
> fixes for non-nulls array, performance is near to lead/lag without support
> of "ignore nulls".
> I have been using the last version for more than one month in a production
> environment with real data and didn't find any bugs, so It is ready for
> committer status.
One thing I worry about the patch is, now the non-nulls array
optimization was removed. Since then I have been thinking about if
there could be other way to optimize searching for non null rows.
Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-06-11 22:52 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-06-11 22:52 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers
> One thing I worry about the patch is, now the non-nulls array
> optimization was removed. Since then I have been thinking about if
> there could be other way to optimize searching for non null rows.
Here is the v12 patch to implement the optimization on top of Oliver's
v11 patch. Only src/backend/executor/nodeWindowAgg.c was modified
(especially ignorenulls_getfuncarginframe). In the patch I created
2-bit not null information array, representing following status for
each row:
UNKNOWN: the row is not determined whether it's NULL or NOT yet.
This is the initial value.
NULL: the row has been determined to be NULL.
NOT NULL: the row has been determined to be NOT NULL.
In ignorenulls_getfuncarginframe:
For the first time window function visits a row in a frame, the row is
fetched using window_gettupleslot() and it is checked whether it is in
the frame using row_is_in_frame(). If it's in the frame and the
information in the array is UNKNOWN, ExecEvalExpr() is executed to
find out if the expression on the function argument is NULL or
not. And the result (NULL or NOT NULL) is stored in the array.
If the information in the array is not UNKNOWN, we can skip calling
ExecEvalExpr() because the information is already in the array.
Note that I do not skip calling window_gettupleslot() and
row_is_in_frame(), skip only calling ExecEvalExpr(), because whether a
row is in a frame or not could be changing as the current row position
moves while processing window functions.
With this technique I observed around 40% speed up in my environment
using the script attached, comparing with Oliver's v11 patch.
v11:
rows duration (msec)
1000 41.019
2000 148.957
3000 248.291
4000 442.478
5000 687.395
v12:
rows duration (msec)
1000 27.515
2000 78.913
3000 174.737
4000 311.412
5000 482.156
The patch is now generated using the standard git format-patch. Also
I have slightly adjusted the coding style so that it aligns with the
one used in nodeWindowAgg.c, and ran pgindent.
Note that I have not modified ignorenulls_getfuncarginpartition yet. I
think we could optimize it using the not null info infrastructure as
well. Will come up with it.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
for i in 1000 2000 3000 4000 5000
do
echo "$i rows: "
pos=`expr $i / 2`
psql -a test <<EOF
\timing
explain analyze
SELECT
x,
nth_value(x,$pos) IGNORE NULLS OVER w
FROM generate_series(1,$i) g(x)
WINDOW w AS (ORDER BY x ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING);
EOF
done | egrep "rows:|Time:" | egrep -v "Planning|Execution"|
sed -e 's/rows: *//' -e 's/Time: //' -e 's/ms//'
Attachments:
[application/octet-stream] v12-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch (51.6K, ../../[email protected]/2-v12-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch)
download | inline diff:
From f1da51ab3c5b2eb5fa07c762ba6e4ccaf30ad208 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 11 Jun 2025 20:47:54 +0900
Subject: [PATCH v12] Add IGNORE NULLS/RESPECT NULLS option to Window
functions.
Add IGNORE NULLS/RESPECT NULLS option to lead, lag, first_value,
last_value and nth_value.
---
doc/src/sgml/func.sgml | 38 +--
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 359 ++++++++++++++++++++++++++-
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 +++++++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++++
15 files changed, 907 insertions(+), 28 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index c67688cbf5f..fd9bdf2f73c 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23533,7 +23533,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23558,7 +23558,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23581,7 +23581,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23595,7 +23595,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23609,7 +23609,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23658,18 +23658,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index ebe85337c28..3a8ad201607 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..11e0a496e25 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,9 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +99,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -198,6 +202,35 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginpartition(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool *isnull, bool *isout);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info map consists of 2-bits array:
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -1263,6 +1296,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -2619,14 +2657,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2720,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3257,235 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * ignorenulls_getfuncarginpartition
+ * For IGNORE NULLS, get the next nonnull * value in the partition, moving
+ * forward or backward until we find a value or reach the partition's end.
+ */
+static Datum
+ignorenulls_getfuncarginpartition(WindowObject winobj, int argno,
+ int relpos, int seektype, bool *isnull,
+ bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ bool gottuple;
+ int64 abs_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ abs_pos = winstate->currentpos;
+ break;
+ case WINDOW_SEEK_HEAD:
+ abs_pos = 0;
+ break;
+ case WINDOW_SEEK_TAIL:
+ spool_tuples(winstate, -1);
+ abs_pos = winstate->spooled_rows - 1 + relpos;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ abs_pos += forward;
+ gottuple = window_gettupleslot(winobj, abs_pos, slot);
+
+ if (!gottuple)
+ {
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+
+ if (!*isnull)
+ ++notnull_offset;
+ } while (notnull_offset < notnull_relpos);
+
+ return datum;
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ int inframe;
+ int v;
+
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ inframe = row_is_in_frame(winstate, abs_pos, slot);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3388,6 +3654,10 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ return ignorenulls_getfuncarginpartition(
+ winobj, argno, relpos, seektype, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3476,6 +3746,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3670,3 +3944,84 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 26a3e050086..9705c8ef2e8 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2572,6 +2572,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 0b5652071d1..84d92332053 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -632,7 +632,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -730,7 +730,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -765,7 +765,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15760,7 +15760,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15793,7 +15793,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16389,6 +16390,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17826,6 +17833,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17944,6 +17952,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index dd00ab420b8..02b7216e9d0 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -444,6 +444,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 7d3b4198f26..a013cd2c1ac 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -577,6 +577,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -600,6 +611,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
[text/plain] ignore_nulls_bench2.sh (407B, ../../[email protected]/3-ignore_nulls_bench2.sh)
download | inline:
for i in 1000 2000 3000 4000 5000
do
echo "$i rows: "
pos=`expr $i / 2`
psql -a test <<EOF
\timing
explain analyze
SELECT
x,
nth_value(x,$pos) IGNORE NULLS OVER w
FROM generate_series(1,$i) g(x)
WINDOW w AS (ORDER BY x ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING);
EOF
done | egrep "rows:|Time:" | egrep -v "Planning|Execution"|
sed -e 's/rows: *//' -e 's/Time: //' -e 's/ms//'
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-06-19 06:21 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-06-19 06:21 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
> Here is the v12 patch to implement the optimization on top of Oliver's
> v11 patch. Only src/backend/executor/nodeWindowAgg.c was modified
> (especially ignorenulls_getfuncarginframe). In the patch I created
> 2-bit not null information array, representing following status for
> each row:
>
> UNKNOWN: the row is not determined whether it's NULL or NOT yet.
> This is the initial value.
> NULL: the row has been determined to be NULL.
> NOT NULL: the row has been determined to be NOT NULL.
>
> In ignorenulls_getfuncarginframe:
>
> For the first time window function visits a row in a frame, the row is
> fetched using window_gettupleslot() and it is checked whether it is in
> the frame using row_is_in_frame(). If it's in the frame and the
> information in the array is UNKNOWN, ExecEvalExpr() is executed to
> find out if the expression on the function argument is NULL or
> not. And the result (NULL or NOT NULL) is stored in the array.
>
> If the information in the array is not UNKNOWN, we can skip calling
> ExecEvalExpr() because the information is already in the array.
>
> Note that I do not skip calling window_gettupleslot() and
> row_is_in_frame(), skip only calling ExecEvalExpr(), because whether a
> row is in a frame or not could be changing as the current row position
> moves while processing window functions.
>
> With this technique I observed around 40% speed up in my environment
> using the script attached, comparing with Oliver's v11 patch.
>
> v11:
> rows duration (msec)
> 1000 41.019
> 2000 148.957
> 3000 248.291
> 4000 442.478
> 5000 687.395
>
> v12:
> rows duration (msec)
> 1000 27.515
> 2000 78.913
> 3000 174.737
> 4000 311.412
> 5000 482.156
>
> The patch is now generated using the standard git format-patch. Also
> I have slightly adjusted the coding style so that it aligns with the
> one used in nodeWindowAgg.c, and ran pgindent.
>
> Note that I have not modified ignorenulls_getfuncarginpartition yet. I
> think we could optimize it using the not null info infrastructure as
> well. Will come up with it.
Attached is the v13 patch to address this: i.e. optimize window
functions (lead/lag) working in a partition. In summary I get 40x
speed up comparing with v12 patch. Here is the test script.
EXPLAIN ANALYZE
SELECT lead(x, 5000) IGNORE NULLS OVER ()
FROM generate_series(1,10000) g(x);
This looks for the 5k th row in a partition including 10k rows.
The average duration of 3 trials are:
v12: 2563.665 ms
v13: 126.259 ms
So I got 40x speed up with the v13 patch. In v12, we needed to scan
10k row partition over and over again. In v13 I used the same NULL/NOT
NULL cache infrastructure created in v12, and we need to scan the
partition only once. This is the reason for the speed up.
Also I removed ignorenulls_getfuncarginpartition(), which was the work
horse for null treatment for window functions working for partitions
in v12 patch. Basically it was a copy and modified version of
WinGetFuncArgInPartition(), thus had quite a few code duplication. In
v13, instead I modified WinGetFuncArgInPartition() so that it can
handle directly null treatment procedures.
BTW I am still not satisfied by the performance improvement for window
functions for frames, that was only 40%. I will study the code to look
for more optimization.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v13-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch (52.9K, ../../[email protected]/2-v13-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch)
download | inline diff:
From 872588099b49623f24fc97bd2418292d4bbe685c Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 19 Jun 2025 14:40:41 +0900
Subject: [PATCH v13] Add IGNORE NULLS/RESPECT NULLS option to Window
functions.
Add IGNORE NULLS/RESPECT NULLS option to lead, lag, first_value,
last_value and nth_value.
---
doc/src/sgml/func.sgml | 38 ++-
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 403 +++++++++++++++++++++++++--
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 +++++++++++++++++++++
src/test/regress/sql/window.sql | 147 ++++++++++
15 files changed, 928 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 8d7d9a2f3e8..6b7c22a2109 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23533,7 +23533,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23558,7 +23558,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23581,7 +23581,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23595,7 +23595,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23609,7 +23609,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23658,18 +23658,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index ebe85337c28..3a8ad201607 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..20355effb95 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,9 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +99,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -198,6 +202,33 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -1263,6 +1294,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -2619,14 +2655,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2718,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3255,274 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ do
+ {
+ int inframe;
+ int v;
+
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ inframe = row_is_in_frame(winstate, abs_pos, slot);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3681,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3723,57 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
- {
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3825,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 26a3e050086..9705c8ef2e8 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2572,6 +2572,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 50f53159d58..547b7f7e465 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15738,7 +15738,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15771,7 +15771,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16367,6 +16368,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17793,6 +17800,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17911,6 +17919,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ba12678d1cb..8bedaf2f750 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 01510b01b64..2f0a77d2cc2 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -577,6 +577,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -600,6 +611,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-06-25 07:19 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-06-25 07:19 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers
> BTW I am still not satisfied by the performance improvement for window
> functions for frames, that was only 40%. I will study the code to look
> for more optimization.
So I come up with more optimization for window functions working on
frames (i.e. first_value, last_value and nth_value). Attached v14
patch does it.
There are 3 major functions used here.
1) window_gettupleslot (get a row)
2) row_is_in_frame (check whether row is in frame or not)
3) ExecEvalExpr (evaluate arg on the row)
In v12 (and v13), we eliminate #3 in some cases but the saving was
only 40%. In v14, I found some cases where we don't need to call
#1. row_is_in_frame requires a row ("tuple" argument), which is
provided by #1. However row_is_in_frame actually uses the row argument
only when frame clause is "RANGE" or "GROUPS" and frame end is
"CURRENT ROW". In other cases it does not use "tuple" argument at
all. So I check the frame clause and the frame end, and if they are
not the case, I can omit #1. Plus if the not null cache for the row
has been already created, we can omit #3 as well. The optimization
contributes to the performance. I observe 2.7x (1k rows case) to 5.2x
(3k rows case) speed up when I compare the performance of v13 patch
and v14 patch using the same script (see attached).
v13:
rows duration (msec)
1000 34.740
2000 91.169
3000 205.847
4000 356.142
5000 557.063
v14:
rows duration (msec)
1000 12.807
2000 21.782
3000 39.248
4000 69.123
5000 101.220
I am not sure how the case where frame clause is "RANGE" or "GROUPS"
and frame end is "CURRENT ROW" is majority of window function use
cases. If it's majority, the optimization in v14 does not help much
because v14 does not optimize the case. However if it's not, the v14
patch is close to commitable form, I think. Comments are welcome.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
for i in 1000 2000 3000 4000 5000
do
echo "$i rows: "
pos=`expr $i / 2`
psql -a test <<EOF
\timing
explain analyze
SELECT
x,
nth_value(x,$pos) IGNORE NULLS OVER w
FROM generate_series(1,$i) g(x)
WINDOW w AS (ORDER BY x ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING);
EOF
done | egrep "rows:|Time:" | egrep -v "Planning|Execution"|
sed -e 's/rows: *//' -e 's/Time: //' -e 's/ms//'
Attachments:
[application/octet-stream] v14-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch (53.8K, ../../[email protected]/2-v14-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch)
download | inline diff:
From f47fee700e97f0f5f2537676671885f376619871 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 25 Jun 2025 14:55:33 +0900
Subject: [PATCH v14] Add IGNORE NULLS/RESPECT NULLS option to Window
functions.
Add IGNORE NULLS/RESPECT NULLS option to lead, lag, first_value,
last_value and nth_value.
---
doc/src/sgml/func.sgml | 38 ++-
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 437 +++++++++++++++++++++++++--
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 +++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++
15 files changed, 962 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 224d4fe5a9f..a95fdbab031 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23543,7 +23543,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23568,7 +23568,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23591,7 +23591,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23605,7 +23605,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23619,7 +23619,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23668,18 +23668,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index ebe85337c28..3a8ad201607 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..ba40cde550c 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,9 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +99,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -198,6 +202,33 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -1263,6 +1294,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -2619,14 +2655,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2718,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3255,308 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+ int frameOptions;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ frameOptions = winstate->frameOptions;
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+ bool gottuple = false;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /*
+ * row_is_in_frame requires slot if following frame options are set.
+ */
+ if (frameOptions & FRAMEOPTION_END_CURRENT_ROW &&
+ frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+ gottuple = true;
+ }
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winstate, abs_pos, slot);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!gottuple)
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+ }
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!gottuple)
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+ }
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3715,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3757,57 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
- {
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3859,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 26a3e050086..9705c8ef2e8 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2572,6 +2572,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 50f53159d58..547b7f7e465 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15738,7 +15738,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15771,7 +15771,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16367,6 +16368,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17793,6 +17800,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17911,6 +17919,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ba12678d1cb..8bedaf2f750 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
[text/plain] ignore_nulls_bench2.sh (407B, ../../[email protected]/3-ignore_nulls_bench2.sh)
download | inline:
for i in 1000 2000 3000 4000 5000
do
echo "$i rows: "
pos=`expr $i / 2`
psql -a test <<EOF
\timing
explain analyze
SELECT
x,
nth_value(x,$pos) IGNORE NULLS OVER w
FROM generate_series(1,$i) g(x)
WINDOW w AS (ORDER BY x ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING);
EOF
done | egrep "rows:|Time:" | egrep -v "Planning|Execution"|
sed -e 's/rows: *//' -e 's/Time: //' -e 's/ms//'
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-06-30 05:25 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-06-30 05:25 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers
Attached is the v15 patch to fix CFbot complains.
Other than that, nothing has been changed since v14.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v15-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch (54.0K, ../../[email protected]/2-v15-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch)
download | inline diff:
From fb63e9a73c475be518c393d438b6dddb3b4ca833 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Mon, 30 Jun 2025 14:21:48 +0900
Subject: [PATCH v15] Add IGNORE NULLS/RESPECT NULLS option to Window
functions.
Add IGNORE NULLS/RESPECT NULLS option to lead, lag, first_value,
last_value and nth_value.
---
doc/src/sgml/func.sgml | 38 ++-
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 447 +++++++++++++++++++++++++--
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 +++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++
15 files changed, 972 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 224d4fe5a9f..a95fdbab031 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23543,7 +23543,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23568,7 +23568,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23591,7 +23591,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23605,7 +23605,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23619,7 +23619,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23668,18 +23668,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index ebe85337c28..3a8ad201607 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..9265c2754a7 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,9 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +99,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -198,6 +202,33 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -1263,6 +1294,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -2619,14 +2655,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2718,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3255,308 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+ int frameOptions;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ frameOptions = winstate->frameOptions;
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+ bool gottuple = false;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /*
+ * row_is_in_frame requires slot if following frame options are set.
+ */
+ if (frameOptions & FRAMEOPTION_END_CURRENT_ROW &&
+ frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+ gottuple = true;
+ }
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winstate, abs_pos, slot);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!gottuple)
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+ }
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!gottuple)
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+ }
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3715,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3757,67 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
- {
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ if (abs_pos < 0)
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ datum = 0;
+ break;
+ }
+
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3869,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 26a3e050086..9705c8ef2e8 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2572,6 +2572,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 50f53159d58..547b7f7e465 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15738,7 +15738,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15771,7 +15771,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16367,6 +16368,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17793,6 +17800,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17911,6 +17919,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ba12678d1cb..8bedaf2f750 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-06-30 08:08 Krasiyan Andreev <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Krasiyan Andreev @ 2025-06-30 08:08 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
Hi,
Patch applies and compiles, all included tests passed and performance gain
is really impressive. I have been using the latest versions for months with
real data and didn't find any bugs, so It is definitely ready for committer
status.
На пн, 30.06.2025 г. в 8:26 Tatsuo Ishii <[email protected]> написа:
> Attached is the v15 patch to fix CFbot complains.
> Other than that, nothing has been changed since v14.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-07-01 00:19 Tatsuo Ishii <[email protected]>
parent: Krasiyan Andreev <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-07-01 00:19 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
Krasiyan,
> Hi,
> Patch applies and compiles, all included tests passed and performance gain
> is really impressive. I have been using the latest versions for months with
> real data and didn't find any bugs, so It is definitely ready for committer
> status.
Thanks for testing the patch. The CF status has been already set to
"ready for committer". I just changed the target version from 18 to
19.
> I am not sure how the case where frame clause is "RANGE" or "GROUPS"
> and frame end is "CURRENT ROW" is majority of window function use
> cases. If it's majority, the optimization in v14 does not help much
> because v14 does not optimize the case. However if it's not, the v14
> patch is close to commitable form, I think. Comments are welcome.
Have you tested cases where the frame option is "RANGE" or "GROUPS"
and the frame end is "CURRENT ROW"? I am asking because in these cases
the optimization in the v14 (and v15) patches do not apply and you may
not be satisfied by the performance.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-07-07 05:37 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-07-07 05:37 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers
Attached is the v16 patch.
In this patch I have changed row_is_in_frame() API from:
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
to:
static int row_is_in_frame(WindowObject winobj, int64 pos,
TupleTableSlot *slot, bool fetch_tuple);
The function is used to decide whether a row specified by pos is in a
frame or not. Previously we needed to always pass "slot" parameter
which is the row in question, fetched by window_gettupleslot. If
IGNORE NULLS option is not passed to window functions, this is fine
because they need to return the row anyway.
However if IGNORE NULLS specified, we need to throw away null rows
until we find the non null row requested by the caller. In reality,
not in all window frames it is required to pass a row to
row_is_in_frame: only when specific frame options are specified. For
example RANGE or GROUP options plus CURRENT ROW frame end option. In
previous patch, I explicitly checked these frame options before
calling row_is_in_frame. However I dislike the way because it's a
layer abstraction violation.
So in this patch I added "fetch_tuple" option to row_is_in_frame so
that it fetches row itself when necessary. A caller now don't need to
fetch the row to pass if fetch_tuple is false.
This way, not only we can avoid the layer violation problem, but
performance is enhanced because tuple is fetched only when it's
necessary.
Note that now the first argument of row_is_in_frame has been changed
from WindowAggState to WindowObject so that row_is_in_frame can call
window_gettupleslot inside.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v16-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch (56.3K, ../../[email protected]/2-v16-0001-Add-IGNORE-NULLS-RESPECT-NULLS-option-to-Window-.patch)
download | inline diff:
From 5825053d159e77fc475c0201eea27b23144fcd37 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Mon, 7 Jul 2025 14:12:15 +0900
Subject: [PATCH v16] Add IGNORE NULLS/RESPECT NULLS option to Window
functions.
Add IGNORE NULLS/RESPECT NULLS option to lead, lag, first_value,
last_value and nth_value.
---
doc/src/sgml/func.sgml | 38 ++-
doc/src/sgml/syntax.sgml | 10 +-
src/backend/catalog/sql_features.txt | 2 +-
src/backend/executor/nodeWindowAgg.c | 454 ++++++++++++++++++++++++---
src/backend/optimizer/util/clauses.c | 1 +
src/backend/parser/gram.y | 19 +-
src/backend/parser/parse_func.c | 9 +
src/backend/utils/adt/ruleutils.c | 7 +-
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +
src/include/parser/kwlist.h | 2 +
src/include/windowapi.h | 6 +
src/test/regress/expected/window.out | 311 ++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++
15 files changed, 969 insertions(+), 61 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 810b2b50f0d..5f3d9e85d4b 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23543,7 +23543,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23568,7 +23568,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23591,7 +23591,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23605,7 +23605,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23619,7 +23619,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23668,18 +23668,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index ebe85337c28..3a8ad201607 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..8def00425f6 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,9 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ int ignore_nulls; /* ignore nulls */
} WindowObjectData;
/*
@@ -96,6 +99,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -182,8 +186,8 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
- TupleTableSlot *slot);
+static int row_is_in_frame(WindowObject winobj, int64 pos,
+ TupleTableSlot *slot, bool fetch_tuple);
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -198,6 +202,33 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -942,7 +973,7 @@ eval_windowaggregates(WindowAggState *winstate)
* Exit loop if no more rows can be in frame. Skip aggregation if
* current row is not in frame but there might be more in the frame.
*/
- ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
+ ret = row_is_in_frame(agg_winobj, winstate->aggregatedupto, agg_row_slot, false);
if (ret < 0)
break;
if (ret == 0)
@@ -1263,6 +1294,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -1412,8 +1448,8 @@ release_partition(WindowAggState *winstate)
* to our window framing rule
*
* The caller must have already determined that the row is in the partition
- * and fetched it into a slot. This function just encapsulates the framing
- * rules.
+ * and fetched it into a slot if fetch_tuple is false.
+.* This function just encapsulates the framing rules.
*
* Returns:
* -1, if the row is out of frame and no succeeding rows can be in frame
@@ -1423,8 +1459,9 @@ release_partition(WindowAggState *winstate)
* May clobber winstate->temp_slot_2.
*/
static int
-row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
+row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot, bool fetch_tuple)
{
+ WindowAggState *winstate = winobj->winstate;
int frameOptions = winstate->frameOptions;
Assert(pos >= 0); /* else caller error */
@@ -1453,9 +1490,13 @@ row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
else if (frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
{
/* following row that is not peer is out of frame */
- if (pos > winstate->currentpos &&
- !are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
- return -1;
+ if (pos > winstate->currentpos)
+ {
+ if (fetch_tuple)
+ window_gettupleslot(winobj, pos, slot);
+ if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
+ return -1;
+ }
}
else
Assert(false);
@@ -2619,14 +2660,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2723,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3260,290 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winobj, abs_pos, slot, true);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3702,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3744,67 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
- {
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ if (abs_pos < 0)
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ datum = 0;
+ break;
+ }
+
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3856,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3624,7 +4008,7 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
goto out_of_frame;
/* The code above does not detect all out-of-frame cases, so check */
- if (row_is_in_frame(winstate, abs_pos, slot) <= 0)
+ if (row_is_in_frame(winobj, abs_pos, slot, false) <= 0)
goto out_of_frame;
if (isout)
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index f45131c34c5..2765d794d20 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2572,6 +2572,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 70a0d832a11..27adff5f9d6 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15796,7 +15796,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15829,7 +15829,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16425,6 +16426,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17851,6 +17858,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17969,6 +17977,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 28e2e8dc0fd..5933e64e9b3 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-07-16 04:44 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-07-16 04:44 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers
Currently the patch set include some regression test additions. I
wanted to expand the test coverage and ended up an idea: generate new
test cases from the existing window function regression test
(window.sql).
Attached "insert_ignore_nulls.sh" script reads window.sql and inserts
"ignore nulls" before "over" clause of each window functions that
accept IGNORE NULLS option. This way, although the generated test
cases do not cover the case where NULL is included, at least covers
all non NULL cases, which is better than nothing, I think.
I replaced the existing window.sql with the modified one, and ran the
regression test. Indeed the test failed because expected file is for
non IGNORE NULLS options. However, the differences should be just for
SQL statements, not the output of the SQL statements since the data
set used does not include NULLs. I did an eyeball check the diff and
the result was what I expected.
For those who are interested this test, I attached some files.
insert_ignore_nulls.sh: shell script to insert "ignore nulls"
window.sql: modified regression script by insert_ignore_nulls.sh
window.diff: diff of original window.out and modified window.out
Question is, how could we put this kind of test into core if it worth
the effort? The simplest idea is just adding the modified window.sql
to the end of existing window.sql and update window.out. Thoughts?
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
#! /bin/sh
# Script to generate test cases for IGNORE NULLS option of some window
# functions (lead, lag, first_value, last_value, and nth_value) in the
# regression test SQL (window.sql). This script supposed to read
# window.sql from stdin and inserts "ignore null" before "over" on all
# occurrences of the window functions that accept IGNORE NULLS, then
# print it to stdout. Since currently the data set used for the
# existing regression test (window.sql) does not include NULLs, the
# test result using the results of this script is expected be
# identical to the existing results of existing window.sql (of course
# except the SQL statements).
sed -r \
-e "s/lead\(([^)]*)\) over/lead(\1) ignore nulls over/g" \
-e "s/lag\(([^)]*)\) over/lag(\1) ignore nulls over/g" \
-e "s/first_value\(([^)]*)\) over/first_value(\1) ignore nulls over/g" \
-e "s/last_value\(([^)]*)\) over/last_value(\1) ignore nulls over/g" \
-e "s/nth_value\(([^)]*)\) over/nth_value(\1) ignore nulls over/g"
--
-- WINDOW FUNCTIONS
--
CREATE TEMPORARY TABLE empsalary (
depname varchar,
empno bigint,
salary int,
enroll_date date
);
INSERT INTO empsalary VALUES
('develop', 10, 5200, '2007-08-01'),
('sales', 1, 5000, '2006-10-01'),
('personnel', 5, 3500, '2007-12-10'),
('sales', 4, 4800, '2007-08-08'),
('personnel', 2, 3900, '2006-12-23'),
('develop', 7, 4200, '2008-01-01'),
('develop', 9, 4500, '2008-01-01'),
('sales', 3, 4800, '2007-08-01'),
('develop', 8, 6000, '2006-10-01'),
('develop', 11, 5200, '2007-08-15');
SELECT depname, empno, salary, sum(salary) OVER (PARTITION BY depname) FROM empsalary ORDER BY depname, salary;
SELECT depname, empno, salary, rank() OVER (PARTITION BY depname ORDER BY salary) FROM empsalary;
-- with GROUP BY
SELECT four, ten, SUM(SUM(four)) OVER (PARTITION BY four), AVG(ten) FROM tenk1
GROUP BY four, ten ORDER BY four, ten;
SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS (PARTITION BY depname);
SELECT depname, empno, salary, rank() OVER w FROM empsalary WINDOW w AS (PARTITION BY depname ORDER BY salary) ORDER BY rank() OVER w;
-- empty window specification
SELECT COUNT(*) OVER () FROM tenk1 WHERE unique2 < 10;
SELECT COUNT(*) OVER w FROM tenk1 WHERE unique2 < 10 WINDOW w AS ();
-- no window operation
SELECT four FROM tenk1 WHERE FALSE WINDOW w AS (PARTITION BY ten);
-- cumulative aggregate
SELECT sum(four) OVER (PARTITION BY ten ORDER BY unique2) AS sum_1, ten, four FROM tenk1 WHERE unique2 < 10;
SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10;
SELECT rank() OVER (PARTITION BY four ORDER BY ten) AS rank_1, ten, four FROM tenk1 WHERE unique2 < 10;
SELECT dense_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT percent_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT cume_dist() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT ntile(3) OVER (ORDER BY ten, four), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT ntile(NULL) OVER (ORDER BY ten, four), ten, four FROM tenk1 LIMIT 2;
SELECT lag(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT lag(ten, four) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT lag(ten, four, 0) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT lag(ten, four, 0.7) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten;
SELECT lead(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT lead(ten * 2, 1) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT lead(ten * 2, 1, -1) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT lead(ten * 2, 1, -1.4) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten;
SELECT first_value(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
-- last_value returns the last row of the frame, which is CURRENT ROW in ORDER BY window.
SELECT last_value(four) OVER (ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
SELECT last_value(ten) OVER (PARTITION BY four), ten, four FROM
(SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s
ORDER BY four, ten;
SELECT nth_value(ten, four + 1) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s;
SELECT ten, two, sum(hundred) AS gsum, sum(sum(hundred)) OVER (PARTITION BY two ORDER BY ten) AS wsum
FROM tenk1 GROUP BY ten, two;
SELECT count(*) OVER (PARTITION BY four), four FROM (SELECT * FROM tenk1 WHERE two = 1)s WHERE unique2 < 10;
SELECT (count(*) OVER (PARTITION BY four ORDER BY ten) +
sum(hundred) OVER (PARTITION BY four ORDER BY ten))::varchar AS cntsum
FROM tenk1 WHERE unique2 < 10;
-- opexpr with different windows evaluation.
SELECT * FROM(
SELECT count(*) OVER (PARTITION BY four ORDER BY ten) +
sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS total,
count(*) OVER (PARTITION BY four ORDER BY ten) AS fourcount,
sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS twosum
FROM tenk1
)sub
WHERE total <> fourcount + twosum;
SELECT avg(four) OVER (PARTITION BY four ORDER BY thousand / 100) FROM tenk1 WHERE unique2 < 10;
SELECT ten, two, sum(hundred) AS gsum, sum(sum(hundred)) OVER win AS wsum
FROM tenk1 GROUP BY ten, two WINDOW win AS (PARTITION BY two ORDER BY ten);
-- more than one window with GROUP BY
SELECT sum(salary),
row_number() OVER (ORDER BY depname),
sum(sum(salary)) OVER (ORDER BY depname DESC)
FROM empsalary GROUP BY depname;
-- identical windows with different names
SELECT sum(salary) OVER w1, count(*) OVER w2
FROM empsalary WINDOW w1 AS (ORDER BY salary), w2 AS (ORDER BY salary);
-- subplan
SELECT lead(ten, (SELECT two FROM tenk1 WHERE s.unique2 = unique2)) OVER (PARTITION BY four ORDER BY ten)
FROM tenk1 s WHERE unique2 < 10;
-- empty table
SELECT count(*) OVER (PARTITION BY four) FROM (SELECT * FROM tenk1 WHERE FALSE)s;
-- mixture of agg/wfunc in the same window
SELECT sum(salary) OVER w, rank() OVER w FROM empsalary WINDOW w AS (PARTITION BY depname ORDER BY salary DESC);
-- strict aggs
SELECT empno, depname, salary, bonus, depadj, MIN(bonus) OVER (ORDER BY empno), MAX(depadj) OVER () FROM(
SELECT *,
CASE WHEN enroll_date < '2008-01-01' THEN 2008 - extract(YEAR FROM enroll_date) END * 500 AS bonus,
CASE WHEN
AVG(salary) OVER (PARTITION BY depname) < salary
THEN 200 END AS depadj FROM empsalary
)s;
-- window function over ungrouped agg over empty row set (bug before 9.1)
SELECT SUM(COUNT(f1)) OVER () FROM int4_tbl WHERE f1=42;
-- window function with ORDER BY an expression involving aggregates (9.1 bug)
select ten,
sum(unique1) + sum(unique2) as res,
rank() over (order by sum(unique1) + sum(unique2)) as rank
from tenk1
group by ten order by ten;
-- window and aggregate with GROUP BY expression (9.2 bug)
explain (costs off)
select first_value(max(x)) over (), y
from (select unique1 as x, ten+four as y from tenk1) ss
group by y;
-- window functions returning pass-by-ref values from different rows
select x, lag(x, 1) ignore nulls over (order by x), lead(x, 3) ignore nulls over (order by x)
from (select x::numeric as x from generate_series(1,10) x);
-- test non-default frame specifications
SELECT four, ten,
sum(ten) over (partition by four order by ten),
last_value(ten) ignore nulls over (partition by four order by ten)
FROM (select distinct ten, four from tenk1) ss;
SELECT four, ten,
sum(ten) over (partition by four order by ten range between unbounded preceding and current row),
last_value(ten) ignore nulls over (partition by four order by ten range between unbounded preceding and current row)
FROM (select distinct ten, four from tenk1) ss;
SELECT four, ten,
sum(ten) over (partition by four order by ten range between unbounded preceding and unbounded following),
last_value(ten) ignore nulls over (partition by four order by ten range between unbounded preceding and unbounded following)
FROM (select distinct ten, four from tenk1) ss;
SELECT four, ten/4 as two,
sum(ten/4) over (partition by four order by ten/4 range between unbounded preceding and current row),
last_value(ten/4) ignore nulls over (partition by four order by ten/4 range between unbounded preceding and current row)
FROM (select distinct ten, four from tenk1) ss;
SELECT four, ten/4 as two,
sum(ten/4) over (partition by four order by ten/4 rows between unbounded preceding and current row),
last_value(ten/4) ignore nulls over (partition by four order by ten/4 rows between unbounded preceding and current row)
FROM (select distinct ten, four from tenk1) ss;
SELECT sum(unique1) over (order by four range between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 2 preceding and 2 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude no others),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT first_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT first_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT first_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT last_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT last_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT last_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 2 preceding and 1 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between 1 following and 3 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between unbounded preceding and 1 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (w range between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
SELECT sum(unique1) over (w range between unbounded preceding and current row exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
SELECT sum(unique1) over (w range between unbounded preceding and current row exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
SELECT sum(unique1) over (w range between unbounded preceding and current row exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
SELECT first_value(unique1) ignore nulls over w,
nth_value(unique1, 2) ignore nulls over w AS nth_2,
last_value(unique1) ignore nulls over w, unique1, four
FROM tenk1 WHERE unique1 < 10
WINDOW w AS (order by four range between current row and unbounded following);
SELECT sum(unique1) over
(order by unique1
rows (SELECT unique1 FROM tenk1 ORDER BY unique1 LIMIT 1) + 1 PRECEDING),
unique1
FROM tenk1 WHERE unique1 < 10;
CREATE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following) as sum_rows
FROM generate_series(1, 10) i;
SELECT * FROM v_window;
SELECT pg_get_viewdef('v_window');
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude current row) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
SELECT pg_get_viewdef('v_window');
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude group) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
SELECT pg_get_viewdef('v_window');
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude ties) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
SELECT pg_get_viewdef('v_window');
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude no others) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
SELECT pg_get_viewdef('v_window');
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i groups between 1 preceding and 1 following) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
SELECT pg_get_viewdef('v_window');
DROP VIEW v_window;
CREATE TEMP VIEW v_window AS
SELECT i, min(i) over (order by i range between '1 day' preceding and '10 days' following) as min_i
FROM generate_series(now(), now()+'100 days'::interval, '1 hour') i;
SELECT pg_get_viewdef('v_window');
-- RANGE offset PRECEDING/FOLLOWING tests
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four desc range between 2::int8 preceding and 1::int2 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude no others),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 6::int2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 6::int2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (partition by four order by unique1 range between 5::int8 preceding and 6::int2 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (partition by four order by unique1 range between 5::int8 preceding and 6::int2 following
exclude current row),unique1, four
FROM tenk1 WHERE unique1 < 10;
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following),
salary, enroll_date from empsalary;
select sum(salary) over (order by enroll_date desc range between '1 year'::interval preceding and '1 year'::interval following),
salary, enroll_date from empsalary;
select sum(salary) over (order by enroll_date desc range between '1 year'::interval following and '1 year'::interval following),
salary, enroll_date from empsalary;
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following
exclude current row), salary, enroll_date from empsalary;
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following
exclude group), salary, enroll_date from empsalary;
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following
exclude ties), salary, enroll_date from empsalary;
select first_value(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
lead(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
nth_value(salary, 1) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
salary from empsalary;
select last_value(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
lag(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
salary from empsalary;
select first_value(salary) ignore nulls over(order by salary range between 1000 following and 3000 following
exclude current row),
lead(salary) ignore nulls over(order by salary range between 1000 following and 3000 following exclude ties),
nth_value(salary, 1) ignore nulls over(order by salary range between 1000 following and 3000 following
exclude ties),
salary from empsalary;
select last_value(salary) ignore nulls over(order by salary range between 1000 following and 3000 following
exclude group),
lag(salary) ignore nulls over(order by salary range between 1000 following and 3000 following exclude group),
salary from empsalary;
select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude ties),
last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following),
salary, enroll_date from empsalary;
select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude ties),
last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude ties),
salary, enroll_date from empsalary;
select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude group),
last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude group),
salary, enroll_date from empsalary;
select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude current row),
last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude current row),
salary, enroll_date from empsalary;
-- RANGE offset PRECEDING/FOLLOWING with null values
select x, y,
first_value(y) ignore nulls over w,
last_value(y) ignore nulls over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x asc nulls first range between 2 preceding and 2 following);
select x, y,
first_value(y) ignore nulls over w,
last_value(y) ignore nulls over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x asc nulls last range between 2 preceding and 2 following);
select x, y,
first_value(y) ignore nulls over w,
last_value(y) ignore nulls over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x desc nulls first range between 2 preceding and 2 following);
select x, y,
first_value(y) ignore nulls over w,
last_value(y) ignore nulls over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x desc nulls last range between 2 preceding and 2 following);
-- There is a syntactic ambiguity in the SQL standard. Since
-- UNBOUNDED is a non-reserved word, it could be the name of a
-- function parameter and be used as an expression. There is a
-- grammar hack to resolve such cases as the keyword. The following
-- tests record this behavior.
CREATE FUNCTION unbounded_syntax_test1a(x int) RETURNS TABLE (a int, b int, c int)
LANGUAGE SQL
BEGIN ATOMIC
SELECT sum(unique1) over (rows between x preceding and x following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
END;
CREATE FUNCTION unbounded_syntax_test1b(x int) RETURNS TABLE (a int, b int, c int)
LANGUAGE SQL
AS $$
SELECT sum(unique1) over (rows between x preceding and x following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
$$;
-- These will apply the argument to the window specification inside the function.
SELECT * FROM unbounded_syntax_test1a(2);
SELECT * FROM unbounded_syntax_test1b(2);
CREATE FUNCTION unbounded_syntax_test2a(unbounded int) RETURNS TABLE (a int, b int, c int)
LANGUAGE SQL
BEGIN ATOMIC
SELECT sum(unique1) over (rows between unbounded preceding and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
END;
CREATE FUNCTION unbounded_syntax_test2b(unbounded int) RETURNS TABLE (a int, b int, c int)
LANGUAGE SQL
AS $$
SELECT sum(unique1) over (rows between unbounded preceding and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
$$;
-- These will not apply the argument but instead treat UNBOUNDED as a keyword.
SELECT * FROM unbounded_syntax_test2a(2);
SELECT * FROM unbounded_syntax_test2b(2);
DROP FUNCTION unbounded_syntax_test1a, unbounded_syntax_test1b,
unbounded_syntax_test2a, unbounded_syntax_test2b;
-- Other tests with token UNBOUNDED in potentially problematic position
CREATE FUNCTION unbounded(x int) RETURNS int LANGUAGE SQL IMMUTABLE RETURN x;
SELECT sum(unique1) over (rows between 1 preceding and 1 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between unbounded(1) preceding and unbounded(1) following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (rows between unbounded.x preceding and unbounded.x following),
unique1, four
FROM tenk1, (values (1)) as unbounded(x) WHERE unique1 < 10;
DROP FUNCTION unbounded;
-- Check overflow behavior for various integer sizes
select x, last_value(x) ignore nulls over (order by x::smallint range between current row and 2147450884 following)
from generate_series(32764, 32766) x;
select x, last_value(x) ignore nulls over (order by x::smallint desc range between current row and 2147450885 following)
from generate_series(-32766, -32764) x;
select x, last_value(x) ignore nulls over (order by x range between current row and 4 following)
from generate_series(2147483644, 2147483646) x;
select x, last_value(x) ignore nulls over (order by x desc range between current row and 5 following)
from generate_series(-2147483646, -2147483644) x;
select x, last_value(x) ignore nulls over (order by x range between current row and 4 following)
from generate_series(9223372036854775804, 9223372036854775806) x;
select x, last_value(x) ignore nulls over (order by x desc range between current row and 5 following)
from generate_series(-9223372036854775806, -9223372036854775804) x;
-- Test in_range for other numeric datatypes
create temp table numerics(
id int,
f_float4 float4,
f_float8 float8,
f_numeric numeric
);
insert into numerics values
(0, '-infinity', '-infinity', '-infinity'),
(1, -3, -3, -3),
(2, -1, -1, -1),
(3, 0, 0, 0),
(4, 1.1, 1.1, 1.1),
(5, 1.12, 1.12, 1.12),
(6, 2, 2, 2),
(7, 100, 100, 100),
(8, 'infinity', 'infinity', 'infinity'),
(9, 'NaN', 'NaN', 'NaN');
select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float4 range between
1 preceding and 1 following);
select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float4 range between
1 preceding and 1.1::float4 following);
select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float4 range between
'inf' preceding and 'inf' following);
select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float4 range between
'inf' preceding and 'inf' preceding);
select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float4 range between
'inf' following and 'inf' following);
select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float4 range between
1.1 preceding and 'NaN' following); -- error, NaN disallowed
select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float8 range between
1 preceding and 1 following);
select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float8 range between
1 preceding and 1.1::float8 following);
select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float8 range between
'inf' preceding and 'inf' following);
select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float8 range between
'inf' preceding and 'inf' preceding);
select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float8 range between
'inf' following and 'inf' following);
select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_float8 range between
1.1 preceding and 'NaN' following); -- error, NaN disallowed
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
1 preceding and 1 following);
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
1 preceding and 1.1::numeric following);
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
1 preceding and 1.1::float8 following); -- currently unsupported
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
'inf' preceding and 'inf' following);
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
'inf' preceding and 'inf' preceding);
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
'inf' following and 'inf' following);
select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from numerics
window w as (order by f_numeric range between
1.1 preceding and 'NaN' following); -- error, NaN disallowed
-- Test in_range for other datetime datatypes
create temp table datetimes(
id int,
f_time time,
f_timetz timetz,
f_interval interval,
f_timestamptz timestamptz,
f_timestamp timestamp
);
insert into datetimes values
(0, '10:00', '10:00 BST', '-infinity', '-infinity', '-infinity'),
(1, '11:00', '11:00 BST', '1 year', '2000-10-19 10:23:54+01', '2000-10-19 10:23:54'),
(2, '12:00', '12:00 BST', '2 years', '2001-10-19 10:23:54+01', '2001-10-19 10:23:54'),
(3, '13:00', '13:00 BST', '3 years', '2001-10-19 10:23:54+01', '2001-10-19 10:23:54'),
(4, '14:00', '14:00 BST', '4 years', '2002-10-19 10:23:54+01', '2002-10-19 10:23:54'),
(5, '15:00', '15:00 BST', '5 years', '2003-10-19 10:23:54+01', '2003-10-19 10:23:54'),
(6, '15:00', '15:00 BST', '5 years', '2004-10-19 10:23:54+01', '2004-10-19 10:23:54'),
(7, '17:00', '17:00 BST', '7 years', '2005-10-19 10:23:54+01', '2005-10-19 10:23:54'),
(8, '18:00', '18:00 BST', '8 years', '2006-10-19 10:23:54+01', '2006-10-19 10:23:54'),
(9, '19:00', '19:00 BST', '9 years', '2007-10-19 10:23:54+01', '2007-10-19 10:23:54'),
(10, '20:00', '20:00 BST', '10 years', '2008-10-19 10:23:54+01', '2008-10-19 10:23:54'),
(11, '21:00', '21:00 BST', 'infinity', 'infinity', 'infinity');
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time range between
'70 min'::interval preceding and '2 hours'::interval following);
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time desc range between
'70 min' preceding and '2 hours' following);
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time desc range between
'-70 min' preceding and '2 hours' following); -- error, negative offset disallowed
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time range between
'infinity'::interval preceding and 'infinity'::interval following);
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time range between
'infinity'::interval preceding and 'infinity'::interval preceding);
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time range between
'infinity'::interval following and 'infinity'::interval following);
select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_time range between
'-infinity'::interval following and
'infinity'::interval following); -- error, negative offset disallowed
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz range between
'70 min'::interval preceding and '2 hours'::interval following);
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz desc range between
'70 min' preceding and '2 hours' following);
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz desc range between
'70 min' preceding and '-2 hours' following); -- error, negative offset disallowed
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz range between
'infinity'::interval preceding and 'infinity'::interval following);
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz range between
'infinity'::interval preceding and 'infinity'::interval preceding);
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz range between
'infinity'::interval following and 'infinity'::interval following);
select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timetz range between
'infinity'::interval following and
'-infinity'::interval following); -- error, negative offset disallowed
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval range between
'1 year'::interval preceding and '1 year'::interval following);
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval desc range between
'1 year' preceding and '1 year' following);
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval desc range between
'-1 year' preceding and '1 year' following); -- error, negative offset disallowed
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval range between
'infinity'::interval preceding and 'infinity'::interval following);
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval range between
'infinity'::interval preceding and 'infinity'::interval preceding);
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval range between
'infinity'::interval following and 'infinity'::interval following);
select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_interval range between
'-infinity'::interval following and
'infinity'::interval following); -- error, negative offset disallowed
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz range between
'1 year'::interval preceding and '1 year'::interval following);
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz desc range between
'1 year' preceding and '1 year' following);
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz desc range between
'1 year' preceding and '-1 year' following); -- error, negative offset disallowed
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz range between
'infinity'::interval preceding and 'infinity'::interval following);
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz range between
'infinity'::interval preceding and 'infinity'::interval preceding);
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz range between
'infinity'::interval following and 'infinity'::interval following);
select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamptz range between
'-infinity'::interval following and
'infinity'::interval following); -- error, negative offset disallowed
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp range between
'1 year'::interval preceding and '1 year'::interval following);
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp desc range between
'1 year' preceding and '1 year' following);
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp desc range between
'-1 year' preceding and '1 year' following); -- error, negative offset disallowed
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp range between
'infinity'::interval preceding and 'infinity'::interval following);
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp range between
'infinity'::interval preceding and 'infinity'::interval preceding);
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp range between
'infinity'::interval following and 'infinity'::interval following);
select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
from datetimes
window w as (order by f_timestamp range between
'-infinity'::interval following and
'infinity'::interval following); -- error, negative offset disallowed
-- RANGE offset PRECEDING/FOLLOWING error cases
select sum(salary) over (order by enroll_date, salary range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
select sum(salary) over (range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
select sum(salary) over (order by depname range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
select max(enroll_date) over (order by enroll_date range between 1 preceding and 2 following
exclude ties), salary, enroll_date from empsalary;
select max(enroll_date) over (order by salary range between -1 preceding and 2 following
exclude ties), salary, enroll_date from empsalary;
select max(enroll_date) over (order by salary range between 1 preceding and -2 following
exclude ties), salary, enroll_date from empsalary;
select max(enroll_date) over (order by salary range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
select max(enroll_date) over (order by enroll_date range between '1 year'::interval preceding and '-2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
-- GROUPS tests
SELECT sum(unique1) over (order by four groups between unbounded preceding and current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between unbounded preceding and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 1 preceding and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 1 following and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between unbounded preceding and 2 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 0 preceding and 0 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following
exclude current row), unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following
exclude group), unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following
exclude ties), unique1, four
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following),unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following exclude current row), unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following exclude group), unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following exclude ties), unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
select first_value(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
lead(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
nth_value(salary, 1) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
salary, enroll_date from empsalary;
select last_value(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
lag(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
salary, enroll_date from empsalary;
select first_value(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following
exclude current row),
lead(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following exclude ties),
nth_value(salary, 1) ignore nulls over(order by enroll_date groups between 1 following and 3 following
exclude ties),
salary, enroll_date from empsalary;
select last_value(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following
exclude group),
lag(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following exclude group),
salary, enroll_date from empsalary;
-- Show differences in offset interpretation between ROWS, RANGE, and GROUPS
WITH cte (x) AS (
SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x rows between 1 preceding and 1 following);
WITH cte (x) AS (
SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x range between 1 preceding and 1 following);
WITH cte (x) AS (
SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x groups between 1 preceding and 1 following);
WITH cte (x) AS (
select 1 union all select 1 union all select 1 union all
SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x rows between 1 preceding and 1 following);
WITH cte (x) AS (
select 1 union all select 1 union all select 1 union all
SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x range between 1 preceding and 1 following);
WITH cte (x) AS (
select 1 union all select 1 union all select 1 union all
SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x groups between 1 preceding and 1 following);
-- with UNION
SELECT count(*) OVER (PARTITION BY four) FROM (SELECT * FROM tenk1 UNION ALL SELECT * FROM tenk2)s LIMIT 0;
-- check some degenerate cases
create temp table t1 (f1 int, f2 int8);
insert into t1 values (1,1),(1,2),(2,2);
select f1, sum(f1) over (partition by f1
range between 1 preceding and 1 following)
from t1 where f1 = f2; -- error, must have order by
explain (costs off)
select f1, sum(f1) over (partition by f1 order by f2
range between 1 preceding and 1 following)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1 order by f2
range between 1 preceding and 1 following)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1, f1 order by f2
range between 2 preceding and 1 preceding)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1, f2 order by f2
range between 1 following and 2 following)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1
groups between 1 preceding and 1 following)
from t1 where f1 = f2; -- error, must have order by
explain (costs off)
select f1, sum(f1) over (partition by f1 order by f2
groups between 1 preceding and 1 following)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1 order by f2
groups between 1 preceding and 1 following)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1, f1 order by f2
groups between 2 preceding and 1 preceding)
from t1 where f1 = f2;
select f1, sum(f1) over (partition by f1, f2 order by f2
groups between 1 following and 2 following)
from t1 where f1 = f2;
-- ordering by a non-integer constant is allowed
SELECT rank() OVER (ORDER BY length('abc'));
-- can't order by another window function
SELECT rank() OVER (ORDER BY rank() OVER (ORDER BY random()));
-- some other errors
SELECT * FROM empsalary WHERE row_number() OVER (ORDER BY salary) < 10;
SELECT * FROM empsalary INNER JOIN tenk1 ON row_number() OVER (ORDER BY salary) < 10;
SELECT rank() OVER (ORDER BY 1), count(*) FROM empsalary GROUP BY 1;
SELECT * FROM rank() OVER (ORDER BY random());
DELETE FROM empsalary WHERE (rank() OVER (ORDER BY random())) > 10;
DELETE FROM empsalary RETURNING rank() OVER (ORDER BY random());
SELECT count(*) OVER w FROM tenk1 WINDOW w AS (ORDER BY unique1), w AS (ORDER BY unique1);
SELECT rank() OVER (PARTITION BY four, ORDER BY ten) FROM tenk1;
SELECT count() OVER () FROM tenk1;
SELECT generate_series(1, 100) OVER () FROM empsalary;
SELECT ntile(0) OVER (ORDER BY ten), ten, four FROM tenk1;
SELECT nth_value(four, 0) OVER (ORDER BY ten), ten, four FROM tenk1;
-- filter
SELECT sum(salary), row_number() OVER (ORDER BY depname), sum(
sum(salary) FILTER (WHERE enroll_date > '2007-01-01')
) FILTER (WHERE depname <> 'sales') OVER (ORDER BY depname DESC) AS "filtered_sum",
depname
FROM empsalary GROUP BY depname;
--
-- Test SupportRequestOptimizeWindowClause's ability to de-duplicate
-- WindowClauses
--
-- Ensure WindowClause frameOptions are changed so that only a single
-- WindowAgg exists in the plan.
EXPLAIN (COSTS OFF)
SELECT
empno,
depname,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date) rn,
rank() OVER (PARTITION BY depname ORDER BY enroll_date ROWS BETWEEN
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) rnk,
dense_rank() OVER (PARTITION BY depname ORDER BY enroll_date RANGE BETWEEN
CURRENT ROW AND CURRENT ROW) drnk,
ntile(10) OVER (PARTITION BY depname ORDER BY enroll_date RANGE BETWEEN
CURRENT ROW AND UNBOUNDED FOLLOWING) nt,
percent_rank() OVER (PARTITION BY depname ORDER BY enroll_date ROWS BETWEEN
CURRENT ROW AND UNBOUNDED FOLLOWING) pr,
cume_dist() OVER (PARTITION BY depname ORDER BY enroll_date RANGE BETWEEN
CURRENT ROW AND UNBOUNDED FOLLOWING) cd
FROM empsalary;
-- Ensure WindowFuncs which cannot support their WindowClause's frameOptions
-- being changed are untouched
EXPLAIN (COSTS OFF, VERBOSE)
SELECT
empno,
depname,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date) rn,
rank() OVER (PARTITION BY depname ORDER BY enroll_date ROWS BETWEEN
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) rnk,
count(*) OVER (PARTITION BY depname ORDER BY enroll_date RANGE BETWEEN
CURRENT ROW AND CURRENT ROW) cnt
FROM empsalary;
-- Ensure the above query gives us the expected results
SELECT
empno,
depname,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date) rn,
rank() OVER (PARTITION BY depname ORDER BY enroll_date ROWS BETWEEN
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) rnk,
count(*) OVER (PARTITION BY depname ORDER BY enroll_date RANGE BETWEEN
CURRENT ROW AND CURRENT ROW) cnt
FROM empsalary;
-- Test pushdown of quals into a subquery containing window functions
-- pushdown is safe because all PARTITION BY clauses include depname:
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT depname,
sum(salary) OVER (PARTITION BY depname) depsalary,
min(salary) OVER (PARTITION BY depname || 'A', depname) depminsalary
FROM empsalary) emp
WHERE depname = 'sales';
-- pushdown is unsafe because there's a PARTITION BY clause without depname:
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT depname,
sum(salary) OVER (PARTITION BY enroll_date) enroll_salary,
min(salary) OVER (PARTITION BY depname) depminsalary
FROM empsalary) emp
WHERE depname = 'sales';
-- Test window function run conditions are properly pushed down into the
-- WindowAgg
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
row_number() OVER (ORDER BY empno) rn
FROM empsalary) emp
WHERE rn < 3;
-- The following 3 statements should result the same result.
SELECT * FROM
(SELECT empno,
row_number() OVER (ORDER BY empno) rn
FROM empsalary) emp
WHERE rn < 3;
SELECT * FROM
(SELECT empno,
row_number() OVER (ORDER BY empno) rn
FROM empsalary) emp
WHERE 3 > rn;
SELECT * FROM
(SELECT empno,
row_number() OVER (ORDER BY empno) rn
FROM empsalary) emp
WHERE 2 >= rn;
-- Ensure r <= 3 is pushed down into the run condition of the window agg
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
rank() OVER (ORDER BY salary DESC) r
FROM empsalary) emp
WHERE r <= 3;
SELECT * FROM
(SELECT empno,
salary,
rank() OVER (ORDER BY salary DESC) r
FROM empsalary) emp
WHERE r <= 3;
-- Ensure dr = 1 is converted to dr <= 1 to get all rows leading up to dr = 1
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
dense_rank() OVER (ORDER BY salary DESC) dr
FROM empsalary) emp
WHERE dr = 1;
SELECT * FROM
(SELECT empno,
salary,
dense_rank() OVER (ORDER BY salary DESC) dr
FROM empsalary) emp
WHERE dr = 1;
-- Check COUNT() and COUNT(*)
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER (ORDER BY salary DESC) c
FROM empsalary) emp
WHERE c <= 3;
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER (ORDER BY salary DESC) c
FROM empsalary) emp
WHERE c <= 3;
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(empno) OVER (ORDER BY salary DESC) c
FROM empsalary) emp
WHERE c <= 3;
SELECT * FROM
(SELECT empno,
salary,
count(empno) OVER (ORDER BY salary DESC) c
FROM empsalary) emp
WHERE c <= 3;
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER (ORDER BY salary DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) c
FROM empsalary) emp
WHERE c >= 3;
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER () c
FROM empsalary) emp
WHERE 11 <= c;
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER (ORDER BY salary DESC) c,
dense_rank() OVER (ORDER BY salary DESC) dr
FROM empsalary) emp
WHERE dr = 1;
-- Ensure we get a run condition when there's a PARTITION BY clause
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
depname,
row_number() OVER (PARTITION BY depname ORDER BY empno) rn
FROM empsalary) emp
WHERE rn < 3;
-- and ensure we get the correct results from the above plan
SELECT * FROM
(SELECT empno,
depname,
row_number() OVER (PARTITION BY depname ORDER BY empno) rn
FROM empsalary) emp
WHERE rn < 3;
-- ensure that "unused" subquery columns are not removed when the column only
-- exists in the run condition
EXPLAIN (COSTS OFF)
SELECT empno, depname FROM
(SELECT empno,
depname,
row_number() OVER (PARTITION BY depname ORDER BY empno) rn
FROM empsalary) emp
WHERE rn < 3;
-- likewise with count(empno) instead of row_number()
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
depname,
salary,
count(empno) OVER (PARTITION BY depname ORDER BY salary DESC) c
FROM empsalary) emp
WHERE c <= 3;
-- and again, check the results are what we expect.
SELECT * FROM
(SELECT empno,
depname,
salary,
count(empno) OVER (PARTITION BY depname ORDER BY salary DESC) c
FROM empsalary) emp
WHERE c <= 3;
-- Ensure we get the correct run condition when the window function is both
-- monotonically increasing and decreasing.
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
depname,
salary,
count(empno) OVER () c
FROM empsalary) emp
WHERE c = 1;
-- Try another case with a WindowFunc with a byref return type
SELECT * FROM
(SELECT row_number() OVER (PARTITION BY salary) AS rn,
lead(depname) OVER (PARTITION BY salary) || ' Department' AS n_dep
FROM empsalary) emp
WHERE rn < 1;
-- Some more complex cases with multiple window clauses
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT *,
count(salary) OVER (PARTITION BY depname || '') c1, -- w1
row_number() OVER (PARTITION BY depname) rn, -- w2
count(*) OVER (PARTITION BY depname) c2, -- w2
count(*) OVER (PARTITION BY '' || depname) c3, -- w3
ntile(2) OVER (PARTITION BY depname) nt -- w2
FROM empsalary
) e WHERE rn <= 1 AND c1 <= 3 AND nt < 2;
-- Ensure we correctly filter out all of the run conditions from each window
SELECT * FROM
(SELECT *,
count(salary) OVER (PARTITION BY depname || '') c1, -- w1
row_number() OVER (PARTITION BY depname) rn, -- w2
count(*) OVER (PARTITION BY depname) c2, -- w2
count(*) OVER (PARTITION BY '' || depname) c3, -- w3
ntile(2) OVER (PARTITION BY depname) nt -- w2
FROM empsalary
) e WHERE rn <= 1 AND c1 <= 3 AND nt < 2;
-- Ensure we remove references to reduced outer joins as nulling rels in run
-- conditions
EXPLAIN (COSTS OFF)
SELECT 1 FROM
(SELECT ntile(e2.salary) OVER (PARTITION BY e1.depname) AS c
FROM empsalary e1 LEFT JOIN empsalary e2 ON TRUE
WHERE e1.empno = e2.empno) s
WHERE s.c = 1;
-- Ensure the run condition optimization is used in cases where the WindowFunc
-- has a Var from another query level
EXPLAIN (COSTS OFF)
SELECT 1 FROM
(SELECT ntile(s1.x) OVER () AS c
FROM (SELECT (SELECT 1) AS x) AS s1) s
WHERE s.c = 1;
-- Tests to ensure we don't push down the run condition when it's not valid to
-- do so.
-- Ensure we don't push down when the frame options show that the window
-- function is not monotonically increasing
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER (ORDER BY salary DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) c
FROM empsalary) emp
WHERE c <= 3;
-- Ensure we don't push down when the window function's monotonic properties
-- don't match that of the clauses.
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(*) OVER (ORDER BY salary) c
FROM empsalary) emp
WHERE 3 <= c;
-- Ensure we don't use a run condition when there's a volatile function in the
-- WindowFunc
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count(random()) OVER (ORDER BY empno DESC) c
FROM empsalary) emp
WHERE c = 1;
-- Ensure we don't use a run condition when the WindowFunc contains subplans
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT empno,
salary,
count((SELECT 1)) OVER (ORDER BY empno DESC) c
FROM empsalary) emp
WHERE c = 1;
-- Test Sort node collapsing
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT depname,
sum(salary) OVER (PARTITION BY depname order by empno) depsalary,
min(salary) OVER (PARTITION BY depname, empno order by enroll_date) depminsalary
FROM empsalary) emp
WHERE depname = 'sales';
-- Ensure that the evaluation order of the WindowAggs results in the WindowAgg
-- with the same sort order that's required by the ORDER BY is evaluated last.
EXPLAIN (COSTS OFF)
SELECT empno,
enroll_date,
depname,
sum(salary) OVER (PARTITION BY depname order by empno) depsalary,
min(salary) OVER (PARTITION BY depname order by enroll_date) depminsalary
FROM empsalary
ORDER BY depname, empno;
-- As above, but with an adjusted ORDER BY to ensure the above plan didn't
-- perform only 2 sorts by accident.
EXPLAIN (COSTS OFF)
SELECT empno,
enroll_date,
depname,
sum(salary) OVER (PARTITION BY depname order by empno) depsalary,
min(salary) OVER (PARTITION BY depname order by enroll_date) depminsalary
FROM empsalary
ORDER BY depname, enroll_date;
SET enable_hashagg TO off;
-- Ensure we don't get a sort for both DISTINCT and ORDER BY. We expect the
-- sort for the DISTINCT to provide presorted input for the ORDER BY.
EXPLAIN (COSTS OFF)
SELECT DISTINCT
empno,
enroll_date,
depname,
sum(salary) OVER (PARTITION BY depname order by empno) depsalary,
min(salary) OVER (PARTITION BY depname order by enroll_date) depminsalary
FROM empsalary
ORDER BY depname, enroll_date;
-- As above but adjust the ORDER BY clause to help ensure the plan with the
-- minimum amount of sorting wasn't a fluke.
EXPLAIN (COSTS OFF)
SELECT DISTINCT
empno,
enroll_date,
depname,
sum(salary) OVER (PARTITION BY depname order by empno) depsalary,
min(salary) OVER (PARTITION BY depname order by enroll_date) depminsalary
FROM empsalary
ORDER BY depname, empno;
RESET enable_hashagg;
-- Test Sort node reordering
EXPLAIN (COSTS OFF)
SELECT
lead(1) OVER (PARTITION BY depname ORDER BY salary, enroll_date),
lag(1) OVER (PARTITION BY depname ORDER BY salary,enroll_date,empno)
FROM empsalary;
-- Test incremental sorting
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT depname,
empno,
salary,
enroll_date,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date) AS first_emp,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date DESC) AS last_emp
FROM empsalary) emp
WHERE first_emp = 1 OR last_emp = 1;
SELECT * FROM
(SELECT depname,
empno,
salary,
enroll_date,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date) AS first_emp,
row_number() OVER (PARTITION BY depname ORDER BY enroll_date DESC) AS last_emp
FROM empsalary) emp
WHERE first_emp = 1 OR last_emp = 1;
-- cleanup
DROP TABLE empsalary;
-- test user-defined window function with named args and default args
CREATE FUNCTION nth_value_def(val anyelement, n integer = 1) RETURNS anyelement
LANGUAGE internal WINDOW IMMUTABLE STRICT AS 'window_nth_value';
SELECT nth_value_def(n := 2, val := ten) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten) s;
SELECT nth_value_def(ten) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten) s;
--
-- Test the basic moving-aggregate machinery
--
-- create aggregates that record the series of transform calls (these are
-- intentionally not true inverses)
CREATE FUNCTION logging_sfunc_nonstrict(text, anyelement) RETURNS text AS
$$ SELECT COALESCE($1, '') || '*' || quote_nullable($2) $$
LANGUAGE SQL IMMUTABLE;
CREATE FUNCTION logging_msfunc_nonstrict(text, anyelement) RETURNS text AS
$$ SELECT COALESCE($1, '') || '+' || quote_nullable($2) $$
LANGUAGE SQL IMMUTABLE;
CREATE FUNCTION logging_minvfunc_nonstrict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '-' || quote_nullable($2) $$
LANGUAGE SQL IMMUTABLE;
CREATE AGGREGATE logging_agg_nonstrict (anyelement)
(
stype = text,
sfunc = logging_sfunc_nonstrict,
mstype = text,
msfunc = logging_msfunc_nonstrict,
minvfunc = logging_minvfunc_nonstrict
);
CREATE AGGREGATE logging_agg_nonstrict_initcond (anyelement)
(
stype = text,
sfunc = logging_sfunc_nonstrict,
mstype = text,
msfunc = logging_msfunc_nonstrict,
minvfunc = logging_minvfunc_nonstrict,
initcond = 'I',
minitcond = 'MI'
);
CREATE FUNCTION logging_sfunc_strict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '*' || quote_nullable($2) $$
LANGUAGE SQL STRICT IMMUTABLE;
CREATE FUNCTION logging_msfunc_strict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '+' || quote_nullable($2) $$
LANGUAGE SQL STRICT IMMUTABLE;
CREATE FUNCTION logging_minvfunc_strict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '-' || quote_nullable($2) $$
LANGUAGE SQL STRICT IMMUTABLE;
CREATE AGGREGATE logging_agg_strict (text)
(
stype = text,
sfunc = logging_sfunc_strict,
mstype = text,
msfunc = logging_msfunc_strict,
minvfunc = logging_minvfunc_strict
);
CREATE AGGREGATE logging_agg_strict_initcond (anyelement)
(
stype = text,
sfunc = logging_sfunc_strict,
mstype = text,
msfunc = logging_msfunc_strict,
minvfunc = logging_minvfunc_strict,
initcond = 'I',
minitcond = 'MI'
);
-- test strict and non-strict cases
SELECT
p::text || ',' || i::text || ':' || COALESCE(v::text, 'NULL') AS row,
logging_agg_nonstrict(v) over wnd as nstrict,
logging_agg_nonstrict_initcond(v) over wnd as nstrict_init,
logging_agg_strict(v::text) over wnd as strict,
logging_agg_strict_initcond(v) over wnd as strict_init
FROM (VALUES
(1, 1, NULL),
(1, 2, 'a'),
(1, 3, 'b'),
(1, 4, NULL),
(1, 5, NULL),
(1, 6, 'c'),
(2, 1, NULL),
(2, 2, 'x'),
(3, 1, 'z')
) AS t(p, i, v)
WINDOW wnd AS (PARTITION BY P ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY p, i;
-- and again, but with filter
SELECT
p::text || ',' || i::text || ':' ||
CASE WHEN f THEN COALESCE(v::text, 'NULL') ELSE '-' END as row,
logging_agg_nonstrict(v) filter(where f) over wnd as nstrict_filt,
logging_agg_nonstrict_initcond(v) filter(where f) over wnd as nstrict_init_filt,
logging_agg_strict(v::text) filter(where f) over wnd as strict_filt,
logging_agg_strict_initcond(v) filter(where f) over wnd as strict_init_filt
FROM (VALUES
(1, 1, true, NULL),
(1, 2, false, 'a'),
(1, 3, true, 'b'),
(1, 4, false, NULL),
(1, 5, false, NULL),
(1, 6, false, 'c'),
(2, 1, false, NULL),
(2, 2, true, 'x'),
(3, 1, true, 'z')
) AS t(p, i, f, v)
WINDOW wnd AS (PARTITION BY p ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY p, i;
-- test that volatile arguments disable moving-aggregate mode
SELECT
i::text || ':' || COALESCE(v::text, 'NULL') as row,
logging_agg_strict(v::text)
over wnd as inverse,
logging_agg_strict(v::text || CASE WHEN random() < 0 then '?' ELSE '' END)
over wnd as noinverse
FROM (VALUES
(1, 'a'),
(2, 'b'),
(3, 'c')
) AS t(i, v)
WINDOW wnd AS (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY i;
SELECT
i::text || ':' || COALESCE(v::text, 'NULL') as row,
logging_agg_strict(v::text) filter(where true)
over wnd as inverse,
logging_agg_strict(v::text) filter(where random() >= 0)
over wnd as noinverse
FROM (VALUES
(1, 'a'),
(2, 'b'),
(3, 'c')
) AS t(i, v)
WINDOW wnd AS (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY i;
-- test that non-overlapping windows don't use inverse transitions
SELECT
logging_agg_strict(v::text) OVER wnd
FROM (VALUES
(1, 'a'),
(2, 'b'),
(3, 'c')
) AS t(i, v)
WINDOW wnd AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND CURRENT ROW)
ORDER BY i;
-- test that returning NULL from the inverse transition functions
-- restarts the aggregation from scratch. The second aggregate is supposed
-- to test cases where only some aggregates restart, the third one checks
-- that one aggregate restarting doesn't cause others to restart.
CREATE FUNCTION sum_int_randrestart_minvfunc(int4, int4) RETURNS int4 AS
$$ SELECT CASE WHEN random() < 0.2 THEN NULL ELSE $1 - $2 END $$
LANGUAGE SQL STRICT;
CREATE AGGREGATE sum_int_randomrestart (int4)
(
stype = int4,
sfunc = int4pl,
mstype = int4,
msfunc = int4pl,
minvfunc = sum_int_randrestart_minvfunc
);
WITH
vs AS (
SELECT i, (random() * 100)::int4 AS v
FROM generate_series(1, 100) AS i
),
sum_following AS (
SELECT i, SUM(v) OVER
(ORDER BY i DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS s
FROM vs
)
SELECT DISTINCT
sum_following.s = sum_int_randomrestart(v) OVER fwd AS eq1,
-sum_following.s = sum_int_randomrestart(-v) OVER fwd AS eq2,
100*3+(vs.i-1)*3 = length(logging_agg_nonstrict(''::text) OVER fwd) AS eq3
FROM vs
JOIN sum_following ON sum_following.i = vs.i
WINDOW fwd AS (
ORDER BY vs.i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
);
--
-- Test various built-in aggregates that have moving-aggregate support
--
-- test inverse transition functions handle NULLs properly
SELECT i,AVG(v::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,AVG(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,AVG(v::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,AVG(v::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1.5),(2,2.5),(3,NULL),(4,NULL)) t(i,v);
SELECT i,AVG(v::interval) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,'1 sec'),(2,'2 sec'),(3,NULL),(4,NULL)) t(i,v);
-- moving aggregates over infinite intervals
SELECT x
,avg(x) OVER(ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING ) as curr_next_avg
,avg(x) OVER(ROWS BETWEEN 1 PRECEDING AND CURRENT ROW ) as prev_curr_avg
,sum(x) OVER(ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING ) as curr_next_sum
,sum(x) OVER(ROWS BETWEEN 1 PRECEDING AND CURRENT ROW ) as prev_curr_sum
FROM (VALUES (NULL::interval),
('infinity'::interval),
('-2147483648 days -2147483648 months -9223372036854775807 usecs'), -- extreme interval value
('-infinity'::interval),
('2147483647 days 2147483647 months 9223372036854775806 usecs'), -- extreme interval value
('infinity'::interval),
('6 days'::interval),
('7 days'::interval),
(NULL::interval),
('-infinity'::interval)) v(x);
--should fail.
SELECT x, avg(x) OVER(ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING)
FROM (VALUES (NULL::interval),
('3 days'::interval),
('infinity'::timestamptz - now()),
('6 days'::interval),
('-infinity'::interval)) v(x);
--should fail.
SELECT x, sum(x) OVER(ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING)
FROM (VALUES (NULL::interval),
('3 days'::interval),
('infinity'::timestamptz - now()),
('6 days'::interval),
('-infinity'::interval)) v(x);
SELECT i,SUM(v::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::money) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,'1.10'),(2,'2.20'),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::interval) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,'1 sec'),(2,'2 sec'),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1.1),(2,2.2),(3,NULL),(4,NULL)) t(i,v);
SELECT SUM(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1.01),(2,2),(3,3)) v(i,n);
SELECT i,COUNT(v) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,COUNT(*) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT VAR_POP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_POP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_POP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_POP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_SAMP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_SAMP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_SAMP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VAR_SAMP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VARIANCE(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VARIANCE(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VARIANCE(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT VARIANCE(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT STDDEV_POP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_POP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_POP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_POP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_SAMP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_SAMP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_SAMP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV_SAMP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
SELECT STDDEV(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT STDDEV(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT STDDEV(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
SELECT STDDEV(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
-- test that inverse transition functions work with various frame options
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND CURRENT ROW)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,3),(4,4)) t(i,v);
-- ensure aggregate over numeric properly recovers from NaN values
SELECT a, b,
SUM(b) OVER(ORDER BY A ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
FROM (VALUES(1,1::numeric),(2,2),(3,'NaN'),(4,3),(5,4)) t(a,b);
-- It might be tempting for someone to add an inverse trans function for
-- float and double precision. This should not be done as it can give incorrect
-- results. This test should fail if anyone ever does this without thinking too
-- hard about it.
SELECT to_char(SUM(n::float8) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING),'999999999999999999999D9')
FROM (VALUES(1,1e20),(2,1)) n(i,n);
SELECT i, b, bool_and(b) OVER w, bool_or(b) OVER w
FROM (VALUES (1,true), (2,true), (3,false), (4,false), (5,true)) v(i,b)
WINDOW w AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING);
--
-- Test WindowAgg costing takes into account the number of rows that need to
-- be fetched before the first row can be output.
--
-- Ensure we get a cheap start up plan as the WindowAgg can output the first
-- row after reading 1 row from the join.
EXPLAIN (COSTS OFF)
SELECT COUNT(*) OVER (ORDER BY t1.unique1)
FROM tenk1 t1 INNER JOIN tenk1 t2 ON t1.unique1 = t2.tenthous
LIMIT 1;
-- Ensure we get a cheap total plan. Lack of ORDER BY in the WindowClause
-- means that all rows must be read from the join, so a cheap startup plan
-- isn't a good choice.
EXPLAIN (COSTS OFF)
SELECT COUNT(*) OVER ()
FROM tenk1 t1 INNER JOIN tenk1 t2 ON t1.unique1 = t2.tenthous
WHERE t2.two = 1
LIMIT 1;
-- Ensure we get a cheap total plan. This time use UNBOUNDED FOLLOWING, which
-- needs to read all join rows to output the first WindowAgg row.
EXPLAIN (COSTS OFF)
SELECT COUNT(*) OVER (ORDER BY t1.unique1 ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
FROM tenk1 t1 INNER JOIN tenk1 t2 ON t1.unique1 = t2.tenthous
LIMIT 1;
-- Ensure we get a cheap total plan. This time use 10000 FOLLOWING so we need
-- to read all join rows.
EXPLAIN (COSTS OFF)
SELECT COUNT(*) OVER (ORDER BY t1.unique1 ROWS BETWEEN UNBOUNDED PRECEDING AND 10000 FOLLOWING)
FROM tenk1 t1 INNER JOIN tenk1 t2 ON t1.unique1 = t2.tenthous
LIMIT 1;
-- Tests for problems with failure to walk or mutate expressions
-- within window frame clauses.
-- test walker (fails with collation error if expressions are not walked)
SELECT array_agg(i) OVER w
FROM generate_series(1,5) i
WINDOW w AS (ORDER BY i ROWS BETWEEN (('foo' < 'foobar')::integer) PRECEDING AND CURRENT ROW);
-- test mutator (fails when inlined if expressions are not mutated)
CREATE FUNCTION pg_temp.f(group_size BIGINT) RETURNS SETOF integer[]
AS $$
SELECT array_agg(s) OVER w
FROM generate_series(1,5) s
WINDOW w AS (ORDER BY s ROWS BETWEEN CURRENT ROW AND GROUP_SIZE FOLLOWING)
$$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
-- IGNORE NULLS tests
CREATE TEMPORARY TABLE planets (
name text,
distance text,
orbit integer
);
INSERT INTO planets VALUES
('mercury', 'close', 88),
('venus', 'close', 224),
('earth', 'close', NULL),
('mars', 'close', NULL),
('jupiter', 'close', 4332),
('saturn', 'far', 24491),
('uranus', 'far', NULL),
('neptune', 'far', 60182),
('pluto', 'far', 90560),
('xyzzy', 'far', NULL);
-- test ruleutils
CREATE VIEW planets_view AS
SELECT name,
orbit,
lag(orbit) OVER w AS lag,
lag(orbit) RESPECT NULLS OVER w AS lag_respect,
lag(orbit) IGNORE NULLS OVER w AS lag_ignore
FROM planets
WINDOW w AS (ORDER BY name)
;
SELECT pg_get_viewdef('planets_view');
-- lag
SELECT name,
orbit,
lag(orbit) OVER w AS lag,
lag(orbit) RESPECT NULLS OVER w AS lag_respect,
lag(orbit) IGNORE NULLS OVER w AS lag_ignore
FROM planets
WINDOW w AS (ORDER BY name)
;
-- lead
SELECT name,
orbit,
lead(orbit) OVER w AS lead,
lead(orbit) RESPECT NULLS OVER w AS lead_respect,
lead(orbit) IGNORE NULLS OVER w AS lead_ignore
FROM planets
WINDOW w AS (ORDER BY name)
;
-- first_value
SELECT name,
orbit,
first_value(orbit) RESPECT NULLS OVER w1,
first_value(orbit) IGNORE NULLS OVER w1,
first_value(orbit) RESPECT NULLS OVER w2,
first_value(orbit) IGNORE NULLS OVER w2
FROM planets
WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
;
-- nth_value
SELECT name,
orbit,
nth_value(orbit, 2) RESPECT NULLS OVER w1,
nth_value(orbit, 2) IGNORE NULLS OVER w1,
nth_value(orbit, 2) RESPECT NULLS OVER w2,
nth_value(orbit, 2) IGNORE NULLS OVER w2
FROM planets
WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
;
-- last_value
SELECT name,
orbit,
last_value(orbit) RESPECT NULLS OVER w1,
last_value(orbit) IGNORE NULLS OVER w1,
last_value(orbit) RESPECT NULLS OVER w2,
last_value(orbit) IGNORE NULLS OVER w2
FROM planets
WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
;
-- exclude current row
SELECT name,
orbit,
first_value(orbit) IGNORE NULLS OVER w,
last_value(orbit) IGNORE NULLS OVER w,
nth_value(orbit, 2) IGNORE NULLS OVER w,
lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
FROM planets
WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
;
-- valid and invalid functions
SELECT sum(orbit) OVER () FROM planets; -- succeeds
SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
SELECT row_number() OVER () FROM planets; -- succeeds
SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
-- test two consecutive nulls
update planets set orbit=null where name='jupiter';
SELECT name,
orbit,
first_value(orbit) IGNORE NULLS OVER w,
last_value(orbit) IGNORE NULLS OVER w,
nth_value(orbit, 2) IGNORE NULLS OVER w,
lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
FROM planets
WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
;
-- test partitions
SELECT name,
distance,
orbit,
first_value(orbit) IGNORE NULLS OVER w,
last_value(orbit) IGNORE NULLS OVER w,
nth_value(orbit, 2) IGNORE NULLS OVER w,
lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
FROM planets
WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
;
-- nth_value without nulls
SELECT x,
nth_value(x,2) IGNORE NULLS OVER w
FROM generate_series(1,5) g(x)
WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
SELECT x,
nth_value(x,2) IGNORE NULLS OVER w
FROM generate_series(1,5) g(x)
WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
--cleanup
DROP TABLE planets CASCADE;
Attachments:
[text/plain] insert_ignore_nulls.sh (1011B, ../../[email protected]/2-insert_ignore_nulls.sh)
download | inline:
#! /bin/sh
# Script to generate test cases for IGNORE NULLS option of some window
# functions (lead, lag, first_value, last_value, and nth_value) in the
# regression test SQL (window.sql). This script supposed to read
# window.sql from stdin and inserts "ignore null" before "over" on all
# occurrences of the window functions that accept IGNORE NULLS, then
# print it to stdout. Since currently the data set used for the
# existing regression test (window.sql) does not include NULLs, the
# test result using the results of this script is expected be
# identical to the existing results of existing window.sql (of course
# except the SQL statements).
sed -r \
-e "s/lead\(([^)]*)\) over/lead(\1) ignore nulls over/g" \
-e "s/lag\(([^)]*)\) over/lag(\1) ignore nulls over/g" \
-e "s/first_value\(([^)]*)\) over/first_value(\1) ignore nulls over/g" \
-e "s/last_value\(([^)]*)\) over/last_value(\1) ignore nulls over/g" \
-e "s/nth_value\(([^)]*)\) over/nth_value(\1) ignore nulls over/g"
[text/plain] window.sql (78.3K, ../../[email protected]/3-window.sql)
download
[text/x-patch] window.diff (20.7K, ../../[email protected]/4-window.diff)
download | inline diff:
662c662
< select x, lag(x, 1) over (order by x), lead(x, 3) over (order by x)
---
> select x, lag(x, 1) ignore nulls over (order by x), lead(x, 3) ignore nulls over (order by x)
681c681
< last_value(ten) over (partition by four order by ten)
---
> last_value(ten) ignore nulls over (partition by four order by ten)
709c709
< last_value(ten) over (partition by four order by ten range between unbounded preceding and current row)
---
> last_value(ten) ignore nulls over (partition by four order by ten range between unbounded preceding and current row)
737c737
< last_value(ten) over (partition by four order by ten range between unbounded preceding and unbounded following)
---
> last_value(ten) ignore nulls over (partition by four order by ten range between unbounded preceding and unbounded following)
765c765
< last_value(ten/4) over (partition by four order by ten/4 range between unbounded preceding and current row)
---
> last_value(ten/4) ignore nulls over (partition by four order by ten/4 range between unbounded preceding and current row)
793c793
< last_value(ten/4) over (partition by four order by ten/4 rows between unbounded preceding and current row)
---
> last_value(ten/4) ignore nulls over (partition by four order by ten/4 rows between unbounded preceding and current row)
938c938
< SELECT first_value(unique1) over (ORDER BY four rows between current row and 2 following exclude current row),
---
> SELECT first_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude current row),
955c955
< SELECT first_value(unique1) over (ORDER BY four rows between current row and 2 following exclude group),
---
> SELECT first_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude group),
972c972
< SELECT first_value(unique1) over (ORDER BY four rows between current row and 2 following exclude ties),
---
> SELECT first_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude ties),
989c989
< SELECT last_value(unique1) over (ORDER BY four rows between current row and 2 following exclude current row),
---
> SELECT last_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude current row),
1006c1006
< SELECT last_value(unique1) over (ORDER BY four rows between current row and 2 following exclude group),
---
> SELECT last_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude group),
1023c1023
< SELECT last_value(unique1) over (ORDER BY four rows between current row and 2 following exclude ties),
---
> SELECT last_value(unique1) ignore nulls over (ORDER BY four rows between current row and 2 following exclude ties),
1159,1161c1159,1161
< SELECT first_value(unique1) over w,
< nth_value(unique1, 2) over w AS nth_2,
< last_value(unique1) over w, unique1, four
---
> SELECT first_value(unique1) ignore nulls over w,
> nth_value(unique1, 2) ignore nulls over w AS nth_2,
> last_value(unique1) ignore nulls over w, unique1, four
1631,1633c1631,1633
< select first_value(salary) over(order by salary range between 1000 preceding and 1000 following),
< lead(salary) over(order by salary range between 1000 preceding and 1000 following),
< nth_value(salary, 1) over(order by salary range between 1000 preceding and 1000 following),
---
> select first_value(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
> lead(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
> nth_value(salary, 1) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
1649,1650c1649,1650
< select last_value(salary) over(order by salary range between 1000 preceding and 1000 following),
< lag(salary) over(order by salary range between 1000 preceding and 1000 following),
---
> select last_value(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
> lag(salary) ignore nulls over(order by salary range between 1000 preceding and 1000 following),
1666c1666
< select first_value(salary) over(order by salary range between 1000 following and 3000 following
---
> select first_value(salary) ignore nulls over(order by salary range between 1000 following and 3000 following
1668,1669c1668,1669
< lead(salary) over(order by salary range between 1000 following and 3000 following exclude ties),
< nth_value(salary, 1) over(order by salary range between 1000 following and 3000 following
---
> lead(salary) ignore nulls over(order by salary range between 1000 following and 3000 following exclude ties),
> nth_value(salary, 1) ignore nulls over(order by salary range between 1000 following and 3000 following
1686c1686
< select last_value(salary) over(order by salary range between 1000 following and 3000 following
---
> select last_value(salary) ignore nulls over(order by salary range between 1000 following and 3000 following
1688c1688
< lag(salary) over(order by salary range between 1000 following and 3000 following exclude group),
---
> lag(salary) ignore nulls over(order by salary range between 1000 following and 3000 following exclude group),
1704c1704
< select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1706c1706
< last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following),
---
> last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following),
1722c1722
< select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1724c1724
< last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1741c1741
< select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1743c1743
< last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1760c1760
< select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> select first_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1762c1762
< last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
---
> last_value(salary) ignore nulls over(order by enroll_date range between unbounded preceding and '1 year'::interval following
1781,1782c1781,1782
< first_value(y) over w,
< last_value(y) over w
---
> first_value(y) ignore nulls over w,
> last_value(y) ignore nulls over w
1801,1802c1801,1802
< first_value(y) over w,
< last_value(y) over w
---
> first_value(y) ignore nulls over w,
> last_value(y) ignore nulls over w
1821,1822c1821,1822
< first_value(y) over w,
< last_value(y) over w
---
> first_value(y) ignore nulls over w,
> last_value(y) ignore nulls over w
1841,1842c1841,1842
< first_value(y) over w,
< last_value(y) over w
---
> first_value(y) ignore nulls over w,
> last_value(y) ignore nulls over w
2001c2001
< select x, last_value(x) over (order by x::smallint range between current row and 2147450884 following)
---
> select x, last_value(x) ignore nulls over (order by x::smallint range between current row and 2147450884 following)
2010c2010
< select x, last_value(x) over (order by x::smallint desc range between current row and 2147450885 following)
---
> select x, last_value(x) ignore nulls over (order by x::smallint desc range between current row and 2147450885 following)
2019c2019
< select x, last_value(x) over (order by x range between current row and 4 following)
---
> select x, last_value(x) ignore nulls over (order by x range between current row and 4 following)
2028c2028
< select x, last_value(x) over (order by x desc range between current row and 5 following)
---
> select x, last_value(x) ignore nulls over (order by x desc range between current row and 5 following)
2037c2037
< select x, last_value(x) over (order by x range between current row and 4 following)
---
> select x, last_value(x) ignore nulls over (order by x range between current row and 4 following)
2046c2046
< select x, last_value(x) over (order by x desc range between current row and 5 following)
---
> select x, last_value(x) ignore nulls over (order by x desc range between current row and 5 following)
2073c2073
< select id, f_float4, first_value(id) over w, last_value(id) over w
---
> select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2091c2091
< select id, f_float4, first_value(id) over w, last_value(id) over w
---
> select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2109c2109
< select id, f_float4, first_value(id) over w, last_value(id) over w
---
> select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2127c2127
< select id, f_float4, first_value(id) over w, last_value(id) over w
---
> select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2145c2145
< select id, f_float4, first_value(id) over w, last_value(id) over w
---
> select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2163c2163
< select id, f_float4, first_value(id) over w, last_value(id) over w
---
> select id, f_float4, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2168c2168
< select id, f_float8, first_value(id) over w, last_value(id) over w
---
> select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2186c2186
< select id, f_float8, first_value(id) over w, last_value(id) over w
---
> select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2204c2204
< select id, f_float8, first_value(id) over w, last_value(id) over w
---
> select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2222c2222
< select id, f_float8, first_value(id) over w, last_value(id) over w
---
> select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2240c2240
< select id, f_float8, first_value(id) over w, last_value(id) over w
---
> select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2258c2258
< select id, f_float8, first_value(id) over w, last_value(id) over w
---
> select id, f_float8, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2263c2263
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2281c2281
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2299c2299
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2307c2307
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2325c2325
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2343c2343
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2361c2361
< select id, f_numeric, first_value(id) over w, last_value(id) over w
---
> select id, f_numeric, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2388c2388
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2408c2408
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2428c2428
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2433c2433
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2453c2453
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2473c2473
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2493c2493
< select id, f_time, first_value(id) over w, last_value(id) over w
---
> select id, f_time, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2499c2499
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2519c2519
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2539c2539
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2544c2544
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2564c2564
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2584c2584
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2604c2604
< select id, f_timetz, first_value(id) over w, last_value(id) over w
---
> select id, f_timetz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2610c2610
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2630c2630
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2650c2650
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2655c2655
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2675c2675
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2695c2695
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2715c2715
< select id, f_interval, first_value(id) over w, last_value(id) over w
---
> select id, f_interval, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2721c2721
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2741c2741
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2761c2761
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2766c2766
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2786c2786
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2806c2806
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2826c2826
< select id, f_timestamptz, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamptz, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2832c2832
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2852c2852
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2872c2872
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2877c2877
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2897c2897
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2917c2917
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
2937c2937
< select id, f_timestamp, first_value(id) over w, last_value(id) over w
---
> select id, f_timestamp, first_value(id) ignore nulls over w, last_value(id) ignore nulls over w
3253,3255c3253,3255
< select first_value(salary) over(order by enroll_date groups between 1 preceding and 1 following),
< lead(salary) over(order by enroll_date groups between 1 preceding and 1 following),
< nth_value(salary, 1) over(order by enroll_date groups between 1 preceding and 1 following),
---
> select first_value(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
> lead(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
> nth_value(salary, 1) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
3271,3272c3271,3272
< select last_value(salary) over(order by enroll_date groups between 1 preceding and 1 following),
< lag(salary) over(order by enroll_date groups between 1 preceding and 1 following),
---
> select last_value(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
> lag(salary) ignore nulls over(order by enroll_date groups between 1 preceding and 1 following),
3288c3288
< select first_value(salary) over(order by enroll_date groups between 1 following and 3 following
---
> select first_value(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following
3290,3291c3290,3291
< lead(salary) over(order by enroll_date groups between 1 following and 3 following exclude ties),
< nth_value(salary, 1) over(order by enroll_date groups between 1 following and 3 following
---
> lead(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following exclude ties),
> nth_value(salary, 1) ignore nulls over(order by enroll_date groups between 1 following and 3 following
3308c3308
< select last_value(salary) over(order by enroll_date groups between 1 following and 3 following
---
> select last_value(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following
3310c3310
< lag(salary) over(order by enroll_date groups between 1 following and 3 following exclude group),
---
> lag(salary) ignore nulls over(order by enroll_date groups between 1 following and 3 following exclude group),
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-07-25 07:49 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-07-25 07:49 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: [email protected]
Attached are the v17 patches for adding RESPECT/IGNORE NULLS options
defined in the standard to some window functions. FROM FIRST/LAST
options are not considered in the patch (yet).
This time I split the patch into 6
patches for reviewer's convenience. Also each patch has a short commit
message to explain the patch.
0001: parse and analysis
0002: rewriter
0003: planner
0004: executor
0005: documents
0006: tests
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v17-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch (8.3K, ../../[email protected]/2-v17-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch)
download | inline diff:
From b88424b08f6767b3902048b009fb23fe05301fe0 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 25 Jul 2025 16:22:47 +0900
Subject: [PATCH v17 1/6] Modify parse/analysis modules to accept
RESPECT/IGNORE NULLS option.
Following changes have been made to parse//analysis modules.
- add IGNORE_P RESPECT_P keywords
- add "null_treatment" to func_expr after filter_clause and before
over_clause as the SQL standard requries. null_treatment is resolved
to either PARSER_IGNORE_NULLS, PARSER_RESPECT_NULLS or
NO_NULLTREATMENT
- add "ignore_nulls" to WindowFunc and FuncCall
---
src/backend/parser/gram.y | 19 ++++++++++++++-----
src/backend/parser/parse_func.c | 9 +++++++++
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +++++++++++++
src/include/parser/kwlist.h | 2 ++
5 files changed, 39 insertions(+), 5 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 73345bb3c70..5136e0992d2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15803,7 +15803,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15836,7 +15836,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16432,6 +16433,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17858,6 +17865,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17976,6 +17984,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 86a236bd58b..9ec6d79d834 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
[application/octet-stream] v17-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch (977B, ../../[email protected]/3-v17-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch)
download | inline diff:
From 37fddc904ea66153e2ead1d40c2337854d61109f Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 25 Jul 2025 16:22:47 +0900
Subject: [PATCH v17 2/6] Modify get_windowfunc_expr_helper to handle IGNORE
NULLS option.
---
src/backend/utils/adt/ruleutils.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
--
2.25.1
[application/octet-stream] v17-0003-Modify-eval_const_expressions_mutator-to-handle-.patch (853B, ../../[email protected]/4-v17-0003-Modify-eval_const_expressions_mutator-to-handle-.patch)
download | inline diff:
From 2a1eac74e7b1fbb4e53bf617c12e940f9bfbedb7 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 25 Jul 2025 16:22:47 +0900
Subject: [PATCH v17 3/6] Modify eval_const_expressions_mutator to handle
IGNORE NULLS option.
---
src/backend/optimizer/util/clauses.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 6f0b338d2cd..92e0f14fa54 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2576,6 +2576,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
--
2.25.1
[application/octet-stream] v17-0004-Modify-executor-and-window-functions-to-handle-I.patch (22.0K, ../../[email protected]/5-v17-0004-Modify-executor-and-window-functions-to-handle-I.patch)
download | inline diff:
From 50ee381124728a89440c08a4ce16b9014ac6e2fc Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 25 Jul 2025 16:22:47 +0900
Subject: [PATCH v17 4/6] Modify executor and window functions to handle IGNORE
NULLS.
Following changes have been made to executor and window functions
modules.
- New window function API WinCheckAndInitializeNullTreatment() is
added. Window functions should call this to express if they accept a
null treatment clause or not. If they do not, an error is raised in
this function. Built-in window functions are modified to call it.
- WinGetFuncArgInPartition is modified to handle IGNORE NULLS.
- WinGetFuncArgInFrame is modified to handle IGNORE NULLS. The actual
workhorse for this is ignorenulls_getfuncarginframe.
- While searching not null rows, to not scan tuples over and over
again, "notnull_info" cache module added. This holds 2-bit info for
each tuple, to keep whether the tuple has already been checked if it
is not yet checked, null or not null. The notnull_info is added to
WindowObjectData.
---
src/backend/executor/nodeWindowAgg.c | 458 +++++++++++++++++++++++++--
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/windowapi.h | 6 +
3 files changed, 439 insertions(+), 35 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..85883958e8e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,13 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ /*
+ * Null treatment options. One of: NO_NULLTREATMENT, PARSER_IGNORE_NULLS,
+ * PARSER_RESPECT_NULLS or IGNORE_NULLS.
+ */
+ int ignore_nulls;
} WindowObjectData;
/*
@@ -96,6 +103,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -182,8 +190,8 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
- TupleTableSlot *slot);
+static int row_is_in_frame(WindowObject winobj, int64 pos,
+ TupleTableSlot *slot, bool fetch_tuple);
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -198,6 +206,33 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -942,7 +977,7 @@ eval_windowaggregates(WindowAggState *winstate)
* Exit loop if no more rows can be in frame. Skip aggregation if
* current row is not in frame but there might be more in the frame.
*/
- ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
+ ret = row_is_in_frame(agg_winobj, winstate->aggregatedupto, agg_row_slot, false);
if (ret < 0)
break;
if (ret == 0)
@@ -1263,6 +1298,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -1412,8 +1452,8 @@ release_partition(WindowAggState *winstate)
* to our window framing rule
*
* The caller must have already determined that the row is in the partition
- * and fetched it into a slot. This function just encapsulates the framing
- * rules.
+ * and fetched it into a slot if fetch_tuple is false.
+.* This function just encapsulates the framing rules.
*
* Returns:
* -1, if the row is out of frame and no succeeding rows can be in frame
@@ -1423,8 +1463,9 @@ release_partition(WindowAggState *winstate)
* May clobber winstate->temp_slot_2.
*/
static int
-row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
+row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot, bool fetch_tuple)
{
+ WindowAggState *winstate = winobj->winstate;
int frameOptions = winstate->frameOptions;
Assert(pos >= 0); /* else caller error */
@@ -1453,9 +1494,13 @@ row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
else if (frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
{
/* following row that is not peer is out of frame */
- if (pos > winstate->currentpos &&
- !are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
- return -1;
+ if (pos > winstate->currentpos)
+ {
+ if (fetch_tuple)
+ window_gettupleslot(winobj, pos, slot);
+ if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
+ return -1;
+ }
}
else
Assert(false);
@@ -2619,14 +2664,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2727,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3264,290 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winobj, abs_pos, slot, true);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3706,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3748,67 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
- {
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ if (abs_pos < 0)
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ datum = 0;
+ break;
+ }
+
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3860,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3624,7 +4012,7 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
goto out_of_frame;
/* The code above does not detect all out-of-frame cases, so check */
- if (row_is_in_frame(winstate, abs_pos, slot) <= 0)
+ if (row_is_in_frame(winobj, abs_pos, slot, false) <= 0)
goto out_of_frame;
if (isout)
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
--
2.25.1
[application/octet-stream] v17-0005-Modify-documents-to-add-null-treatment-clause.patch (8.5K, ../../[email protected]/6-v17-0005-Modify-documents-to-add-null-treatment-clause.patch)
download | inline diff:
From f90eac87f324fd4888bfca4ced7f11b525275475 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 25 Jul 2025 16:22:47 +0900
Subject: [PATCH v17 5/6] Modify documents to add null treatment clause.
---
doc/src/sgml/func.sgml | 38 +++++++++++++++++-----------
doc/src/sgml/syntax.sgml | 10 +++++---
src/backend/catalog/sql_features.txt | 2 +-
3 files changed, 30 insertions(+), 20 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index de5b5929ee0..4106e1768d8 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23543,7 +23543,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23568,7 +23568,7 @@ SELECT count(*) FROM sometable;
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -23591,7 +23591,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23605,7 +23605,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23619,7 +23619,7 @@ SELECT count(*) FROM sometable;
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -23668,18 +23668,26 @@ SELECT count(*) FROM sometable;
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
diff --git a/doc/src/sgml/syntax.sgml b/doc/src/sgml/syntax.sgml
index 916189a7d68..237d7306fe8 100644
--- a/doc/src/sgml/syntax.sgml
+++ b/doc/src/sgml/syntax.sgml
@@ -1834,8 +1834,8 @@ FROM generate_series(1,10) AS s(i);
The syntax of a window function call is one of the following:
<synopsis>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
-<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
+<replaceable>function_name</replaceable> (<optional><replaceable>expression</replaceable> <optional>, <replaceable>expression</replaceable> ... </optional></optional>) <optional>null treatment</optional> [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER <replaceable>window_name</replaceable>
<replaceable>function_name</replaceable> ( * ) [ FILTER ( WHERE <replaceable>filter_clause</replaceable> ) ] OVER ( <replaceable class="parameter">window_definition</replaceable> )
</synopsis>
@@ -1873,7 +1873,9 @@ EXCLUDE NO OTHERS
<para>
Here, <replaceable>expression</replaceable> represents any value
- expression that does not itself contain window function calls.
+ expression that does not itself contain window function calls. Some
+ non-aggregate functions allow a <literal>null treatment</literal> clause,
+ described in <xref linkend="functions-window"/>.
</para>
<para>
@@ -2048,7 +2050,7 @@ EXCLUDE NO OTHERS
<para>
The built-in window functions are described in <xref
- linkend="functions-window-table"/>. Other window functions can be added by
+ linkend="functions-window-table"/>. Other window functions can be added by
the user. Also, any built-in or user-defined general-purpose or
statistical aggregate can be used as a window function. (Ordered-set
and hypothetical-set aggregates cannot presently be used as window functions.)
diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt
index ebe85337c28..3a8ad201607 100644
--- a/src/backend/catalog/sql_features.txt
+++ b/src/backend/catalog/sql_features.txt
@@ -518,7 +518,7 @@ T612 Advanced OLAP operations YES
T613 Sampling YES
T614 NTILE function YES
T615 LEAD and LAG functions YES
-T616 Null treatment option for LEAD and LAG functions NO
+T616 Null treatment option for LEAD and LAG functions YES
T617 FIRST_VALUE and LAST_VALUE functions YES
T618 NTH_VALUE function NO function exists, but some options missing
T619 Nested window functions NO
--
2.25.1
[application/octet-stream] v17-0006-Modify-window-function-regression-tests-to-test-.patch (18.4K, ../../[email protected]/7-v17-0006-Modify-window-function-regression-tests-to-test-.patch)
download | inline diff:
From 77b39553d575c8e24b99e165a5d315a717051638 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 25 Jul 2025 16:22:48 +0900
Subject: [PATCH v17 6/6] Modify window function regression tests to test null
treatment clause.
---
src/test/regress/expected/window.out | 311 +++++++++++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++++++
2 files changed, 458 insertions(+)
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-08-16 09:33 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-08-16 09:33 UTC (permalink / raw)
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; +Cc: [email protected]
Attached are the v18 patches for adding RESPECT/IGNORE NULLS options
to some window functions. Recent changes to doc/src/sgml/func.sgml
required v17 to be rebased. Other than that, nothing has been changed.
Oliver, do you have any comments on the patches?
> Attached are the v17 patches for adding RESPECT/IGNORE NULLS options
> defined in the standard to some window functions. FROM FIRST/LAST
> options are not considered in the patch (yet).
>
> This time I split the patch into 6
> patches for reviewer's convenience. Also each patch has a short commit
> message to explain the patch.
>
> 0001: parse and analysis
> 0002: rewriter
> 0003: planner
> 0004: executor
> 0005: documents
> 0006: tests
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v18-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch (8.3K, ../../[email protected]/2-v18-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch)
download | inline diff:
From 4db6a5d9a69b82af870232233e4ef77ac8b4153e Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 16 Aug 2025 18:29:23 +0900
Subject: [PATCH v18 1/6] Modify parse/analysis modules to accept
RESPECT/IGNORE NULLS option.
Following changes have been made to parse//analysis modules.
- add IGNORE_P RESPECT_P keywords
- add "null_treatment" to func_expr after filter_clause and before
over_clause as the SQL standard requries. null_treatment is resolved
to either PARSER_IGNORE_NULLS, PARSER_RESPECT_NULLS or
NO_NULLTREATMENT
- add "ignore_nulls" to WindowFunc and FuncCall
---
src/backend/parser/gram.y | 19 ++++++++++++++-----
src/backend/parser/parse_func.c | 9 +++++++++
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +++++++++++++
src/include/parser/kwlist.h | 2 ++
5 files changed, 39 insertions(+), 5 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index db43034b9db..e7847a7b7a5 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15791,7 +15791,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15824,7 +15824,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16420,6 +16421,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17846,6 +17853,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17964,6 +17972,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 583bbbf232f..3772c514b1e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -514,6 +515,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -834,6 +842,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 86a236bd58b..9ec6d79d834 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
[application/octet-stream] v18-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch (977B, ../../[email protected]/3-v18-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch)
download | inline diff:
From 4ca53852c6af568fd5aca25f2dc01593338fc561 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 16 Aug 2025 18:29:23 +0900
Subject: [PATCH v18 2/6] Modify get_windowfunc_expr_helper to handle IGNORE
NULLS option.
---
src/backend/utils/adt/ruleutils.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3d6e6bdbfd2..4e837d2afea 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
--
2.25.1
[application/octet-stream] v18-0003-Modify-eval_const_expressions_mutator-to-handle-.patch (853B, ../../[email protected]/4-v18-0003-Modify-eval_const_expressions_mutator-to-handle-.patch)
download | inline diff:
From 8a95f6fd4ab49e97cbd54d55278fe91bdcf30e54 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 16 Aug 2025 18:29:23 +0900
Subject: [PATCH v18 3/6] Modify eval_const_expressions_mutator to handle
IGNORE NULLS option.
---
src/backend/optimizer/util/clauses.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 6f0b338d2cd..92e0f14fa54 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2576,6 +2576,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
--
2.25.1
[application/octet-stream] v18-0004-Modify-executor-and-window-functions-to-handle-I.patch (22.0K, ../../[email protected]/5-v18-0004-Modify-executor-and-window-functions-to-handle-I.patch)
download | inline diff:
From 163577c9191d63b80083a153fe2604227c3f9420 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 16 Aug 2025 18:29:23 +0900
Subject: [PATCH v18 4/6] Modify executor and window functions to handle IGNORE
NULLS.
Following changes have been made to executor and window functions
modules.
- New window function API WinCheckAndInitializeNullTreatment() is
added. Window functions should call this to express if they accept a
null treatment clause or not. If they do not, an error is raised in
this function. Built-in window functions are modified to call it.
- WinGetFuncArgInPartition is modified to handle IGNORE NULLS.
- WinGetFuncArgInFrame is modified to handle IGNORE NULLS. The actual
workhorse for this is ignorenulls_getfuncarginframe.
- While searching not null rows, to not scan tuples over and over
again, "notnull_info" cache module added. This holds 2-bit info for
each tuple, to keep whether the tuple has already been checked if it
is not yet checked, null or not null. The notnull_info is added to
WindowObjectData.
---
src/backend/executor/nodeWindowAgg.c | 458 +++++++++++++++++++++++++--
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/windowapi.h | 6 +
3 files changed, 439 insertions(+), 35 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..85883958e8e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,13 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+ /*
+ * Null treatment options. One of: NO_NULLTREATMENT, PARSER_IGNORE_NULLS,
+ * PARSER_RESPECT_NULLS or IGNORE_NULLS.
+ */
+ int ignore_nulls;
} WindowObjectData;
/*
@@ -96,6 +103,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ int ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -182,8 +190,8 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
- TupleTableSlot *slot);
+static int row_is_in_frame(WindowObject winobj, int64 pos,
+ TupleTableSlot *slot, bool fetch_tuple);
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -198,6 +206,33 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -942,7 +977,7 @@ eval_windowaggregates(WindowAggState *winstate)
* Exit loop if no more rows can be in frame. Skip aggregation if
* current row is not in frame but there might be more in the frame.
*/
- ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
+ ret = row_is_in_frame(agg_winobj, winstate->aggregatedupto, agg_row_slot, false);
if (ret < 0)
break;
if (ret == 0)
@@ -1263,6 +1298,11 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
}
@@ -1412,8 +1452,8 @@ release_partition(WindowAggState *winstate)
* to our window framing rule
*
* The caller must have already determined that the row is in the partition
- * and fetched it into a slot. This function just encapsulates the framing
- * rules.
+ * and fetched it into a slot if fetch_tuple is false.
+.* This function just encapsulates the framing rules.
*
* Returns:
* -1, if the row is out of frame and no succeeding rows can be in frame
@@ -1423,8 +1463,9 @@ release_partition(WindowAggState *winstate)
* May clobber winstate->temp_slot_2.
*/
static int
-row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
+row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot, bool fetch_tuple)
{
+ WindowAggState *winstate = winobj->winstate;
int frameOptions = winstate->frameOptions;
Assert(pos >= 0); /* else caller error */
@@ -1453,9 +1494,13 @@ row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
else if (frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
{
/* following row that is not peer is out of frame */
- if (pos > winstate->currentpos &&
- !are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
- return -1;
+ if (pos > winstate->currentpos)
+ {
+ if (fetch_tuple)
+ window_gettupleslot(winobj, pos, slot);
+ if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
+ return -1;
+ }
}
else
Assert(false);
@@ -2619,14 +2664,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2727,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3264,290 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winobj, abs_pos, slot, true);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+#define INIT_NOT_NULL_INFO_NUM 128 /* initial number of notnull info members */
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3706,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3748,67 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
- {
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ if (abs_pos < 0)
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ datum = 0;
+ break;
+ }
+
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3860,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3624,7 +4012,7 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
goto out_of_frame;
/* The code above does not detect all out-of-frame cases, so check */
- if (row_is_in_frame(winstate, abs_pos, slot) <= 0)
+ if (row_is_in_frame(winobj, abs_pos, slot, false) <= 0)
goto out_of_frame;
if (isout)
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
--
2.25.1
[application/octet-stream] v18-0005-Modify-documents-to-add-null-treatment-clause.patch (4.9K, ../../[email protected]/6-v18-0005-Modify-documents-to-add-null-treatment-clause.patch)
download | inline diff:
From f38a1fd9ecbabbe6fe975e039d7b29931a2831e0 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 16 Aug 2025 18:29:23 +0900
Subject: [PATCH v18 5/6] Modify documents to add null treatment clause.
---
doc/src/sgml/func/func-window.sgml | 38 ++++++++++++++++++------------
1 file changed, 23 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/func/func-window.sgml b/doc/src/sgml/func/func-window.sgml
index cce0165b952..bcf755c9ebc 100644
--- a/doc/src/sgml/func/func-window.sgml
+++ b/doc/src/sgml/func/func-window.sgml
@@ -140,7 +140,7 @@
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -165,7 +165,7 @@
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -188,7 +188,7 @@
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -202,7 +202,7 @@
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -216,7 +216,7 @@
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -265,18 +265,26 @@
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
--
2.25.1
[application/octet-stream] v18-0006-Modify-window-function-regression-tests-to-test-.patch (18.4K, ../../[email protected]/7-v18-0006-Modify-window-function-regression-tests-to-test-.patch)
download | inline diff:
From c9b09bd64a43c4155380eabd67433a91966588fd Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 16 Aug 2025 18:29:23 +0900
Subject: [PATCH v18 6/6] Modify window function regression tests to test null
treatment clause.
---
src/test/regress/expected/window.out | 311 +++++++++++++++++++++++++++
src/test/regress/sql/window.sql | 147 +++++++++++++
2 files changed, 458 insertions(+)
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..f929d81bc8a 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,314 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..1f8c8669436 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,150 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.25.1
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-08-16 19:19 Oliver Ford <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Oliver Ford @ 2025-08-16 19:19 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Sat, Aug 16, 2025 at 10:33 AM Tatsuo Ishii <[email protected]> wrote:
> Attached are the v18 patches for adding RESPECT/IGNORE NULLS options
> to some window functions. Recent changes to doc/src/sgml/func.sgml
> required v17 to be rebased. Other than that, nothing has been changed.
>
> Oliver, do you have any comments on the patches?
>
Looks good, tried it on the nth_value test script from a bit ago - I added
a 1 million rows test and it takes an average of 12 seconds on my i7.
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-08-16 22:34 Tatsuo Ishii <[email protected]>
parent: Oliver Ford <[email protected]>
1 sibling, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-08-16 22:34 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> Attached are the v18 patches for adding RESPECT/IGNORE NULLS options
>> to some window functions. Recent changes to doc/src/sgml/func.sgml
>> required v17 to be rebased. Other than that, nothing has been changed.
>>
>> Oliver, do you have any comments on the patches?
>>
>
> Looks good, tried it on the nth_value test script from a bit ago - I added
> a 1 million rows test and it takes an average of 12 seconds on my i7.
Thanks.
I have moved the CF entry from PG19-1 to PG19-2 as PG19-1 has been
already closed on July 31. Hope this help CF bot to catch the v18
patches.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-09-12 09:53 Tatsuo Ishii <[email protected]>
parent: Oliver Ford <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-09-12 09:53 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> Attached are the v18 patches for adding RESPECT/IGNORE NULLS options
>> to some window functions. Recent changes to doc/src/sgml/func.sgml
>> required v17 to be rebased. Other than that, nothing has been changed.
>>
>> Oliver, do you have any comments on the patches?
>>
>
> Looks good, tried it on the nth_value test script from a bit ago - I added
> a 1 million rows test and it takes an average of 12 seconds on my i7.
I would like to push the patch by the end of this month or early in
October if there's no objection.
Comments/suggestions are welcome.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-09-15 08:08 Chao Li <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Chao Li @ 2025-09-15 08:08 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Overall LGTM. Just a few small comments:
> On Sep 12, 2025, at 17:53, Tatsuo Ishii <[email protected]> wrote:
>
>
> Comments/suggestions are welcome.
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>
1 - 0001
```
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : 0);
```
Should we use the constant NO_NULLTREATMENT here for 0?
2 - 0001
```
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
```
Maybe we can use “uint8” type for “ignore_nulls”. Because the previous two are both of type “bool”, an uint8 will just fit to the padding bytes, so that new field won’t add extra memory to the structure.
3 - 0004
```
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
+ memset(perfuncstate->winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
}
```
Where in “if” and “memset()”, we can just use “winobj”.
4 - 0004
```
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
```
“Procform” is assigned but not used.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-09-17 09:18 Tatsuo Ishii <[email protected]>
parent: Chao Li <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-09-17 09:18 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Overall LGTM. Just a few small comments:
> 1 - 0001
> ```
> --- a/src/backend/parser/parse_func.c
> +++ b/src/backend/parser/parse_func.c
> @@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
> bool agg_star = (fn ? fn->agg_star : false);
> bool agg_distinct = (fn ? fn->agg_distinct : false);
> bool func_variadic = (fn ? fn->func_variadic : false);
> + int ignore_nulls = (fn ? fn->ignore_nulls : 0);
> ```
>
> Should we use the constant NO_NULLTREATMENT here for 0?
Good suggestion. Will fix.
> 2 - 0001
> ```
> --- a/src/include/nodes/primnodes.h
> +++ b/src/include/nodes/primnodes.h
> @@ -579,6 +579,17 @@ typedef struct GroupingFunc
> * Collation information is irrelevant for the query jumbling, as is the
> * internal state information of the node like "winstar" and "winagg".
> */
> +
> +/*
> + * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
> + * which is then converted to IGNORE_NULLS if the window function allows the
> + * null treatment clause.
> + */
> +#define NO_NULLTREATMENT 0
> +#define PARSER_IGNORE_NULLS 1
> +#define PARSER_RESPECT_NULLS 2
> +#define IGNORE_NULLS 3
> +
> typedef struct WindowFunc
> {
> Expr xpr;
> @@ -602,6 +613,8 @@ typedef struct WindowFunc
> bool winstar pg_node_attr(query_jumble_ignore);
> /* is function a simple aggregate? */
> bool winagg pg_node_attr(query_jumble_ignore);
> + /* ignore nulls. One of the Null Treatment options */
> + int ignore_nulls;
> ```
>
> Maybe we can use “uint8” type for “ignore_nulls”. Because the previous two are both of type “bool”, an uint8 will just fit to the padding bytes, so that new field won’t add extra memory to the structure.
If we change the data type for ignore_nulls in WindowFunc, we may also
want to change it elsewhere (FuncCall, WindowObjectData,
WindowStatePerFuncData) for consistency?
> 3 - 0004
> ```
> winobj->markpos = -1;
> winobj->seekpos = -1;
> +
> + /* reset null map */
> + if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
> + memset(perfuncstate->winobj->notnull_info, 0,
> + NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
> }
> ```
> Where in “if” and “memset()”, we can just use “winobj”.
Good catch. Will fix.
> 4 - 0004
> ```
> + if (!HeapTupleIsValid(proctup))
> + elog(ERROR, "cache lookup failed for function %u", funcid);
> + procform = (Form_pg_proc) GETSTRUCT(proctup);
> + elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
> + NameStr(procform->proname));
> ```
>
> “Procform” is assigned but not used.
I think procform is used in the following elog(ERROR, ...).
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-09-24 05:39 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-09-24 05:39 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Attached is the updated v19 patches. Mostly applied changes suggested
by Chao.
>> Overall LGTM. Just a few small comments:
>
>> 1 - 0001
>> ```
>> --- a/src/backend/parser/parse_func.c
>> +++ b/src/backend/parser/parse_func.c
>> @@ -98,6 +98,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
>> bool agg_star = (fn ? fn->agg_star : false);
>> bool agg_distinct = (fn ? fn->agg_distinct : false);
>> bool func_variadic = (fn ? fn->func_variadic : false);
>> + int ignore_nulls = (fn ? fn->ignore_nulls : 0);
>> ```
>>
>> Should we use the constant NO_NULLTREATMENT here for 0?
>
> Good suggestion. Will fix.
Done.
>> 2 - 0001
>> ```
>> --- a/src/include/nodes/primnodes.h
>> +++ b/src/include/nodes/primnodes.h
>> @@ -579,6 +579,17 @@ typedef struct GroupingFunc
>> * Collation information is irrelevant for the query jumbling, as is the
>> * internal state information of the node like "winstar" and "winagg".
>> */
>> +
>> +/*
>> + * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
>> + * which is then converted to IGNORE_NULLS if the window function allows the
>> + * null treatment clause.
>> + */
>> +#define NO_NULLTREATMENT 0
>> +#define PARSER_IGNORE_NULLS 1
>> +#define PARSER_RESPECT_NULLS 2
>> +#define IGNORE_NULLS 3
>> +
>> typedef struct WindowFunc
>> {
>> Expr xpr;
>> @@ -602,6 +613,8 @@ typedef struct WindowFunc
>> bool winstar pg_node_attr(query_jumble_ignore);
>> /* is function a simple aggregate? */
>> bool winagg pg_node_attr(query_jumble_ignore);
>> + /* ignore nulls. One of the Null Treatment options */
>> + int ignore_nulls;
>> ```
>>
>> Maybe we can use “uint8” type for “ignore_nulls”. Because the previous two are both of type “bool”, an uint8 will just fit to the padding bytes, so that new field won’t add extra memory to the structure.
>
> If we change the data type for ignore_nulls in WindowFunc, we may also
> want to change it elsewhere (FuncCall, WindowObjectData,
> WindowStatePerFuncData) for consistency?
I tried to change all "int ignore_nulls;" to "uint8 ignore_nulls;" but
gen_node_support.pl dislikes it and complains like:
could not handle type "uint8" in struct "FuncCall" field "ignore_nulls"
>> 3 - 0004
>> ```
>> winobj->markpos = -1;
>> winobj->seekpos = -1;
>> +
>> + /* reset null map */
>> + if (perfuncstate->winobj->ignore_nulls == IGNORE_NULLS)
>> + memset(perfuncstate->winobj->notnull_info, 0,
>> + NN_POS_TO_BYTES(perfuncstate->winobj->num_notnull_info));
>> }
>> ```
>> Where in “if” and “memset()”, we can just use “winobj”.
>
> Good catch. Will fix.
Done.
>> 4 - 0004
>> ```
>> + if (!HeapTupleIsValid(proctup))
>> + elog(ERROR, "cache lookup failed for function %u", funcid);
>> + procform = (Form_pg_proc) GETSTRUCT(proctup);
>> + elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
>> + NameStr(procform->proname));
>> ```
>>
>> “Procform” is assigned but not used.
>
> I think procform is used in the following elog(ERROR, ...).
I added more tests for functions (rank(), dense_rank(),
percent_rank(), cume_dist() and ntile()) that do not support
RESPECT/IGNORE NULLS options to confirm that they throw errors if the
options are given. Previously there was only test cases for
row_number().
Also I have made small cosmetic changes to executor/nodeWindowAgg.c to
make too long lines shorter.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v19-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch (8.3K, ../../[email protected]/2-v19-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch)
download | inline diff:
From f860f4cc377b66c1f5a407476fa616e5bcf2b855 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 24 Sep 2025 14:15:19 +0900
Subject: [PATCH v19 1/6] Modify parse/analysis modules to accept
RESPECT/IGNORE NULLS option.
Following changes have been made to parse//analysis modules.
- add IGNORE_P RESPECT_P keywords
- add "null_treatment" to func_expr after filter_clause and before
over_clause as the SQL standard requries. null_treatment is resolved
to either PARSER_IGNORE_NULLS, PARSER_RESPECT_NULLS or
NO_NULLTREATMENT
- add "ignore_nulls" to WindowFunc and FuncCall
---
src/backend/parser/gram.y | 19 ++++++++++++++-----
src/backend/parser/parse_func.c | 9 +++++++++
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +++++++++++++
src/include/parser/kwlist.h | 2 ++
5 files changed, 39 insertions(+), 5 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9fd48acb1f8..e7ad62c5e5e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -631,7 +631,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -729,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -764,7 +764,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15792,7 +15792,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15825,7 +15825,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16421,6 +16422,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17847,6 +17854,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17965,6 +17973,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index c43020a769d..778d69c6f3c 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -100,6 +100,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : NO_NULLTREATMENT);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -518,6 +519,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -840,6 +848,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4ed14fc5b78..95caa375fb8 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -452,6 +452,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
[application/octet-stream] v19-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch (977B, ../../[email protected]/3-v19-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch)
download | inline diff:
From b8e9eb7fd806d31a2156a8b1197d8defa6a8b0f6 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 24 Sep 2025 14:15:19 +0900
Subject: [PATCH v19 2/6] Modify get_windowfunc_expr_helper to handle IGNORE
NULLS option.
---
src/backend/utils/adt/ruleutils.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 0408a95941d..867f4aea591 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11090,7 +11090,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
--
2.43.0
[application/octet-stream] v19-0003-Modify-eval_const_expressions_mutator-to-handle-.patch (853B, ../../[email protected]/4-v19-0003-Modify-eval_const_expressions_mutator-to-handle-.patch)
download | inline diff:
From 43a985fbef41e8a61a70dac546df5984bca27360 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 24 Sep 2025 14:15:19 +0900
Subject: [PATCH v19 3/6] Modify eval_const_expressions_mutator to handle
IGNORE NULLS option.
---
src/backend/optimizer/util/clauses.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index f49bde7595b..81d768ff2a2 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2578,6 +2578,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
--
2.43.0
[application/octet-stream] v19-0004-Modify-executor-and-window-functions-to-handle-I.patch (22.1K, ../../[email protected]/5-v19-0004-Modify-executor-and-window-functions-to-handle-I.patch)
download | inline diff:
From 66dcc0ab8d6702d348d6796de7a0c70158a466c1 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 24 Sep 2025 14:15:19 +0900
Subject: [PATCH v19 4/6] Modify executor and window functions to handle IGNORE
NULLS.
Following changes have been made to executor and window functions
modules.
- New window function API WinCheckAndInitializeNullTreatment() is
added. Window functions should call this to express if they accept a
null treatment clause or not. If they do not, an error is raised in
this function. Built-in window functions are modified to call it.
- WinGetFuncArgInPartition is modified to handle IGNORE NULLS.
- WinGetFuncArgInFrame is modified to handle IGNORE NULLS. The actual
workhorse for this is ignorenulls_getfuncarginframe.
- While searching not null rows, to not scan tuples over and over
again, "notnull_info" cache module added. This holds 2-bit info for
each tuple, to keep whether the tuple has already been checked if it
is not yet checked, null or not null. The notnull_info is added to
WindowObjectData.
---
src/backend/executor/nodeWindowAgg.c | 467 +++++++++++++++++++++++++--
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/windowapi.h | 6 +
3 files changed, 448 insertions(+), 35 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..b459c980803 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,14 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+
+ /*
+ * Null treatment options. One of: NO_NULLTREATMENT, PARSER_IGNORE_NULLS,
+ * PARSER_RESPECT_NULLS or IGNORE_NULLS.
+ */
+ int ignore_nulls;
} WindowObjectData;
/*
@@ -96,6 +104,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ uint8 ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -182,8 +191,8 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
- TupleTableSlot *slot);
+static int row_is_in_frame(WindowObject winobj, int64 pos,
+ TupleTableSlot *slot, bool fetch_tuple);
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -198,6 +207,34 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull,
+ bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -942,7 +979,8 @@ eval_windowaggregates(WindowAggState *winstate)
* Exit loop if no more rows can be in frame. Skip aggregation if
* current row is not in frame but there might be more in the frame.
*/
- ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
+ ret = row_is_in_frame(agg_winobj, winstate->aggregatedupto,
+ agg_row_slot, false);
if (ret < 0)
break;
if (ret == 0)
@@ -1263,6 +1301,12 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ memset(winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(
+ perfuncstate->winobj->num_notnull_info));
}
}
@@ -1412,8 +1456,8 @@ release_partition(WindowAggState *winstate)
* to our window framing rule
*
* The caller must have already determined that the row is in the partition
- * and fetched it into a slot. This function just encapsulates the framing
- * rules.
+ * and fetched it into a slot if fetch_tuple is false.
+.* This function just encapsulates the framing rules.
*
* Returns:
* -1, if the row is out of frame and no succeeding rows can be in frame
@@ -1423,8 +1467,10 @@ release_partition(WindowAggState *winstate)
* May clobber winstate->temp_slot_2.
*/
static int
-row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
+row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot,
+ bool fetch_tuple)
{
+ WindowAggState *winstate = winobj->winstate;
int frameOptions = winstate->frameOptions;
Assert(pos >= 0); /* else caller error */
@@ -1453,9 +1499,13 @@ row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
else if (frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
{
/* following row that is not peer is out of frame */
- if (pos > winstate->currentpos &&
- !are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
- return -1;
+ if (pos > winstate->currentpos)
+ {
+ if (fetch_tuple)
+ window_gettupleslot(winobj, pos, slot);
+ if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
+ return -1;
+ }
}
else
Assert(false);
@@ -2619,14 +2669,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2732,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3269,294 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth
+ (winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winobj, abs_pos, slot, true);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates,
+ argno), econtext,
+ isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth
+ (winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+/* initial number of notnull info members */
+#define INIT_NOT_NULL_INFO_NUM 128
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3715,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3757,67 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
- {
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ if (abs_pos < 0)
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ datum = 0;
+ break;
+ }
+
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3869,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3624,7 +4021,7 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
goto out_of_frame;
/* The code above does not detect all out-of-frame cases, so check */
- if (row_is_in_frame(winstate, abs_pos, slot) <= 0)
+ if (row_is_in_frame(winobj, abs_pos, slot, false) <= 0)
goto out_of_frame;
if (isout)
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
--
2.43.0
[application/octet-stream] v19-0005-Modify-documents-to-add-null-treatment-clause.patch (4.9K, ../../[email protected]/6-v19-0005-Modify-documents-to-add-null-treatment-clause.patch)
download | inline diff:
From e632d9e9261b4983393828b09d079a950f9b68ca Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 24 Sep 2025 14:15:19 +0900
Subject: [PATCH v19 5/6] Modify documents to add null treatment clause.
---
doc/src/sgml/func/func-window.sgml | 38 ++++++++++++++++++------------
1 file changed, 23 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/func/func-window.sgml b/doc/src/sgml/func/func-window.sgml
index cce0165b952..bcf755c9ebc 100644
--- a/doc/src/sgml/func/func-window.sgml
+++ b/doc/src/sgml/func/func-window.sgml
@@ -140,7 +140,7 @@
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -165,7 +165,7 @@
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -188,7 +188,7 @@
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -202,7 +202,7 @@
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -216,7 +216,7 @@
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -265,18 +265,26 @@
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
--
2.43.0
[application/octet-stream] v19-0006-Modify-window-function-regression-tests-to-test-.patch (21.4K, ../../[email protected]/7-v19-0006-Modify-window-function-regression-tests-to-test-.patch)
download | inline diff:
From 6818690e8fcdfdac51fe5d48aedf1db6fb21a9eb Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 24 Sep 2025 14:15:19 +0900
Subject: [PATCH v19 6/6] Modify window function regression tests to test null
treatment clause.
---
src/test/regress/expected/window.out | 406 +++++++++++++++++++++++++++
src/test/regress/sql/window.sql | 162 +++++++++++
2 files changed, 568 insertions(+)
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..a595fa28ce1 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,409 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT rank() OVER () FROM planets; -- succeeds
+ rank
+------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT rank() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function rank does not allow RESPECT/IGNORE NULLS
+SELECT rank() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function rank does not allow RESPECT/IGNORE NULLS
+SELECT dense_rank() OVER () FROM planets; -- succeeds
+ dense_rank
+------------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT dense_rank() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function dense_rank does not allow RESPECT/IGNORE NULLS
+SELECT dense_rank() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function dense_rank does not allow RESPECT/IGNORE NULLS
+SELECT percent_rank() OVER () FROM planets; -- succeeds
+ percent_rank
+--------------
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+(10 rows)
+
+SELECT percent_rank() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function percent_rank does not allow RESPECT/IGNORE NULLS
+SELECT percent_rank() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function percent_rank does not allow RESPECT/IGNORE NULLS
+SELECT cume_dist() OVER () FROM planets; -- succeeds
+ cume_dist
+-----------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT cume_dist() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function cume_dist does not allow RESPECT/IGNORE NULLS
+SELECT cume_dist() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function cume_dist does not allow RESPECT/IGNORE NULLS
+SELECT ntile(1) OVER () FROM planets; -- succeeds
+ ntile
+-------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT ntile(1) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function ntile does not allow RESPECT/IGNORE NULLS
+SELECT ntile(1) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function ntile does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..85fc621c8db 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,165 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT rank() OVER () FROM planets; -- succeeds
+SELECT rank() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT rank() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT dense_rank() OVER () FROM planets; -- succeeds
+SELECT dense_rank() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT dense_rank() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT percent_rank() OVER () FROM planets; -- succeeds
+SELECT percent_rank() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT percent_rank() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT cume_dist() OVER () FROM planets; -- succeeds
+SELECT cume_dist() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT cume_dist() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT ntile(1) OVER () FROM planets; -- succeeds
+SELECT ntile(1) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT ntile(1) IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-09-24 12:24 Oliver Ford <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Oliver Ford @ 2025-09-24 12:24 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Wed, Sep 24, 2025 at 6:39 AM Tatsuo Ishii <[email protected]> wrote:
> I tried to change all "int ignore_nulls;" to "uint8 ignore_nulls;" but
> gen_node_support.pl dislikes it and complains like:
>
> could not handle type "uint8" in struct "FuncCall" field "ignore_nulls"
>
>
uint8 was missing in one place in that perl script. The attached patch
silences it for uint8/uint16.
Attachments:
[application/x-patch] 0001-Add-uint8-uint16-to-gen_node_support.patch (847B, ../../CAGMVOdtM_Ur=DO2MbddNKRMsN5+BbFqTs679rt7hfiWzjMo0ag@mail.gmail.com/3-0001-Add-uint8-uint16-to-gen_node_support.patch)
download | inline diff:
From 428d094b3fe209bd398f1356e81191cedf97951b Mon Sep 17 00:00:00 2001
From: Oliver Ford <[email protected]>
Date: Wed, 24 Sep 2025 12:53:42 +0100
Subject: [PATCH] Add uint8/uint16 to gen_node_support
---
src/backend/nodes/gen_node_support.pl | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/backend/nodes/gen_node_support.pl b/src/backend/nodes/gen_node_support.pl
index 9ecddb14231..b24713f000b 100644
--- a/src/backend/nodes/gen_node_support.pl
+++ b/src/backend/nodes/gen_node_support.pl
@@ -1030,7 +1030,9 @@ _read${n}(void)
print $off "\tWRITE_INT_FIELD($f);\n";
print $rff "\tREAD_INT_FIELD($f);\n" unless $no_read;
}
- elsif ($t eq 'uint32'
+ elsif ($t eq 'uint8'
+ || $t eq 'uint16'
+ || $t eq 'uint32'
|| $t eq 'bits32'
|| $t eq 'BlockNumber'
|| $t eq 'Index'
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-09-24 13:18 Tatsuo Ishii <[email protected]>
parent: Oliver Ford <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-09-24 13:18 UTC (permalink / raw)
To: [email protected]; [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Hi Oliver,
> On Wed, Sep 24, 2025 at 6:39 AM Tatsuo Ishii <[email protected]> wrote:
>
>> I tried to change all "int ignore_nulls;" to "uint8 ignore_nulls;" but
>> gen_node_support.pl dislikes it and complains like:
>>
>> could not handle type "uint8" in struct "FuncCall" field "ignore_nulls"
>>
>>
> uint8 was missing in one place in that perl script. The attached patch
> silences it for uint8/uint16.
Thank you for the patch. (I noticed int8 is also missing).
I have looked into the commit 964d01ae90c3 which was made by Peter. I
have quick read through the discussion to know why uint8/uint16 (and
int8) are missing in gen_node_support.pl. Unfortunately I have no
clear idea why these data types are missing in the script.
Peter,
Maybe you wanted to limit the data types that are actually used at
that point? If so, probably we should only add uint8 support this time
(uint8 is only needed to implement $Subject for now). What do you
think?
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-02 12:15 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-02 12:15 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Thank you for the patch. (I noticed int8 is also missing).
>
> I have looked into the commit 964d01ae90c3 which was made by Peter. I
> have quick read through the discussion to know why uint8/uint16 (and
> int8) are missing in gen_node_support.pl. Unfortunately I have no
> clear idea why these data types are missing in the script.
>
> Peter,
> Maybe you wanted to limit the data types that are actually used at
> that point? If so, probably we should only add uint8 support this time
> (uint8 is only needed to implement $Subject for now). What do you
> think?
I decided not to include the fix to gen_node_support.pl for now and
commit the patch without it. We could revisit it later on.
So here is the commit message I would like to propose.
For the technical part please look at the message.
Non technical part:
First of all the author is Oliver (no doubt). I would like to be
listed as a co-author since I wrote the not null cache part.
Next is reviewers. Actually the first effor to implement null
treatment clause was back to 9.3 era (2013) at least. After that
multiple trials to implemnt the feature happend but they had faded
away. I think we don't need to include all of those who joined the old
discussions as reviewers. So I started to check from the discussion:
https://postgr.es/m/flat/CAGMVOdsbtRwE_4+v8zjH1d9xfovDeQAGLkP_B6k69_VoFEgX-A@mail.gmail.com
because it's refered to by the commit fest entry.
Oliver and others, I love to hear your comment!
BTW, Oliver's last patch made the CF bot to misunderstand the patch
contents, which was not actually the main patch. So I attach the same
patch as v20.
----------------------------------------------------------------------
Add IGNORE NULLS/RESPECT NULLS option to Window functions.
Add IGNORE NULLS/RESPECT NULLS option (null treatment clause) to lead,
lag, first_value, last_value and nth_value window functions. If
unspecified, the default is RESPECT NULLS which includes NULL values
in any result calculation. IGNORE NULLS ignores NULL values.
Built-in window functions are modified to call new API
WinCheckAndInitializeNullTreatment() to indicate whether they accept
IGNORE NULLS/RESPECT NULLS option or not (the API can be called by
user defined window functions as well). If WinGetFuncArgInPartition's
allowNullTreatment argument is true and IGNORE NULLS option is given,
WinGetFuncArgInPartition() or WinGetFuncArgInFrame() will return
evaluated function's argument expression on specified non NULL row (if
it exists) in the partition or the frame.
When IGNORE NULLS option is given, window functions need to visit and
evaluate same rows over and over again to look for non null rows. To
mitigate the issue, 2-bit not null information array is created while
executing window functions to remember whether the row has been
already evaluated to NULL or NOT NULL. If already evaluated, we could
skip some the evaluation work, thus we could get better performance.
Author: Oliver Ford <[email protected]>
Co-authored-by: Tatsuo Ishii <[email protected]>
Reviewed-by: Andrew Gierth <[email protected]>
Reviewed-by: Tom Lane <[email protected]>
Reviewed-by: David Fetter <[email protected]>
Reviewed-by: Vik Fearing <[email protected]>
Reviewed-by: "David G. Johnston" <[email protected]>
Reviewed-by: Krasiyan Andreev <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Discussion: https://postgr.es/m/flat/CAGMVOdsbtRwE_4+v8zjH1d9xfovDeQAGLkP_B6k69_VoFEgX-A@mail.gmail.com
----------------------------------------------------------------------
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v20-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch (8.3K, ../../[email protected]/2-v20-0001-Modify-parse-analysis-modules-to-accept-RESPECT-.patch)
download | inline diff:
From cb9575f920fae82e2a0e7b40f243fb6980054244 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 2 Oct 2025 21:06:40 +0900
Subject: [PATCH v20 1/6] Modify parse/analysis modules to accept
RESPECT/IGNORE NULLS option.
Following changes have been made to parse//analysis modules.
- add IGNORE_P RESPECT_P keywords
- add "null_treatment" to func_expr after filter_clause and before
over_clause as the SQL standard requries. null_treatment is resolved
to either PARSER_IGNORE_NULLS, PARSER_RESPECT_NULLS or
NO_NULLTREATMENT
- add "ignore_nulls" to WindowFunc and FuncCall
---
src/backend/parser/gram.y | 19 ++++++++++++++-----
src/backend/parser/parse_func.c | 9 +++++++++
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/primnodes.h | 13 +++++++++++++
src/include/parser/kwlist.h | 2 ++
5 files changed, 39 insertions(+), 5 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index f1def67ac7c..57bf7a7c7f2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -632,7 +632,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
-%type <ival> opt_window_exclusion_clause
+%type <ival> null_treatment opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
%type <boolean> opt_unique_null_treatment
@@ -730,7 +730,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -765,7 +765,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -15805,7 +15805,7 @@ func_application: func_name '(' ')'
* (Note that many of the special SQL functions wouldn't actually make any
* sense as functional index entries, but we ignore that consideration here.)
*/
-func_expr: func_application within_group_clause filter_clause over_clause
+func_expr: func_application within_group_clause filter_clause null_treatment over_clause
{
FuncCall *n = (FuncCall *) $1;
@@ -15838,7 +15838,8 @@ func_expr: func_application within_group_clause filter_clause over_clause
n->agg_within_group = true;
}
n->agg_filter = $3;
- n->over = $4;
+ n->ignore_nulls = $4;
+ n->over = $5;
$$ = (Node *) n;
}
| json_aggregate_func filter_clause over_clause
@@ -16434,6 +16435,12 @@ filter_clause:
/*
* Window Definitions
*/
+null_treatment:
+ IGNORE_P NULLS_P { $$ = PARSER_IGNORE_NULLS; }
+ | RESPECT_P NULLS_P { $$ = PARSER_RESPECT_NULLS; }
+ | /*EMPTY*/ { $$ = NO_NULLTREATMENT; }
+ ;
+
window_clause:
WINDOW window_definition_list { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -17861,6 +17868,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_P
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17979,6 +17987,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESPECT_P
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index c43020a769d..778d69c6f3c 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -100,6 +100,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
bool agg_star = (fn ? fn->agg_star : false);
bool agg_distinct = (fn ? fn->agg_distinct : false);
bool func_variadic = (fn ? fn->func_variadic : false);
+ int ignore_nulls = (fn ? fn->ignore_nulls : NO_NULLTREATMENT);
CoercionForm funcformat = (fn ? fn->funcformat : COERCE_EXPLICIT_CALL);
bool could_be_projection;
Oid rettype;
@@ -518,6 +519,13 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
errmsg("%s is not an ordered-set aggregate, so it cannot have WITHIN GROUP",
NameListToString(funcname)),
parser_errposition(pstate, location)));
+
+ /* It also can't treat nulls as a window function */
+ if (ignore_nulls != NO_NULLTREATMENT)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("aggregate functions do not accept RESPECT/IGNORE NULLS"),
+ parser_errposition(pstate, location)));
}
}
else if (fdresult == FUNCDETAIL_WINDOWFUNC)
@@ -840,6 +848,7 @@ ParseFuncOrColumn(ParseState *pstate, List *funcname, List *fargs,
wfunc->winstar = agg_star;
wfunc->winagg = (fdresult == FUNCDETAIL_AGGREGATE);
wfunc->aggfilter = agg_filter;
+ wfunc->ignore_nulls = ignore_nulls;
wfunc->runCondition = NIL;
wfunc->location = location;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ac0e02a1db7..87c1086ec99 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -453,6 +453,7 @@ typedef struct FuncCall
List *agg_order; /* ORDER BY (list of SortBy) */
Node *agg_filter; /* FILTER clause, if any */
struct WindowDef *over; /* OVER clause, if any */
+ int ignore_nulls; /* ignore nulls for window function */
bool agg_within_group; /* ORDER BY appeared in WITHIN GROUP */
bool agg_star; /* argument was really '*' */
bool agg_distinct; /* arguments were labeled DISTINCT */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 6dfca3cb35b..e9d8bf74145 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -579,6 +579,17 @@ typedef struct GroupingFunc
* Collation information is irrelevant for the query jumbling, as is the
* internal state information of the node like "winstar" and "winagg".
*/
+
+/*
+ * Null Treatment options. If specified, initially set to PARSER_IGNORE_NULLS
+ * which is then converted to IGNORE_NULLS if the window function allows the
+ * null treatment clause.
+ */
+#define NO_NULLTREATMENT 0
+#define PARSER_IGNORE_NULLS 1
+#define PARSER_RESPECT_NULLS 2
+#define IGNORE_NULLS 3
+
typedef struct WindowFunc
{
Expr xpr;
@@ -602,6 +613,8 @@ typedef struct WindowFunc
bool winstar pg_node_attr(query_jumble_ignore);
/* is function a simple aggregate? */
bool winagg pg_node_attr(query_jumble_ignore);
+ /* ignore nulls. One of the Null Treatment options */
+ int ignore_nulls;
/* token location, or -1 if unknown */
ParseLoc location;
} WindowFunc;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index a4af3f717a1..84182eaaae2 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -202,6 +202,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore", IGNORE_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -378,6 +379,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("respect", RESPECT_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
[application/octet-stream] v20-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch (976B, ../../[email protected]/3-v20-0002-Modify-get_windowfunc_expr_helper-to-handle-IGNO.patch)
download | inline diff:
From 5e6c7b9306211384933b6c22af816c02ca9c5504 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 2 Oct 2025 21:06:40 +0900
Subject: [PATCH v20 2/6] Modify get_windowfunc_expr_helper to handle IGNORE
NULLS option.
---
src/backend/utils/adt/ruleutils.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index c6d83d67b87..21663af6979 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -11091,7 +11091,12 @@ get_windowfunc_expr_helper(WindowFunc *wfunc, deparse_context *context,
get_rule_expr((Node *) wfunc->aggfilter, context, false);
}
- appendStringInfoString(buf, ") OVER ");
+ appendStringInfoString(buf, ") ");
+
+ if (wfunc->ignore_nulls == PARSER_IGNORE_NULLS)
+ appendStringInfoString(buf, "IGNORE NULLS ");
+
+ appendStringInfoString(buf, "OVER ");
if (context->windowClause)
{
--
2.43.0
[application/octet-stream] v20-0003-Modify-eval_const_expressions_mutator-to-handle-.patch (852B, ../../[email protected]/4-v20-0003-Modify-eval_const_expressions_mutator-to-handle-.patch)
download | inline diff:
From 1ecce664b714f4a03c57990b1be6a72dd6f26103 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 2 Oct 2025 21:06:40 +0900
Subject: [PATCH v20 3/6] Modify eval_const_expressions_mutator to handle
IGNORE NULLS option.
---
src/backend/optimizer/util/clauses.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index f49bde7595b..81d768ff2a2 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2578,6 +2578,7 @@ eval_const_expressions_mutator(Node *node,
newexpr->winref = expr->winref;
newexpr->winstar = expr->winstar;
newexpr->winagg = expr->winagg;
+ newexpr->ignore_nulls = expr->ignore_nulls;
newexpr->location = expr->location;
return (Node *) newexpr;
--
2.43.0
[application/octet-stream] v20-0004-Modify-executor-and-window-functions-to-handle-I.patch (22.1K, ../../[email protected]/5-v20-0004-Modify-executor-and-window-functions-to-handle-I.patch)
download | inline diff:
From cb0805b83d612d845a083d75094cddd22464e3dc Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 2 Oct 2025 21:06:40 +0900
Subject: [PATCH v20 4/6] Modify executor and window functions to handle IGNORE
NULLS.
Following changes have been made to executor and window functions
modules.
- New window function API WinCheckAndInitializeNullTreatment() is
added. Window functions should call this to express if they accept a
null treatment clause or not. If they do not, an error is raised in
this function. Built-in window functions are modified to call it.
- WinGetFuncArgInPartition is modified to handle IGNORE NULLS.
- WinGetFuncArgInFrame is modified to handle IGNORE NULLS. The actual
workhorse for this is ignorenulls_getfuncarginframe.
- While searching not null rows, to not scan tuples over and over
again, "notnull_info" cache module added. This holds 2-bit info for
each tuple, to keep whether the tuple has already been checked if it
is not yet checked, null or not null. The notnull_info is added to
WindowObjectData.
---
src/backend/executor/nodeWindowAgg.c | 467 +++++++++++++++++++++++++--
src/backend/utils/adt/windowfuncs.c | 10 +
src/include/windowapi.h | 6 +
3 files changed, 448 insertions(+), 35 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 9a1acce2b5d..b459c980803 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,6 +69,14 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
+ uint8 *notnull_info; /* not null info */
+ int num_notnull_info; /* track size of the notnull_info array */
+
+ /*
+ * Null treatment options. One of: NO_NULLTREATMENT, PARSER_IGNORE_NULLS,
+ * PARSER_RESPECT_NULLS or IGNORE_NULLS.
+ */
+ int ignore_nulls;
} WindowObjectData;
/*
@@ -96,6 +104,7 @@ typedef struct WindowStatePerFuncData
bool plain_agg; /* is it just a plain aggregate function? */
int aggno; /* if so, index of its WindowStatePerAggData */
+ uint8 ignore_nulls; /* ignore nulls */
WindowObject winobj; /* object used in window function API */
} WindowStatePerFuncData;
@@ -182,8 +191,8 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
- TupleTableSlot *slot);
+static int row_is_in_frame(WindowObject winobj, int64 pos,
+ TupleTableSlot *slot, bool fetch_tuple);
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -198,6 +207,34 @@ static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype,
+ bool set_mark, bool *isnull,
+ bool *isout);
+static Datum gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull,
+ bool *isout);
+static void init_notnull_info(WindowObject winobj);
+static void grow_notnull_info(WindowObject winobj, int64 pos);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos);
+static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+
+/*
+ * Not null info bit array consists of 2-bit items
+ */
+#define NN_UNKNOWN 0x00 /* value not calculated yet */
+#define NN_NULL 0x01 /* NULL */
+#define NN_NOTNULL 0x02 /* NOT NULL */
+#define NN_MASK 0x03 /* mask for NOT NULL MAP */
+#define NN_BITS_PER_MEMBER 2 /* number of bit in not null map */
+/* number of items per variable */
+#define NN_ITEM_PER_VAR (BITS_PER_BYTE / NN_BITS_PER_MEMBER)
+/* convert map position to byte offset */
+#define NN_POS_TO_BYTES(pos) ((pos) / NN_ITEM_PER_VAR)
+/* bytes offset to map position */
+#define NN_BYTES_TO_POS(bytes) ((bytes) * NN_ITEM_PER_VAR)
+/* caculate shift bits */
+#define NN_SHIFT(pos) ((pos) % NN_ITEM_PER_VAR) * NN_BITS_PER_MEMBER
/*
* initialize_windowaggregate
@@ -942,7 +979,8 @@ eval_windowaggregates(WindowAggState *winstate)
* Exit loop if no more rows can be in frame. Skip aggregation if
* current row is not in frame but there might be more in the frame.
*/
- ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
+ ret = row_is_in_frame(agg_winobj, winstate->aggregatedupto,
+ agg_row_slot, false);
if (ret < 0)
break;
if (ret == 0)
@@ -1263,6 +1301,12 @@ begin_partition(WindowAggState *winstate)
winobj->markpos = -1;
winobj->seekpos = -1;
+
+ /* reset null map */
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ memset(winobj->notnull_info, 0,
+ NN_POS_TO_BYTES(
+ perfuncstate->winobj->num_notnull_info));
}
}
@@ -1412,8 +1456,8 @@ release_partition(WindowAggState *winstate)
* to our window framing rule
*
* The caller must have already determined that the row is in the partition
- * and fetched it into a slot. This function just encapsulates the framing
- * rules.
+ * and fetched it into a slot if fetch_tuple is false.
+.* This function just encapsulates the framing rules.
*
* Returns:
* -1, if the row is out of frame and no succeeding rows can be in frame
@@ -1423,8 +1467,10 @@ release_partition(WindowAggState *winstate)
* May clobber winstate->temp_slot_2.
*/
static int
-row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
+row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot,
+ bool fetch_tuple)
{
+ WindowAggState *winstate = winobj->winstate;
int frameOptions = winstate->frameOptions;
Assert(pos >= 0); /* else caller error */
@@ -1453,9 +1499,13 @@ row_is_in_frame(WindowAggState *winstate, int64 pos, TupleTableSlot *slot)
else if (frameOptions & (FRAMEOPTION_RANGE | FRAMEOPTION_GROUPS))
{
/* following row that is not peer is out of frame */
- if (pos > winstate->currentpos &&
- !are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
- return -1;
+ if (pos > winstate->currentpos)
+ {
+ if (fetch_tuple)
+ window_gettupleslot(winobj, pos, slot);
+ if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
+ return -1;
+ }
}
else
Assert(false);
@@ -2619,14 +2669,17 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
elog(ERROR, "WindowFunc with winref %u assigned to WindowAgg with winref %u",
wfunc->winref, node->winref);
- /* Look for a previous duplicate window function */
+ /*
+ * Look for a previous duplicate window function, which needs the same
+ * ignore_nulls value
+ */
for (i = 0; i <= wfuncno; i++)
{
if (equal(wfunc, perfunc[i].wfunc) &&
!contain_volatile_functions((Node *) wfunc))
break;
}
- if (i <= wfuncno)
+ if (i <= wfuncno && wfunc->ignore_nulls == perfunc[i].ignore_nulls)
{
/* Found a match to an existing entry, so just mark it */
wfuncstate->wfuncno = i;
@@ -2679,6 +2732,8 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->argstates = wfuncstate->args;
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
+ winobj->ignore_nulls = wfunc->ignore_nulls;
+ init_notnull_info(winobj);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3214,12 +3269,294 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
+/*
+ * get tupple and evaluate in a partition
+ */
+static Datum
+gettuple_eval_partition(WindowObject winobj, int argno,
+ int64 abs_pos, bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+
+ winstate = winobj->winstate;
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+ }
+
+ if (isout)
+ *isout = false;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth
+ (winobj->argstates, argno),
+ econtext, isnull);
+}
+
+/*
+ * ignorenulls_getfuncarginframe
+ * For IGNORE NULLS, get the next nonnull value in the frame, moving forward
+ * or backward until we find a value or reach the frame's end.
+ */
+static Datum
+ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ ExprContext *econtext;
+ TupleTableSlot *slot;
+ Datum datum;
+ int64 abs_pos;
+ int64 mark_pos;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+ econtext = winstate->ss.ps.ps_ExprContext;
+ slot = winstate->temp_slot_1;
+ datum = (Datum) 0;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+
+ switch (seektype)
+ {
+ case WINDOW_SEEK_CURRENT:
+ elog(ERROR, "WINDOW_SEEK_CURRENT is not supported for WinGetFuncArgInFrame");
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ case WINDOW_SEEK_HEAD:
+ /* rejecting relpos < 0 is easy and simplifies code below */
+ if (relpos < 0)
+ goto out_of_frame;
+ update_frameheadpos(winstate);
+ abs_pos = winstate->frameheadpos;
+ mark_pos = winstate->frameheadpos;
+ forward = 1;
+ break;
+ case WINDOW_SEEK_TAIL:
+ /* rejecting relpos > 0 is easy and simplifies code below */
+ if (relpos > 0)
+ goto out_of_frame;
+ update_frametailpos(winstate);
+ abs_pos = winstate->frametailpos - 1;
+ mark_pos = 0; /* keep compiler quiet */
+ forward = -1;
+ break;
+ default:
+ elog(ERROR, "unrecognized window seek type: %d", seektype);
+ abs_pos = mark_pos = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /*
+ * Get the next nonnull value in the frame, moving forward or backward
+ * until we find a value or reach the frame's end.
+ */
+ do
+ {
+ int inframe;
+ int v;
+
+ /*
+ * Check apparent out of frame case. We need to do this because we
+ * may not call window_gettupleslot before row_is_in_frame, which
+ * supposes abs_pos is never negative.
+ */
+ if (abs_pos < 0)
+ goto out_of_frame;
+
+ /* check whether row is in frame */
+ inframe = row_is_in_frame(winobj, abs_pos, slot, true);
+ if (inframe == -1)
+ goto out_of_frame;
+ else if (inframe == 0)
+ goto advance;
+
+ if (isout)
+ *isout = false;
+
+ v = get_notnull_info(winobj, abs_pos);
+ if (v == NN_NULL) /* this row is known to be NULL */
+ goto advance;
+
+ else if (v == NN_UNKNOWN) /* need to check NULL or not */
+ {
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth(winobj->argstates,
+ argno), econtext,
+ isnull);
+ if (!*isnull)
+ notnull_offset++;
+
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ }
+ else /* this row is known to be NOT NULL */
+ {
+ notnull_offset++;
+ if (notnull_offset > notnull_relpos)
+ {
+ /* to prepare exiting this loop, datum needs to be set */
+ if (!window_gettupleslot(winobj, abs_pos, slot))
+ goto out_of_frame;
+
+ econtext->ecxt_outertuple = slot;
+ datum = ExecEvalExpr(
+ (ExprState *) list_nth
+ (winobj->argstates, argno),
+ econtext, isnull);
+ }
+ }
+advance:
+ abs_pos += forward;
+ } while (notnull_offset <= notnull_relpos);
+
+ if (set_mark)
+ WinSetMarkPosition(winobj, mark_pos);
+
+ return datum;
+
+out_of_frame:
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+
+/*
+ * init_notnull_info
+ * Initialize non null map.
+ */
+static void
+init_notnull_info(WindowObject winobj)
+{
+/* initial number of notnull info members */
+#define INIT_NOT_NULL_INFO_NUM 128
+
+ if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+
+ winobj->notnull_info = palloc0(size);
+ winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ }
+}
+
+/*
+ * grow_notnull_info
+ * expand notnull_info if necessary.
+ * pos: not null info position
+*/
+static void
+grow_notnull_info(WindowObject winobj, int64 pos)
+{
+ if (pos >= winobj->num_notnull_info)
+ {
+ for (;;)
+ {
+ Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
+ Size newsize = oldsize * 2;
+
+ winobj->notnull_info =
+ repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info > pos)
+ break;
+ }
+ }
+}
+
+/*
+ * get_notnull_info
+ * retrieve a map
+ * pos: map position
+ */
+static uint8
+get_notnull_info(WindowObject winobj, int64 pos)
+{
+ uint8 mb;
+ int64 bpos;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ return (mb >> (NN_SHIFT(pos))) & NN_MASK;
+}
+
+/*
+ * put_notnull_info
+ * update map
+ * pos: map position
+ */
+static void
+put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+{
+ uint8 mb;
+ int64 bpos;
+ uint8 val = isnull ? NN_NULL : NN_NOTNULL;
+ int shift;
+
+ grow_notnull_info(winobj, pos);
+ bpos = NN_POS_TO_BYTES(pos);
+ mb = winobj->notnull_info[bpos];
+ shift = NN_SHIFT(pos);
+ mb &= ~(NN_MASK << shift); /* clear map */
+ mb |= (val << shift); /* update map */
+ winobj->notnull_info[bpos] = mb;
+}
/***********************************************************************
* API exposed to window functions
***********************************************************************/
+/*
+ * WinCheckAndInitializeNullTreatment
+ * Check null treatment clause and sets ignore_nulls
+ *
+ * Window functions should call this to check if they are being called with
+ * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ */
+void
+WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo)
+{
+ if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
+ {
+ HeapTuple proctup;
+ Form_pg_proc procform;
+ Oid funcid;
+
+ funcid = fcinfo->flinfo->fn_oid;
+ proctup = SearchSysCache1(PROCOID,
+ ObjectIdGetDatum(funcid));
+ if (!HeapTupleIsValid(proctup))
+ elog(ERROR, "cache lookup failed for function %u", funcid);
+ procform = (Form_pg_proc) GETSTRUCT(proctup);
+ elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
+ NameStr(procform->proname));
+ }
+ else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ winobj->ignore_nulls = IGNORE_NULLS;
+
+}
+
/*
* WinGetPartitionLocalMemory
* Get working memory that lives till end of partition processing
@@ -3378,23 +3715,37 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
bool *isnull, bool *isout)
{
WindowAggState *winstate;
- ExprContext *econtext;
- TupleTableSlot *slot;
- bool gottuple;
int64 abs_pos;
+ Datum datum;
+ bool null_treatment = false;
+ int notnull_offset;
+ int notnull_relpos;
+ int forward;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- econtext = winstate->ss.ps.ps_ExprContext;
- slot = winstate->temp_slot_1;
+
+ if (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0)
+ {
+ null_treatment = true;
+ notnull_offset = 0;
+ notnull_relpos = abs(relpos);
+ forward = relpos > 0 ? 1 : -1;
+ }
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
- abs_pos = winstate->currentpos + relpos;
+ if (null_treatment)
+ abs_pos = winstate->currentpos;
+ else
+ abs_pos = winstate->currentpos + relpos;
break;
case WINDOW_SEEK_HEAD:
- abs_pos = relpos;
+ if (null_treatment)
+ abs_pos = 0;
+ else
+ abs_pos = relpos;
break;
case WINDOW_SEEK_TAIL:
spool_tuples(winstate, -1);
@@ -3406,25 +3757,67 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
}
- gottuple = window_gettupleslot(winobj, abs_pos, slot);
-
- if (!gottuple)
- {
- if (isout)
- *isout = true;
- *isnull = true;
- return (Datum) 0;
- }
- else
+ if (!null_treatment) /* IGNORE NULLS is not specified */
{
- if (isout)
- *isout = false;
- if (set_mark)
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (!*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return datum;
}
+
+ /*
+ * Get the next nonnull value in the partition, moving forward or backward
+ * until we find a value or reach the partition's end.
+ */
+ do
+ {
+ abs_pos += forward;
+ if (abs_pos < 0)
+ {
+ /* out of partition */
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ datum = 0;
+ break;
+ }
+
+ switch (get_notnull_info(winobj, abs_pos))
+ {
+ case NN_NOTNULL: /* this row is known to be NOT NULL */
+ notnull_offset++;
+ if (notnull_offset >= notnull_relpos)
+ {
+ /* prepare to exit this loop */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ }
+ break;
+ case NN_NULL: /* this row is known to be NULL */
+ if (isout)
+ *isout = false;
+ *isnull = true;
+ datum = 0;
+ break;
+ default: /* need to check NULL or not */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, isout);
+ if (*isout) /* out of partition? */
+ return datum;
+
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
+ break;
+ }
+ } while (notnull_offset < notnull_relpos);
+
+ if (!*isout && set_mark)
+ WinSetMarkPosition(winobj, abs_pos);
+
+ return datum;
}
/*
@@ -3476,6 +3869,10 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (winobj->ignore_nulls == IGNORE_NULLS)
+ return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
+ set_mark, isnull, isout);
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3624,7 +4021,7 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
goto out_of_frame;
/* The code above does not detect all out-of-frame cases, so check */
- if (row_is_in_frame(winstate, abs_pos, slot) <= 0)
+ if (row_is_in_frame(winobj, abs_pos, slot, false) <= 0)
goto out_of_frame;
if (isout)
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index bb35f3bc4a9..969f02aa59b 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -86,6 +86,7 @@ window_row_number(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
int64 curpos = WinGetCurrentPosition(winobj);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
WinSetMarkPosition(winobj, curpos);
PG_RETURN_INT64(curpos + 1);
}
@@ -141,6 +142,7 @@ window_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -203,6 +205,7 @@ window_dense_rank(PG_FUNCTION_ARGS)
rank_context *context;
bool up;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
WinGetPartitionLocalMemory(winobj, sizeof(rank_context));
@@ -266,6 +269,7 @@ window_percent_rank(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -335,6 +339,7 @@ window_cume_dist(PG_FUNCTION_ARGS)
int64 totalrows = WinGetPartitionRowCount(winobj);
Assert(totalrows > 0);
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
up = rank_up(winobj);
context = (rank_context *)
@@ -413,6 +418,7 @@ window_ntile(PG_FUNCTION_ARGS)
WindowObject winobj = PG_WINDOW_OBJECT();
ntile_context *context;
+ WinCheckAndInitializeNullTreatment(winobj, false, fcinfo);
context = (ntile_context *)
WinGetPartitionLocalMemory(winobj, sizeof(ntile_context));
@@ -535,6 +541,7 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -652,6 +659,7 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -673,6 +681,7 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -696,6 +705,7 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
+ WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index cb2ece166b6..20cfd9e9dd9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -28,6 +28,8 @@
#ifndef WINDOWAPI_H
#define WINDOWAPI_H
+#include "fmgr.h"
+
/* values of "seektype" */
#define WINDOW_SEEK_CURRENT 0
#define WINDOW_SEEK_HEAD 1
@@ -41,6 +43,10 @@ typedef struct WindowObjectData *WindowObject;
#define WindowObjectIsValid(winobj) \
((winobj) != NULL && IsA(winobj, WindowObjectData))
+extern void WinCheckAndInitializeNullTreatment(WindowObject winobj,
+ bool allowNullTreatment,
+ FunctionCallInfo fcinfo);
+
extern void *WinGetPartitionLocalMemory(WindowObject winobj, Size sz);
extern int64 WinGetCurrentPosition(WindowObject winobj);
--
2.43.0
[application/octet-stream] v20-0005-Modify-documents-to-add-null-treatment-clause.patch (4.9K, ../../[email protected]/6-v20-0005-Modify-documents-to-add-null-treatment-clause.patch)
download | inline diff:
From 112f9cd77e5fdcd78692d12b101dd5b46ea05d84 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 2 Oct 2025 21:06:40 +0900
Subject: [PATCH v20 5/6] Modify documents to add null treatment clause.
---
doc/src/sgml/func/func-window.sgml | 38 ++++++++++++++++++------------
1 file changed, 23 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/func/func-window.sgml b/doc/src/sgml/func/func-window.sgml
index cce0165b952..bcf755c9ebc 100644
--- a/doc/src/sgml/func/func-window.sgml
+++ b/doc/src/sgml/func/func-window.sgml
@@ -140,7 +140,7 @@
</indexterm>
<function>lag</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -165,7 +165,7 @@
</indexterm>
<function>lead</function> ( <parameter>value</parameter> <type>anycompatible</type>
<optional>, <parameter>offset</parameter> <type>integer</type>
- <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> )
+ <optional>, <parameter>default</parameter> <type>anycompatible</type> </optional></optional> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anycompatible</returnvalue>
</para>
<para>
@@ -188,7 +188,7 @@
<indexterm>
<primary>first_value</primary>
</indexterm>
- <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>first_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -202,7 +202,7 @@
<indexterm>
<primary>last_value</primary>
</indexterm>
- <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> )
+ <function>last_value</function> ( <parameter>value</parameter> <type>anyelement</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -216,7 +216,7 @@
<indexterm>
<primary>nth_value</primary>
</indexterm>
- <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> )
+ <function>nth_value</function> ( <parameter>value</parameter> <type>anyelement</type>, <parameter>n</parameter> <type>integer</type> ) <optional> <parameter>null treatment</parameter> </optional>
<returnvalue>anyelement</returnvalue>
</para>
<para>
@@ -265,18 +265,26 @@
Other frame specifications can be used to obtain other effects.
</para>
+ <para>
+ The <literal>null treatment</literal> option must be one of:
+<synopsis>
+ RESPECT NULLS
+ IGNORE NULLS
+</synopsis>
+ If unspecified, the default is <literal>RESPECT NULLS</literal> which includes NULL
+ values in any result calculation. <literal>IGNORE NULLS</literal> ignores NULL values.
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
+ </para>
+
<note>
<para>
- The SQL standard defines a <literal>RESPECT NULLS</literal> or
- <literal>IGNORE NULLS</literal> option for <function>lead</function>, <function>lag</function>,
- <function>first_value</function>, <function>last_value</function>, and
- <function>nth_value</function>. This is not implemented in
- <productname>PostgreSQL</productname>: the behavior is always the
- same as the standard's default, namely <literal>RESPECT NULLS</literal>.
- Likewise, the standard's <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
- option for <function>nth_value</function> is not implemented: only the
- default <literal>FROM FIRST</literal> behavior is supported. (You can achieve
- the result of <literal>FROM LAST</literal> by reversing the <literal>ORDER BY</literal>
+ The SQL standard defines a <literal>FROM FIRST</literal> or <literal>FROM LAST</literal>
+ option for <function>nth_value</function>. This is not implemented in
+ <productname>PostgreSQL</productname>: only the default <literal>FROM FIRST</literal>
+ behavior is supported. (You can achieve the result of <literal>FROM LAST</literal> by
+ reversing the <literal>ORDER BY</literal>
ordering.)
</para>
</note>
--
2.43.0
[application/octet-stream] v20-0006-Modify-window-function-regression-tests-to-test-.patch (21.4K, ../../[email protected]/7-v20-0006-Modify-window-function-regression-tests-to-test-.patch)
download | inline diff:
From 041e95f6b11f4c9cfcd88480621aa70b8d1d753e Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 2 Oct 2025 21:06:40 +0900
Subject: [PATCH v20 6/6] Modify window function regression tests to test null
treatment clause.
---
src/test/regress/expected/window.out | 406 +++++++++++++++++++++++++++
src/test/regress/sql/window.sql | 162 +++++++++++
2 files changed, 568 insertions(+)
diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out
index b86b668f433..a595fa28ce1 100644
--- a/src/test/regress/expected/window.out
+++ b/src/test/regress/expected/window.out
@@ -5453,3 +5453,409 @@ SELECT * FROM pg_temp.f(2);
{5}
(5 rows)
+-- IGNORE NULLS tests
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+NOTICE: view "planets_view" will be a temporary view
+SELECT pg_get_viewdef('planets_view');
+ pg_get_viewdef
+--------------------------------------------------
+ SELECT name, +
+ orbit, +
+ lag(orbit) OVER w AS lag, +
+ lag(orbit) OVER w AS lag_respect, +
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore+
+ FROM planets +
+ WINDOW w AS (ORDER BY name);
+(1 row)
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lag | lag_respect | lag_ignore
+---------+-------+-------+-------------+------------
+ earth | | | |
+ jupiter | 4332 | | |
+ mars | | 4332 | 4332 | 4332
+ mercury | 88 | | | 4332
+ neptune | 60182 | 88 | 88 | 88
+ pluto | 90560 | 60182 | 60182 | 60182
+ saturn | 24491 | 90560 | 90560 | 90560
+ uranus | | 24491 | 24491 | 24491
+ venus | 224 | | | 24491
+ xyzzy | | 224 | 224 | 224
+(10 rows)
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+ name | orbit | lead | lead_respect | lead_ignore
+---------+-------+-------+--------------+-------------
+ earth | | 4332 | 4332 | 4332
+ jupiter | 4332 | | | 88
+ mars | | 88 | 88 | 88
+ mercury | 88 | 60182 | 60182 | 60182
+ neptune | 60182 | 90560 | 90560 | 90560
+ pluto | 90560 | 24491 | 24491 | 24491
+ saturn | 24491 | | | 224
+ uranus | | 224 | 224 | 224
+ venus | 224 | | |
+ xyzzy | | | |
+(10 rows)
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | first_value | first_value | first_value
+---------+-------+-------------+-------------+-------------+-------------
+ earth | | | 4332 | | 4332
+ jupiter | 4332 | | 4332 | | 4332
+ mars | | | 4332 | | 4332
+ mercury | 88 | | 4332 | 4332 | 4332
+ neptune | 60182 | | 4332 | | 88
+ pluto | 90560 | | 4332 | 88 | 88
+ saturn | 24491 | | 4332 | 60182 | 60182
+ uranus | | | 4332 | 90560 | 90560
+ venus | 224 | | 4332 | 24491 | 24491
+ xyzzy | | | 4332 | | 224
+(10 rows)
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | nth_value | nth_value | nth_value | nth_value
+---------+-------+-----------+-----------+-----------+-----------
+ earth | | 4332 | 88 | 4332 |
+ jupiter | 4332 | 4332 | 88 | 4332 | 88
+ mars | | 4332 | 88 | 4332 | 88
+ mercury | 88 | 4332 | 88 | | 88
+ neptune | 60182 | 4332 | 88 | 88 | 60182
+ pluto | 90560 | 4332 | 88 | 60182 | 60182
+ saturn | 24491 | 4332 | 88 | 90560 | 90560
+ uranus | | 4332 | 88 | 24491 | 24491
+ venus | 224 | 4332 | 88 | | 224
+ xyzzy | | 4332 | 88 | 224 |
+(10 rows)
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | last_value | last_value | last_value | last_value
+---------+-------+------------+------------+------------+------------
+ earth | | | 224 | | 4332
+ jupiter | 4332 | | 224 | 88 | 88
+ mars | | | 224 | 60182 | 60182
+ mercury | 88 | | 224 | 90560 | 90560
+ neptune | 60182 | | 224 | 24491 | 24491
+ pluto | 90560 | | 224 | | 24491
+ saturn | 24491 | | 224 | 224 | 224
+ uranus | | | 224 | | 224
+ venus | 224 | | 224 | | 224
+ xyzzy | | | 224 | | 224
+(10 rows)
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | 4332 | 4332 | | 4332 |
+ jupiter | 4332 | 88 | 88 | | 88 |
+ mars | | 4332 | 60182 | 88 | 88 | 4332
+ mercury | 88 | 4332 | 90560 | 60182 | 60182 | 4332
+ neptune | 60182 | 88 | 24491 | 90560 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 24491 | | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+ sum
+--------
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+ 179877
+(10 rows)
+
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) RESPECT NULLS OVER () FROM planets;
+ ^
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: aggregate functions do not accept RESPECT/IGNORE NULLS
+LINE 1: SELECT sum(orbit) IGNORE NULLS OVER () FROM planets;
+ ^
+SELECT row_number() OVER () FROM planets; -- succeeds
+ row_number
+------------
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+ 10
+(10 rows)
+
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function row_number does not allow RESPECT/IGNORE NULLS
+SELECT rank() OVER () FROM planets; -- succeeds
+ rank
+------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT rank() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function rank does not allow RESPECT/IGNORE NULLS
+SELECT rank() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function rank does not allow RESPECT/IGNORE NULLS
+SELECT dense_rank() OVER () FROM planets; -- succeeds
+ dense_rank
+------------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT dense_rank() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function dense_rank does not allow RESPECT/IGNORE NULLS
+SELECT dense_rank() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function dense_rank does not allow RESPECT/IGNORE NULLS
+SELECT percent_rank() OVER () FROM planets; -- succeeds
+ percent_rank
+--------------
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+ 0
+(10 rows)
+
+SELECT percent_rank() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function percent_rank does not allow RESPECT/IGNORE NULLS
+SELECT percent_rank() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function percent_rank does not allow RESPECT/IGNORE NULLS
+SELECT cume_dist() OVER () FROM planets; -- succeeds
+ cume_dist
+-----------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT cume_dist() RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function cume_dist does not allow RESPECT/IGNORE NULLS
+SELECT cume_dist() IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function cume_dist does not allow RESPECT/IGNORE NULLS
+SELECT ntile(1) OVER () FROM planets; -- succeeds
+ ntile
+-------
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+ 1
+(10 rows)
+
+SELECT ntile(1) RESPECT NULLS OVER () FROM planets; -- fails
+ERROR: function ntile does not allow RESPECT/IGNORE NULLS
+SELECT ntile(1) IGNORE NULLS OVER () FROM planets; -- fails
+ERROR: function ntile does not allow RESPECT/IGNORE NULLS
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+-------+-------------+------------+-----------+-------------+------------
+ earth | | | | | 88 |
+ jupiter | | 88 | 88 | | 88 |
+ mars | | 88 | 60182 | 60182 | 88 |
+ mercury | 88 | 88 | 90560 | 60182 | 60182 |
+ neptune | 60182 | 88 | 24491 | 60182 | 90560 | 88
+ pluto | 90560 | 88 | 24491 | 60182 | 24491 | 60182
+ saturn | 24491 | 60182 | 224 | 90560 | 224 | 90560
+ uranus | | 90560 | 224 | 24491 | 224 | 24491
+ venus | 224 | 24491 | 224 | 224 | | 24491
+ xyzzy | | 224 | 224 | | | 224
+(10 rows)
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+ name | distance | orbit | first_value | last_value | nth_value | lead_ignore | lag_ignore
+---------+----------+-------+-------------+------------+-----------+-------------+------------
+ earth | close | | | | | 88 |
+ jupiter | close | | 88 | 88 | | 88 |
+ mars | close | | 88 | 224 | 224 | 88 |
+ mercury | close | 88 | 88 | 224 | 224 | 224 |
+ venus | close | 224 | 88 | 224 | 224 | | 88
+ neptune | far | 60182 | 60182 | 24491 | 90560 | 90560 |
+ pluto | far | 90560 | 60182 | 24491 | 90560 | 24491 | 60182
+ saturn | far | 24491 | 60182 | 24491 | 90560 | | 90560
+ uranus | far | | 90560 | 24491 | 24491 | | 24491
+ xyzzy | far | | 24491 | 24491 | | | 24491
+(10 rows)
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+ x | nth_value
+---+-----------
+ 1 | 3
+ 2 | 3
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+ x | nth_value
+---+-----------
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 3
+ 5 | 4
+(5 rows)
+
+--cleanup
+DROP TABLE planets CASCADE;
+NOTICE: drop cascades to view planets_view
diff --git a/src/test/regress/sql/window.sql b/src/test/regress/sql/window.sql
index 02f105f070e..85fc621c8db 100644
--- a/src/test/regress/sql/window.sql
+++ b/src/test/regress/sql/window.sql
@@ -1958,3 +1958,165 @@ $$ LANGUAGE SQL STABLE;
EXPLAIN (costs off) SELECT * FROM pg_temp.f(2);
SELECT * FROM pg_temp.f(2);
+
+-- IGNORE NULLS tests
+
+CREATE TEMPORARY TABLE planets (
+ name text,
+ distance text,
+ orbit integer
+);
+
+INSERT INTO planets VALUES
+ ('mercury', 'close', 88),
+ ('venus', 'close', 224),
+ ('earth', 'close', NULL),
+ ('mars', 'close', NULL),
+ ('jupiter', 'close', 4332),
+ ('saturn', 'far', 24491),
+ ('uranus', 'far', NULL),
+ ('neptune', 'far', 60182),
+ ('pluto', 'far', 90560),
+ ('xyzzy', 'far', NULL);
+
+-- test ruleutils
+CREATE VIEW planets_view AS
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+SELECT pg_get_viewdef('planets_view');
+
+-- lag
+SELECT name,
+ orbit,
+ lag(orbit) OVER w AS lag,
+ lag(orbit) RESPECT NULLS OVER w AS lag_respect,
+ lag(orbit) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- lead
+SELECT name,
+ orbit,
+ lead(orbit) OVER w AS lead,
+ lead(orbit) RESPECT NULLS OVER w AS lead_respect,
+ lead(orbit) IGNORE NULLS OVER w AS lead_ignore
+FROM planets
+WINDOW w AS (ORDER BY name)
+;
+
+-- first_value
+SELECT name,
+ orbit,
+ first_value(orbit) RESPECT NULLS OVER w1,
+ first_value(orbit) IGNORE NULLS OVER w1,
+ first_value(orbit) RESPECT NULLS OVER w2,
+ first_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value
+SELECT name,
+ orbit,
+ nth_value(orbit, 2) RESPECT NULLS OVER w1,
+ nth_value(orbit, 2) IGNORE NULLS OVER w1,
+ nth_value(orbit, 2) RESPECT NULLS OVER w2,
+ nth_value(orbit, 2) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- last_value
+SELECT name,
+ orbit,
+ last_value(orbit) RESPECT NULLS OVER w1,
+ last_value(orbit) IGNORE NULLS OVER w1,
+ last_value(orbit) RESPECT NULLS OVER w2,
+ last_value(orbit) IGNORE NULLS OVER w2
+FROM planets
+WINDOW w1 AS (ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),
+ w2 AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- exclude current row
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW)
+;
+
+-- valid and invalid functions
+SELECT sum(orbit) OVER () FROM planets; -- succeeds
+SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT sum(orbit) IGNORE NULLS OVER () FROM planets; -- fails
+SELECT row_number() OVER () FROM planets; -- succeeds
+SELECT row_number() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT row_number() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT rank() OVER () FROM planets; -- succeeds
+SELECT rank() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT rank() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT dense_rank() OVER () FROM planets; -- succeeds
+SELECT dense_rank() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT dense_rank() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT percent_rank() OVER () FROM planets; -- succeeds
+SELECT percent_rank() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT percent_rank() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT cume_dist() OVER () FROM planets; -- succeeds
+SELECT cume_dist() RESPECT NULLS OVER () FROM planets; -- fails
+SELECT cume_dist() IGNORE NULLS OVER () FROM planets; -- fails
+SELECT ntile(1) OVER () FROM planets; -- succeeds
+SELECT ntile(1) RESPECT NULLS OVER () FROM planets; -- fails
+SELECT ntile(1) IGNORE NULLS OVER () FROM planets; -- fails
+
+-- test two consecutive nulls
+update planets set orbit=null where name='jupiter';
+SELECT name,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- test partitions
+SELECT name,
+ distance,
+ orbit,
+ first_value(orbit) IGNORE NULLS OVER w,
+ last_value(orbit) IGNORE NULLS OVER w,
+ nth_value(orbit, 2) IGNORE NULLS OVER w,
+ lead(orbit, 1) IGNORE NULLS OVER w AS lead_ignore,
+ lag(orbit, 1) IGNORE NULLS OVER w AS lag_ignore
+FROM planets
+WINDOW w AS (PARTITION BY distance ORDER BY name ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING)
+;
+
+-- nth_value without nulls
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING EXCLUDE CURRENT ROW);
+SELECT x,
+ nth_value(x,2) IGNORE NULLS OVER w
+FROM generate_series(1,5) g(x)
+WINDOW w AS (ORDER BY x ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING);
+
+--cleanup
+DROP TABLE planets CASCADE;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-02 15:36 Oliver Ford <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Oliver Ford @ 2025-10-02 15:36 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; Krasiyan Andreev <[email protected]>; Tom Lane <[email protected]>; Vik Fearing <[email protected]>; [email protected]; David Fetter <[email protected]>; [email protected]
On Thu, 2 Oct 2025, 13:16 Tatsuo Ishii, <[email protected]> wrote:
> > Thank you for the patch. (I noticed int8 is also missing).
> >
> > I have looked into the commit 964d01ae90c3 which was made by Peter. I
> > have quick read through the discussion to know why uint8/uint16 (and
> > int8) are missing in gen_node_support.pl. Unfortunately I have no
> > clear idea why these data types are missing in the script.
> >
> > Peter,
> > Maybe you wanted to limit the data types that are actually used at
> > that point? If so, probably we should only add uint8 support this time
> > (uint8 is only needed to implement $Subject for now). What do you
> > think?
>
> I decided not to include the fix to gen_node_support.pl for now and
> commit the patch without it. We could revisit it later on.
>
> So here is the commit message I would like to propose.
>
> For the technical part please look at the message.
>
> Non technical part:
> First of all the author is Oliver (no doubt). I would like to be
> listed as a co-author since I wrote the not null cache part.
>
> Next is reviewers. Actually the first effor to implement null
> treatment clause was back to 9.3 era (2013) at least. After that
> multiple trials to implemnt the feature happend but they had faded
> away. I think we don't need to include all of those who joined the old
> discussions as reviewers. So I started to check from the discussion:
>
> https://postgr.es/m/flat/CAGMVOdsbtRwE_4+v8zjH1d9xfovDeQAGLkP_B6k69_VoFEgX-A@mail.gmail.com
> because it's refered to by the commit fest entry.
>
> Oliver and others, I love to hear your comment!
Looks great, so glad this is finally going in.
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-03 01:06 Tatsuo Ishii <[email protected]>
parent: Oliver Ford <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-03 01:06 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> On Thu, 2 Oct 2025, 13:16 Tatsuo Ishii, <[email protected]> wrote:
>
>> > Thank you for the patch. (I noticed int8 is also missing).
>> >
>> > I have looked into the commit 964d01ae90c3 which was made by Peter. I
>> > have quick read through the discussion to know why uint8/uint16 (and
>> > int8) are missing in gen_node_support.pl. Unfortunately I have no
>> > clear idea why these data types are missing in the script.
>> >
>> > Peter,
>> > Maybe you wanted to limit the data types that are actually used at
>> > that point? If so, probably we should only add uint8 support this time
>> > (uint8 is only needed to implement $Subject for now). What do you
>> > think?
>>
>> I decided not to include the fix to gen_node_support.pl for now and
>> commit the patch without it. We could revisit it later on.
>>
>> So here is the commit message I would like to propose.
>>
>> For the technical part please look at the message.
>>
>> Non technical part:
>> First of all the author is Oliver (no doubt). I would like to be
>> listed as a co-author since I wrote the not null cache part.
>>
>> Next is reviewers. Actually the first effor to implement null
>> treatment clause was back to 9.3 era (2013) at least. After that
>> multiple trials to implemnt the feature happend but they had faded
>> away. I think we don't need to include all of those who joined the old
>> discussions as reviewers. So I started to check from the discussion:
>>
>> https://postgr.es/m/flat/CAGMVOdsbtRwE_4+v8zjH1d9xfovDeQAGLkP_B6k69_VoFEgX-A@mail.gmail.com
>> because it's refered to by the commit fest entry.
>>
>> Oliver and others, I love to hear your comment!
>
>
> Looks great, so glad this is finally going in.
I have just pushed the patch (plus patches for syntax.sgml and
sql_features.txt. They were missued after I splitted the patch).
Thank you for your effort!
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-05 15:59 Tom Lane <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 2 replies; 91+ messages in thread
From: Tom Lane @ 2025-10-05 15:59 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Tatsuo Ishii <[email protected]> writes:
> I have just pushed the patch (plus patches for syntax.sgml and
> sql_features.txt. They were missued after I splitted the patch).
Coverity is not very happy with this patch.
It's complaining that the result of window_gettupleslot
is not checked, which seems valid:
1503 {
1504 if (fetch_tuple)
>>> CID 1666587: Error handling issues (CHECKED_RETURN)
>>> Calling "window_gettupleslot" without checking return value (as is done elsewhere 8 out of 9 times).
1505 window_gettupleslot(winobj, pos, slot);
1506 if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
1507 return -1;
and also that WinGetFuncArgInPartition is dereferencing
a possibly-null "isout" pointer at several places, including
>>> Dereferencing null pointer "isout".
3806 if (*isout) /* out of partition? */
>>> Dereferencing null pointer "isout".
3817 if (!*isout && set_mark)
3818 WinSetMarkPosition(winobj, abs_pos);
>>> Dereferencing null pointer "isout".
3817 if (!*isout && set_mark)
3818 WinSetMarkPosition(winobj, abs_pos);
The latter complaints seem to be because some places in
WinGetFuncArgInPartition check for nullness of that pointer
and some do not. That looks like at least a latent bug
to me. If it isn't, the function's comment needs to be
expanded to say when it's legal to pass isout == NULL.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-05 16:20 Álvaro Herrera <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 2 replies; 91+ messages in thread
From: Álvaro Herrera @ 2025-10-05 16:20 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2025-Oct-03, Tatsuo Ishii wrote:
> I have just pushed the patch (plus patches for syntax.sgml and
> sql_features.txt. They were missued after I splitted the patch).
> Thank you for your effort!
I just noticed this compiler warning in a CI run,
[16:06:29.920] ../src/backend/executor/nodeWindowAgg.c:3820:16: warning: ‘datum’ may be used uninitialized [-Wmaybe-uninitialized]
[16:06:29.920] 3820 | return datum;
[16:06:29.920] | ^~~~~
[16:06:29.920] ../src/backend/executor/nodeWindowAgg.c:3719:25: note: ‘datum’ was declared here
[16:06:29.920] 3719 | Datum datum;
[16:06:29.920] | ^~~~~
The logic in this function looks somewhat wicked.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-05 16:35 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tom Lane @ 2025-10-05 16:35 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
I wrote:
> ... also that WinGetFuncArgInPartition is dereferencing
> a possibly-null "isout" pointer at several places
Looking around, there is only one in-core caller of
WinGetFuncArgInPartition, and it does pass a valid "isout" pointer,
explaining why this inconsistency wasn't obvious in testing.
There are outside callers though according to Debian Code Search,
and at least PostGIS is one that passes a null pointer.
As Alvaro notes nearby, this function is ridiculously complicated
already. I'm tempted to remove the API allowance for isout == NULL,
and thereby simplify the code slightly, rather than complicate it more
by continuing to allow that. We'd have to warn the PostGIS people
about the API change though.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-05 16:51 Tom Lane <[email protected]>
parent: Álvaro Herrera <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tom Lane @ 2025-10-05 16:51 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: Tatsuo Ishii <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
=?utf-8?Q?=C3=81lvaro?= Herrera <[email protected]> writes:
> I just noticed this compiler warning in a CI run,
> [16:06:29.920] ../src/backend/executor/nodeWindowAgg.c:3820:16: warning: ‘datum’ may be used uninitialized [-Wmaybe-uninitialized]
> [16:06:29.920] 3820 | return datum;
> [16:06:29.920] | ^~~~~
Yeah, I can easily believe that a compiler running at relatively low
optimization level wouldn't make the connection that the NN_NOTNULL
case must perform the "prepare to exit this loop" bit if the loop
will be exited this time. But there's another thing that is
confusing: the NN_NULL case certainly looks like it's expecting
to exit the loop, but that "break" will only get out of the switch
not the loop. Moreover, the NN_NULL case looks like it'd fail
to notice end-of-frame. And it's not entirely clear what the
default case thinks it's doing either.
In short, this loop is impossible to understand, and the lack of
comments doesn't help. Even if it's not actually buggy, it
needs to be rewritten in a way that helps readers and compilers
see that it's not buggy. I think it might help to separate the
detection of null-ness and fetching of the datum value (if required)
from the loop control logic.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-05 23:51 Tatsuo Ishii <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-05 23:51 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Thank you for the report!
> Coverity is not very happy with this patch.
> It's complaining that the result of window_gettupleslot
> is not checked, which seems valid:
>
> 1503 {
> 1504 if (fetch_tuple)
>>>> CID 1666587: Error handling issues (CHECKED_RETURN)
>>>> Calling "window_gettupleslot" without checking return value (as is done elsewhere 8 out of 9 times).
> 1505 window_gettupleslot(winobj, pos, slot);
> 1506 if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
> 1507 return -1;
Yes, I forgot to check the return value of window_gettupleslot.
> and also that WinGetFuncArgInPartition is dereferencing
> a possibly-null "isout" pointer at several places, including
>
>>>> Dereferencing null pointer "isout".
> 3806 if (*isout) /* out of partition? */
>
>>>> Dereferencing null pointer "isout".
> 3817 if (!*isout && set_mark)
> 3818 WinSetMarkPosition(winobj, abs_pos);
>
>>>> Dereferencing null pointer "isout".
> 3817 if (!*isout && set_mark)
> 3818 WinSetMarkPosition(winobj, abs_pos);
>
> The latter complaints seem to be because some places in
> WinGetFuncArgInPartition check for nullness of that pointer
> and some do not. That looks like at least a latent bug
> to me.
Agreed.
Attached is a patch to fix the issue.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v1-0001-Fix-Coverity-issues-reported-in-commit-25a30bbd42.patch (2.2K, ../../[email protected]/2-v1-0001-Fix-Coverity-issues-reported-in-commit-25a30bbd42.patch)
download | inline diff:
From 28f5332b50a7cf11f16d35b2860628e58077c239 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Mon, 6 Oct 2025 08:45:03 +0900
Subject: [PATCH v1] Fix Coverity issues reported in commit 25a30bbd423.
This commit fixes several issues pointed out by Coverity.
- In row_is_in_frame(), return value of window_gettupleslot() was not checked.
- WinGetFuncArgInPartition() tried to derefference "isout" pointer
even if it's NULL in some places.
Discussion: https://postgr.es/m/202510051612.gw67jlc2iqpw%40alvherre.pgsql
---
src/backend/executor/nodeWindowAgg.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index cf667c81211..7ccf1160dca 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -1501,8 +1501,9 @@ row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot,
/* following row that is not peer is out of frame */
if (pos > winstate->currentpos)
{
- if (fetch_tuple)
- window_gettupleslot(winobj, pos, slot);
+ if (fetch_tuple) /* need to fetch tuple? */
+ if (!window_gettupleslot(winobj, pos, slot))
+ return -1;
if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
return -1;
}
@@ -3761,7 +3762,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
{
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, isout);
- if (!*isout && set_mark)
+ if (isout && !*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
return datum;
}
@@ -3803,7 +3804,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
default: /* need to check NULL or not */
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, isout);
- if (*isout) /* out of partition? */
+ if (isout && *isout) /* out of partition? */
return datum;
if (!*isnull)
@@ -3814,7 +3815,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
}
} while (notnull_offset < notnull_relpos);
- if (!*isout && set_mark)
+ if (isout && !*isout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
return datum;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-06 00:09 Tatsuo Ishii <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-06 00:09 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> will be exited this time. But there's another thing that is
> confusing: the NN_NULL case certainly looks like it's expecting
> to exit the loop, but that "break" will only get out of the switch
> not the loop.
You mean "NN_NOTNULL" case? if (notnull_offset >= notnull_relpos),
then following "while (notnull_offset < notnull_relpos)" does not
satisfy the continuous condition of the while loop and exits the loop.
I can add "goto" to explicitly exit the loop if we want.
> Moreover, the NN_NULL case looks like it'd fail
> to notice end-of-frame. And it's not entirely clear what the
> default case thinks it's doing either.
WinGetFuncArgInPartition() does not care about frame, no?
> In short, this loop is impossible to understand, and the lack of
> comments doesn't help. Even if it's not actually buggy, it
> needs to be rewritten in a way that helps readers and compilers
> see that it's not buggy. I think it might help to separate the
> detection of null-ness and fetching of the datum value (if required)
> from the loop control logic.
Thanks for the idea. Let me think if I could change the loop to be
easier to understand.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-06 00:28 Tatsuo Ishii <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-06 00:28 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Looking around, there is only one in-core caller of
> WinGetFuncArgInPartition, and it does pass a valid "isout" pointer,
> explaining why this inconsistency wasn't obvious in testing.
> There are outside callers though according to Debian Code Search,
> and at least PostGIS is one that passes a null pointer.
>
> As Alvaro notes nearby, this function is ridiculously complicated
> already. I'm tempted to remove the API allowance for isout == NULL,
> and thereby simplify the code slightly, rather than complicate it more
> by continuing to allow that. We'd have to warn the PostGIS people
> about the API change though.
It think it's a good idea.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-06 09:34 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-06 09:34 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Thank you for the report!
>
>> Coverity is not very happy with this patch.
>> It's complaining that the result of window_gettupleslot
>> is not checked, which seems valid:
>>
>> 1503 {
>> 1504 if (fetch_tuple)
>>>>> CID 1666587: Error handling issues (CHECKED_RETURN)
>>>>> Calling "window_gettupleslot" without checking return value (as is done elsewhere 8 out of 9 times).
>> 1505 window_gettupleslot(winobj, pos, slot);
>> 1506 if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
>> 1507 return -1;
>
> Yes, I forgot to check the return value of window_gettupleslot.
>
>> and also that WinGetFuncArgInPartition is dereferencing
>> a possibly-null "isout" pointer at several places, including
>>
>>>>> Dereferencing null pointer "isout".
>> 3806 if (*isout) /* out of partition? */
>>
>>>>> Dereferencing null pointer "isout".
>> 3817 if (!*isout && set_mark)
>> 3818 WinSetMarkPosition(winobj, abs_pos);
>>
>>>>> Dereferencing null pointer "isout".
>> 3817 if (!*isout && set_mark)
>> 3818 WinSetMarkPosition(winobj, abs_pos);
>>
>> The latter complaints seem to be because some places in
>> WinGetFuncArgInPartition check for nullness of that pointer
>> and some do not. That looks like at least a latent bug
>> to me.
>
> Agreed.
>
> Attached is a patch to fix the issue.
Please disregard the v1 patch. It includes a bug: If
WinGetFuncArgInPartition() is called with set_mark == true and isout
== NULL, WinSetMarkPosition() is not called by
WinGetFuncArgInPartition().
I will post v2 patch.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-07 02:28 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-07 02:28 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> Thank you for the report!
>>
>>> Coverity is not very happy with this patch.
>>> It's complaining that the result of window_gettupleslot
>>> is not checked, which seems valid:
>>>
>>> 1503 {
>>> 1504 if (fetch_tuple)
>>>>>> CID 1666587: Error handling issues (CHECKED_RETURN)
>>>>>> Calling "window_gettupleslot" without checking return value (as is done elsewhere 8 out of 9 times).
>>> 1505 window_gettupleslot(winobj, pos, slot);
>>> 1506 if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
>>> 1507 return -1;
>>
>> Yes, I forgot to check the return value of window_gettupleslot.
>>
>>> and also that WinGetFuncArgInPartition is dereferencing
>>> a possibly-null "isout" pointer at several places, including
>>>
>>>>>> Dereferencing null pointer "isout".
>>> 3806 if (*isout) /* out of partition? */
>>>
>>>>>> Dereferencing null pointer "isout".
>>> 3817 if (!*isout && set_mark)
>>> 3818 WinSetMarkPosition(winobj, abs_pos);
>>>
>>>>>> Dereferencing null pointer "isout".
>>> 3817 if (!*isout && set_mark)
>>> 3818 WinSetMarkPosition(winobj, abs_pos);
>>>
>>> The latter complaints seem to be because some places in
>>> WinGetFuncArgInPartition check for nullness of that pointer
>>> and some do not. That looks like at least a latent bug
>>> to me.
>>
>> Agreed.
>>
>> Attached is a patch to fix the issue.
>
> Please disregard the v1 patch. It includes a bug: If
> WinGetFuncArgInPartition() is called with set_mark == true and isout
> == NULL, WinSetMarkPosition() is not called by
> WinGetFuncArgInPartition().
>
> I will post v2 patch.
Attached is the v2 patch.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v2-0001-Fix-Coverity-issues-reported-in-commit-25a30bbd42.patch (4.4K, ../../[email protected]/2-v2-0001-Fix-Coverity-issues-reported-in-commit-25a30bbd42.patch)
download | inline diff:
From daf0113128eab7a55baf53fc676f863fba3914e8 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Tue, 7 Oct 2025 11:22:51 +0900
Subject: [PATCH v2] Fix Coverity issues reported in commit 25a30bbd423.
This commit fixes several issues pointed out by Coverity.
- In row_is_in_frame(), return value of window_gettupleslot() was not
checked.
- WinGetFuncArgInPartition() tried to derefference "isout" pointer
even if it's NULL in some places.
Moreover, in WinGetFuncArgInPartition refactor the do...while loop so
that the codes inside the loop simpler. Also simplify the case when
abs_pos < 0.
Discussion: https://postgr.es/m/202510051612.gw67jlc2iqpw%40alvherre.pgsql
---
src/backend/executor/nodeWindowAgg.c | 77 ++++++++++++++--------------
1 file changed, 38 insertions(+), 39 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index cf667c81211..0698aae37a7 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -1501,8 +1501,9 @@ row_is_in_frame(WindowObject winobj, int64 pos, TupleTableSlot *slot,
/* following row that is not peer is out of frame */
if (pos > winstate->currentpos)
{
- if (fetch_tuple)
- window_gettupleslot(winobj, pos, slot);
+ if (fetch_tuple) /* need to fetch tuple? */
+ if (!window_gettupleslot(winobj, pos, slot))
+ return -1;
if (!are_peers(winstate, slot, winstate->ss.ss_ScanTupleSlot))
return -1;
}
@@ -3721,6 +3722,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
int notnull_offset;
int notnull_relpos;
int forward;
+ bool myisout;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
@@ -3759,63 +3761,60 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
if (!null_treatment) /* IGNORE NULLS is not specified */
{
+ /* get tupple and evaluate in a partition */
datum = gettuple_eval_partition(winobj, argno,
- abs_pos, isnull, isout);
- if (!*isout && set_mark)
+ abs_pos, isnull, &myisout);
+ if (!myisout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
+ if (isout)
+ *isout = myisout;
return datum;
}
+ myisout = false;
+ datum = 0;
+
/*
* Get the next nonnull value in the partition, moving forward or backward
* until we find a value or reach the partition's end.
*/
do
{
+ int nn_info; /* NOT NULL info */
+
abs_pos += forward;
- if (abs_pos < 0)
- {
- /* out of partition */
- if (isout)
- *isout = true;
- *isnull = true;
- datum = 0;
+ if (abs_pos < 0) /* apparently out of partition */
break;
- }
- switch (get_notnull_info(winobj, abs_pos))
+ /* check NOT NULL cached info */
+ nn_info = get_notnull_info(winobj, abs_pos);
+ if (nn_info == NN_NOTNULL) /* this row is known to be NOT NULL */
+ notnull_offset++;
+
+ else if (nn_info == NN_NULL) /* this row is known to be NULL */
+ continue; /* keep on moving forward or backward */
+
+ else /* need to check NULL or not */
{
- case NN_NOTNULL: /* this row is known to be NOT NULL */
- notnull_offset++;
- if (notnull_offset >= notnull_relpos)
- {
- /* prepare to exit this loop */
- datum = gettuple_eval_partition(winobj, argno,
- abs_pos, isnull, isout);
- }
- break;
- case NN_NULL: /* this row is known to be NULL */
- if (isout)
- *isout = false;
- *isnull = true;
- datum = 0;
- break;
- default: /* need to check NULL or not */
- datum = gettuple_eval_partition(winobj, argno,
- abs_pos, isnull, isout);
- if (*isout) /* out of partition? */
- return datum;
-
- if (!*isnull)
- notnull_offset++;
- /* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ /* get tupple and evaluate in a partition */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, &myisout);
+ if (myisout) /* out of partition? */
break;
+ if (!*isnull)
+ notnull_offset++;
+ /* record the row status */
+ put_notnull_info(winobj, abs_pos, *isnull);
}
} while (notnull_offset < notnull_relpos);
- if (!*isout && set_mark)
+ /* get tupple and evaluate in a partition */
+ datum = gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, &myisout);
+ if (!myisout && set_mark)
WinSetMarkPosition(winobj, abs_pos);
+ if (isout)
+ *isout = myisout;
return datum;
}
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-07 20:14 Paul Ramsey <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Paul Ramsey @ 2025-10-07 20:14 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Mon, Oct 6, 2025 at 7:29 PM Tatsuo Ishii <[email protected]> wrote:
>
> > Please disregard the v1 patch. It includes a bug: If
> > WinGetFuncArgInPartition() is called with set_mark == true and isout
> > == NULL, WinSetMarkPosition() is not called by
> > WinGetFuncArgInPartition().
> >
> > I will post v2 patch.
>
> Attached is the v2 patch.
>
Thanks! This passes regression, and reads right to my eye and (most
important to me) allows PostGIS to run under Pg19 again.
Thanks,
P
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-07 23:50 Tatsuo Ishii <[email protected]>
parent: Álvaro Herrera <[email protected]>
1 sibling, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-07 23:50 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> I just noticed this compiler warning in a CI run,
>
> [16:06:29.920] ../src/backend/executor/nodeWindowAgg.c:3820:16: warning: ‘datum’ may be used uninitialized [-Wmaybe-uninitialized]
> [16:06:29.920] 3820 | return datum;
> [16:06:29.920] | ^~~~~
> [16:06:29.920] ../src/backend/executor/nodeWindowAgg.c:3719:25: note: ‘datum’ was declared here
> [16:06:29.920] 3719 | Datum datum;
> [16:06:29.920] | ^~~~~
>
> The logic in this function looks somewhat wicked.
Thanks for the report. I believe the warning is eliminated in the v2
patch[1].
Best regards,
[1] https://www.postgresql.org/message-id/20251007.112832.740065769089328041.ishii%40postgresql.org
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-08 00:31 Tatsuo Ishii <[email protected]>
parent: Paul Ramsey <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-08 00:31 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> Attached is the v2 patch.
>>
>
> Thanks! This passes regression, and reads right to my eye and (most
> important to me) allows PostGIS to run under Pg19 again.
Thank you for the review! I have just pushed the v2 patch.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-09 15:19 Tom Lane <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tom Lane @ 2025-10-09 15:19 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Tatsuo Ishii <[email protected]> writes:
> Thank you for the review! I have just pushed the v2 patch.
While I'd paid basically zero attention to this patch (the claim
in the commit message that I reviewed it is a flight of fancy),
I've been forced to look through it as a consequence of the mop-up
that's been happening to silence compiler warnings. There are a
couple of points that I think were not well done:
1. WinCheckAndInitializeNullTreatment really needs a rethink.
You cannot realistically assume that existing user-defined window
functions will be fixed to call that. I think it should be set up
so that if the window function fails to call that, then something in
mainline execution of nodeWindowAgg.c throws an error when there had
been a RESPECT/IGNORE NULLS option. With that idea, you could drop
the allowNullTreatment argument and just have the window functions
that support this syntax call something named along the lines of
WinAllowNullTreatmentOption. Also the error is certainly user-facing,
so using elog() was quite inappropriate. It should be ereport with an
errcode of (probably) ERRCODE_FEATURE_NOT_SUPPORTED. Rolling your
own implementation of get_func_name() wasn't great either.
Alternatively, you could just drop the entire concept of throwing an
error for that. What's the point? The implementation is entirely
within nodeWindowAgg.c and does not depend in any way on the
cooperation of the window function. I do not in any case like the
documentation's wording
+ This option is only allowed for the following functions: <function>lag</function>,
+ <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
+ <function>nth_value</function>.
as this fails to account for the possibility of user-defined window
functions. IMO we could drop the error check altogether and rewrite
the docs along the lines of "Not all window functions pay attention
to this option. Of the built-in window functions, only blah blah
and blah do."
2. AFAICS there is only one notnull_info array, which amounts to
assuming that the window function will have only one argument position
that it calls WinGetFuncArgInFrame or WinGetFuncArgInPartition for.
That may be true for the built-in functions but it seems mighty
restrictive for extensions. Worse yet, there's no check, so that
you'd just get silently wrong answers if two or more arguments are
evaluated. I think there ought to be a separate array for each argno;
of course only created if the window function actually asks for
evaluations of a particular argno.
regards, tom lane
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-11 00:07 Tatsuo Ishii <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 4 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-11 00:07 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> While I'd paid basically zero attention to this patch (the claim
> in the commit message that I reviewed it is a flight of fancy),
Sorry, I added you as a reviewer because you had joined past
discussions regarding this feature. Next time I will add reviewers
only those who actually looked into a patch.
> I've been forced to look through it as a consequence of the mop-up
> that's been happening to silence compiler warnings. There are a
> couple of points that I think were not well done:
>
> 1. WinCheckAndInitializeNullTreatment really needs a rethink.
> You cannot realistically assume that existing user-defined window
> functions will be fixed to call that.
Currently if WinCheckAndInitializeNullTreatment is not called,
RESPECT/IGNORE NULLS option is disregarded and WinGetFuncArgInFrame or
WinGetFuncArgInPartition works as if RESPECT/IGNORE NULLS option is
not given. So I thought it's safe even if existing user-defined window
functions are not fixed.
> I think it should be set up
> so that if the window function fails to call that, then something in
> mainline execution of nodeWindowAgg.c throws an error when there had
> been a RESPECT/IGNORE NULLS option. With that idea, you could drop
> the allowNullTreatment argument and just have the window functions
> that support this syntax call something named along the lines of
> WinAllowNullTreatmentOption.
Does that mean all user defined window functions start to fail after
upgrading to PostgreSQL 19? I am not sure if it's acceptable for
extension developers and their users.
> Also the error is certainly user-facing,
> so using elog() was quite inappropriate. It should be ereport with an
> errcode of (probably) ERRCODE_FEATURE_NOT_SUPPORTED. Rolling your
> own implementation of get_func_name() wasn't great either.
I overlooked the elog() call and "own implementation of
get_func_name()". Will fix.
> Alternatively, you could just drop the entire concept of throwing an
> error for that. What's the point?
If we do that, extensions would need to be re-tested against IGNORE
NULLS option case. I might be wrong but I guess some of (or many of)
extension developers do not plan (or have no time to work on it for
now) to utilize IGNORE NULLS option for their extensions.
For buil-in window functions. I don't want to create test cases how
built-in window functions, that are not allowed IGNORE NULLS option,
behave against IGNORE NULLS option. Instead I prefer to throw an error
as it is done today.
> The implementation is entirely
> within nodeWindowAgg.c and does not depend in any way on the
> cooperation of the window function. I do not in any case like the
> documentation's wording
>
> + This option is only allowed for the following functions: <function>lag</function>,
> + <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
> + <function>nth_value</function>.
> as this fails to account for the possibility of user-defined window
> functions.
The page explains only built-in window functions. Thus for me it's not
that strange that it does not say anything about user defined window
functions.
> IMO we could drop the error check altogether and rewrite
> the docs along the lines of "Not all window functions pay attention
> to this option. Of the built-in window functions, only blah blah
> and blah do."
>
> 2. AFAICS there is only one notnull_info array, which amounts to
> assuming that the window function will have only one argument position
> that it calls WinGetFuncArgInFrame or WinGetFuncArgInPartition for.
> That may be true for the built-in functions but it seems mighty
> restrictive for extensions. Worse yet, there's no check, so that
> you'd just get silently wrong answers if two or more arguments are
> evaluated. I think there ought to be a separate array for each argno;
> of course only created if the window function actually asks for
> evaluations of a particular argno.
I missed that. Thank you for pointed it out. I agree it would be
better allow to use multiple argument positions that calls
WinGetFuncArgInFrame or WinGetFuncArgInPartition in
extensions. Attached is a PoC patch for that.
Currently there's an issue with the patch, however.
SELECT x, y, mywindowfunc2(x, y, 2) IGNORE NULLS OVER w FROM g
WINDOW w AS (ORDER BY y);
psql:test2.sql:9: ERROR: cannot fetch row before WindowObject's mark position
mywindowfunc2 is a user defined window function, taking 3 arguments. x
and y are expected to be evaluated to integer. The third argument is
relative offset to current row. In the query above x and y are
retrieved using two WinGetFuncArgInPartition() calls. The data set
(table "g") looks like below.
x | y
----+---
| 1
| 2
10 | 3
20 | 4
(4 rows)
I think the cause of the error is:
(1) WinGetFuncArgInPartition keep on fetching column x until it's
evalued to not null and placed in the second row (in this case that's
x==20). In WinGetFuncArgInPartition WinSetMarkPosition is called at
abs_pos==3.
(2) WinGetFuncArgInPartition tries to fetch column y at row 0. Since
the mark was set to at row 3, the error occurred.
To avoid the error, we could call WinGetFuncArgInPartition with
set_mark = false (and call WinSetMarkPosition separately) but I am not
sure if it's an acceptable solution.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v1-0001-Allow-multiple-WinGetFuncArgInPartition-Frame-cal.patch (7.3K, ../../[email protected]/2-v1-0001-Allow-multiple-WinGetFuncArgInPartition-Frame-cal.patch)
download | inline diff:
From 1a96b19b6f12c5a19ba11c9d4f5ea82f04733e39 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Fri, 10 Oct 2025 20:53:17 +0900
Subject: [PATCH v1] Allow multiple WinGetFuncArgInPartition/Frame calls with
IGNORE NULLS option.
Previously it was assumed that there's only one call to
WinGetFuncArgInPartition/Frame in a window function when IGNORE NULLS
option is specified. To allow multiple calls to them,
winobj->notnull_info is modified from "uint8 *" to "uint8 **" so that
winobj->notnull_info could store pointers to not null info that
correspond to each function argument.
---
src/backend/executor/nodeWindowAgg.c | 67 +++++++++++++++++-----------
1 file changed, 42 insertions(+), 25 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index e6a53f95391..c7cb97a0643 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,7 +69,7 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
- uint8 *notnull_info; /* not null info */
+ uint8 **notnull_info; /* not null info for each func args */
int num_notnull_info; /* track size of the notnull_info array */
/*
@@ -214,10 +214,10 @@ static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
static Datum gettuple_eval_partition(WindowObject winobj, int argno,
int64 abs_pos, bool *isnull,
bool *isout);
-static void init_notnull_info(WindowObject winobj);
-static void grow_notnull_info(WindowObject winobj, int64 pos);
-static uint8 get_notnull_info(WindowObject winobj, int64 pos);
-static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+static void init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate);
+static void grow_notnull_info(WindowObject winobj, int64 pos, int argno);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos, int argno);
+static void put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull);
/*
* Not null info bit array consists of 2-bit items
@@ -1304,9 +1304,14 @@ begin_partition(WindowAggState *winstate)
/* reset null map */
if (winobj->ignore_nulls == IGNORE_NULLS)
- memset(winobj->notnull_info, 0,
- NN_POS_TO_BYTES(
- perfuncstate->winobj->num_notnull_info));
+ {
+ int numargs = perfuncstate->numArguments;
+
+ for (int j = 0; j < numargs; j++)
+ memset(winobj->notnull_info[j], 0,
+ NN_POS_TO_BYTES(
+ perfuncstate->winobj->num_notnull_info));
+ }
}
}
@@ -2734,7 +2739,7 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
winobj->ignore_nulls = wfunc->ignore_nulls;
- init_notnull_info(winobj);
+ init_notnull_info(winobj, perfuncstate);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3386,7 +3391,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
if (isout)
*isout = false;
- v = get_notnull_info(winobj, abs_pos);
+ v = get_notnull_info(winobj, abs_pos, argno);
if (v == NN_NULL) /* this row is known to be NULL */
goto advance;
@@ -3404,7 +3409,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
else /* this row is known to be NOT NULL */
{
@@ -3444,16 +3449,20 @@ out_of_frame:
* Initialize non null map.
*/
static void
-init_notnull_info(WindowObject winobj)
+init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate)
{
/* initial number of notnull info members */
#define INIT_NOT_NULL_INFO_NUM 128
+ int numargs = perfuncstate->numArguments;
if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
{
Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
- winobj->notnull_info = palloc0(size);
+ winobj->notnull_info = palloc(sizeof(uint8 *) * numargs);
+ for (int i = 0; i < numargs; i++)
+ /* allocate notnull_info for each argument */
+ winobj->notnull_info[i] = palloc0(size);
winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
}
}
@@ -3462,9 +3471,10 @@ init_notnull_info(WindowObject winobj)
* grow_notnull_info
* expand notnull_info if necessary.
* pos: not null info position
+ * argno: argument number
*/
static void
-grow_notnull_info(WindowObject winobj, int64 pos)
+grow_notnull_info(WindowObject winobj, int64 pos, int argno)
{
if (pos >= winobj->num_notnull_info)
{
@@ -3473,8 +3483,8 @@ grow_notnull_info(WindowObject winobj, int64 pos)
Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
Size newsize = oldsize * 2;
- winobj->notnull_info =
- repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->notnull_info[argno] =
+ repalloc0(winobj->notnull_info[argno], oldsize, newsize);
winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
if (winobj->num_notnull_info > pos)
break;
@@ -3486,16 +3496,19 @@ grow_notnull_info(WindowObject winobj, int64 pos)
* get_notnull_info
* retrieve a map
* pos: map position
+ * argno: argument number
*/
static uint8
-get_notnull_info(WindowObject winobj, int64 pos)
+get_notnull_info(WindowObject winobj, int64 pos, int argno)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
return (mb >> (NN_SHIFT(pos))) & NN_MASK;
}
@@ -3503,22 +3516,26 @@ get_notnull_info(WindowObject winobj, int64 pos)
* put_notnull_info
* update map
* pos: map position
+ * argno: argument number
+ * isnull: indicate NULL or NOT
*/
static void
-put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
uint8 val = isnull ? NN_NULL : NN_NOTNULL;
int shift;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
shift = NN_SHIFT(pos);
mb &= ~(NN_MASK << shift); /* clear map */
mb |= (val << shift); /* update map */
- winobj->notnull_info[bpos] = mb;
+ mbp[bpos] = mb;
}
/***********************************************************************
@@ -3787,7 +3804,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
/* check NOT NULL cached info */
- nn_info = get_notnull_info(winobj, abs_pos);
+ nn_info = get_notnull_info(winobj, abs_pos, argno);
if (nn_info == NN_NOTNULL) /* this row is known to be NOT NULL */
notnull_offset++;
else if (nn_info == NN_NULL) /* this row is known to be NULL */
@@ -3802,7 +3819,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
if (!*isnull)
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
} while (notnull_offset < notnull_relpos);
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-11 05:42 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
3 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-11 05:42 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> Also the error is certainly user-facing,
>> so using elog() was quite inappropriate. It should be ereport with an
>> errcode of (probably) ERRCODE_FEATURE_NOT_SUPPORTED. Rolling your
>> own implementation of get_func_name() wasn't great either.
>
> I overlooked the elog() call and "own implementation of
> get_func_name()". Will fix.
Attached is a trivial patch to fix that. I am going to push it if
there's no objection.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v1-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch (1.9K, ../../[email protected]/2-v1-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch)
download | inline diff:
From 61b4393bd12ad286e735a3bdf793443ecbf3b1aa Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sat, 11 Oct 2025 14:31:12 +0900
Subject: [PATCH v1] Use ereport rather than elog in
WinCheckAndInitializeNullTreatment.
Previously WinCheckAndInitializeNullTreatment() used elog() to emit an
error message. ereport() should be used instead because it's a
user-facing error.
Also fix WinCheckAndInitializeNullTreatment() to use existing
get_func_name() to get a function's name, rather than own
implementation.
Reported-by: Tom Lane <[email protected]>
Author: Tatsuo Ishii <[email protected]>
Discussion: https://postgr.es/m/2952409.1760023154%40sss.pgh.pa.us
---
src/backend/executor/nodeWindowAgg.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index e6a53f95391..892bdbd5ef5 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -3540,22 +3540,15 @@ WinCheckAndInitializeNullTreatment(WindowObject winobj,
{
if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
{
- HeapTuple proctup;
- Form_pg_proc procform;
- Oid funcid;
-
- funcid = fcinfo->flinfo->fn_oid;
- proctup = SearchSysCache1(PROCOID,
- ObjectIdGetDatum(funcid));
- if (!HeapTupleIsValid(proctup))
- elog(ERROR, "cache lookup failed for function %u", funcid);
- procform = (Form_pg_proc) GETSTRUCT(proctup);
- elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
- NameStr(procform->proname));
+ char *funcname = get_func_name(fcinfo->flinfo->fn_oid);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function %s does not allow RESPECT/IGNORE NULLS",
+ funcname)));
}
else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
winobj->ignore_nulls = IGNORE_NULLS;
-
}
/*
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-11 05:57 Chao Li <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Chao Li @ 2025-10-11 05:57 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> On Oct 11, 2025, at 13:42, Tatsuo Ishii <[email protected]> wrote:
>
>>> Also the error is certainly user-facing,
>>> so using elog() was quite inappropriate. It should be ereport with an
>>> errcode of (probably) ERRCODE_FEATURE_NOT_SUPPORTED. Rolling your
>>> own implementation of get_func_name() wasn't great either.
>>
>> I overlooked the elog() call and "own implementation of
>> get_func_name()". Will fix.
>
> Attached is a trivial patch to fix that. I am going to push it if
> there's no objection.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
> <v1-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch>
I just take a quick look at the patch, a tiny comment is:
```
+ char *funcname = get_func_name(fcinfo->flinfo->fn_oid);
```
This can be a “const char *”.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-11 09:11 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
3 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-11 09:11 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> I think it should be set up
>> so that if the window function fails to call that, then something in
>> mainline execution of nodeWindowAgg.c throws an error when there had
>> been a RESPECT/IGNORE NULLS option. With that idea, you could drop
>> the allowNullTreatment argument and just have the window functions
>> that support this syntax call something named along the lines of
>> WinAllowNullTreatmentOption.
>
> Does that mean all user defined window functions start to fail after
> upgrading to PostgreSQL 19? I am not sure if it's acceptable for
> extension developers and their users.
Probably I misunderstood what you said. Now I realize what you are
suggesting was, throwing an error *only* when a RESPECT/IGNORE NULLS
option is given and the function did not call
WinAllowNullTreatmentOption. If the option is not given, no error is
thrown even if WinAllowNullTreatmentOption is not called. I am okay
with this direction. I will post a patch for this.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-12 08:39 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-12 08:39 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Probably I misunderstood what you said. Now I realize what you are
> suggesting was, throwing an error *only* when a RESPECT/IGNORE NULLS
> option is given and the function did not call
> WinAllowNullTreatmentOption. If the option is not given, no error is
> thrown even if WinAllowNullTreatmentOption is not called. I am okay
> with this direction. I will post a patch for this.
While I was implementing this, I realized that in order to check if
WinAllowNullTreatmentOption had been called or not, it's necessary to
call the window function at least once in mainline execution of
nodeWindowAgg.c (I supposed in eval_windowfunction). I don't like this
because I expected to throw an error *before* calling the window
function.
So I studied your idea:
> Alternatively, you could just drop the entire concept of throwing an
> error for that.
Attached is a patch to implement this. Previously window functions
should call WinCheckAndInitializeNullTreatment with
allowNullTreatment==true if they accept a null treatment
clause. Otherwise, they are called as if null treatment clause is not
specified. With the patch, window functions accept a null treatment
clause as specified without calling
WinCheckAndInitializeNullTreatment.
There's one thing which might be different from what you suggested
is, I want to give window functions a method to stat that they do not
want accept a null treatment clause. For this purpose
WinCheckAndInitializeNullTreatment (with allowNullTreatment==false)
can be called.(Alternatively we could eliminate allowNullTreatment
argument and rename it something like WinDisallowNullTreatmentOption).
Some of built-in window functions that do not accept a null treatment
clause call this in the patch. This way, we do not need to test the
case when the functions are given a null treatment option except just
they throw an error. User defined functions would call the function
for the same purpose as built-in window functions.
> The implementation is entirely
> within nodeWindowAgg.c and does not depend in any way on the
> cooperation of the window function.
I am not sure. For example built-in lead function's behavior (with
IGNORE NULLS option) is defined by the standard. Unlike RESPECT NULLS
case, the expected behavior may not be obvious. According the
standard:
1. If lead's "offset" option is 0, the argument evaluated on current
row is returned regardless the value is NULL or NOT.
2. Otherwise, returns the value evaluated on a row which is nth NOT NULL.
For me, 2 is obvious but 1 was not so obvious because I thought that
lead() returns only non NULL value (except there's no non null values
or specified offset is out of partition).
Thus lead() calls WinGetFuncArgInPartition with
seektype==WINDOW_SEEK_CURRENT. Thus WinGetFuncArgInPartition with
WINDOW_SEEK_CURRENT is implemented in a way to satisfy the lead()
semantics above. This means if someone tries to implement a new window
function calling WinGetFuncArgInPartition with WINDOW_SEEK_CURRENT,
the function must has the same semantics as lead(). I think there's a
cooperation between nodeWindowAgg.c and window functions.
> + This option is only allowed for the following functions: <function>lag</function>,
> + <function>lead</function>, <function>first_value</function>, <function>last_value</function>,
> + <function>nth_value</function>.
>
> as this fails to account for the possibility of user-defined window
> functions. IMO we could drop the error check altogether and rewrite
> the docs along the lines of "Not all window functions pay attention
> to this option. Of the built-in window functions, only blah blah
> and blah do."
Fixing docs are not included in the patch (yet).
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v1-0001-Allow-window-functions-to-accept-a-null-treatment.patch (4.3K, ../../[email protected]/2-v1-0001-Allow-window-functions-to-accept-a-null-treatment.patch)
download | inline diff:
From 925c527b4e776d49292a0fd747dcb16357418703 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sun, 12 Oct 2025 16:21:40 +0900
Subject: [PATCH v1] Allow window functions to accept a null treatment clause
by default.
Previously window functions (either built-in or user defined) should
call WinCheckAndInitializeNullTreatment with allowNullTreatment==true
if they accept a null treatment clause (RESPECT NULLS/IGNORE
NULLS). Otherwise, they are called as if null treatment clause is not
specified.
This commit changes the behavior so that window functions accept a
null treatment clause as specified without calling
WinCheckAndInitializeNullTreatment.
If window functions do not want accept a null treatment clause, call
WinCheckAndInitializeNullTreatment with allowNullTreatment==false.
---
src/backend/executor/nodeWindowAgg.c | 17 +++++++++++------
src/backend/utils/adt/windowfuncs.c | 4 ----
2 files changed, 11 insertions(+), 10 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index e6a53f95391..c2e2ca69347 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -1303,7 +1303,8 @@ begin_partition(WindowAggState *winstate)
winobj->seekpos = -1;
/* reset null map */
- if (winobj->ignore_nulls == IGNORE_NULLS)
+ if (winobj->ignore_nulls == IGNORE_NULLS ||
+ winobj->ignore_nulls == PARSER_IGNORE_NULLS)
memset(winobj->notnull_info, 0,
NN_POS_TO_BYTES(
perfuncstate->winobj->num_notnull_info));
@@ -3530,8 +3531,10 @@ put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
* WinCheckAndInitializeNullTreatment
* Check null treatment clause and sets ignore_nulls
*
- * Window functions should call this to check if they are being called with
- * a null treatment clause when they don't allow it, or to set ignore_nulls.
+ * Window functions call this if they do not accept a null treatment clause
+ * with allowNullTreatment==false. It's not mandatory but they can call this
+ * with allowNullTreatment==true to explicitly stat that they accept a a null
+ * treatment clause.
*/
void
WinCheckAndInitializeNullTreatment(WindowObject winobj,
@@ -3555,7 +3558,6 @@ WinCheckAndInitializeNullTreatment(WindowObject winobj,
}
else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
winobj->ignore_nulls = IGNORE_NULLS;
-
}
/*
@@ -3727,7 +3729,9 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
- null_treatment = (winobj->ignore_nulls == IGNORE_NULLS && relpos != 0);
+ null_treatment = ((winobj->ignore_nulls == IGNORE_NULLS ||
+ winobj->ignore_nulls == PARSER_IGNORE_NULLS) &&
+ relpos != 0);
switch (seektype)
{
@@ -3866,7 +3870,8 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
- if (winobj->ignore_nulls == IGNORE_NULLS)
+ if (winobj->ignore_nulls == IGNORE_NULLS ||
+ winobj->ignore_nulls == PARSER_IGNORE_NULLS)
return ignorenulls_getfuncarginframe(winobj, argno, relpos, seektype,
set_mark, isnull, isout);
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 969f02aa59b..7e936a8bfbc 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -541,7 +541,6 @@ leadlag_common(FunctionCallInfo fcinfo,
bool isnull;
bool isout;
- WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
if (withoffset)
{
offset = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
@@ -659,7 +658,6 @@ window_first_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
- WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_HEAD, true,
&isnull, NULL);
@@ -681,7 +679,6 @@ window_last_value(PG_FUNCTION_ARGS)
Datum result;
bool isnull;
- WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
result = WinGetFuncArgInFrame(winobj, 0,
0, WINDOW_SEEK_TAIL, true,
&isnull, NULL);
@@ -705,7 +702,6 @@ window_nth_value(PG_FUNCTION_ARGS)
bool isnull;
int32 nth;
- WinCheckAndInitializeNullTreatment(winobj, true, fcinfo);
nth = DatumGetInt32(WinGetFuncArgCurrent(winobj, 1, &isnull));
if (isnull)
PG_RETURN_NULL();
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-12 12:05 Tatsuo Ishii <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-12 12:05 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Hi Tom,
> While I'd paid basically zero attention to this patch (the claim
> in the commit message that I reviewed it is a flight of fancy),
> I've been forced to look through it as a consequence of the mop-up
> that's been happening to silence compiler warnings.
Sorry for taking up your time to fix the compiler warnings. I haven't
noticed your commit 71540dcdcb2 until today. Next time I will try to
fix warnings found by buildfarm.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-13 04:43 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
3 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-13 04:43 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> 2. AFAICS there is only one notnull_info array, which amounts to
>> assuming that the window function will have only one argument position
>> that it calls WinGetFuncArgInFrame or WinGetFuncArgInPartition for.
>> That may be true for the built-in functions but it seems mighty
>> restrictive for extensions. Worse yet, there's no check, so that
>> you'd just get silently wrong answers if two or more arguments are
>> evaluated. I think there ought to be a separate array for each argno;
>> of course only created if the window function actually asks for
>> evaluations of a particular argno.
>
> I missed that. Thank you for pointed it out. I agree it would be
> better allow to use multiple argument positions that calls
> WinGetFuncArgInFrame or WinGetFuncArgInPartition in
> extensions. Attached is a PoC patch for that.
>
> Currently there's an issue with the patch, however.
>
> SELECT x, y, mywindowfunc2(x, y, 2) IGNORE NULLS OVER w FROM g
> WINDOW w AS (ORDER BY y);
> psql:test2.sql:9: ERROR: cannot fetch row before WindowObject's mark position
>
> mywindowfunc2 is a user defined window function, taking 3 arguments. x
> and y are expected to be evaluated to integer. The third argument is
> relative offset to current row. In the query above x and y are
> retrieved using two WinGetFuncArgInPartition() calls. The data set
> (table "g") looks like below.
>
> x | y
> ----+---
> | 1
> | 2
> 10 | 3
> 20 | 4
> (4 rows)
>
> I think the cause of the error is:
>
> (1) WinGetFuncArgInPartition keep on fetching column x until it's
> evalued to not null and placed in the second row (in this case that's
> x==20). In WinGetFuncArgInPartition WinSetMarkPosition is called at
> abs_pos==3.
>
> (2) WinGetFuncArgInPartition tries to fetch column y at row 0. Since
> the mark was set to at row 3, the error occurred.
>
> To avoid the error, we could call WinGetFuncArgInPartition with
> set_mark = false (and call WinSetMarkPosition separately) but I am not
> sure if it's an acceptable solution.
Attached is a v2 patch to fix the "cannot fetch row before
WindowObject's mark position" error, by tweaking the logic to
calculate the set mark position in WinGetFuncArgInPartition.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v2-0001-Allow-multiple-WinGetFuncArgInPartition-Frame-cal.patch (8.9K, ../../[email protected]/2-v2-0001-Allow-multiple-WinGetFuncArgInPartition-Frame-cal.patch)
download | inline diff:
From 2913e8189adf9d75bdefef3c62ad83dabf2fa5cb Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Mon, 13 Oct 2025 13:39:04 +0900
Subject: [PATCH v2] Allow multiple WinGetFuncArgInPartition/Frame calls with
IGNORE NULLS option.
Previously it was assumed that there's only one call to
WinGetFuncArgInPartition/Frame in a window function when IGNORE NULLS
option is specified. To allow multiple calls to them,
winobj->notnull_info is modified from "uint8 *" to "uint8 **" so that
winobj->notnull_info could store pointers to not null info that
correspond to each function argument.
Also fix the set mark position logic in WinGetFuncArgInPartition to
not raise a "cannot fetch row before WindowObject's mark position"
error in IGNORE NULLS case.
---
src/backend/executor/nodeWindowAgg.c | 87 +++++++++++++++++++---------
1 file changed, 61 insertions(+), 26 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index e6a53f95391..d25ab51ca82 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,7 +69,7 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
- uint8 *notnull_info; /* not null info */
+ uint8 **notnull_info; /* not null info for each func args */
int num_notnull_info; /* track size of the notnull_info array */
/*
@@ -214,10 +214,10 @@ static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
static Datum gettuple_eval_partition(WindowObject winobj, int argno,
int64 abs_pos, bool *isnull,
bool *isout);
-static void init_notnull_info(WindowObject winobj);
-static void grow_notnull_info(WindowObject winobj, int64 pos);
-static uint8 get_notnull_info(WindowObject winobj, int64 pos);
-static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+static void init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate);
+static void grow_notnull_info(WindowObject winobj, int64 pos, int argno);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos, int argno);
+static void put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull);
/*
* Not null info bit array consists of 2-bit items
@@ -1304,9 +1304,14 @@ begin_partition(WindowAggState *winstate)
/* reset null map */
if (winobj->ignore_nulls == IGNORE_NULLS)
- memset(winobj->notnull_info, 0,
- NN_POS_TO_BYTES(
- perfuncstate->winobj->num_notnull_info));
+ {
+ int numargs = perfuncstate->numArguments;
+
+ for (int j = 0; j < numargs; j++)
+ memset(winobj->notnull_info[j], 0,
+ NN_POS_TO_BYTES(
+ perfuncstate->winobj->num_notnull_info));
+ }
}
}
@@ -2734,7 +2739,7 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
winobj->ignore_nulls = wfunc->ignore_nulls;
- init_notnull_info(winobj);
+ init_notnull_info(winobj, perfuncstate);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3386,7 +3391,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
if (isout)
*isout = false;
- v = get_notnull_info(winobj, abs_pos);
+ v = get_notnull_info(winobj, abs_pos, argno);
if (v == NN_NULL) /* this row is known to be NULL */
goto advance;
@@ -3404,7 +3409,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
else /* this row is known to be NOT NULL */
{
@@ -3444,16 +3449,20 @@ out_of_frame:
* Initialize non null map.
*/
static void
-init_notnull_info(WindowObject winobj)
+init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate)
{
/* initial number of notnull info members */
#define INIT_NOT_NULL_INFO_NUM 128
+ int numargs = perfuncstate->numArguments;
if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
{
Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
- winobj->notnull_info = palloc0(size);
+ winobj->notnull_info = palloc(sizeof(uint8 *) * numargs);
+ for (int i = 0; i < numargs; i++)
+ /* allocate notnull_info for each argument */
+ winobj->notnull_info[i] = palloc0(size);
winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
}
}
@@ -3462,9 +3471,10 @@ init_notnull_info(WindowObject winobj)
* grow_notnull_info
* expand notnull_info if necessary.
* pos: not null info position
+ * argno: argument number
*/
static void
-grow_notnull_info(WindowObject winobj, int64 pos)
+grow_notnull_info(WindowObject winobj, int64 pos, int argno)
{
if (pos >= winobj->num_notnull_info)
{
@@ -3473,8 +3483,8 @@ grow_notnull_info(WindowObject winobj, int64 pos)
Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
Size newsize = oldsize * 2;
- winobj->notnull_info =
- repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->notnull_info[argno] =
+ repalloc0(winobj->notnull_info[argno], oldsize, newsize);
winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
if (winobj->num_notnull_info > pos)
break;
@@ -3486,16 +3496,19 @@ grow_notnull_info(WindowObject winobj, int64 pos)
* get_notnull_info
* retrieve a map
* pos: map position
+ * argno: argument number
*/
static uint8
-get_notnull_info(WindowObject winobj, int64 pos)
+get_notnull_info(WindowObject winobj, int64 pos, int argno)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
return (mb >> (NN_SHIFT(pos))) & NN_MASK;
}
@@ -3503,22 +3516,26 @@ get_notnull_info(WindowObject winobj, int64 pos)
* put_notnull_info
* update map
* pos: map position
+ * argno: argument number
+ * isnull: indicate NULL or NOT
*/
static void
-put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
uint8 val = isnull ? NN_NULL : NN_NOTNULL;
int shift;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
shift = NN_SHIFT(pos);
mb &= ~(NN_MASK << shift); /* clear map */
mb |= (val << shift); /* update map */
- winobj->notnull_info[bpos] = mb;
+ mbp[bpos] = mb;
}
/***********************************************************************
@@ -3717,6 +3734,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
{
WindowAggState *winstate;
int64 abs_pos;
+ int64 mark_pos;
Datum datum;
bool null_treatment;
int notnull_offset;
@@ -3772,6 +3790,23 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
forward = relpos > 0 ? 1 : -1;
myisout = false;
datum = 0;
+ /*
+ * IGNORE NULLS + WINDOW_SEEK_CURRENT + relpos > 0 case, we would fetch
+ * beyond the current row + relpos to find out the target row. If we mark
+ * at abs_pos, next call to WinGetFuncArgInPartition or
+ * WinGetFuncArgInFrame (in case when a window function have multiple
+ * args) could fail with "cannot fetch row before WindowObject's mark
+ * position". So keep the mark position at currentpos.
+ */
+ if (seektype == WINDOW_SEEK_CURRENT && relpos > 0)
+ mark_pos = winstate->currentpos;
+ else
+ /*
+ * For other cases we have no idea what position of row callers would
+ * fetch next time. Also for relpos < 0 case (we go backward), we
+ * cannot set mark either. For those cases we always set mark at 0.
+ */
+ mark_pos = 0;
/*
* Get the next nonnull value in the partition, moving forward or backward
@@ -3787,7 +3822,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
/* check NOT NULL cached info */
- nn_info = get_notnull_info(winobj, abs_pos);
+ nn_info = get_notnull_info(winobj, abs_pos, argno);
if (nn_info == NN_NOTNULL) /* this row is known to be NOT NULL */
notnull_offset++;
else if (nn_info == NN_NULL) /* this row is known to be NULL */
@@ -3802,7 +3837,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
if (!*isnull)
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
} while (notnull_offset < notnull_relpos);
@@ -3810,7 +3845,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
if (!myisout && set_mark)
- WinSetMarkPosition(winobj, abs_pos);
+ WinSetMarkPosition(winobj, mark_pos);
if (isout)
*isout = myisout;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-13 04:49 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
3 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-13 04:49 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> 2. AFAICS there is only one notnull_info array, which amounts to
>> assuming that the window function will have only one argument position
>> that it calls WinGetFuncArgInFrame or WinGetFuncArgInPartition for.
>> That may be true for the built-in functions but it seems mighty
>> restrictive for extensions. Worse yet, there's no check, so that
>> you'd just get silently wrong answers if two or more arguments are
>> evaluated. I think there ought to be a separate array for each argno;
>> of course only created if the window function actually asks for
>> evaluations of a particular argno.
>
> I missed that. Thank you for pointed it out. I agree it would be
> better allow to use multiple argument positions that calls
> WinGetFuncArgInFrame or WinGetFuncArgInPartition in
> extensions. Attached is a PoC patch for that.
>
> Currently there's an issue with the patch, however.
>
> SELECT x, y, mywindowfunc2(x, y, 2) IGNORE NULLS OVER w FROM g
> WINDOW w AS (ORDER BY y);
> psql:test2.sql:9: ERROR: cannot fetch row before WindowObject's mark position
>
> mywindowfunc2 is a user defined window function, taking 3 arguments. x
> and y are expected to be evaluated to integer. The third argument is
> relative offset to current row. In the query above x and y are
> retrieved using two WinGetFuncArgInPartition() calls. The data set
> (table "g") looks like below.
>
> x | y
> ----+---
> | 1
> | 2
> 10 | 3
> 20 | 4
> (4 rows)
>
> I think the cause of the error is:
>
> (1) WinGetFuncArgInPartition keep on fetching column x until it's
> evalued to not null and placed in the second row (in this case that's
> x==20). In WinGetFuncArgInPartition WinSetMarkPosition is called at
> abs_pos==3.
>
> (2) WinGetFuncArgInPartition tries to fetch column y at row 0. Since
> the mark was set to at row 3, the error occurred.
>
> To avoid the error, we could call WinGetFuncArgInPartition with
> set_mark = false (and call WinSetMarkPosition separately) but I am not
> sure if it's an acceptable solution.
Attached is a v2 patch to fix the "cannot fetch row before
WindowObject's mark position" error, by tweaking the logic to
calculate the set mark position in WinGetFuncArgInPartition.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v2-0001-Allow-multiple-WinGetFuncArgInPartition-Frame-cal.patch (8.9K, ../../[email protected]/2-v2-0001-Allow-multiple-WinGetFuncArgInPartition-Frame-cal.patch)
download | inline diff:
From 692f7d8b6c82cc76a7a9915f3adcca00fa1fd9ba Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Mon, 13 Oct 2025 13:48:54 +0900
Subject: [PATCH v2] Allow multiple WinGetFuncArgInPartition/Frame calls with
IGNORE NULLS option.
Previously it was assumed that there's only one call to
WinGetFuncArgInPartition/Frame in a window function when IGNORE NULLS
option is specified. To allow multiple calls to them,
winobj->notnull_info is modified from "uint8 *" to "uint8 **" so that
winobj->notnull_info could store pointers to not null info that
correspond to each function argument.
Also fix the set mark position logic in WinGetFuncArgInPartition to
not raise a "cannot fetch row before WindowObject's mark position"
error in IGNORE NULLS case.
---
src/backend/executor/nodeWindowAgg.c | 87 +++++++++++++++++++---------
1 file changed, 61 insertions(+), 26 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index e6a53f95391..d25ab51ca82 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,7 +69,7 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
- uint8 *notnull_info; /* not null info */
+ uint8 **notnull_info; /* not null info for each func args */
int num_notnull_info; /* track size of the notnull_info array */
/*
@@ -214,10 +214,10 @@ static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
static Datum gettuple_eval_partition(WindowObject winobj, int argno,
int64 abs_pos, bool *isnull,
bool *isout);
-static void init_notnull_info(WindowObject winobj);
-static void grow_notnull_info(WindowObject winobj, int64 pos);
-static uint8 get_notnull_info(WindowObject winobj, int64 pos);
-static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+static void init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate);
+static void grow_notnull_info(WindowObject winobj, int64 pos, int argno);
+static uint8 get_notnull_info(WindowObject winobj, int64 pos, int argno);
+static void put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull);
/*
* Not null info bit array consists of 2-bit items
@@ -1304,9 +1304,14 @@ begin_partition(WindowAggState *winstate)
/* reset null map */
if (winobj->ignore_nulls == IGNORE_NULLS)
- memset(winobj->notnull_info, 0,
- NN_POS_TO_BYTES(
- perfuncstate->winobj->num_notnull_info));
+ {
+ int numargs = perfuncstate->numArguments;
+
+ for (int j = 0; j < numargs; j++)
+ memset(winobj->notnull_info[j], 0,
+ NN_POS_TO_BYTES(
+ perfuncstate->winobj->num_notnull_info));
+ }
}
}
@@ -2734,7 +2739,7 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
winobj->ignore_nulls = wfunc->ignore_nulls;
- init_notnull_info(winobj);
+ init_notnull_info(winobj, perfuncstate);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3386,7 +3391,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
if (isout)
*isout = false;
- v = get_notnull_info(winobj, abs_pos);
+ v = get_notnull_info(winobj, abs_pos, argno);
if (v == NN_NULL) /* this row is known to be NULL */
goto advance;
@@ -3404,7 +3409,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
else /* this row is known to be NOT NULL */
{
@@ -3444,16 +3449,20 @@ out_of_frame:
* Initialize non null map.
*/
static void
-init_notnull_info(WindowObject winobj)
+init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate)
{
/* initial number of notnull info members */
#define INIT_NOT_NULL_INFO_NUM 128
+ int numargs = perfuncstate->numArguments;
if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
{
Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
- winobj->notnull_info = palloc0(size);
+ winobj->notnull_info = palloc(sizeof(uint8 *) * numargs);
+ for (int i = 0; i < numargs; i++)
+ /* allocate notnull_info for each argument */
+ winobj->notnull_info[i] = palloc0(size);
winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
}
}
@@ -3462,9 +3471,10 @@ init_notnull_info(WindowObject winobj)
* grow_notnull_info
* expand notnull_info if necessary.
* pos: not null info position
+ * argno: argument number
*/
static void
-grow_notnull_info(WindowObject winobj, int64 pos)
+grow_notnull_info(WindowObject winobj, int64 pos, int argno)
{
if (pos >= winobj->num_notnull_info)
{
@@ -3473,8 +3483,8 @@ grow_notnull_info(WindowObject winobj, int64 pos)
Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
Size newsize = oldsize * 2;
- winobj->notnull_info =
- repalloc0(winobj->notnull_info, oldsize, newsize);
+ winobj->notnull_info[argno] =
+ repalloc0(winobj->notnull_info[argno], oldsize, newsize);
winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
if (winobj->num_notnull_info > pos)
break;
@@ -3486,16 +3496,19 @@ grow_notnull_info(WindowObject winobj, int64 pos)
* get_notnull_info
* retrieve a map
* pos: map position
+ * argno: argument number
*/
static uint8
-get_notnull_info(WindowObject winobj, int64 pos)
+get_notnull_info(WindowObject winobj, int64 pos, int argno)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
return (mb >> (NN_SHIFT(pos))) & NN_MASK;
}
@@ -3503,22 +3516,26 @@ get_notnull_info(WindowObject winobj, int64 pos)
* put_notnull_info
* update map
* pos: map position
+ * argno: argument number
+ * isnull: indicate NULL or NOT
*/
static void
-put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
uint8 val = isnull ? NN_NULL : NN_NOTNULL;
int shift;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
shift = NN_SHIFT(pos);
mb &= ~(NN_MASK << shift); /* clear map */
mb |= (val << shift); /* update map */
- winobj->notnull_info[bpos] = mb;
+ mbp[bpos] = mb;
}
/***********************************************************************
@@ -3717,6 +3734,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
{
WindowAggState *winstate;
int64 abs_pos;
+ int64 mark_pos;
Datum datum;
bool null_treatment;
int notnull_offset;
@@ -3772,6 +3790,23 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
forward = relpos > 0 ? 1 : -1;
myisout = false;
datum = 0;
+ /*
+ * IGNORE NULLS + WINDOW_SEEK_CURRENT + relpos > 0 case, we would fetch
+ * beyond the current row + relpos to find out the target row. If we mark
+ * at abs_pos, next call to WinGetFuncArgInPartition or
+ * WinGetFuncArgInFrame (in case when a window function have multiple
+ * args) could fail with "cannot fetch row before WindowObject's mark
+ * position". So keep the mark position at currentpos.
+ */
+ if (seektype == WINDOW_SEEK_CURRENT && relpos > 0)
+ mark_pos = winstate->currentpos;
+ else
+ /*
+ * For other cases we have no idea what position of row callers would
+ * fetch next time. Also for relpos < 0 case (we go backward), we
+ * cannot set mark either. For those cases we always set mark at 0.
+ */
+ mark_pos = 0;
/*
* Get the next nonnull value in the partition, moving forward or backward
@@ -3787,7 +3822,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
/* check NOT NULL cached info */
- nn_info = get_notnull_info(winobj, abs_pos);
+ nn_info = get_notnull_info(winobj, abs_pos, argno);
if (nn_info == NN_NOTNULL) /* this row is known to be NOT NULL */
notnull_offset++;
else if (nn_info == NN_NULL) /* this row is known to be NULL */
@@ -3802,7 +3837,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
if (!*isnull)
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
} while (notnull_offset < notnull_relpos);
@@ -3810,7 +3845,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
if (!myisout && set_mark)
- WinSetMarkPosition(winobj, abs_pos);
+ WinSetMarkPosition(winobj, mark_pos);
if (isout)
*isout = myisout;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-13 05:39 Tatsuo Ishii <[email protected]>
parent: Chao Li <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-13 05:39 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>>>> Also the error is certainly user-facing,
>>>> so using elog() was quite inappropriate. It should be ereport with an
>>>> errcode of (probably) ERRCODE_FEATURE_NOT_SUPPORTED. Rolling your
>>>> own implementation of get_func_name() wasn't great either.
>>>
>>> I overlooked the elog() call and "own implementation of
>>> get_func_name()". Will fix.
>>
>> Attached is a trivial patch to fix that. I am going to push it if
>> there's no objection.
>>
>> Best regards,
>> --
>> Tatsuo Ishii
>> SRA OSS K.K.
>> English: http://www.sraoss.co.jp/index_en/
>> Japanese:http://www.sraoss.co.jp
>> <v1-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch>
>
>
> I just take a quick look at the patch, a tiny comment is:
>
> ```
> + char *funcname = get_func_name(fcinfo->flinfo->fn_oid);
> ```
>
> This can be a “const char *”.
Thanks for the review. In addition to the point, I added an assertion
which is called by all other window function API. Also added check to
the return value of get_func_name() because it could return NULL. V2
patch attached.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v2-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch (2.2K, ../../[email protected]/2-v2-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch)
download | inline diff:
From 976de4af0951a49e9fa55fad03f0582a4e77e7f5 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Mon, 13 Oct 2025 14:32:01 +0900
Subject: [PATCH v2] Use ereport rather than elog in
WinCheckAndInitializeNullTreatment.
Previously WinCheckAndInitializeNullTreatment() used elog() to emit an
error message. ereport() should be used instead because it's a
user-facing error. Also use existing get_func_name() to get a
function's name, rather than own implementation.
In addition to them, add an assertion to validate winobj parameter,
just like other window function API.
Reported-by: Tom Lane <[email protected]>
Author: Tatsuo Ishii <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Discussion: https://postgr.es/m/2952409.1760023154%40sss.pgh.pa.us
---
src/backend/executor/nodeWindowAgg.c | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index e6a53f95391..47e00be7b49 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -3538,24 +3538,20 @@ WinCheckAndInitializeNullTreatment(WindowObject winobj,
bool allowNullTreatment,
FunctionCallInfo fcinfo)
{
+ Assert(WindowObjectIsValid(winobj));
if (winobj->ignore_nulls != NO_NULLTREATMENT && !allowNullTreatment)
{
- HeapTuple proctup;
- Form_pg_proc procform;
- Oid funcid;
+ const char *funcname = get_func_name(fcinfo->flinfo->fn_oid);
- funcid = fcinfo->flinfo->fn_oid;
- proctup = SearchSysCache1(PROCOID,
- ObjectIdGetDatum(funcid));
- if (!HeapTupleIsValid(proctup))
- elog(ERROR, "cache lookup failed for function %u", funcid);
- procform = (Form_pg_proc) GETSTRUCT(proctup);
- elog(ERROR, "function %s does not allow RESPECT/IGNORE NULLS",
- NameStr(procform->proname));
+ if (!funcname)
+ elog(ERROR, "could not get function name");
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function %s does not allow RESPECT/IGNORE NULLS",
+ funcname)));
}
else if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
winobj->ignore_nulls = IGNORE_NULLS;
-
}
/*
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-13 11:22 Álvaro Herrera <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Álvaro Herrera @ 2025-10-13 11:22 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2025-Oct-13, Tatsuo Ishii wrote:
> Thanks for the review. In addition to the point, I added an assertion
> which is called by all other window function API. Also added check to
> the return value of get_func_name() because it could return NULL. V2
> patch attached.
Hmm, this change made me realize that all or almost all the calls to
get_func_name() would crash if it were to return a NULL value. I found
no caller that checks the return value for nullness. I wonder why do we
allow it to return NULL at all ... it might be better to just
elog(ERROR) if the cache entry is not found.
I think it was already wrong as introduced by 31c775adeb22.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"La espina, desde que nace, ya pincha" (Proverbio africano)
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-14 00:01 Tatsuo Ishii <[email protected]>
parent: Álvaro Herrera <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-14 00:01 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Hmm, this change made me realize that all or almost all the calls to
> get_func_name() would crash if it were to return a NULL value. I found
> no caller that checks the return value for nullness. I wonder why do we
> allow it to return NULL at all ... it might be better to just
> elog(ERROR) if the cache entry is not found.
I agree it's better but what about user defined functions? Some of
them might already check the return value to emit their own error
messages, I don't know. If so, modifying get_func_name() could break
them. Maybe invent something like get_func_name_with_error(calling
elog(ERROR)) and gradually update our code?
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-14 10:21 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-14 10:21 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>>>> I overlooked the elog() call and "own implementation of
>>>> get_func_name()". Will fix.
>>>
>>> Attached is a trivial patch to fix that. I am going to push it if
>>> there's no objection.
>>>
>>> Best regards,
>>> --
>>> Tatsuo Ishii
>>> SRA OSS K.K.
>>> English: http://www.sraoss.co.jp/index_en/
>>> Japanese:http://www.sraoss.co.jp
>>> <v1-0001-Use-ereport-rather-than-elog-in-WinCheckAndInitia.patch>
>>
>>
>> I just take a quick look at the patch, a tiny comment is:
>>
>> ```
>> + char *funcname = get_func_name(fcinfo->flinfo->fn_oid);
>> ```
>>
>> This can be a “const char *”.
>
> Thanks for the review. In addition to the point, I added an assertion
> which is called by all other window function API. Also added check to
> the return value of get_func_name() because it could return NULL. V2
> patch attached.
V2 patch pushed. Thanks.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-16 06:19 Michael Paquier <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Michael Paquier @ 2025-10-16 06:19 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Tue, Oct 14, 2025 at 07:21:13PM +0900, Tatsuo Ishii wrote:
> V2 patch pushed. Thanks.
Coverity thinks that this code has still some incorrect bits, and I
think that it is right to think so even on today's HEAD at
02c171f63fca.
In WinGetFuncArgInPartition()@nodeWindowAgg.c, we have the following
loop (keeping only the relevant parts:
do
{
[...]
else /* need to check NULL or not */
{
/* get tuple and evaluate in partition */
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
if (myisout) /* out of partition? */
break;
if (!*isnull)
notnull_offset++;
/* record the row status */
put_notnull_info(winobj, abs_pos, *isnull);
}
} while (notnull_offset < notnull_relpos);
/* get tuple and evaluate in partition */
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
And Coverity is telling that there is no point in setting a datum in
this else condition to just override its value when we exit the while
loop. To me, it's a sigh that this code's logic could be simplified.
In passing, gettuple_eval_partition() is under-documented for me. Its
name refers to the fact that it gets a tuple and evaluates a
partition. Its top comment tells the same thing as the name of the
function, so it's a bit hard to say why it is useful with the code
written this way, and how others many benefit when attempting to reuse
it, or if it even makes sense to reuse it for other purposes.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-16 10:17 Tatsuo Ishii <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-16 10:17 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Thanks for the report.
> Coverity thinks that this code has still some incorrect bits, and I
> think that it is right to think so even on today's HEAD at
> 02c171f63fca.
>
> In WinGetFuncArgInPartition()@nodeWindowAgg.c, we have the following
> loop (keeping only the relevant parts:
> do
> {
> [...]
> else /* need to check NULL or not */
> {
> /* get tuple and evaluate in partition */
> datum = gettuple_eval_partition(winobj, argno,
> abs_pos, isnull, &myisout);
> if (myisout) /* out of partition? */
> break;
> if (!*isnull)
> notnull_offset++;
> /* record the row status */
> put_notnull_info(winobj, abs_pos, *isnull);
> }
> } while (notnull_offset < notnull_relpos);
>
> /* get tuple and evaluate in partition */
> datum = gettuple_eval_partition(winobj, argno,
> abs_pos, isnull, &myisout);
>
> And Coverity is telling that there is no point in setting a datum in
> this else condition to just override its value when we exit the while
> loop. To me, it's a sigh that this code's logic could be simplified.
To fix the issue, I think we can change:
> datum = gettuple_eval_partition(winobj, argno,
> abs_pos, isnull, &myisout);
to:
(void) gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
This explicitely stats that we ignore the return value from
gettuple_eval_partition. I hope coverity understands this.
> In passing, gettuple_eval_partition() is under-documented for me. Its
> name refers to the fact that it gets a tuple and evaluates a
> partition. Its top comment tells the same thing as the name of the
> function, so it's a bit hard to say why it is useful with the code
> written this way, and how others many benefit when attempting to reuse
> it, or if it even makes sense to reuse it for other purposes.
What about changing the comment this way?
/* gettuple_eval_partition
* get tuple in a patition and evaluate the window function's argument
* expression on it.
*/
Attached is the patch for above.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v1-0001-Fix-coverity-complaint-about-WinGetFuncArgInParti.patch (2.4K, ../../[email protected]/2-v1-0001-Fix-coverity-complaint-about-WinGetFuncArgInParti.patch)
download | inline diff:
From c18dcbccff7bd7c95295beedabb2116cbb3b3be2 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Thu, 16 Oct 2025 19:00:56 +0900
Subject: [PATCH v1] Fix coverity complaint about WinGetFuncArgInPartition.
Coverity complains that the return value from gettuple_eval_partition
in a do..while loop in WinGetFuncArgInPartition is overwritten when
exiting the while loop. This commit tries to fix the issue by changing
gettuple_eval_partition call to:
(void) gettuple_eval_partition()
explicitly stating that we discard the return value.
Also enhance some comments for easier code reading.
Discussion: https://postgr.es/m/aPCOabSE4VfJLaky%40paquier.xyz
---
src/backend/executor/nodeWindowAgg.c | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 47e00be7b49..aa145d4e1a9 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -3270,8 +3270,9 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return true;
}
-/*
- * get tuple and evaluate in partition
+/* gettuple_eval_partition
+ * get tuple in a patition and evaluate the window function's argument
+ * expression on it.
*/
static Datum
gettuple_eval_partition(WindowObject winobj, int argno,
@@ -3790,9 +3791,15 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
continue; /* keep on moving forward or backward */
else /* need to check NULL or not */
{
- /* get tuple and evaluate in partition */
- datum = gettuple_eval_partition(winobj, argno,
- abs_pos, isnull, &myisout);
+ /*
+ * NOT NULL info does not exist yet. Get tuple and evaluate func
+ * arg in partition. We ignore the return value from
+ * gettuple_eval_partition because we are just interested in
+ * whether we are inside or outside of partition, NULL or NOT
+ * NULL.
+ */
+ (void) gettuple_eval_partition(winobj, argno,
+ abs_pos, isnull, &myisout);
if (myisout) /* out of partition? */
break;
if (!*isnull)
@@ -3802,7 +3809,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
}
} while (notnull_offset < notnull_relpos);
- /* get tuple and evaluate in partition */
+ /* get tuple and evaluate func arg in partition */
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
if (!myisout && set_mark)
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-16 11:50 Chao Li <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Chao Li @ 2025-10-16 11:50 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> On Oct 16, 2025, at 18:17, Tatsuo Ishii <[email protected]> wrote:
>
> Thanks for the report.
>
>> Coverity thinks that this code has still some incorrect bits, and I
>> think that it is right to think so even on today's HEAD at
>> 02c171f63fca.
>>
>> In WinGetFuncArgInPartition()@nodeWindowAgg.c, we have the following
>> loop (keeping only the relevant parts:
>> do
>> {
>> [...]
>> else /* need to check NULL or not */
>> {
>> /* get tuple and evaluate in partition */
>> datum = gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>> if (myisout) /* out of partition? */
>> break;
>> if (!*isnull)
>> notnull_offset++;
>> /* record the row status */
>> put_notnull_info(winobj, abs_pos, *isnull);
>> }
>> } while (notnull_offset < notnull_relpos);
>>
>> /* get tuple and evaluate in partition */
>> datum = gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>>
>> And Coverity is telling that there is no point in setting a datum in
>> this else condition to just override its value when we exit the while
>> loop. To me, it's a sigh that this code's logic could be simplified.
>
> To fix the issue, I think we can change:
>
>> datum = gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>
> to:
>
> (void) gettuple_eval_partition(winobj, argno,
> abs_pos, isnull, &myisout);
>
> This explicitely stats that we ignore the return value from
> gettuple_eval_partition. I hope coverity understands this.
>
>>
I think Coverity is complaining about the redundant call to gettuple_eval_partition().
In the “else” clause, the function is called, then when “if (myisout)” is satisfied, it will break out the while loop. After that, the function is immediately called again, so “datum” is overwritten. But I haven’t spent time thinking about how to fix.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-19 00:38 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-19 00:38 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> Thanks for the report.
>
>> Coverity thinks that this code has still some incorrect bits, and I
>> think that it is right to think so even on today's HEAD at
>> 02c171f63fca.
>>
>> In WinGetFuncArgInPartition()@nodeWindowAgg.c, we have the following
>> loop (keeping only the relevant parts:
>> do
>> {
>> [...]
>> else /* need to check NULL or not */
>> {
>> /* get tuple and evaluate in partition */
>> datum = gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>> if (myisout) /* out of partition? */
>> break;
>> if (!*isnull)
>> notnull_offset++;
>> /* record the row status */
>> put_notnull_info(winobj, abs_pos, *isnull);
>> }
>> } while (notnull_offset < notnull_relpos);
>>
>> /* get tuple and evaluate in partition */
>> datum = gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>>
>> And Coverity is telling that there is no point in setting a datum in
>> this else condition to just override its value when we exit the while
>> loop. To me, it's a sigh that this code's logic could be simplified.
>
> To fix the issue, I think we can change:
>
>> datum = gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>
> to:
>
> (void) gettuple_eval_partition(winobj, argno,
> abs_pos, isnull, &myisout);
>
> This explicitely stats that we ignore the return value from
> gettuple_eval_partition. I hope coverity understands this.
>
>> In passing, gettuple_eval_partition() is under-documented for me. Its
>> name refers to the fact that it gets a tuple and evaluates a
>> partition. Its top comment tells the same thing as the name of the
>> function, so it's a bit hard to say why it is useful with the code
>> written this way, and how others many benefit when attempting to reuse
>> it, or if it even makes sense to reuse it for other purposes.
>
> What about changing the comment this way?
>
> /* gettuple_eval_partition
> * get tuple in a patition and evaluate the window function's argument
> * expression on it.
> */
>
> Attached is the patch for above.
Patch pushed with minor comment tweaks.
Thanks.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-19 01:02 Tatsuo Ishii <[email protected]>
parent: Chao Li <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-19 01:02 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> On Oct 16, 2025, at 18:17, Tatsuo Ishii <[email protected]> wrote:
>>
>> Thanks for the report.
>>
>>> Coverity thinks that this code has still some incorrect bits, and I
>>> think that it is right to think so even on today's HEAD at
>>> 02c171f63fca.
>>>
>>> In WinGetFuncArgInPartition()@nodeWindowAgg.c, we have the following
>>> loop (keeping only the relevant parts:
>>> do
>>> {
>>> [...]
>>> else /* need to check NULL or not */
>>> {
>>> /* get tuple and evaluate in partition */
>>> datum = gettuple_eval_partition(winobj, argno,
>>> abs_pos, isnull, &myisout);
>>> if (myisout) /* out of partition? */
>>> break;
>>> if (!*isnull)
>>> notnull_offset++;
>>> /* record the row status */
>>> put_notnull_info(winobj, abs_pos, *isnull);
>>> }
>>> } while (notnull_offset < notnull_relpos);
>>>
>>> /* get tuple and evaluate in partition */
>>> datum = gettuple_eval_partition(winobj, argno,
>>> abs_pos, isnull, &myisout);
>>>
>>> And Coverity is telling that there is no point in setting a datum in
>>> this else condition to just override its value when we exit the while
>>> loop. To me, it's a sigh that this code's logic could be simplified.
>>
>> To fix the issue, I think we can change:
>>
>>> datum = gettuple_eval_partition(winobj, argno,
>>> abs_pos, isnull, &myisout);
>>
>> to:
>>
>> (void) gettuple_eval_partition(winobj, argno,
>> abs_pos, isnull, &myisout);
>>
>> This explicitely stats that we ignore the return value from
>> gettuple_eval_partition. I hope coverity understands this.
>>
>>>
>
> I think Coverity is complaining about the redundant call to gettuple_eval_partition().
>
> In the “else” clause, the function is called, then when “if (myisout)” is satisfied, it will break out the while loop. After that, the function is immediately called again, so “datum” is overwritten. But I haven’t spent time thinking about how to fix.
Yes, the function is called again. But I think the cost is cheap in
this case. Inside the function window_gettupleslot() is called. It
could be costly if it spools tuples. But as tuple is already spooled
by the former call of gettuple_eval_partition(), almost no cost is
needed. We could avoid the redundant call by putting more code after
the former function call to return immediately, or introduce a goto
statement or a flag. But I think they will make the code harder to
read and do not worth the trouble. Others may think differently
though.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-19 09:53 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 2 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-19 09:53 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>>> 2. AFAICS there is only one notnull_info array, which amounts to
>>> assuming that the window function will have only one argument position
>>> that it calls WinGetFuncArgInFrame or WinGetFuncArgInPartition for.
>>> That may be true for the built-in functions but it seems mighty
>>> restrictive for extensions. Worse yet, there's no check, so that
>>> you'd just get silently wrong answers if two or more arguments are
>>> evaluated. I think there ought to be a separate array for each argno;
>>> of course only created if the window function actually asks for
>>> evaluations of a particular argno.
>>
>> I missed that. Thank you for pointed it out. I agree it would be
>> better allow to use multiple argument positions that calls
>> WinGetFuncArgInFrame or WinGetFuncArgInPartition in
>> extensions. Attached is a PoC patch for that.
>>
>> Currently there's an issue with the patch, however.
>>
>> SELECT x, y, mywindowfunc2(x, y, 2) IGNORE NULLS OVER w FROM g
>> WINDOW w AS (ORDER BY y);
>> psql:test2.sql:9: ERROR: cannot fetch row before WindowObject's mark position
>>
>> mywindowfunc2 is a user defined window function, taking 3 arguments. x
>> and y are expected to be evaluated to integer. The third argument is
>> relative offset to current row. In the query above x and y are
>> retrieved using two WinGetFuncArgInPartition() calls. The data set
>> (table "g") looks like below.
>>
>> x | y
>> ----+---
>> | 1
>> | 2
>> 10 | 3
>> 20 | 4
>> (4 rows)
>>
>> I think the cause of the error is:
>>
>> (1) WinGetFuncArgInPartition keep on fetching column x until it's
>> evalued to not null and placed in the second row (in this case that's
>> x==20). In WinGetFuncArgInPartition WinSetMarkPosition is called at
>> abs_pos==3.
>>
>> (2) WinGetFuncArgInPartition tries to fetch column y at row 0. Since
>> the mark was set to at row 3, the error occurred.
>>
>> To avoid the error, we could call WinGetFuncArgInPartition with
>> set_mark = false (and call WinSetMarkPosition separately) but I am not
>> sure if it's an acceptable solution.
>
> Attached is a v2 patch to fix the "cannot fetch row before
> WindowObject's mark position" error, by tweaking the logic to
> calculate the set mark position in WinGetFuncArgInPartition.
Attached is a v3 patch which is ready for commit IMO. Major
difference from v2 patch is, now the patch satisfies the request
below.
>>> of course only created if the window function actually asks for
>>> evaluations of a particular argno.
The NOT NULL information array is allocated only when the window
function actually asks for evaluations of a particular argno using
WinGetFuncArgInFrame or WinGetFuncArgInPartition.
If there's no objection, I am going to commit in a few days.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v3-0001-Fix-multi-WinGetFuncArgInFrame-Partition-calls-wi.patch (10.7K, ../../[email protected]/2-v3-0001-Fix-multi-WinGetFuncArgInFrame-Partition-calls-wi.patch)
download | inline diff:
From 888dc5db9e8a8c0bb4b9b1603e5335f209c2d98c Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Sun, 19 Oct 2025 18:37:08 +0900
Subject: [PATCH v3] Fix multi WinGetFuncArgInFrame/Partition calls with IGNORE
NULLS.
Previously it was mistakenly assumed that there's only one window
function argument which needs to be processed by WinGetFuncArgInFrame
or WinGetFuncArgInPartition when IGNORE NULLS option is specified. To
eliminate the limitation, WindowObject->notnull_info is modified from
"uint8 *" to "uint8 **" so that WindowObject->notnull_info could store
pointers to "uint8 *" which holds NOT NULL info corresponding to each
window function argument. Moreover, WindowObject->num_notnull_info is
changed from "int" to "int64 *" so that WindowObject->num_notnull_info
could store the number of NOT NULL info corresponding to each function
argument. Memories for these data structures will be allocated when
WinGetFuncArgInFrame or WinGetFuncArgInPartition is called. Thus no
memory except the pointers is allocated for function arguments which
do not call these functions
Also fix the set mark position logic in WinGetFuncArgInPartition to
not raise a "cannot fetch row before WindowObject's mark position"
error in IGNORE NULLS case.
Reported-by: Tom Lane <[email protected]>
Author: Tatsuo Ishii <[email protected]>
Discussion: https://postgr.es/m/2952409.1760023154%40sss.pgh.pa.us
---
src/backend/executor/nodeWindowAgg.c | 136 +++++++++++++++++++--------
1 file changed, 98 insertions(+), 38 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index aa145d4e1a9..497eb25ea29 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -69,8 +69,10 @@ typedef struct WindowObjectData
int readptr; /* tuplestore read pointer for this fn */
int64 markpos; /* row that markptr is positioned on */
int64 seekpos; /* row that readptr is positioned on */
- uint8 *notnull_info; /* not null info */
- int num_notnull_info; /* track size of the notnull_info array */
+ uint8 **notnull_info; /* not null info for each func args */
+ int64 *num_notnull_info; /* track size (number of tuples in
+ * partition) of the notnull_info array
+ * for each func args */
/*
* Null treatment options. One of: NO_NULLTREATMENT, PARSER_IGNORE_NULLS,
@@ -214,10 +216,14 @@ static Datum ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
static Datum gettuple_eval_partition(WindowObject winobj, int argno,
int64 abs_pos, bool *isnull,
bool *isout);
-static void init_notnull_info(WindowObject winobj);
-static void grow_notnull_info(WindowObject winobj, int64 pos);
-static uint8 get_notnull_info(WindowObject winobj, int64 pos);
-static void put_notnull_info(WindowObject winobj, int64 pos, bool isnull);
+static void init_notnull_info(WindowObject winobj,
+ WindowStatePerFunc perfuncstate);
+static void grow_notnull_info(WindowObject winobj,
+ int64 pos, int argno);
+static uint8 get_notnull_info(WindowObject winobj,
+ int64 pos, int argno);
+static void put_notnull_info(WindowObject winobj,
+ int64 pos, int argno, bool isnull);
/*
* Not null info bit array consists of 2-bit items
@@ -1303,10 +1309,20 @@ begin_partition(WindowAggState *winstate)
winobj->seekpos = -1;
/* reset null map */
- if (winobj->ignore_nulls == IGNORE_NULLS)
- memset(winobj->notnull_info, 0,
- NN_POS_TO_BYTES(
- perfuncstate->winobj->num_notnull_info));
+ if (winobj->ignore_nulls == IGNORE_NULLS ||
+ winobj->ignore_nulls == PARSER_IGNORE_NULLS)
+ {
+ int numargs = perfuncstate->numArguments;
+
+ for (int j = 0; j < numargs; j++)
+ {
+ int n = winobj->num_notnull_info[j];
+
+ if (n > 0)
+ memset(winobj->notnull_info[j], 0,
+ NN_POS_TO_BYTES(n));
+ }
+ }
}
}
@@ -2734,7 +2750,7 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winobj->localmem = NULL;
perfuncstate->winobj = winobj;
winobj->ignore_nulls = wfunc->ignore_nulls;
- init_notnull_info(winobj);
+ init_notnull_info(winobj, perfuncstate);
/* It's a real window function, so set up to call it. */
fmgr_info_cxt(wfunc->winfnoid, &perfuncstate->flinfo,
@@ -3387,7 +3403,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
if (isout)
*isout = false;
- v = get_notnull_info(winobj, abs_pos);
+ v = get_notnull_info(winobj, abs_pos, argno);
if (v == NN_NULL) /* this row is known to be NULL */
goto advance;
@@ -3405,7 +3421,7 @@ ignorenulls_getfuncarginframe(WindowObject winobj, int argno,
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
else /* this row is known to be NOT NULL */
{
@@ -3445,17 +3461,14 @@ out_of_frame:
* Initialize non null map.
*/
static void
-init_notnull_info(WindowObject winobj)
+init_notnull_info(WindowObject winobj, WindowStatePerFunc perfuncstate)
{
-/* initial number of notnull info members */
-#define INIT_NOT_NULL_INFO_NUM 128
+ int numargs = perfuncstate->numArguments;
if (winobj->ignore_nulls == PARSER_IGNORE_NULLS)
{
- Size size = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
-
- winobj->notnull_info = palloc0(size);
- winobj->num_notnull_info = INIT_NOT_NULL_INFO_NUM;
+ winobj->notnull_info = palloc0(sizeof(uint8 *) * numargs);
+ winobj->num_notnull_info = palloc0(sizeof(int64) * numargs);
}
}
@@ -3463,23 +3476,43 @@ init_notnull_info(WindowObject winobj)
* grow_notnull_info
* expand notnull_info if necessary.
* pos: not null info position
+ * argno: argument number
*/
static void
-grow_notnull_info(WindowObject winobj, int64 pos)
+grow_notnull_info(WindowObject winobj, int64 pos, int argno)
{
- if (pos >= winobj->num_notnull_info)
+/* initial number of notnull info members */
+#define INIT_NOT_NULL_INFO_NUM 128
+
+ if (pos >= winobj->num_notnull_info[argno])
{
+ /* We may be called in a short-lived context */
+ MemoryContext oldcontext = MemoryContextSwitchTo
+ (winobj->winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
+
for (;;)
{
- Size oldsize = NN_POS_TO_BYTES(winobj->num_notnull_info);
- Size newsize = oldsize * 2;
+ Size oldsize = NN_POS_TO_BYTES
+ (winobj->num_notnull_info[argno]);
+ Size newsize;
- winobj->notnull_info =
- repalloc0(winobj->notnull_info, oldsize, newsize);
- winobj->num_notnull_info = NN_BYTES_TO_POS(newsize);
- if (winobj->num_notnull_info > pos)
+ if (oldsize == 0) /* memory has not been allocated yet for this
+ * arg */
+ {
+ newsize = NN_POS_TO_BYTES(INIT_NOT_NULL_INFO_NUM);
+ winobj->notnull_info[argno] = palloc0(newsize);
+ }
+ else
+ {
+ newsize = oldsize * 2;
+ winobj->notnull_info[argno] =
+ repalloc0(winobj->notnull_info[argno], oldsize, newsize);
+ }
+ winobj->num_notnull_info[argno] = NN_BYTES_TO_POS(newsize);
+ if (winobj->num_notnull_info[argno] > pos)
break;
}
+ MemoryContextSwitchTo(oldcontext);
}
}
@@ -3487,16 +3520,19 @@ grow_notnull_info(WindowObject winobj, int64 pos)
* get_notnull_info
* retrieve a map
* pos: map position
+ * argno: argument number
*/
static uint8
-get_notnull_info(WindowObject winobj, int64 pos)
+get_notnull_info(WindowObject winobj, int64 pos, int argno)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
return (mb >> (NN_SHIFT(pos))) & NN_MASK;
}
@@ -3504,22 +3540,26 @@ get_notnull_info(WindowObject winobj, int64 pos)
* put_notnull_info
* update map
* pos: map position
+ * argno: argument number
+ * isnull: indicate NULL or NOT
*/
static void
-put_notnull_info(WindowObject winobj, int64 pos, bool isnull)
+put_notnull_info(WindowObject winobj, int64 pos, int argno, bool isnull)
{
+ uint8 *mbp;
uint8 mb;
int64 bpos;
uint8 val = isnull ? NN_NULL : NN_NOTNULL;
int shift;
- grow_notnull_info(winobj, pos);
+ grow_notnull_info(winobj, pos, argno);
bpos = NN_POS_TO_BYTES(pos);
- mb = winobj->notnull_info[bpos];
+ mbp = winobj->notnull_info[argno];
+ mb = mbp[bpos];
shift = NN_SHIFT(pos);
mb &= ~(NN_MASK << shift); /* clear map */
mb |= (val << shift); /* update map */
- winobj->notnull_info[bpos] = mb;
+ mbp[bpos] = mb;
}
/***********************************************************************
@@ -3714,6 +3754,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
{
WindowAggState *winstate;
int64 abs_pos;
+ int64 mark_pos;
Datum datum;
bool null_treatment;
int notnull_offset;
@@ -3770,6 +3811,25 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
myisout = false;
datum = 0;
+ /*
+ * IGNORE NULLS + WINDOW_SEEK_CURRENT + relpos > 0 case, we would fetch
+ * beyond the current row + relpos to find out the target row. If we mark
+ * at abs_pos, next call to WinGetFuncArgInPartition or
+ * WinGetFuncArgInFrame (in case when a window function have multiple
+ * args) could fail with "cannot fetch row before WindowObject's mark
+ * position". So keep the mark position at currentpos.
+ */
+ if (seektype == WINDOW_SEEK_CURRENT && relpos > 0)
+ mark_pos = winstate->currentpos;
+ else
+
+ /*
+ * For other cases we have no idea what position of row callers would
+ * fetch next time. Also for relpos < 0 case (we go backward), we
+ * cannot set mark either. For those cases we always set mark at 0.
+ */
+ mark_pos = 0;
+
/*
* Get the next nonnull value in the partition, moving forward or backward
* until we find a value or reach the partition's end. We cache the
@@ -3784,7 +3844,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
break;
/* check NOT NULL cached info */
- nn_info = get_notnull_info(winobj, abs_pos);
+ nn_info = get_notnull_info(winobj, abs_pos, argno);
if (nn_info == NN_NOTNULL) /* this row is known to be NOT NULL */
notnull_offset++;
else if (nn_info == NN_NULL) /* this row is known to be NULL */
@@ -3805,7 +3865,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
if (!*isnull)
notnull_offset++;
/* record the row status */
- put_notnull_info(winobj, abs_pos, *isnull);
+ put_notnull_info(winobj, abs_pos, argno, *isnull);
}
} while (notnull_offset < notnull_relpos);
@@ -3813,7 +3873,7 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
datum = gettuple_eval_partition(winobj, argno,
abs_pos, isnull, &myisout);
if (!myisout && set_mark)
- WinSetMarkPosition(winobj, abs_pos);
+ WinSetMarkPosition(winobj, mark_pos);
if (isout)
*isout = myisout;
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-20 03:22 Chao Li <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 1 reply; 91+ messages in thread
From: Chao Li @ 2025-10-20 03:22 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> On Oct 19, 2025, at 17:53, Tatsuo Ishii <[email protected]> wrote:
>
>
> If there's no objection, I am going to commit in a few days.
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
> <v3-0001-Fix-multi-WinGetFuncArgInFrame-Partition-calls-wi.patch>
A very trivial commit:
```
+ else
+
+ /*
+ * For other cases we have no idea what position of row callers would
+ * fetch next time. Also for relpos < 0 case (we go backward), we
+ * cannot set mark either. For those cases we always set mark at 0.
+ */
+ mark_pos = 0;
```
The empty line after “else” is not needed.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-20 03:58 Tatsuo Ishii <[email protected]>
parent: Chao Li <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-20 03:58 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> A very trivial commit:
>
> ```
> + else
> +
> + /*
> + * For other cases we have no idea what position of row callers would
> + * fetch next time. Also for relpos < 0 case (we go backward), we
> + * cannot set mark either. For those cases we always set mark at 0.
> + */
> + mark_pos = 0;
> ```
>
> The empty line after “else” is not needed.
That was added by pgindent.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-22 03:14 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
1 sibling, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-22 03:14 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>>>> 2. AFAICS there is only one notnull_info array, which amounts to
>>>> assuming that the window function will have only one argument position
>>>> that it calls WinGetFuncArgInFrame or WinGetFuncArgInPartition for.
>>>> That may be true for the built-in functions but it seems mighty
>>>> restrictive for extensions. Worse yet, there's no check, so that
>>>> you'd just get silently wrong answers if two or more arguments are
>>>> evaluated. I think there ought to be a separate array for each argno;
>>>> of course only created if the window function actually asks for
>>>> evaluations of a particular argno.
>>>
>>> I missed that. Thank you for pointed it out. I agree it would be
>>> better allow to use multiple argument positions that calls
>>> WinGetFuncArgInFrame or WinGetFuncArgInPartition in
>>> extensions. Attached is a PoC patch for that.
>>>
>>> Currently there's an issue with the patch, however.
>>>
>>> SELECT x, y, mywindowfunc2(x, y, 2) IGNORE NULLS OVER w FROM g
>>> WINDOW w AS (ORDER BY y);
>>> psql:test2.sql:9: ERROR: cannot fetch row before WindowObject's mark position
>>>
>>> mywindowfunc2 is a user defined window function, taking 3 arguments. x
>>> and y are expected to be evaluated to integer. The third argument is
>>> relative offset to current row. In the query above x and y are
>>> retrieved using two WinGetFuncArgInPartition() calls. The data set
>>> (table "g") looks like below.
>>>
>>> x | y
>>> ----+---
>>> | 1
>>> | 2
>>> 10 | 3
>>> 20 | 4
>>> (4 rows)
>>>
>>> I think the cause of the error is:
>>>
>>> (1) WinGetFuncArgInPartition keep on fetching column x until it's
>>> evalued to not null and placed in the second row (in this case that's
>>> x==20). In WinGetFuncArgInPartition WinSetMarkPosition is called at
>>> abs_pos==3.
>>>
>>> (2) WinGetFuncArgInPartition tries to fetch column y at row 0. Since
>>> the mark was set to at row 3, the error occurred.
>>>
>>> To avoid the error, we could call WinGetFuncArgInPartition with
>>> set_mark = false (and call WinSetMarkPosition separately) but I am not
>>> sure if it's an acceptable solution.
>>
>> Attached is a v2 patch to fix the "cannot fetch row before
>> WindowObject's mark position" error, by tweaking the logic to
>> calculate the set mark position in WinGetFuncArgInPartition.
>
> Attached is a v3 patch which is ready for commit IMO. Major
> difference from v2 patch is, now the patch satisfies the request
> below.
>
>>>> of course only created if the window function actually asks for
>>>> evaluations of a particular argno.
>
> The NOT NULL information array is allocated only when the window
> function actually asks for evaluations of a particular argno using
> WinGetFuncArgInFrame or WinGetFuncArgInPartition.
>
> If there's no objection, I am going to commit in a few days.
Patch pushed. Thanks.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-22 04:18 David Rowley <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: David Rowley @ 2025-10-22 04:18 UTC (permalink / raw)
To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Mon, 20 Oct 2025 at 16:59, Tatsuo Ishii <[email protected]> wrote:
>
> > A very trivial commit:
> >
> > ```
> > + else
> > +
> > + /*
> > + * For other cases we have no idea what position of row callers would
> > + * fetch next time. Also for relpos < 0 case (we go backward), we
> > + * cannot set mark either. For those cases we always set mark at 0.
> > + */
> > + mark_pos = 0;
> > ```
> >
> > The empty line after “else” is not needed.
>
> That was added by pgindent.
If it's written down somewhere, I can't find it, but the rule we
normally follow here is; don't use braces if the code block has a
single statement without any comments that appear on a separate line.
Otherwise, use braces.
Since your comments are not on the same line as the statement, it
should have braces. I imagine that's why pgindent is "acting weird".
David
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-22 05:49 Tatsuo Ishii <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-22 05:49 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
> On Mon, 20 Oct 2025 at 16:59, Tatsuo Ishii <[email protected]> wrote:
>>
>> > A very trivial commit:
>> >
>> > ```
>> > + else
>> > +
>> > + /*
>> > + * For other cases we have no idea what position of row callers would
>> > + * fetch next time. Also for relpos < 0 case (we go backward), we
>> > + * cannot set mark either. For those cases we always set mark at 0.
>> > + */
>> > + mark_pos = 0;
>> > ```
>> >
>> > The empty line after “else” is not needed.
>>
>> That was added by pgindent.
>
> If it's written down somewhere, I can't find it, but the rule we
> normally follow here is; don't use braces if the code block has a
> single statement without any comments that appear on a separate line.
> Otherwise, use braces.
Oh ok, I didn't know that.
> Since your comments are not on the same line as the statement, it
> should have braces. I imagine that's why pgindent is "acting weird".
Attached is a trivial patch to follow the rule.
Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
Attachments:
[application/octet-stream] v1-0001-Fix-coding-style-with-else.patch (1.3K, ../../[email protected]/2-v1-0001-Fix-coding-style-with-else.patch)
download | inline diff:
From 1c17f071d825c64843e994c137cc35c076e13af9 Mon Sep 17 00:00:00 2001
From: Tatsuo Ishii <[email protected]>
Date: Wed, 22 Oct 2025 14:38:21 +0900
Subject: [PATCH v1] Fix coding style with "else".
The "else" code block having single statement with comments on a
separate line should have been surrounded by braces.
Reported-by: Chao Li <[email protected]>
Suggested-by: David Rowley <[email protected]>
Author: Tatsuo Ishii <[email protected]>
Discussion: https://postgr.es/m/20251020.125847.997839131426057290.ishii%40postgresql.org
---
src/backend/executor/nodeWindowAgg.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 497eb25ea29..88c6bbba259 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -3822,13 +3822,14 @@ WinGetFuncArgInPartition(WindowObject winobj, int argno,
if (seektype == WINDOW_SEEK_CURRENT && relpos > 0)
mark_pos = winstate->currentpos;
else
-
+ {
/*
* For other cases we have no idea what position of row callers would
* fetch next time. Also for relpos < 0 case (we go backward), we
* cannot set mark either. For those cases we always set mark at 0.
*/
mark_pos = 0;
+ }
/*
* Get the next nonnull value in the partition, moving forward or backward
--
2.43.0
^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options
@ 2025-10-23 02:06 Tatsuo Ishii <[email protected]>
parent: Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 91+ messages in thread
From: Tatsuo Ishii @ 2025-10-23 02:06 UTC (permalink / raw)
To: [email protected]; [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
>> On Mon, 20 Oct 2025 at 16:59, Tatsuo Ishii <[email protected]> wrote:
>>>
>>> > A very trivial commit:
>>> >
>>> > ```
>>> > + else
>>> > +
>>> > + /*
>>> > + * For other cases we have no idea what position of row callers would
>>> > + * fetch next time. Also for relpos < 0 case (we go backward), we
>>> > + * cannot set mark either. For those cases we always set mark at 0.
>>> > + */
>>> > + mark_pos = 0;
>>> > ```
>>> >
>>> > The empty line after “else” is not needed.
>>>
>>> That was added by pgindent.
>>
>> If it's written down somewhere, I can't find it, but the rule we
>> normally follow here is; don't use braces if the code block has a
>> single statement without any comments that appear on a separate line.
>> Otherwise, use braces.
>
> Oh ok, I didn't know that.
>
>> Since your comments are not on the same line as the statement, it
>> should have braces. I imagine that's why pgindent is "acting weird".
>
> Attached is a trivial patch to follow the rule.
Patch pushed. Thanks.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 91+ messages in thread
end of thread, other threads:[~2025-10-23 02:06 UTC | newest]
Thread overview: 91+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-07-13 12:52 Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options Oliver Ford <[email protected]>
2018-07-28 18:59 ` David Fetter <[email protected]>
2018-09-18 12:05 ` Krasiyan Andreev <[email protected]>
2018-09-22 20:06 ` Andrew Gierth <[email protected]>
2018-09-23 21:13 ` Tom Lane <[email protected]>
2018-09-23 22:22 ` Andrew Gierth <[email protected]>
2018-09-23 23:10 ` Tom Lane <[email protected]>
2018-09-23 23:46 ` Andrew Gierth <[email protected]>
2018-09-24 00:20 ` Tom Lane <[email protected]>
2018-09-24 00:26 ` Andrew Gierth <[email protected]>
2018-09-25 04:16 ` Andrew Gierth <[email protected]>
2018-09-25 14:07 ` Tom Lane <[email protected]>
2018-09-27 04:40 ` Andrew Gierth <[email protected]>
2020-04-30 18:58 ` Stephen Frost <[email protected]>
2020-04-30 19:50 ` Krasiyan Andreev <[email protected]>
2018-09-27 04:58 ` Andrew Gierth <[email protected]>
2020-03-06 22:50 [PATCH v8 2/6] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 22:50 [PATCH v11 1/9] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 22:50 [PATCH v12 01/11] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 22:50 [PATCH v7 2/6] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 22:50 [PATCH v9 02/11] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 22:50 [PATCH v10 1/9] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 23:12 [PATCH v9 03/11] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 23:12 [PATCH v7 3/6] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 23:12 [PATCH v12 03/11] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 23:12 [PATCH v11 2/9] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 23:12 [PATCH v8 3/6] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2020-03-06 23:12 [PATCH v10 2/9] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2025-03-09 06:39 Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options Tatsuo Ishii <[email protected]>
2025-03-09 20:07 ` Oliver Ford <[email protected]>
2025-03-13 07:49 ` Oliver Ford <[email protected]>
2025-03-29 07:51 ` Krasiyan Andreev <[email protected]>
2025-03-29 11:18 ` Tatsuo Ishii <[email protected]>
2025-06-11 22:52 ` Tatsuo Ishii <[email protected]>
2025-06-19 06:21 ` Tatsuo Ishii <[email protected]>
2025-06-25 07:19 ` Tatsuo Ishii <[email protected]>
2025-06-30 05:25 ` Tatsuo Ishii <[email protected]>
2025-06-30 08:08 ` Krasiyan Andreev <[email protected]>
2025-07-01 00:19 ` Tatsuo Ishii <[email protected]>
2025-07-07 05:37 ` Tatsuo Ishii <[email protected]>
2025-07-16 04:44 ` Tatsuo Ishii <[email protected]>
2025-07-25 07:49 ` Tatsuo Ishii <[email protected]>
2025-08-16 09:33 ` Tatsuo Ishii <[email protected]>
2025-08-16 19:19 ` Oliver Ford <[email protected]>
2025-08-16 22:34 ` Tatsuo Ishii <[email protected]>
2025-09-12 09:53 ` Tatsuo Ishii <[email protected]>
2025-09-15 08:08 ` Chao Li <[email protected]>
2025-09-17 09:18 ` Tatsuo Ishii <[email protected]>
2025-09-24 05:39 ` Tatsuo Ishii <[email protected]>
2025-09-24 12:24 ` Oliver Ford <[email protected]>
2025-09-24 13:18 ` Tatsuo Ishii <[email protected]>
2025-10-02 12:15 ` Tatsuo Ishii <[email protected]>
2025-10-02 15:36 ` Oliver Ford <[email protected]>
2025-10-03 01:06 ` Tatsuo Ishii <[email protected]>
2025-10-05 15:59 ` Tom Lane <[email protected]>
2025-10-05 16:35 ` Tom Lane <[email protected]>
2025-10-06 00:28 ` Tatsuo Ishii <[email protected]>
2025-10-05 23:51 ` Tatsuo Ishii <[email protected]>
2025-10-06 09:34 ` Tatsuo Ishii <[email protected]>
2025-10-07 02:28 ` Tatsuo Ishii <[email protected]>
2025-10-07 20:14 ` Paul Ramsey <[email protected]>
2025-10-08 00:31 ` Tatsuo Ishii <[email protected]>
2025-10-09 15:19 ` Tom Lane <[email protected]>
2025-10-11 00:07 ` Tatsuo Ishii <[email protected]>
2025-10-11 05:42 ` Tatsuo Ishii <[email protected]>
2025-10-11 05:57 ` Chao Li <[email protected]>
2025-10-13 05:39 ` Tatsuo Ishii <[email protected]>
2025-10-13 11:22 ` Álvaro Herrera <[email protected]>
2025-10-14 00:01 ` Tatsuo Ishii <[email protected]>
2025-10-14 10:21 ` Tatsuo Ishii <[email protected]>
2025-10-16 06:19 ` Michael Paquier <[email protected]>
2025-10-16 10:17 ` Tatsuo Ishii <[email protected]>
2025-10-16 11:50 ` Chao Li <[email protected]>
2025-10-19 01:02 ` Tatsuo Ishii <[email protected]>
2025-10-19 00:38 ` Tatsuo Ishii <[email protected]>
2025-10-11 09:11 ` Tatsuo Ishii <[email protected]>
2025-10-12 08:39 ` Tatsuo Ishii <[email protected]>
2025-10-13 04:43 ` Tatsuo Ishii <[email protected]>
2025-10-19 09:53 ` Tatsuo Ishii <[email protected]>
2025-10-20 03:22 ` Chao Li <[email protected]>
2025-10-20 03:58 ` Tatsuo Ishii <[email protected]>
2025-10-22 04:18 ` David Rowley <[email protected]>
2025-10-22 05:49 ` Tatsuo Ishii <[email protected]>
2025-10-23 02:06 ` Tatsuo Ishii <[email protected]>
2025-10-22 03:14 ` Tatsuo Ishii <[email protected]>
2025-10-13 04:49 ` Tatsuo Ishii <[email protected]>
2025-10-12 12:05 ` Tatsuo Ishii <[email protected]>
2025-10-05 16:20 ` Álvaro Herrera <[email protected]>
2025-10-05 16:51 ` Tom Lane <[email protected]>
2025-10-06 00:09 ` Tatsuo Ishii <[email protected]>
2025-10-07 23:50 ` Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox